 |
ImFusion C++ SDK 4.5.0
|
|
ImFusion C++ SDK 4.5.0
|
#include <ImFusion/ML/DataLoaders/CacheDataLoader.h>
Caches the dataset loaded until now. More...
Caches the dataset loaded until now.
Public Member Functions | |
| CacheDataLoader (std::unique_ptr< DataLoader > loader, bool makeExclusiveCPU=true, bool lazy=true, DataItemCompressionMode compressionMode=DataItemCompressionMode::None, bool shuffle=false, int numThreads=1, bool verbose=false, bool shallowCopy=false) | |
| Constructs the caching. | |
| CacheDataLoader (std::unique_ptr< DataLoader > loader, bool makeExclusiveCPU, bool lazy, int compressionLevel, bool shuffle, int numThreads, bool verbose, bool shallowCopy) | |
| CacheDataLoader (DataLoaderList dataLoaders, const Properties &prop) | |
| std::optional< DataItem > | next () override |
| Advance the loader by one iteration. | |
| std::optional< size_t > | size () const override |
| The size of the loader is always known after the cache is fully generated (either lazily or not) If caching lazily, the size is the same as that of the underlying loader (unknown if Dynamic), until the cache is fully generated. | |
| void | reset () override |
| Resets the loader index. | |
| void | reinit () override |
| Forces a cache destruction. | |
| Cardinality | cardinality () const override |
| After caching, the number of items in the cache is known, and thus the cardinality is Fixed. | |
| Public Member Functions inherited from ImFusion::ML::DataLoader | |
| DataLoader (std::string name, bool verbose=false) | |
| Constructor to be used by DataReaders, as they are the first elements in the chain. | |
| DataLoader (std::string name, std::unique_ptr< DataLoader > loader, bool verbose=false) | |
| Constructor to be used by all other derived DataLoaders. | |
| DataLoader (std::string name, DataLoaderList &&dataLoaders, bool verbose=false) | |
| Constructor to be used by all other derived DataLoaders. | |
| const std::string & | name () const |
| Name of the loader. | |
| DataLoader * | singleUnderlyingLoader () const |
| Return a single underlying dataloader Expects there to be only one, returns a nullptr otherwise. | |
| std::unique_ptr< DataLoader > | extractDataloader () |
| Return a single underlying dataloader Expects there to be only one, throws an exception otherwise. | |
| DataLoaderList & | loaders () |
| DataLoaderList const & | loaders () const |
| bool | verbose () const |
| virtual void | setRandomSeed (unsigned int seed) |
| void | configure (const Properties *properties) override |
| Configure this object instance by de-serializing the given Properties. | |
| const std::string & | logDomain () const |
| Log domain for AdvancedParameter. | |
| Public Member Functions inherited from ImFusion::Configurable | |
| virtual void | configuration (Properties *p) const |
| Serialize the current object configuration into the given Properties object. | |
| virtual void | configureDefaults () |
| Retrieve the properties of this object, replaces values with their defaults and sets it again. | |
| void | registerParameter (ParameterBase *param) |
| Register the given Parameter or SubProperty, so that it will be configured during configure()/configuration(). | |
| void | unregisterParameter (const ParameterBase *param) |
| Remove the given Parameter or SubProperty from the list of registered parameters. | |
| Configurable (const Configurable &rhs) | |
| Configurable (Configurable &&rhs) noexcept | |
| Configurable & | operator= (const Configurable &) |
| Configurable & | operator= (Configurable &&) noexcept |
Additional Inherited Members | |
| Public Attributes inherited from ImFusion::ML::DataLoader | |
| LoaderParam< bool > | m_verbose = {"verbose", false, this, ParamRequired::No} |
| If enabled, print log messages with extra information like timings. | |
| Public Attributes inherited from ImFusion::Configurable | |
| Signal | signalParametersChanged |
| Emitted whenever one of the registered Parameters' or SubPropertys' signalValueChanged signal was emitted. | |
| Protected Member Functions inherited from ImFusion::Utils::NotCopyable | |
| NotCopyable (NotCopyable &&) noexcept=default | |
| NotCopyable & | operator= (NotCopyable &&) noexcept=default |
| NotCopyable (const NotCopyable &)=delete | |
| NotCopyable & | operator= (const NotCopyable &)=delete |
| Protected Attributes inherited from ImFusion::ML::DataLoader | |
| std::string | m_name |
| Cardinality | m_cardinality = Cardinality::Fixed |
| Random::Generator | m_randGenerator |
| DataLoaderList | m_dataLoaders |
| Protected Attributes inherited from ImFusion::Configurable | |
| std::vector< Param > | m_params |
| List of all registered Parameter and SubProperty instances. | |
|
explicit |
Constructs the caching.
| loader | previous loader |
| makeExclusiveCPU | releases the GPU memory if true |
| lazy | caches items when requested (otherwise caches the whole dataset at initialization) |
| compressionMode | compression strategy for cached data |
| shuffle | reshuffle the order of the cache every epoch |
| numThreads | the number of items to prefetch from the cache in the background. The cache needs to create copies of the data so fetching is more expensive than one may anticipate. Asynchronous prefetching is enabled only when makeExclusiveCPU is true and numThreads > 1. |
| verbose | enable verbose logging |
| shallowCopy | if true, returns items from the cache without deep-copying the underlying data. This avoids the copy overhead but is potentially unsafe: any in-place modification of the returned item will corrupt the cache. Only enable this if you are certain the pipeline will not modify the data. Ignored when compression is enabled, since deserialization always produces a fresh copy. |
| DataLoaderException | if the nested loader is uncountable |
| std::bad_alloc | if the system runs out of memory while caching the dataset |
|
explicit |
|
overridevirtual |
Advance the loader by one iteration.
The function must pull an item from the underlying node, process it and return it. If the underlyng loader is exhausted, returns an empty optional.
Implements ImFusion::ML::DataLoader.
|
overridevirtual |
The size of the loader is always known after the cache is fully generated (either lazily or not) If caching lazily, the size is the same as that of the underlying loader (unknown if Dynamic), until the cache is fully generated.
After that, the size is known regardless of the underlying loader size.
Implements ImFusion::ML::DataLoader.
|
overridevirtual |
Resets the loader index.
A fully-populated or partially-populated lazy cache is preserved across resets so that cached items are served from memory on subsequent epochs. When the cache is only partially filled (lazy mode), the underlying loader is intentionally not reset so it can continue providing uncached items from where it left off. To force clear the cache, use reinit.
Reimplemented from ImFusion::ML::DataLoader.
|
overridevirtual |
Forces a cache destruction.
The cache is created at the next call of next.
Reimplemented from ImFusion::ML::DataLoader.
|
overridevirtual |
After caching, the number of items in the cache is known, and thus the cardinality is Fixed.
When caching lazily, the cardinality is the same of the underlying loader until the cache is fully generated (first epoch typically), then is Fixed.
Reimplemented from ImFusion::ML::DataLoader.