#include <ImFusion/ML/DataLoaders/PersistentCacheDataLoader.h>

Caches the dataset loaded until now in a persistent manner (on a disk location). More...

Inheritance diagram for ImFusion::ML::PersistentCacheDataLoader:

Detailed Description

Caches the dataset loaded until now in a persistent manner (on a disk location).

Public Member Functions
	PersistentCacheDataLoader (std::unique_ptr< DataLoader > loader, const std::string &location, bool lazy=true, bool reload=false, bool compression=false, bool shuffle=false, bool verbose=false)
	Constructs the caching data loader.
	PersistentCacheDataLoader (DataLoaderList dataLoaders, const Properties &prop)
std::optional< DataItem >	next () override
	Advance the loader by one iteration.
std::optional< size_t >	size () const override
	The size of the loader is always known after the cache is fully generated (either lazily or not) If caching lazily and not reloading, the size is the same as that of the underlying loader (unknown if Dynamic), until the cache is fully generated.
void	reset () override
	Resets the loader index but not the store cache This is to avoid reloading and caching the entire data set every time the iteration is consumed.
void	reinit () override
	Forces a cache destruction, i.e.
Cardinality	cardinality () const override
	After caching, the number of items in the cache is known, and thus the cardinality is Fixed.
Public Member Functions inherited from ImFusion::ML::DataLoader
	DataLoader (std::string name, bool verbose=false)
	Constructor to be used by DataReaders, as they are the first elements in the chain.
	DataLoader (std::string name, std::unique_ptr< DataLoader > loader, bool verbose=false)
	Constructor to be used by all other derived DataLoaders.
	DataLoader (std::string name, DataLoaderList &&dataLoaders, bool verbose=false)
	Constructor to be used by all other derived DataLoaders.
const std::string &	name () const
	Name of the loader.
DataLoader *	singleUnderlyingLoader () const
	Return a single underlying dataloader Expects there to be only one, returns a nullptr otherwise.
std::unique_ptr< DataLoader >	extractDataloader ()
	Return a single underlying dataloader Expects there to be only one, throws an exception otherwise.
DataLoaderList &	loaders ()
DataLoaderList const &	loaders () const
bool	verbose () const
virtual void	setRandomSeed (unsigned int seed)
void	configure (const Properties *properties) override
	Configure this object instance by de-serializing the given Properties.
virtual Status	canBePrefetched () const
	Indicates whether a loader is eligible for prefetching.
const std::string &	logDomain () const
	Log domain for AdvancedParameter.
Public Member Functions inherited from ImFusion::Configurable
virtual void	configuration (Properties *p) const
	Serialize the current object configuration into the given Properties object.
virtual void	configureDefaults ()
	Retrieve the properties of this object, replaces values with their defaults and sets it again.
void	registerParameter (ParameterBase *param)
	Register the given Parameter or SubProperty, so that it will be configured during configure()/configuration().
void	unregisterParameter (const ParameterBase *param)
	Remove the given Parameter or SubProperty from the list of registered parameters.
	Configurable (const Configurable &rhs)
	Configurable (Configurable &&rhs) noexcept
Configurable &	operator= (const Configurable &)
Configurable &	operator= (Configurable &&) noexcept

Additional Inherited Members
Public Attributes inherited from ImFusion::ML::DataLoader
LoaderParam< bool >	m_verbose = {"verbose", false, this, ParamRequired::No}
	If enabled, print log messages with extra information like timings.
Public Attributes inherited from ImFusion::Configurable
Signal	signalParametersChanged
	Emitted whenever one of the registered Parameters' or SubPropertys' signalValueChanged signal was emitted.
Protected Member Functions inherited from ImFusion::Utils::NotCopyable
	NotCopyable (NotCopyable &&) noexcept=default
NotCopyable &	operator= (NotCopyable &&) noexcept=default
	NotCopyable (const NotCopyable &)=delete
NotCopyable &	operator= (const NotCopyable &)=delete
Protected Attributes inherited from ImFusion::ML::DataLoader
std::string	m_name
Cardinality	m_cardinality = Cardinality::Fixed
Random::Generator	m_randGenerator
DataLoaderList	m_dataLoaders
Protected Attributes inherited from ImFusion::Configurable
std::vector< Param >	m_params
	List of all registered Parameter and SubProperty instances.

Constructor & Destructor Documentation

◆ PersistentCacheDataLoader()

ImFusion::ML::PersistentCacheDataLoader::PersistentCacheDataLoader	(	std::unique_ptr< DataLoader >	loader,
		const std::string &	location,
		bool	lazy = true,
		bool	reload = false,
		bool	compression = false,
		bool	shuffle = false,
		bool	verbose = false )

explicit

Constructs the caching data loader.

Parameters

loader	previous loader
location	path to the folder where all the data will be cached
lazy	caches items when requested (otherwise caches the whole dataset at initialization)
reload	try to reload the cache from a previous session
compression	compresses the data when saving on disk
shuffle	reshuffle the order of the cache every epoch

Exceptions

DataLoaderException if the nested loader is uncountable

Member Function Documentation

◆ next()

std::optional< DataItem > ImFusion::ML::PersistentCacheDataLoader::next ( )

overridevirtual

Advance the loader by one iteration.

The function must pull an item from the underlying node, process it and return it. If the underlyng loader is exhausted, returns an empty optional.

Implements ImFusion::ML::DataLoader.

◆ size()

std::optional< size_t > ImFusion::ML::PersistentCacheDataLoader::size ( ) const

overridevirtual

The size of the loader is always known after the cache is fully generated (either lazily or not) If caching lazily and not reloading, the size is the same as that of the underlying loader (unknown if Dynamic), until the cache is fully generated.

After that, the size is known regardless of the underlying loader size.

Implements ImFusion::ML::DataLoader.

◆ reset()

void ImFusion::ML::PersistentCacheDataLoader::reset ( )

overridevirtual

Resets the loader index but not the store cache This is to avoid reloading and caching the entire data set every time the iteration is consumed.

To force clear the cache, use PersistentCacheDataLoader::clearCache().

Reimplemented from ImFusion::ML::DataLoader.

◆ reinit()

void ImFusion::ML::PersistentCacheDataLoader::reinit ( )

overridevirtual

Forces a cache destruction, i.e.

delete all files from the folder The cache is created at the next call of next.

Reimplemented from ImFusion::ML::DataLoader.

◆ cardinality()

Cardinality ImFusion::ML::PersistentCacheDataLoader::cardinality ( ) const

overridevirtual

After caching, the number of items in the cache is known, and thus the cardinality is Fixed.

When caching lazily and not reloading, the cardinality is the same of the underlying loader until the cache is fully generated (first epoch typically), then is Fixed.

Reimplemented from ImFusion::ML::DataLoader.

The documentation for this class was generated from the following file:

ImFusion/ML/DataLoaders/PersistentCacheDataLoader.h

Detailed Description

Public Member Functions

Additional Inherited Members

Constructor & Destructor Documentation

◆ PersistentCacheDataLoader()

Member Function Documentation

◆ next()

◆ size()

◆ reset()

◆ reinit()

◆ cardinality()