ImFusion::ML::DataLoader Class Reference

#include <ImFusion/ML/DataLoader.h>

Interface for all data loaders, establishing the mechanism for loading and processing data. More...

Inheritance diagram for ImFusion::ML::DataLoader:

Detailed Description

Interface for all data loaders, establishing the mechanism for loading and processing data.

A loader is an object working like a simple forward iterator. To advance it, one calls next(). The iteration should stop once the loader stops returning valid objects. Loaders can be linked together. Like in a linked list, each loader has a nested instance of a DataLoader. The idea is to separate logical operations in different loaders, each performing some task on the items produced by the underlying loader. The loaders chain ends when linked to a DataReader, which are responsible to fetch data from source.

A drawback of this design is that it's not always possible to determine how many items will be returned by a loader. Sometimes the number of items produced by a chain of loaders can be known in advanced, sometimes it depends on runtime conditions, or run indefinetly until a stopping criterion is met. We hard code this property in each loader by assigning it a Cardinality.

For this reason the size() function returns the number of items to be loaded only if all the chained loaders are of fixed Cardinality, otherwise the function returns an empty optional.

Since data loaders can be stateful, they must be resettable to their initial state.

Public Member Functions
	DataLoader (std::string name, bool verbose=false)
	Constructor to be used by DataReaders, as they are the first elements in the chain.
	DataLoader (std::string name, std::unique_ptr< DataLoader > loader, bool verbose=false)
	Constructor to be used by all other derived DataLoaders.
	DataLoader (std::string name, DataLoaderList &&dataLoaders, bool verbose=false)
	Constructor to be used by all other derived DataLoaders.
const std::string &	name () const
	Name of the loader.
virtual std::optional< DataItem >	next ()=0
	Advance the loader by one iteration.
virtual std::optional< size_t >	size () const =0
	Size of the loader, has value only for fixed cardinalities.
virtual void	reset ()
	Reset the loader in order to restart the iteration.
virtual void	reinit ()
	Reinitialize the loader state, cleaning it up if necessary Overload this method if your loader has state that must survive reset (i.e.
virtual Cardinality	cardinality () const
	Cardinality of this specific loader node.
DataLoader *	singleUnderlyingLoader () const
	Return a single underlying dataloader Expects there to be only one, returns a nullptr otherwise.
std::unique_ptr< DataLoader >	extractDataloader ()
	Return a single underlying dataloader Expects there to be only one, throws an exception otherwise.
DataLoaderList &	loaders ()
DataLoaderList const &	loaders () const
bool	verbose () const
virtual void	setRandomSeed (unsigned int seed)
void	configure (const Properties *properties) override
	Configure this object instance by de-serializing the given Properties.
virtual Status	canBePrefetched () const
	Indicates whether a loader is eligible for prefetching.
const std::string &	logDomain () const
	Log domain for AdvancedParameter.
Public Member Functions inherited from ImFusion::Configurable
virtual void	configuration (Properties *p) const
	Serialize the current object configuration into the given Properties object.
virtual void	configureDefaults ()
	Retrieve the properties of this object, replaces values with their defaults and sets it again.
void	registerParameter (ParameterBase *param)
	Register the given Parameter or SubProperty, so that it will be configured during configure()/configuration().
void	unregisterParameter (const ParameterBase *param)
	Remove the given Parameter or SubProperty from the list of registered parameters.
	Configurable (const Configurable &rhs)
	Configurable (Configurable &&rhs) noexcept
Configurable &	operator= (const Configurable &)
Configurable &	operator= (Configurable &&) noexcept

Public Attributes
LoaderParam< bool >	m_verbose = {"verbose", false, this, ParamRequired::No}
	If enabled, print log messages with extra information like timings.
Public Attributes inherited from ImFusion::Configurable
Signal	signalParametersChanged
	Emitted whenever one of the registered Parameters' or SubPropertys' signalValueChanged signal was emitted.

Protected Attributes
std::string	m_name
Cardinality	m_cardinality = Cardinality::Fixed
Random::Generator	m_randGenerator
DataLoaderList	m_dataLoaders
Protected Attributes inherited from ImFusion::Configurable
std::vector< Param >	m_params
	List of all registered Parameter and SubProperty instances.

Additional Inherited Members
Protected Member Functions inherited from ImFusion::Utils::NotCopyable
	NotCopyable (NotCopyable &&) noexcept=default
NotCopyable &	operator= (NotCopyable &&) noexcept=default
	NotCopyable (const NotCopyable &)=delete
NotCopyable &	operator= (const NotCopyable &)=delete

Member Function Documentation

next()

virtual std::optional< DataItem > ImFusion::ML::DataLoader::next ( )

pure virtual

Advance the loader by one iteration.

The function must pull an item from the underlying node, process it and return it. If the underlyng loader is exhausted, returns an empty optional.

Implemented in ImFusion::ML::BatchDataLoader, ImFusion::ML::CacheDataLoader, ImFusion::ML::DataListReader, ImFusion::ML::FileReader, ImFusion::ML::FilterDataLoader, ImFusion::ML::FilterEmptyElementContentDataLoader, ImFusion::ML::InterleaveDataLoader, ImFusion::ML::LabelsProjectDataReader, ImFusion::ML::MapDataLoader, ImFusion::ML::PersistentCacheDataLoader, ImFusion::ML::PrefetchDataLoader, ImFusion::ML::PreprocessDataLoader, ImFusion::ML::RandomizeDataLoader, ImFusion::ML::RepeatDataLoader, ImFusion::ML::SampleDataLoader, ImFusion::ML::ShuffleDataLoader, and ImFusion::ML::SplitDataLoader.

size()

virtual std::optional< size_t > ImFusion::ML::DataLoader::size ( ) const

pure virtual

Size of the loader, has value only for fixed cardinalities.

Implemented in ImFusion::ML::BatchDataLoader, ImFusion::ML::CacheDataLoader, ImFusion::ML::DataListReader, ImFusion::ML::FileReader, ImFusion::ML::FilterDataLoader, ImFusion::ML::FilterEmptyElementContentDataLoader, ImFusion::ML::InterleaveDataLoader, ImFusion::ML::LabelsProjectDataReader, ImFusion::ML::MapDataLoader, ImFusion::ML::PersistentCacheDataLoader, ImFusion::ML::PrefetchDataLoader, ImFusion::ML::PreprocessDataLoader, ImFusion::ML::RandomizeDataLoader, ImFusion::ML::RepeatDataLoader, ImFusion::ML::SampleDataLoader, ImFusion::ML::ShuffleDataLoader, and ImFusion::ML::SplitDataLoader.

reset()

virtual void ImFusion::ML::DataLoader::reset ( )

virtual

Reset the loader in order to restart the iteration.

Reimplemented in ImFusion::ML::BatchDataLoader, ImFusion::ML::CacheDataLoader, ImFusion::ML::DataListReader, ImFusion::ML::FileReader, ImFusion::ML::InterleaveDataLoader, ImFusion::ML::LabelsProjectDataReader, ImFusion::ML::MapDataLoader, ImFusion::ML::PersistentCacheDataLoader, ImFusion::ML::PrefetchDataLoader, ImFusion::ML::RandomizeDataLoader, ImFusion::ML::RepeatDataLoader, ImFusion::ML::ShuffleDataLoader, and ImFusion::ML::SplitDataLoader.

reinit()

virtual void ImFusion::ML::DataLoader::reinit ( )

virtual

Reinitialize the loader state, cleaning it up if necessary Overload this method if your loader has state that must survive reset (i.e.

the loader holding a cache)

Reimplemented in ImFusion::ML::CacheDataLoader, ImFusion::ML::DataListReader, ImFusion::ML::PersistentCacheDataLoader, ImFusion::ML::PreprocessDataLoader, and ImFusion::ML::SampleDataLoader.

cardinality()

virtual Cardinality ImFusion::ML::DataLoader::cardinality ( ) const

virtual

Cardinality of this specific loader node.

Reimplemented in ImFusion::ML::CacheDataLoader, ImFusion::ML::DataListReader, ImFusion::ML::DataReader, and ImFusion::ML::PersistentCacheDataLoader.

configure()

void ImFusion::ML::DataLoader::configure ( const Properties * p )

overridevirtual

Configure this object instance by de-serializing the given Properties.

The default implementation will do so automatically for all registered Parameter and SubProperty instances.

See also

configuration() for the inverse functionality

Reimplemented from ImFusion::Configurable.

Reimplemented in ImFusion::ML::FileReader, ImFusion::ML::FilterDataLoader, ImFusion::ML::FilterEmptyElementContentDataLoader, ImFusion::ML::InterleaveDataLoader, ImFusion::ML::MapDataLoader, ImFusion::ML::PreprocessDataLoader, ImFusion::ML::RandomizeDataLoader, ImFusion::ML::RepeatDataLoader, ImFusion::ML::SampleDataLoader, and ImFusion::ML::ShuffleDataLoader.

canBePrefetched()

virtual Status ImFusion::ML::DataLoader::canBePrefetched ( ) const

inlinevirtual

Indicates whether a loader is eligible for prefetching.

Typically, a loader can be prefetched if its code can be executed on any thread. However, if the code to be executed in the prefetcher requires the main thread or acquires state that cannot be released by other threads (such as the Python Global Interpreter Lock), this function must return an error code and a message explaining why.

Returns: A status object indicating whether the loader can be prefetched (OK) or not (error code).

Reimplemented in ImFusion::ML::PreprocessDataLoader, and ImFusion::ML::SampleDataLoader.

---

The documentation for this class was generated from the following file:

• ImFusion/ML/DataLoader.h