FileReader Class Reference

#include <ImFusion/ML/DataReaders/FileReader.h>

Loads data by reading the files from a list of data "columns" (see above). More...

Inheritance diagram for FileReader:

Detailed Description

Loads data by reading the files from a list of data "columns" (see above).

Each data column can have a different KeyMapping specifying the fields of the DataItem to be constructed. The FileReader iterates over the columns "row-wise", meaning at iteration i, the i-th filename in each column is used to load data and create a DataItem. The DataItems produced in this way are then merged together in a single one, and returned by FileReader::next(). If the same field is specified in the KeyMappings of multiple columns, merging the items fails and the loader throws a DataItemException.

Example: Suppose you have a data_list.txt like the following (typically produced by ImFusionLabels) #dataPath labelPath keypointPath 0.imf 0-labels.imf 0-keypoints.json 1.imf 1-labels.imf 1-keypoints.json 2.imf 2-labels.imf 2-keypoints.json ...

And suppose also that the imf files in the first column contain 3 SharedImageSet, representing an MR scan, a co-registered CT scan and some overlap Mask. The files marked as *-label.imf instead contain 2 SharedImageSet representing a segmentation map and a weight map for your training loss. The *.json files instead can contain only a single KeypointElement. In order to setup the FileReader you need to parse this txt files into a vector of FileReaderColumn of size 3.

If you want to load and keep all the data you give the columns the following mappings: columns[0].mapping = {{0, "mr"}, {1, "ct"}, {2, "mask"}}; columns[1].mapping = {{0, "seg"}, {1, "weights"}}; columns[2].mapping = {{0, "keypoints"}};

In case you want to train a simple segmentation model on the CT image, you can discard the rest of the data with the following mappings: columns[0].mapping = {{1, "ct"}}; columns[1].mapping = {{0, "seg"}}; columns[2].mapping = {};

In case you want to train a keypoint localization model on the MR image instead, you can use i.e. the following mappings: columns[0].mapping = {{0, "mr"}}; columns[1].mapping = {}; columns[2].mapping = {{0, "keypoints"}};

... and so on.

Note: You can also parse multiple datalist files and create a list of columns to build the FileReader with.

Public Member Functions
	FileReader (std::vector< FileReaderColumn > dataLists, bool shuffle=false, bool verbose=false)
	FileReader (const Properties &properties)
	constructor for DataLoaderDecoratorFactory
std::optional< DataItem >	next () final
	Advance the loader by one iteration.
std::optional< size_t >	size () const final
	Size of the loader, has value only for fixed cardinalities.
void	reset ()
	Reset the loader in order to restart the iteration.
Public Member Functions inherited from DataReader
	DataReader (std::string name="", bool verbose=false)
Cardinality	cardinality () const override
	Readers have fixed cardinality.
Public Member Functions inherited from DataLoader
	DataLoader (std::string name, bool verbose=false)
	Constructor to be used by DataReaders, as they are the first elements in the chain.
	DataLoader (std::string name, std::unique_ptr< DataLoader > loader, bool verbose=false)
	Constructor to be used by all other derived DataLoaders.
	DataLoader (std::string name, DataLoaderList &&dataLoaders, bool verbose=false)
	Constructor to be used by all other derived DataLoaders.
const std::string &	name () const
	Name of the loader.
virtual void	reinit ()
	Reinitialize the loader state, cleaning it up if necessary Overload this method if your loader has state that must survive reset (i.e.
DataLoader *	singleUnderlyingLoader () const
	Return a single underlying dataloader Expects there to be only one, returns a nullptr otherwise.
std::unique_ptr< DataLoader >	extractDataloader ()
	Return a single underlying dataloader Expects there to be only one, throws an exception otherwise.
DataLoaderList &	loaders ()
DataLoaderList const &	loaders () const
bool	verbose () const
virtual void	setRandomSeed (unsigned int seed)
virtual Status	canBePrefetched () const
	Indicates whether a loader is eligible for prefetching.
const std::string &	logDomain () const
	Log domain for AdvancedParameter.
Public Member Functions inherited from Configurable
virtual void	configuration (Properties *p) const
	Serialize the current object configuration into the given Properties object.
virtual void	configureDefaults ()
	Retrieve the properties of this object, replaces values with their defaults and sets it again.
void	registerParameter (ParameterBase *param)
	Register the given Parameter or SubProperty, so that it will be configured during configure()/configuration().
void	unregisterParameter (const ParameterBase *param)
	Remove the given Parameter or SubProperty from the list of registered parameters.
	Configurable (const Configurable &rhs)
	Configurable (Configurable &&rhs) noexcept
Configurable &	operator= (const Configurable &)
Configurable &	operator= (Configurable &&) noexcept

Static Public Member Functions
static DataItem	load (const std::string &filepath, const KeyMapping &key)
	Load the file at given path and return a DataItem with the data in it as defined in mapping key.

Protected Member Functions
void	configure (const Properties *properties) final
	Configure this object instance by de-serializing the given Properties.
void	init ()
Protected Member Functions inherited from NotCopyable
	NotCopyable (NotCopyable &&) noexcept=default
NotCopyable &	operator= (NotCopyable &&) noexcept=default
	NotCopyable (const NotCopyable &)=delete
NotCopyable &	operator= (const NotCopyable &)=delete

Additional Inherited Members
Public Attributes inherited from DataLoader
LoaderParam< bool >	m_verbose = {"verbose", false, this, ParamRequired::No}
	If enabled, print log messages with extra information like timings.
Public Attributes inherited from Configurable
Signal	signalParametersChanged
	Emitted whenever one of the registered Parameters' or SubPropertys' signalValueChanged signal was emitted.
Protected Attributes inherited from DataLoader
std::string	m_name
Cardinality	m_cardinality = Cardinality::Fixed
Random::Generator	m_randGenerator
DataLoaderList	m_dataLoaders
Protected Attributes inherited from Configurable
std::vector< Param >	m_params
	List of all registered Parameter and SubProperty instances.

Member Function Documentation

next()

std::optional< DataItem > next ( )

finalvirtual

Advance the loader by one iteration.

The function must pull an item from the underlying node, process it and return it. If the underlyng loader is exhausted, returns an empty optional.

Implements DataLoader.

size()

std::optional< size_t > size ( ) const

finalvirtual

Size of the loader, has value only for fixed cardinalities.

Implements DataLoader.

reset()

void reset ( )

virtual

Reset the loader in order to restart the iteration.

Reimplemented from DataLoader.

configure()

void configure ( const Properties * p )

finalprotectedvirtual

Configure this object instance by de-serializing the given Properties.

The default implementation will do so automatically for all registered Parameter and SubProperty instances.

See also

configuration() for the inverse functionality

Reimplemented from DataLoader.

---

The documentation for this class was generated from the following file:

• ImFusion/ML/DataReaders/FileReader.h