ImFusion::ML::MachineLearningModel Class Reference

#include <ImFusion/ML/MachineLearningModel.h>

Class for managing and executing a machine learning model on generic input data. More...

Detailed Description

Class for managing and executing a machine learning model on generic input data.

A model is constituted by a set of different components:

• A Preprocessing pipeline (see Data Pipelines) for preparing the input data.
• An Engine for running the model on the prepared input.
• A Postprocessing pipeline (see Data Pipelines) for modifying the prediction from the engine.

Note

An Engine is a proxy class representing a serialized model from a third party ML framework, such as Torch, ONNX, TensorFlow, etc. Any specific implementation the Engine interface resides in a dedicated plugin, which also wraps the logic and the libraries needed for correctly deserializing and running the model. See Engine for more details.

Splitting the input image into patches and recombining the predictions is controlled by specifying the SplitIntoPatches and RecombinePatches operations respectively in the PreProcessing and PostProcessing sections of the inference.yaml configuration file. Typically, these operations should be the last step in preprocessing and the first step in postprocessing. When set up this way, the MachineLearningModel processes patches lazily: it extracts patches from the input image, runs each patch through the engine, and recombines the predictions on the fly. This approach is especially useful for handling large images that cannot fit entirely in memory. If additional operations are specified after splitting or before recombining, those operations will be applied to each patch as part of the lazy prediction process.

Warning

The MachineLearningModel class is not thread-safe.

Classes
struct	Context

Public Member Functions
DataItem	predict (const DataItem &input)
	Method to execute a generic multiple input/multiple output model The input and output type of a machine learning model is the DataItem, which allows it to give and retrieve an heterogeneous map-type container of the data needed and returned by the model.
DataItem	predict (DataItem &&input)
	Method to execute a generic multiple input/multiple output model The input and output type of a machine learning model is the DataItem, which allows it to give and retrieve an heterogeneous map-type container of the data needed and returned by the model.
std::unique_ptr< SharedImageSet >	predict (const SharedImageSet &images)
	Convenience method to execute a single-input/single-output image-based model.
std::unique_ptr< SharedImageSet >	predict (std::unique_ptr< SharedImageSet > images)
	Convenience method to execute a single-input/single-output image-based model.
const ModelConfiguration &	config () const
	Returns the configuration of this model as const.
ModelConfiguration &	config ()
	Returns the configuration of this model.
const Engine *	engine () const
	Returns a const pointer to the underlying engine.
Engine *	engine ()
	Returns a pointer to the underlying engine. This is useful for setting CPU/GPU mode, querying whether CUDA is available, etc.
const OperationsSequence &	preprocessingSequence () const
	Returns a const reference to the pre-processing operation sequence.
OperationsSequence &	preprocessingSequence ()
	Returns a reference to the pre-processing operation sequence.
const OperationsSequence &	postprocessingSequence () const
	Returns a const pointer to the post-processing operation sequence.
OperationsSequence &	postprocessingSequence ()
	Returns a pointer to the post-processing operation sequence.
void	setProgress (Progress *progress)
	Set the progress.
DataItem	runEngine (const DataItem &input)
	Runs the machine-learning model without any pre-processing or post-processing operations.

Static Public Member Functions
static std::unique_ptr< MachineLearningModel >	create (std::string configPath, PredictionOutput defaultPredictionOutput=PredictionOutput::Unknown) noexcept
	Factory function for creating a machine learning model.
static std::pair< std::unique_ptr< MachineLearningModel >, Status >	createWithStatus (std::string configPath, PredictionOutput defaultPredictionOutput=PredictionOutput::Unknown) noexcept
	Factory function for creating a machine learning model and return it together with its creation status Useful for custom handling of failure cases, since the user can consume the status object.

Protected Member Functions
void	applyPreProcessing (DataItem &input) const
bool	executeFrameByFrame (const DataItem &preprocessedInput, DataItem &outputItem, Progress::Task &task)
bool	executeBatch (const DataItem &preprocessedInput, DataItem &outputItem, Progress::Task &task)
DataItem	setupOutputItemContainers () const
bool	executeFrameByFrameV2 (const DataItem &input, DataItem &outputItem, Progress::Task &task)
bool	executeBatchV2 (const DataItem &input, DataItem &outputItem, Progress::Task &task)
bool	mustSplitImage () const
	MachineLearningModel (std::string configPath, PredictionOutput defaultPredictionOutput=PredictionOutput::Unknown, bool delayEngineLoading=false)
	Constructor from configuration file.
Status	init (std::string configPath, PredictionOutput defaultPredictionOutput=PredictionOutput::Unknown, bool delayEngineLoading=false)
	Protected function that is only used by the MachineLearningModelAlgorithm to delay the loading of the engine.
bool	createEngine ()
	Internal function to create the engine object.

Protected Attributes
std::unique_ptr< ModelConfiguration >	m_config
	Model configuration.
std::unique_ptr< OperationsSequence >	m_preprocessingOp = nullptr
	Pre-processing operation sequence.
std::unique_ptr< OperationsSequence >	m_postprocessingOp = nullptr
	Post-processing operation sequence.
std::unique_ptr< Engine >	m_engine
	The engine running a model serialized by a specific ML framework.
std::unique_ptr< SplitImagesAlgorithm >	m_splitter
	Algorithm for padding/splitting unpadding/recombining images.
std::unique_ptr< OperationsSequence >	m_preprocessingAfterSplitting = nullptr
	Members used by NeuralNetworkV2 implementation.
std::unique_ptr< OperationsSequence >	m_postprocessingBeforeRecombine = nullptr
	Post-processing operation sequence.
std::unique_ptr< Operation >	m_splitOp = nullptr
	Split operation for optionally splitting an input image into patches.
std::unique_ptr< Operation >	m_recombineOp = nullptr
	Recombine operation for recombining predicted patches.
Context	m_context
	Context store for keeping track of relevant data during predict() execution.
Progress *	m_progress = nullptr
	Progress bar.

Constructor & Destructor Documentation

MachineLearningModel()

ImFusion::ML::MachineLearningModel::MachineLearningModel ( std::string configPath, PredictionOutput defaultPredictionOutput = PredictionOutput::Unknown, bool delayEngineLoading = false )

explicitprotected

Constructor from configuration file.

Parameters

configPath	Path to the configuration file used to create ModelConfiguration object owned by the model
defaultPredictionOutput	Parameter used to specify the default prediction output of a model if this is missing from the config file.
delayEngineLoading	Whether to delay loading the engine saved model when predict is called or load it immediately at construction, defaults false.

Note

The prediction output type must be specified either in the constructor or in the configuration file under the key PredictionOutput. If no prediction output is specified the model throws an error, whereas if it is specified in both places, the one from the config file is used. This construct is for supporting older config file where the prediction output type is not specified, and thus might be changed in the future.

Postponing loading the engine is used only in the algorithm/controller pair associated to this class for improving the UI experience (loading the engine saved model takes some time), it should not be used in the SDK as it bypasses checks that all the resources required by an ML model can be acquired without problems.

Exceptions

MLModelException

if any of the model component are not configured correctly.

Member Function Documentation

create()

std::unique_ptr< MachineLearningModel > ImFusion::ML::MachineLearningModel::create ( std::string configPath, PredictionOutput defaultPredictionOutput = PredictionOutput::Unknown )

staticnoexcept

Factory function for creating a machine learning model.

If the resource required by the MachineLearningModel could not be acquired, returns an invalid pointer

Parameters

configPath	Path to the configuration file used to create ModelConfiguration object owned by the model
defaultPredictionOutput	Parameter used to specify the prediction output of a model if this is missing from the config file.

Note

The prediction output type must be specified either in the constructor or in the configuration file under the key PredictionOutput. If no prediction output is specified the model throws an error, whereas if it is specified in both places, the one from the config file is used. This construct is for supporting older config file where the prediction output type is not specified, and thus might be changed in the future.

createWithStatus()

std::pair< std::unique_ptr< MachineLearningModel >, Status > ImFusion::ML::MachineLearningModel::createWithStatus ( std::string configPath, PredictionOutput defaultPredictionOutput = PredictionOutput::Unknown )

staticnoexcept

Factory function for creating a machine learning model and return it together with its creation status Useful for custom handling of failure cases, since the user can consume the status object.

Note

This method doesn't log any (error) message, it is left to the user and the way the status is handled.

predict() [1/4]

DataItem ImFusion::ML::MachineLearningModel::predict

(

const DataItem &

input

)

Method to execute a generic multiple input/multiple output model The input and output type of a machine learning model is the DataItem, which allows it to give and retrieve an heterogeneous map-type container of the data needed and returned by the model.

Parameters

input

Input data item containing all data to be used for inference. A const reference of the input is passed, thus a copy will be made in the proprocessing step. Use the DataItem predict(DataItem&& input) method to avoid this.

Returns

Post-processed prediction

predict() [2/4]

DataItem ImFusion::ML::MachineLearningModel::predict

(

DataItem &&

input

)

Method to execute a generic multiple input/multiple output model The input and output type of a machine learning model is the DataItem, which allows it to give and retrieve an heterogeneous map-type container of the data needed and returned by the model.

Parameters

input

Input data item containing all data to be used for inference. It is passed by rvalue and thus the underlying data might be changed.

Returns

Post-processed prediction

predict() [3/4]

std::unique_ptr< SharedImageSet > ImFusion::ML::MachineLearningModel::predict

(

const SharedImageSet &

images

)

Convenience method to execute a single-input/single-output image-based model.

Parameters

images

Input image set to be used for inference. Passed as const reference, so images is guaranteed to stay unaltered.

Returns

Post-processed prediction images

predict() [4/4]

std::unique_ptr< SharedImageSet > ImFusion::ML::MachineLearningModel::predict

(

std::unique_ptr< SharedImageSet >

images

)

Convenience method to execute a single-input/single-output image-based model.

Parameters

images

Input image set to be used for inference. Passed as a unique_ptr so ownership is assumed and images might be changed.

Returns

Post-processed prediction images

Member Data Documentation

m_preprocessingAfterSplitting

std::unique_ptr<OperationsSequence> ImFusion::ML::MachineLearningModel::m_preprocessingAfterSplitting = nullptr

protected

Members used by NeuralNetworkV2 implementation.

Pre-processing operation sequence

---

The documentation for this class was generated from the following file:

• ImFusion/ML/MachineLearningModel.h