#include <ImFusion/GL/TrackedSharedImageSet.h>

Set of images with arbitrarily sampled tracking data. More...

Inheritance diagram for TrackedSharedImageSet:

Detailed Description

Set of images with arbitrarily sampled tracking data.

The class replaces the matrix assignment within SharedImageSet (each contained image uses its own matrix) by a look-up into TrackingSequence instances. Therefore, directly setting a matrix of an individual frame is deactivated, and the matrices of the individual images are ignored.

Resolving which matrix of which tracking sequence is associated with a particular image depends on several options:

In general, the main tracking sequence (trackingUsed) is used for look-up.
Lookup strategy based on timestamps:
- If useTimestamps is false:
  - A 1:1 lookup is performed: the frame \(i\) is associated directly with tracking matrix \(T_i\).
- If useTimestamps is true:
  - The image timestamp \(t_i - t_{off}\) is used to find the matrix with the closest timestamp, where \(t_{off}\) denotes temporalOffset.
  - If that timestamp lies before the first or after the last in the tracking sequence, the first or last matrix is used, respectively.
  - Otherwise, the matrix is interpolated (using quaternions for the rotation) between the two closest pose matrices.
The final pose \(P\) for a given frame is computed using registration \(R\) and calibration \(C\) matrices:
\[P = R \cdot T \cdot C \]
If there is a relative Tracking Sequence with matrices \(R'\), \(T'_j\), and \(C'\), the final pose becomes:
\[P_i = R \cdot \left( {R'} \cdot T'_j \cdot {C'} \right)^{-1} \cdot T_i \cdot C = R \cdot {C'}^{-1} \cdot {T'_j}^{-1} \cdot {R'}^{-1} \cdot T_i \cdot C \]
- The matrix \(T'_j\) is determined as in step 2, unless relativeToFirst is true, in which case it is simply the first matrix \(T'_0\).

See TrackingSequence for details on the registration/calibration matrices and temporal offset. All matrices are natively stored in TOWORLD convention, and methods called "..FromWorld" will evualate a matrix inversion.

Note: It is advised to only include calibration and registration matrices in the main tracking sequence.

While the class does not strictly enforce identical type, pixel size, spacing, etc. across individual frames, unexpected behavior may result from noncompliance with these assumptions. In particular, all methods exposing size measures in any form (frame pixel or phyiscal size) only access the first frame and ignore all others. I/O operations with noncompliant images may not work.

Public Member Functions
	TrackedSharedImageSet (const TrackedSharedImageSet &other)=delete

std::unique_ptr< TrackedSharedImageSet >	clone (Flags< CloneOptions > cloneOptions=Everything) const
	Create a copy of this TrackedSharedImageSet instance.

void	add (std::shared_ptr< SharedImage > img) override
	Add a shared image to the data set.

void	swapWithImageSet (SharedImageSet &is)
	Swaps all images, timestamps, and properties from `is`, but does not create any tracking sequence.

Geometry::AlignedBox	bounds () const override
	Returns the axis-aligned bounding box of all images in world space.

DataList	children () const override
	Exposes the tracking sequences if exposesTrackingAsChildren is on.

void	setTimestamp (double time, int which=-1) override
	Set specific or focused timestamp in seconds.

void	remove (int pos, int count=1) override
	Adjusts tracking info when images are removed.

void	removeBefore (double timestamp)
	Removes images and tracking samples before the passed timestamp.

void	addTracking (std::unique_ptr< TrackingSequence > ts)
	Add a tracking sequence, transferring ownership.

std::unique_ptr< TrackingSequence >	removeTracking (int num=-1)
	Removes a tracking sequence from the set.

void	clearTrackings ()
	Deletes all tracking sequences.

TrackingSequence *	tracking (int num=-1)
	Returns a tracking sequence If the desired sequence number is not specified, the currently active one is returned.

const TrackingSequence *	tracking (int num=-1) const
	Returns a tracking sequence If the desired sequence number is not specified, the currently active one is returned.

virtual void	setSize (double sizeX, double sizeY, bool doClearCaches=true)
	Set the physical image width and height in mm. Changes the spacing of each frame.

void	setUseTracking (int val)
	Specify which tracking sequence to use.

void	setUseTimestamps (bool val)
	Define if video-tracking timestamp synchronization is to be used.

int	numTracking () const
	Return the number of available tracking sequences.

bool	hasTransformations () const override
	Return true if the images have transformations.

virtual bool	canSetMatricesIndividually () const
	Return true if it is possible to set image matrices individually, i.e.

int	width () const
	Return the width of an image in pixels.

int	height () const
	Return the height of an image in pixels.

double	sizeX () const
	Return the horizontal extent of an image in mm.

double	sizeY () const
	Return the vertical extent of an image in mm.

int	trackingUsed () const
	Return number of active tracking sequence.

bool	useTimestamps () const
	Return whether timestamps are used for tracking synchronization.

void	clearCaches ()
	Will mark all caches as outdated. Called when the main tracking sequence updates.

bool	exposesTrackingAsChildren () const
	Returns whether the inner tracking sequences should be exposed through the CompoundData interface or not.

void	setExposeTrackingAsChildren (bool value)
	Sets whether the inner tracking sequences should be exposed through the CompoundData interface or not.

double	avgTimestepBetweenImageFrames () const
	Caclulate average timestep between frames.

double	medianTimestepBetweenImageFrames () const
	Calculate median timestep between frames.

void	clear () override
	Clears the tracking information and the cached information, resets all member variables to their default values.

void	clearSharedImageSet ()
	Internally calls SharedImageSet::clear, clearing image-data, time-stamps, properties, signals.

void	add (std::shared_ptr< Image > img)
	Add a single image to the data set.

void	add (std::shared_ptr< Image > img, const mat4 &m)
	Add a single image with its transformation matrix.

void	add (const std::vector< std::shared_ptr< SharedImage > > &images)
	Add a list of shared images.

Public Member Functions inherited from SharedImageSet
	SharedImageSet (std::shared_ptr< Image > img)
	Construct from a single Image.

	SharedImageSet (std::shared_ptr< SharedImage > img)
	Construct from a single SharedImage.

	SharedImageSet ()
	Construct an empty SharedImageSet.

SharedImageSet &	operator= (const SharedImageSet &)=delete

std::unique_ptr< SharedImageSet >	clone (Flags< CloneOptions > cloneOptions=Everything) const
	Create a copy of this SharedImageSet instance.

void	add (std::shared_ptr< Image > img)
	Add a single image to the data set.

void	add (std::shared_ptr< Image > img, const mat4 &m)
	Add a single image with its transformation matrix.

void	add (const std::vector< std::shared_ptr< SharedImage > > &images)
	Add a list of shared images.

bool	replace (int pos, std::shared_ptr< SharedImage > img)
	Replace a particular shared image entry, will not do anything if no image of that index exist yet.

void	remove (const SharedImage *img)
	Removes a specific image and deletes it if owned.

std::string	describe () const override
	Short description of the image set.

void	setModalityOverride (std::optional< Data::Modality > modalityOverride)
	Set a modalityOverride.

virtual void	setModality (Modality m)
	Set image modality for all frames.

Modality	modality () const override
	Return the modality of this data.

virtual bool	isMultiModal () const
	Return true if set contains images of different modalities, false otherwise.

virtual void	prepare (bool shiftOnly=false)
	Prepare the image in memory for most common scenarios Integral types are converted to unsigned representation if applicable, double-precision will be converted to single-precision float.

void	uploadGl () const
	Upload all images as OpenGL textures.

bool	hasMem (int which=-1) const
	Query whether an up-to-date memory image is present. If not, a sync is required.

bool	hasGl (int which=-1) const
	Query whether an up-to-date OpenGL image is present. If not, a sync is required.

const DataComponentList &	components () const
	Returns the list of DataComponents for this data.

DataComponentList &	components ()

const DataComponentList &	components (int i) const
	Returns the DataComponents associated to the element with that index.

virtual SharedImage *	get (int which=-1)
	Get specific or selected shared image.

std::shared_ptr< SharedImage >	getShared (int which=-1)
	Get specific or selected shared image.

MemImage *	mem (int which=-1)
	Get specific or selected image in memory If required image will be synchronized.

GlImage *	gl (int which=-1)
	Get specific or selected OpenGL image If required image will be synchronized.

const std::vector< std::shared_ptr< SharedImage > > &	images ()
	Return entire vector of images.

std::vector< std::shared_ptr< SharedImage > >	selectedImages (Selection::NonePolicy noneSelectionHandling)
	Return vector of all selected images.

virtual const SharedImage *	get (int which=-1) const
	Get specific or selected shared image.

std::shared_ptr< const SharedImage >	getShared (int which=-1) const
	Get specific or selected shared image.

const Image *	img (int which=-1) const
	Get specific or selected image descriptor.

const MemImage *	mem (int which=-1) const
	Get specific or selected image in memory If required image will be synchronized.

const GlImage *	gl (int which=-1) const
	Get specific or selected OpenGL image If required image will be synchronized.

std::vector< std::shared_ptr< const SharedImage > >	images () const
	Return entire vector of images.

std::vector< std::shared_ptr< const SharedImage > >	selectedImages (Selection::NonePolicy noneSelectionHandling) const
	Return vector of all selected images.

virtual void	setDeformation (std::shared_ptr< Deformation > def, int which=-1)
	Sets a deformation for specified or focus image.

virtual std::shared_ptr< Deformation >	deformation (int which=-1) const
	Returns the deformation for the specified or focused image, or 0 if not deformation is set.

virtual void	setMask (std::shared_ptr< Mask > mask, int which=-1)
	Sets a mask for specified or focus image.

virtual std::shared_ptr< Mask >	mask (int which=-1) const
	Returns the mask for the specified or focused image, or 0 if not mask is set.

virtual mat4	matrixFromWorld () const
	Get matrix mapping from the world coordinate system to the data coordinate system.

virtual mat4	matrixToWorld () const
	Get matrix mapping from the data coordinate system to the world coordinate system.

virtual void	setMatrixFromWorld (const mat4 &m)
	Set matrix mapping from the world coordinate system to the data coordinate system.

virtual void	setMatrixToWorld (const mat4 &m)
	Set matrix mapping from the data coordinate system to the world coordinate system.

double	timestamp (int which=-1) const
	Get specific or focused timestamp in seconds.

const std::vector< double >	timestamps () const
	Get the timestamps sequence.

bool	hasTimestamps () const
	Return if all image frames have timestamps.

int	size () const override
	Return the size of the image set.

bool	allEqual () const
	Return if all image descriptors are the same.

Kind	kind () const override
	Return the kind of data contained here.

Properties *	properties ()
	Return auxiliary data structures stored in Properties.

const Properties *	properties () const
	Return auxiliary data structures stored in Properties.

Public Member Functions inherited from Data
	Data (const std::string &name="")

	Data (const Data &other)

Data &	operator= (const Data &other)

virtual	~Data ()
	Mandatory virtual destructor.

const std::string &	name () const
	Return the name of this data.

void	setName (const std::string &name)
	Sets the name of this data.

virtual bool	isAnnotationType () const
	Return whether this data type is visualized through an annotation (e.g. mesh)

const DataComponentList &	components () const
	Returns the list of DataComponents for this data.

DataComponentList &	components ()

Public Member Functions inherited from SignalReceiver
	SignalReceiver ()=default
	Default constructor.

	SignalReceiver (const SignalReceiver &other)
	Copy constructor, does not copy any existing signal connections from other.

SignalReceiver &	operator= (SignalReceiver rhs)
	Assignment operator, disconnects all existing connections, does not copy any existing signal connections from rhs.

virtual	~SignalReceiver ()
	Virtual destructor disconnects from all connected signals.

Public Member Functions inherited from ElementwiseDataComponentData
ElementwiseDataComponentList &	elementwiseComponents ()
	Returns elementwise data component list.

const ElementwiseDataComponentList &	elementwiseComponents () const
	Returns elementwise data component list.

DataComponentList &	mutableComponents (int i)
	Returns the DataComponents associated to the element with that index, creating it if necessary.

const DataComponentList &	components (int i) const
	Returns the DataComponents associated to the element with that index.

Public Member Functions inherited from Selectable
	Selectable (const Selectable &)

	Selectable (Selectable &&)

Selectable &	operator= (const Selectable &)

Selectable &	operator= (Selectable &&)

virtual Selection	selection () const

virtual int	focus () const

virtual void	setSelection (const Selection &sel)
	Set the underlying selection.

virtual void	setFocus (int index)
	Set focus of the selection.

Public Attributes
Signal< const TrackedSharedImageSet * >	signalUsedTrackingSequenceChanged
	Signal emitted when the used tracking sequence changed or reset due to deletion.

Public Attributes inherited from Data
Signal< const Data * >	signalDeleted
	Signal emitted when this instance is deleted.

Signal< const Data * >	signalMatrixChanged
	Signal emitted when the transformation of this Data has changed.

Signal< std::string >	signalNameChanged
	Signal emitted when the name changed.

Public Attributes inherited from Selectable
ProtectedSignal	signalSelectionChanged
	Signals that the selection or size might have changed.

Public Attributes inherited from CompoundData
Signal< Data * >	signalChildAdded
	Signal emitted when a new Data instance has been added to children()

Signal< Data * >	signalChildMoved
	Signal emitted when a Data instance has changed its position in children()

Signal< Data * >	signalChildAboutToBeRemoved
	Signal emitted when a Data instance is about to be removed or taken from children()

Protected Member Functions
void	convertToGray (TypedImage< unsigned char > **img) const
	Convert a 3 or 4-channel image to gray-scale, see createGrayscale.

TrackedSharedImageSet *	createCloneContainer () const override
	Internal method for creating the container object holding the clone.

void	cloneInternal (SharedImageSet &other, Flags< CloneOptions > cloneFlags) const override
	Internal method for cloning which should be called from derived container classes.

Protected Member Functions inherited from SharedImageSet
virtual void	swapWith (SharedImageSet &other)
	Internal method for swapping the contents of this container with `other`.

Protected Member Functions inherited from Data
void	swapWith (Data &other)
	Swaps the data and emits a matrix and name changed signal for both.

Protected Member Functions inherited from SignalReceiver
void	disconnectAll ()
	Disconnects all existing connections.

Image matrix access
Pose::TransformationConvention	matrixConvention () const override

void	setMatrix (const mat4 &m, int which, bool updateAll=false) override
	Attempts to update tracking matrices.

mat4	matrix (int which) const override
	Get specific (`which` >=0) or focused (`which` ==-1) image matrix.

void	setMatrix (const mat4 &m) override
	This method just issues a warning since matrices cannot be edited here.

mat4	matrix () const override
	Get the focused image matrix.

void	setMatrixFromWorld (const mat4 &m, int which, bool updateAll=false) override
	Attempts to update tracking matrices.

void	setMatrixToWorld (const mat4 &m, int which, bool updateAll=false) override
	Attempts to update tracking matrices, directly calls setMatrix(const mat4&, int, bool).

mat4	matrixFromWorld (int which) const override
	Get specific (`which` >=0) or focused (`which` ==-1) image matrix mapping from the world coordinate system. Inverse of matrix.

mat4	matrixToWorld (int which) const override
	Get specific (`which` >=0) or focused (`which` ==-1) image matrix mapping to the world coordinate system. Same as matrix.

void	setRegistration (const mat4 &m)
	Sets the registration matrix on the active tracking sequence and invokes transformation listeners.

Additional Inherited Members
Public Types inherited from SharedImageSet
enum	CloneOptions { Everything = 0 , NoImageData = 1 << 0 , NoDataComponents = 1 << 1 , ShallowImageCopy = 1 << 2 , NoSelection = 1 << 3 , NoTracking = 1 << 4 }
	Bitset enumeration to configure which aspects of a SharedImageSet to copy. More...

Public Types inherited from Data
enum	Kind { UNKNOWN = 0 , IMAGE = 1 , VOLUME = 2 , IMAGESET = 3 , VOLUMESET = 4 , IMAGESTREAM = 5 , VOLUMESTREAM = 6 , POINTSET = 7 , SURFACE = 8 , TRACKINGSTREAM = 9 , LIVETRACKINGSTREAM = TRACKINGSTREAM , TRACKINGDATA = 10 , TREE = 11 , TENSOR = 12 , POLYDATASTREAM = 13 , STEREOIMAGESET = 14 , STEREOIMAGESTREAM = 15 , VOLUMETRICMESH = 16 }
	Kind of data. More...

enum	Modality { NA = 0 , XRAY = 1 , CT = 2 , MRI = 3 , ULTRASOUND = 4 , VIDEO = 5 , NM = 6 , OCT = 7 , LABEL = 8 , DISTANCE = 9 }
	Image modality of the data. More...

Static Public Member Functions inherited from Data
static std::string	modalityString (Data::Modality m)
	Return the name of an image modality.

static Data::Modality	stringToModality (const std::string &s)
	Returns the modality corresponding to a modality string or NA if nothing matches.

Protected Attributes inherited from SharedImageSet
std::vector< std::shared_ptr< SharedImage > >	m_images
	Vector of shared image pointers.

std::vector< double >	m_timestamps
	Optional timestamps of the image frames.

Properties	m_properties
	Auxiliary information properties.

Geometry::AlignedBox	m_bounds
	Axis-aligned bounding box of all images in world space.

bool	m_updateBounds = true
	Flag specifying whether the bounds should be re-computed.

std::optional< Data::Modality >	m_modalityOverride
	Overrides the modality of the containing SharedImages. This is used e.g. for the ConeBeamData concept in CT.

Protected Attributes inherited from Data
mat4	m_matrix
	Transformation matrix.

std::recursive_mutex *	m_matrixMutex
	Used to internally synchronize access to the matrix of data.

DataComponentList	m_dataComponentList
	The list of DataComponents for this data.

Protected Attributes inherited from ElementwiseDataComponentData
ElementwiseDataComponentList	m_elementwiseComponents
	Encodes elementwise data components.

Protected Attributes inherited from Selectable
std::mutex	m_selectionMutex
	Mutex protecting the selection.

Selection	m_sel

Member Function Documentation

◆ add()

void add ( std::shared_ptr< SharedImage > img )

overridevirtual

Add a shared image to the data set.

This method checks if the added images have matrix different from identity, and if so, assumes that the user is deliberately adding these images with matrices and wants to retain them. In many cases it is not trivial to understand unambiguously what the users wants given the images and tracking samples already there, and to keep images and TrackingSequence instances synchronized, so the method will in such cases check the following conditions:

The image set is not using timestamps.
The image set is empty and there is no TrackingSequence there (it will be created), or there is exactly one TrackingSequence with as many matrices as images in the set. If both conditions are met, the matrix is added to the TrackingSequence. It is taken as-is, and not inverted (note that TrackedSharedImageSet uses TOWORLD convention). Note that these criteria are stricter than canSetMatricesIndividually.

Reimplemented from SharedImageSet.

◆ swapWithImageSet()

void swapWithImageSet ( SharedImageSet & is )

Swaps all images, timestamps, and properties from is, but does not create any tracking sequence.

Used for transferring recorded sequence data into this data structure.

◆ bounds()

Geometry::AlignedBox bounds ( ) const

overridevirtual

Returns the axis-aligned bounding box of all images in world space.

Reimplemented from SharedImageSet.

Reimplemented in UltrasoundSweep.

◆ children()

DataList children ( ) const

overridevirtual

Exposes the tracking sequences if exposesTrackingAsChildren is on.

Implements CompoundData.

◆ matrixConvention()

Pose::TransformationConvention matrixConvention ( ) const

inlineoverridevirtual

Image matrices map from the coordinate system of the image frames to to the world coordinate system, internally using a TrackingSequence.

Reimplemented from SharedImageSet.

◆ setMatrix() [1/2]

void setMatrix	(	const mat4 &	m,
		int	which,
		bool	updateAll = false )

overridevirtual

Attempts to update tracking matrices.

If updateAll is true, will update the image set's registration matrix so that querying the matrix of image which (focus if -1) will return m afterwards.
If no timestamps are used, follows the same logic as add(shared_ptr<SharedImage>) and adds the matrix to the (potentially newly created) tracking sequence if one-to-one mapping is unambiguous.
If canSetMatricesIndividually is true, updates the tracking sequence's raw matrix so that querying the matrix of image \which will return m afterwards.
Does nothing and issues a warning otherwise.

Reimplemented from SharedImageSet.

◆ matrix() [1/2]

mat4 matrix ( int which ) const

overridevirtual

Get specific (which >=0) or focused (which ==-1) image matrix.

Reimplemented from SharedImageSet.

◆ setMatrix() [2/2]

void setMatrix ( const mat4 & m )

overridevirtual

This method just issues a warning since matrices cannot be edited here.

Reimplemented from SharedImageSet.

◆ matrix() [2/2]

mat4 matrix ( ) const

overridevirtual

Get the focused image matrix.

Reimplemented from SharedImageSet.

◆ setMatrixFromWorld()

void setMatrixFromWorld	(	const mat4 &	m,
		int	which,
		bool	updateAll = false )

overridevirtual

Attempts to update tracking matrices.

Calls internally setMatrix(const mat4&, int, bool) using the inverse of m.

Reimplemented from SharedImageSet.

◆ setMatrixToWorld()

void setMatrixToWorld	(	const mat4 &	m,
		int	which,
		bool	updateAll = false )

overridevirtual

Attempts to update tracking matrices, directly calls setMatrix(const mat4&, int, bool).

Reimplemented from SharedImageSet.

◆ matrixFromWorld()

mat4 matrixFromWorld ( int which ) const

overridevirtual

Get specific (which >=0) or focused (which ==-1) image matrix mapping from the world coordinate system. Inverse of matrix.

Reimplemented from SharedImageSet.

◆ matrixToWorld()

mat4 matrixToWorld ( int which ) const

overridevirtual

Get specific (which >=0) or focused (which ==-1) image matrix mapping to the world coordinate system. Same as matrix.

Reimplemented from SharedImageSet.

◆ setTimestamp()

void setTimestamp	(	double	time,
		int	which = -1 )

overridevirtual

Set specific or focused timestamp in seconds.

Reimplemented from SharedImageSet.

◆ remove()

void remove	(	int	pos,
		int	count = 1 )

overridevirtual

Adjusts tracking info when images are removed.

Reimplemented from SharedImageSet.

◆ removeTracking()

std::unique_ptr< TrackingSequence > removeTracking ( int num = -1 )

Removes a tracking sequence from the set.

If no tracking number is specified, the currently active tracking is removed.

◆ setSize()

virtual void setSize	(	double	sizeX,
		double	sizeY,
		bool	doClearCaches = true )

virtual

Set the physical image width and height in mm. Changes the spacing of each frame.

Reimplemented in UltrasoundSweep.

◆ hasTransformations()

bool hasTransformations ( ) const

overridevirtual

Return true if the images have transformations.

Reimplemented from SharedImageSet.

◆ canSetMatricesIndividually()

virtual bool canSetMatricesIndividually ( ) const

virtual

Return true if it is possible to set image matrices individually, i.e.

when there is a one-to-one mapping between images and matrices contained in the tracked shared image set. This is the case if there as as many images as tracking samples, and if either no timestamps are used at all, or the timestamps match, respectively.

◆ clear()

void clear ( )

overridevirtual

Clears the tracking information and the cached information, resets all member variables to their default values.

Note: call TrackedSharedImageSet::clearSharedImageSet or SharedImageSet::clear to retain the legacy behavior

Reimplemented from SharedImageSet.

◆ createCloneContainer()

TrackedSharedImageSet * createCloneContainer ( ) const

overrideprotectedvirtual

Internal method for creating the container object holding the clone.

Reimplemented from SharedImageSet.

Reimplemented in UltrasoundSweep.

◆ cloneInternal()

void cloneInternal	(	SharedImageSet &	other,
		Flags< CloneOptions >	cloneOptions ) const

overrideprotectedvirtual

Internal method for cloning which should be called from derived container classes.

Reimplemented from SharedImageSet.

The documentation for this class was generated from the following file:

ImFusion/GL/TrackedSharedImageSet.h

Detailed Description

Public Member Functions

Public Attributes

Protected Member Functions

Image matrix access

Additional Inherited Members

Member Function Documentation

◆ add()

◆ swapWithImageSet()

◆ bounds()

◆ children()

◆ matrixConvention()

◆ setMatrix() [1/2]

◆ matrix() [1/2]

◆ setMatrix() [2/2]

◆ matrix() [2/2]

◆ setMatrixFromWorld()

◆ setMatrixToWorld()

◆ matrixFromWorld()

◆ matrixToWorld()

◆ setTimestamp()

◆ remove()

◆ removeTracking()

◆ setSize()

◆ hasTransformations()

◆ canSetMatricesIndividually()

◆ clear()

◆ createCloneContainer()

◆ cloneInternal()