Multi Modal Object Tracking

Tracks rigid object motion through a sequence of registered RGB-D frames using sparse features, optical flow, and 3D correspondence refinement.

Input

Two or three image sets (same frame count and image size):

  • Color — RGB UByte images, at least two frames; frame 0 is the reference.

  • Depth — single-channel Float depth maps aligned with the color frames.

  • Mask (optional) — single-channel label image; when provided, only features on the labeled object (default label 1) are used.

Each color and depth set needs a camera calibration data component (or equivalent RGB-D calibration) so depth can be lifted to 3D.

Output

Registered point clouds and object poses relative to the reference frame. The controller can show live color, depth, and tracking views while processing runs in the background.

Description

For each frame after the reference, the algorithm detects features (optionally restricted by the mask), matches them to the previous keyframe window, lifts matches to 3D using depth, and estimates a rigid transform with RANSAC Kabsch refinement. Optical flow propagates features between keyframes. A reference mesh can be built from the tracked object for visualization.

Press Compute to start sequential processing; press again to stop an active run.

Tracking parameters

Early stopping match count percentage threshold — fraction of reference keypoints matched before the keyframe search stops early.

Minimum match count — minimum number of accepted feature matches required to track a frame.

Max keyframe count — maximum keyframes kept in the sliding window.

Optical flow uncertainty threshold — discard optical-flow correspondences above this uncertainty.

RANSAC Threshold — inlier distance threshold (millimeters) for the Kabsch RANSAC loop.

Max RANSAC iterations — upper bound on RANSAC iterations per frame.

Crop point clouds to label — when a mask is used, limit output point clouds to labeled pixels.

Detector, matcher, and pruner settings

The properties panel exposes three nested groups from the Feature Detection pipeline. They run on each keyframe comparison before matches are lifted to 3D and passed to the RANSAC step above.

RIDE with Mask Detector Parameters

Learning-based RIDE keypoints and descriptors (see the RIDE entry under Feature Detection). When a mask input is provided, detection is limited to pixels with Mask label (default 1).

  • Max features — upper bound on keypoints per frame (tracking default 4000).

  • Rotation invariance — rotate descriptors to a canonical orientation (tracking default: off).

  • Dense non maximum suppression — suppress nearby keypoints on a dense grid (tracking default: on).

  • Dense non maximum suppression radius — NMS radius in pixels (tracking default 3).

Matcher Parameters

Brute-force descriptor matching between the current frame and a keyframe in the sliding window. See the Brute Force matcher section in Feature Detection.

  • Norm — distance measure for descriptor comparison (tracking default: DotProduct, suited to RIDE).

  • Cross check — keep a match only if each keypoint is the other’s best match (tracking default: on).

  • Ratio threshold — Lowe ratio test; matches above this ratio are rejected.

Pruner Parameters

Grid-based motion statistics (GMS) pruning to remove matches inconsistent with a smooth 2D motion model. See the Grid-based Motion Statistics section in Feature Detection.

  • Consider rotation — allow rotational motion when scoring match consistency.

  • Consider scale — allow scale change in the motion model.

  • Threshold factor — multiplied with the median motion statistic to set the inlier cutoff; higher values prune more aggressively (tracking default 3, lower than the standalone Feature Detection default).