Marker Hand-Eye Calibration

Calibrates the rigid transform between a camera viewing a calibration board and a tracker (for example a robot end effector) that moves rigidly with the sensor.

Input

One of the following:

  • Monocular — an image set showing a calibration board together with a tracking sequence, or a tracked shared image set that already combines images and tracker poses.

  • Stereo — a stereo image set (left and right views) together with a tracking sequence.

Images must show a board configured in marker configuration. Per-eye camera calibration data component is required for monocular and stereo input. Stereo input additionally needs a stereo camera calibration data component on the stereo data.

Output

The computed hand-eye transform (camera to tracker target) and registration (calibration board to tracker) are written to the log. Optionally, output tracking sequences with the calibration applied.

Description

The workflow is similar to hand-eye calibration: relate a moving sensor frame to a hand (tracker) frame using synchronized poses. Here, camera poses are estimated from detected calibration markers instead of supplying a separate camera tracking sequence.

Configure the calibration board, then run the algorithm. The controller can overlay marker detections and reprojections on the images to assess quality. For stereo data, separate overlays are available for the left and right camera as well as stereo-only reprojection checks.

Calibration type — Eye in Hand or Eye on Base; see hand-eye calibration for the coordinate-frame layout.

Stream info tracker — defines how the tracker sequence transforms are interpreted (for example Hand to Base).

Match timestamps — when enabled, pairs camera and tracker samples by timestamp; otherwise pairs by frame index.

Optimize scale — optimizes a scale factor for the physical marker board size when detections allow it.

Optimize scale using stereo reprojection — for stereo input with Optimize scale enabled, refines board scale from left/right reprojection consistency instead of monocular scale optimization alone.

Marker detection and frame selection

Marker max reprojection error — maximum mean marker reprojection error (in pixels) for a frame to be used. Frames above this threshold are treated as invalid when building the camera tracking sequence from detections.

Marker min detections — minimum number of detected marker corners required in a frame for that frame to contribute to calibration.

Hand-eye computation

These options are passed to the internal hand-eye calibration step (see hand-eye calibration for the underlying equations):

Min relative angle — minimum angular difference between two pose samples for them to form a valid pair in the calibration.

Pose inlier threshold — maximum pose-consistency error for a sample pair to count as an inlier.

Use RANSAC — when enabled, repeatedly estimates calibration from random pose pairs and keeps the best result.

Non-linear refinement — refines the hand-eye result with a non-linear optimization on pose consistency.

Refinement and output

Refine MRE (not pose) — when enabled, refines hand-eye calibration, registration, and (optionally) scale or time offset by minimizing marker reprojection error rather than pose error alone.

Refine time offset — when Match timestamps and Refine MRE (not pose) are enabled, also optimizes the temporal offset between camera and tracker streams.

Apply calibration to input — when enabled, applies the computed hand-eye calibration to the input tracked image set after a successful run.

Visualization

The controller can overlay Detections, Reprojections (Camera), Reprojections (Hand-Eye), and (for stereo) Reprojections (Stereo), as well as optional views of the camera and tracker tracking streams before and after calibration.