Multi-View Reconstruction (MapAnything)

Guide to 3D reconstruction from unposed RGB images using the MapAnything model.

Collaboration diagram for Multi-View Reconstruction (MapAnything):

Guide to 3D reconstruction from unposed RGB images using the MapAnything model.

Overview

The MultiViewReconstructionMapAnythingAlgorithm provides 3D reconstruction from N >= 1 unposed RGB images. Given images and their pinhole intrinsics, the underlying MapAnything TorchScript model jointly predicts:

• Per-pixel metric Z-depth maps
• Camera-to-world 4x4 poses
• Recovered intrinsics at the output resolution
• Per-pixel confidence scores
• Binary validity masks

The algorithm outputs depth maps and coloured world-space point clouds. Both are uniformly scaled by 100 relative to the model's metric output so the reconstruction fits the display coordinate system better.

Usage

The following example demonstrates basic usage of the MultiViewReconstructionMapAnythingAlgorithm. The input is a SharedImageSet containing N >= 1 same-resolution RGB images with an attached CameraCalibrationDataComponent.

#include <ImFusion/Base/DataList.h>
#include <ImFusion/Base/PointCloud.h>
#include <ImFusion/Base/SharedImageSet.h>
#include <ImFusion/Vision/CameraCalibrationDataComponent.h>
#include <ImFusion/Vision/MultiViewReconstructionMapAnythingAlgorithm.h>
 
using namespace ImFusion;
 
// Assume inputImages is a std::unique_ptr<SharedImageSet> with N >= 2 RGB images.
// Attach camera intrinsics:
auto calib = std::make_unique<CameraCalibrationDataComponent>();
calib->p_K = K;  // 3x3 pinhole intrinsic matrix for the original image resolution
inputImages->components().add(std::move(calib));
 
// Create and configure the algorithm
MultiViewReconstructionMapAnythingAlgorithm algorithm(inputImages.get());
algorithm.p_exportPointClouds = true;   // produce world-space point clouds
 
// Run reconstruction
algorithm.compute();
 
// Extract results
OwningDataList output = algorithm.takeOutput();
 
// Depth maps (SharedImageSet with N float images, Z-depth scaled for display)
auto depthMaps = output.extractFirstImage();
 
// Point clouds (one per view, in world space; same display scaling as depth maps)
auto pointClouds = output.extractAll<PointCloud>(Data::POINTSET);

Model Details

The MapAnything model is a TorchScript module loaded via the ImFusion ML framework. It expects two inputs:

• images: float32 tensor of shape (N, 3, H, W) with RGB pixel values in [0, 255]
• intrinsics: float32 tensor of shape (N, 3, 3) with standard pinhole matrices

The model internally resizes to 518x518 and maps outputs back to the input resolution. Any input resolution is accepted, and all views must share the same resolution.

Output coordinate conventions:

• Depth: Z-depth along the optical axis, in display units (model's metric depth multiplied by 100)
• Poses: 4x4 camera-to-world transformation matrices, with translations multiplied by 100 to match the scaled depth maps
• Camera frame: OpenCV convention (Z forward, X right, Y down)

Low-Level Access

For direct access to all model outputs (including confidence and mask tensors), use the MultiViewReconstructionMapAnything utility class:

#include <ImFusion/Vision/MultiViewReconstructionMapAnything.h>
 
MultiViewReconstructionMapAnything reconstruction;
auto result = reconstruction.compute(images);
if (result)
{
    // result->cameraPoses    — ML::Tensor (N, 4, 4)
    // result->depthZ         — ML::Tensor (N, H, W, 1)
    // result->intrinsicsOut  — ML::Tensor (N, 3, 3)
    // result->confidence     — ML::Tensor (N, H, W)
    // result->mask           — ML::Tensor (N, H, W)
}

See also

MultiViewReconstructionMapAnythingAlgorithm, MultiViewReconstructionMapAnything, MapAnythingResult