Stereo Reconstruction

Guide to 3D reconstruction from a sequence of stereo images using the StereoReconstructionAlgorithm class.

Collaboration diagram for Stereo Reconstruction:

Guide to 3D reconstruction from a sequence of stereo images using the StereoReconstructionAlgorithm class.

Stereo Reconstruction Overview

Stereo reconstruction is a computer vision technique that estimates the 3D structure of a scene from two images taken from different viewpoints. By analyzing the disparity (difference in pixel positions) between corresponding points in stereo images, the algorithm can compute depth information and generate 3D reconstruction.

The StereoReconstructionAlgorithm class provides a high-level interface for stereo reconstruction, while the underlying StereoReconstruction class handles the core reconstruction logic.

Basic Stereo Reconstruction Usage

Basic Stereo Reconstruction

The following example demonstrates basic usage of the StereoReconstructionAlgorithm where we assume the input is

1. Left and right stereo images given as leftImages, rightImages of type std::unique_ptr<SharedImageSet>.
2. Left and right intrinsic matrices given as leftK and rightK of type Eigen::Matrix3d
3. Transformation of right camera w.r.t. the left camera given as registration of type Eigen::Matrix4d

#include <ImFusion/Base/SharedImageSet.h>
#include <ImFusion/Base/DataList.h>
#include <ImFusion/Vision/CameraCalibrationDataComponent.h>
#include <ImFusion/Vision/StereoReconstructionAlgorithm.h>
 
using namespace ImFusion;
 
// Attach camera calibration information
auto& leftCalib = leftImages->components().getOrCreate<CameraCalibrationDataComponent>();
leftCalib.p_K = leftK;
leftCalib.p_distortion = dist;
leftCalib.p_camToWorldT = Eigen::Matrix4d::Identity(); // left camera as origin.
 
auto& rightCalib = rightImages->components().getOrCreate<CameraCalibrationDataComponent>();
rightCalib.p_K = rightK;
rightCalib.p_distortion = dist;
rightCalib.p_camToWorldT = registration; // right camera pose w.r.t., left camera. 
 
// Create a StereoSharedImageSet
auto stereoSet = StereoSharedImageSet::create(std::move(leftImages), std::move(rightImages));
 
// Create stereo reconstruction algorithm
StereoReconstructionAlgorithm stereoReconstructor(stereoSet.get());
 
// Configure algorithm parameters
stereoReconstructor.p_stereoReconstructionName = "Stereo RAFT";  // Use learning based stereo RAFT approach
stereoReconstructor.p_exportRectifiedImages = true;              // Export rectified stereo images
stereoReconstructor.p_exportPointClouds = true;                  // Generate 3D point clouds
 
// Run stereo reconstruction
stereoReconstructor.compute();
 
// Extract results
OwningDataList output = stereoReconstructor.takeOutput();
 
// Access disparity and depth images directly from the algorithm
auto outputImages = output.extractAllImages();
 
std::unique_ptr<SharedImageSet> disparityResult = nullptr;
std::unique_ptr<SharedImageSet> depthResult = nullptr;
 
for (auto& output : outputImages)
{
    if (output)
    {
        if (output->name() == "Disparity")
            disparityResult = std::move(output);
        else if (output->name() == "Depth")
            depthResult = std::move(output);
    }
}
 
 
// Process point clouds if enabled
std::vector<std::unique_ptr<PointCloud>> pointCloudsResult;
 
if (stereoReconstructor.p_exportPointClouds) {
    auto pointClouds = outputs.extractAll<PointCloud>(Data::POINTSET);
    for (auto& pc : pointClouds)
    {
        if (pc)
            pointCloudsResult.emplace_back(std::move(pc));
    }
}
// disparityResult contains the disparity
// depthResult contains the depth
// pointCloudsResult contains the reconstructed point cloud

See also

StereoReconstructionAlgorithm

Custom Models

StereoReconstruction allow running your custom models through our machine learning engines.

This is done by tracing or scripting your Torch or exporting your ONNX model and linking it through a configuration file.

Here is an example configuration file in .yaml format for StereoReconstruction:

Version: '8.0'
Type: NeuralNetwork
Name: CustomModel
Description: Stereo Reconstruction
Engine:
  Name: torch
  DisableJITRecompile: true
  ModelFile: <path_to_model_file>
  ForceCPU: false
  Verbose: true
  InputFields: [LeftImage, RightImage]
  OutputFields: [Disparity]
PredictionOutput: [Image]
PreprocessingInputFields: [FirstImage, SecondImage]
PreProcessing:
  - MakeFloat: {}
Sampling:
  - SkipUnpadding: true

We require InputFields and OutputFields to have exactly the same names as mentioned above. In our current release, we only support the Depth output to be exposed to the user, returned as the first element by the model. Any additional outputs should still be appended in OutputFields and PredictionOutput with its return type but they will be discarded.

For further information on the machine learning configuration files, please refer to the Machine Learning Model page in our user documentation.