Stereo Structure

Comprehensive guide to Stereo Image.

Collaboration diagram for Stereo Structure:

Comprehensive guide to Stereo Image.

A stereo camera takes two pictures at the same time from slightly different angles. By comparing these pictures, the camera can understand depth and distance. This page provides detailed information about our stereo structure and its application.

Stereo Structure Overview

A binocular camera, also known as a stereo camera, is an imaging system equipped with two lenses that simulate human binocular vision. By capturing two images from slightly different perspectives, the camera can calculate depth information and reconstruct three-dimensional scenes. This technology is widely used in fields such as robotics, autonomous driving, augmented reality, and industrial inspection. Compared with single-lens cameras, binocular cameras provide richer spatial information, enabling accurate distance measurement, object detection, and environment mapping.

In order to facilitate the utilization of dual images, we introduces a stereo image and stream structure. They are primarily responsible for synchronizing images and storing information related to the stereo camera. We recommend its use in stereo vision applications.

Stereo image

A StereoImage stores a pair of images, taken by a stereo camera at the same time point. It provides access to left and right image individually. It can also be cloned.

Stereo Shared Image Set

A StereoSharedImageSet is the main high-level container for stereo image data set. It should also have a StereoCalibrationDataComponent. A StereoSharedImageSet can be only created through the static create function. Both input image set should have same size. It provides access to both left and right image individually, as well as a clone function. It can also be extended by adding StereoImage. The following example demonstrates how to create it

#include <ImFusion/Base/ImFusionFile.h>
#include <ImFusion/Base/SharedImageSet.h>
#include <ImFusion/Vision/StereoSharedImageSet.h>
#include <ImFusion/Vision/StereoReconstruction.h>
 
using namespace ImFusion;
 
// Load a file with two shared image sets
// User can also load then seperately
ImFusionFile file("TwoImage.imf");
std::optional<OwningDataList> dataList = file.load();
if(!dataList.has_value())
    return;
auto images = dataList->extractAllImages();
if(images.size()!=2)
    return;
// create a StereoSharedImageSet
std::unique_ptr<StereoSharedImageSet>  ssisPtr = StereoSharedImageSet::create(std::move(images[0]), std::move(images[1]));
// always check whether the creation was successful
if(!ssisPtr)
    return;
// save StereoSharedImageSet to imfusion file
ImFusionFile fileOut("stereoImage.imf");
DataList out;
out.add(ssisPtr.get());
fileOut.save(out);

One can also use CreateStereoSharedImageSetAlgorithm to create StereoSharedImageSet.

Stereo calibration data component

A StereoCalibrationDataComponent stores the left to right camera registration matrix between a stereo camera pair. It also supports save and load functions, similar to CameraCalibrationDataComponent

Stereo calibration, rectification and reconstruction

In this section, we introduce three applications of stereo cameras and explain how the StereoSharedImageSet is used in each. Stereo calibration estimates the intrinsic parameters of each camera and the relative position and orientation between them. Rectification aligns the two images onto the same plane, simplifying the search for corresponding points. Finally, 3D reconstruction uses these correspondences to compute depth and generate a three-dimensional representation of the scene. Together, these processes form the foundation of accurate stereo vision systems, widely used in robotics, autonomous driving, and augmented reality.

Stereo calibration

Stereo calibration is the process of determining the intrinsic parameters of each camera (such as focal length and lens distortion) and the extrinsic parameters that describe the relative position and orientation between the two cameras. Accurate calibration is essential for stereo vision systems, as it ensures that the depth and 3D measurements derived from the image pair are precise. This step forms the foundation for subsequent processes like rectification and 3D reconstruction. In ImFusion, a stereo calibration consists two steps: camera calibration for each camera and camera registration. To obtain better calibration results, we also need to follow calibration guidelines, such as capturing clear images, maintaining an appropriate working distance, and ensuring the calibration board appears at multiple angles in the images. For more details, please refer to CameraCalibration. Here we focus on how to conduct camera registration in ImFusion, The following example demonstrates it.

#include <ImFusion/Base/ImFusionFile.h>
#include <ImFusion/Base/SharedImageSet.h>
#include <ImFusion/Vision/CameraRegistrationAlgorithm.h>
#include <ImFusion/Vision/StereoSharedImageSet.h>
 
using namespace ImFusion;
 
// Load a file with two shared image sets
// User can also load then seperately
ImFusionFile file("stereoImage.imf");
std::optional<OwningDataList> dataList = file.load();
if(!dataList.has_value())
    return;
auto images = dataList->extractAllImages();
if(images.size()!=2)
    return;
 
// For CameraRegistration it is important that both camera are already calibrated
auto& camLeft = images[0]->components().getOrCreate<CameraCalibrationDataComponent>();
camLeft.load("camLeft.xml");
auto& camRight = images[1]->components().getOrCreate<CameraCalibrationDataComponent>();
camRight.load("camRight.xml");
std::unique_ptr<StereoSharedImageSet>  ssisPtr = StereoSharedImageSet::create(std::move(images[0]), std::move(images[1]));
if(!ssisPtr)
    return;
 
// configure markers and calibration settings
// please refer to camera calibration for more details  
MarkerConfiguration config;
CameraCalibrationSettings settings;
 
CameraRegistrationAlgorithm alg(ssisPtr.get());
alg.setCalibrationSettings(settings);
alg.setMarkerConfig(config);
alg.compute();
auto resultImg = alg.takeOutput().extractFirst<StereoSharedImageSet>();
if(!resultImg)
    return;
auto reg = resultImg->components().get<StereoCalibrationDataComponent>();
if(!reg)
    return;
reg->save("registration.xml");

We also support two seperate image sets as input. The registration is then stored as a Paremeter p_camToWorldT in the right image set and the position of left camera is assumed to be identity.

stereo rectification

Stereo rectification is the process of transforming these two images so that their corresponding points appear on the same horizontal lines. This makes the search for matches much simpler and more efficient, since we only need to look along one direction instead of across the whole image.By rectifying images, we can create more accurate depth maps and 3D reconstructions. The following example demonstrates how conduct stereo rectification.

#include <ImFusion/Vision/StereoSharedImageSet.h>
#include <ImFusion/Vision/StereoRectification.h>
#include <ImFusion/Vision/CameraCalibrationDataComponent.h>
 
using namespace ImFusion;
 
// Load a file with two shared image sets
// User can also load then seperately
// Both images should contain calibration and the registration information
ImFusionFile file("stereoImage.imf");
std::optional<OwningDataList> dataList = file.load();
if (!dataList.has_value())
    return;
auto images = dataList->extractAll<StereoSharedImageSet>(Data::STEREOIMAGESET);
if (images.empty())
    return;
if (!images[0]->leftCalibration() || !images[0]->rightCalibration() || !images[0]->registration())
    return;
StereoRectification rect(
    images[0]->leftCalibration().value(), images[0]->rightCalibration().value(), images[0]->registration().value(), 1.0, 0);

auto imgRectLeft = rect.rectify(*images[0]->leftImageSet(), StereoImage::Side::Left);
auto imgRectRight = rect.rectify(*images[0]->rightImageSet(), StereoImage::Side::Right);
if(!imgRectLeft || !imgRectRight)
    return;

One can also use StereoRectificationAlgorithm to produce the same result.

stereo reconstruction

Stereo reconstruction is a method used to recover the 3D structure of a scene from two or more images taken from different viewpoints. Just like human eyes use slightly different perspectives to perceive depth, cameras can capture overlapping views that allow us to estimate the distance of objects. The main idea is to find matching points in the images and calculate how much they shift, known as disparity. From this disparity, the depth of each point can be estimated, and a 3D model of the scene can be built. In ImFusion we provide four different methods: Stereo Block Matching, Stereo Semi Global Block Matching, CGI and RAFT

#include <ImFusion/Base/ImFusionFile.h>
#include <ImFusion/Base/SharedImageSet.h>
#include <ImFusion/Core/Filesystem/Path.h>
#include <ImFusion/Vision/CameraCalibrationAlgorithm.h>
#include <ImFusion/Vision/CameraCalibrationDataComponent.h>
#include <ImFusion/Vision/CameraCalibrationSettings.h>
#include <ImFusion/Vision/CameraRegistrationAlgorithm.h>
#include <ImFusion/Vision/StereoReconstructionAlgorithm.h>
#include <ImFusion/Vision/StereoRectification.h>
#include <ImFusion/Vision/StereoSharedImageSet.h>
#include <ImFusion/Vision/StereoUtils.h>
 
using namespace ImFusion;
 
// Load a file with two shared image sets
// User can also load then seperately
// Both images should contain calibration and the registration information
ImFusionFile file("stereoImage.imf");
std::optional<OwningDataList> dataList = file.load();
if (!dataList.has_value())
    return;
auto images = dataList->extractAll<StereoSharedImageSet>(Data::STEREOIMAGESET);
if (images.empty())
    return;
if (!images[0]->leftCalibration() || !images[0]->rightCalibration() || !images[0]->registration())
    return;
StereoRectification rect(
    images[0]->leftCalibration().value(), images[0]->rightCalibration().value(), images[0]->registration().value(), 1.0, 0);
auto rectInfo = rect.rectificationInformation();
 
// We provide four different reconstruction methods, default RAFT
auto stereo = StereoReconstructionFactory::get().create("Stereo RAFT");
if (!stereo)
    return;

auto imgDepth = std::make_unique<SharedImageSet>();
auto imgDisp = std::make_unique<SharedImageSet>();
auto imgRectLeft = rect.rectify(*images[0]->leftImageSet(), StereoImage::Side::Left);
auto imgRectRight = rect.rectify(*images[0]->rightImageSet(), StereoImage::Side::Right);
 
for (int i = 0; i < images[0]->size(); ++i)
{
    auto imgDispTmp = stereo->compute(*imgRectLeft->get(i), *imgRectRight->get(i));
    if (!imgDispTmp)
        return;
    auto imgDepthTmp = StereoUtils::computeDepthFromDisparity(*imgDispTmp, rectInfo.Q, nullptr, -1);
    if (!imgDepthTmp)
        return;
    imgDepth->add(std::move(imgDepthTmp));
}

One can also use StereoReconstructionAlgorithmAlgorithm to produce the same result.

See also

StereoSharedImageSet, StereoCalibrationDataComponent, CameraRegistrationAlgorithm, StereoRectificationAlgorithm, StereoReconstructionAlgorithm