Custom RGBD Stream

Comprehensive guide to implementing a CustomRGBDStream with the ImFusion RGB-D Plugin.

Collaboration diagram for Custom RGBD Stream:

Comprehensive guide to implementing a CustomRGBDStream with the ImFusion RGB-D Plugin.

This page provides step-by-step instructions and code snippets for implementing your own RGBDStream subclass to interface with custom or unsupported RGB-D sensors. The example below demonstrates how to implement the required methods for device enumeration, format selection, calibration, and frame acquisition.

Overview

To integrate a new RGB-D sensor, derive a class from RGBDStream and implement all required virtual methods. This enables your sensor to be used seamlessly with ImFusion's algorithms and visualization tools.

Implementation Steps

Class Definition

Start by defining your class and inheriting from RGBDStream:

class CustomRGBDStream : public ImFusion::RGBDStream
{
public:
    CustomRGBDStream();
    ~CustomRGBDStream() override;
 
    static std::vector<std::unique_ptr<CustomRGBDStream>> enumerate();
 
    // Format selection
    bool depthFormats(std::vector<FrameInfo>& formats) override;
    bool colorFormats(std::vector<FrameInfo>& formats) override;
    bool depthFormat(FrameInfo& format) const override;
    bool colorFormat(FrameInfo& format) const override;
    bool setDepthFormat(const FrameInfo& format) override;
    bool setColorFormat(const FrameInfo& format) override;
 
    // Sensor properties
    bool supportsColor() const override;
    std::string name() const override;
    std::string uuid() override;
    std::string serialNumber() const override;
 
    // Image sizes
    int depthImageWidth() const override;
    int depthImageHeight() const override;
    int colorImageWidth() const override;
    int colorImageHeight() const override;
 
    // Calibration
    bool depthK(mat3& K) const override;
    bool depthDist(Eigen::Matrix<double, 5, 1>& dist) const override;
    bool colorK(mat3& K) const override;
    bool colorDist(Eigen::Matrix<double, 5, 1>& dist) const override;
    bool depthToColorT(mat4& T) const override;
 
protected:
    bool openImpl() override;
    bool closeImpl() override;
    bool startImpl() override;
    bool stopImpl() override;
    std::optional<WorkContinuation> doWork() override;
    bool defaultColorResolution(int& width, int& height) override;
    bool defaultDepthResolution(int& width, int& height) override;
 
private:
    // Internal state and buffers
    FrameInfo m_colorFormat;
    FrameInfo m_depthFormat;
    mat3 m_depthK;
    vec5 m_depthDist;
    mat3 m_colorK;
    vec5 m_colorDist;
    mat4 m_depthToColorT;
    int m_frame = 0;
    std::shared_ptr<TypedImage<unsigned char>> m_colorImg;
    std::shared_ptr<TypedImage<float>> m_depthImg;
    bool m_running = false;
};

Device Enumeration

Implement a static method to enumerate available devices:

std::vector<std::unique_ptr<CustomRGBDStream>> CustomRGBDStream::enumerate()
{
    std::vector<std::unique_ptr<CustomRGBDStream>> streams;
    // Use sensor-specific code to discover devices
    streams.push_back(std::make_unique<CustomRGBDStream>());
    return streams;
}

Format Support

Report supported formats and allow selection:

bool CustomRGBDStream::depthFormats(std::vector<FrameInfo>& formats)
{
    formats.clear();
    formats.push_back(m_depthFormat);
    return true;
}
 
bool CustomRGBDStream::colorFormats(std::vector<FrameInfo>& formats)
{
    formats.clear();
    formats.push_back(m_colorFormat);
    return true;
}
 
bool CustomRGBDStream::setDepthFormat(const FrameInfo& format)
{
    // Configure the sensor if needed
    return format == m_depthFormat;
}
 
bool CustomRGBDStream::setColorFormat(const FrameInfo& format)
{
    // Configure the sensor if needed
    return format == m_colorFormat;
}

Calibration and Metadata

Implement methods to provide camera intrinsics, distortion, and transformation:

bool CustomRGBDStream::depthK(mat3& K) const { K = m_depthK; return true; }
bool CustomRGBDStream::colorK(mat3& K) const { K = m_colorK; return true; }
bool CustomRGBDStream::depthDist(Eigen::Matrix<double, 5, 1>& dist) const { dist = m_depthDist; return true; }
bool CustomRGBDStream::colorDist(Eigen::Matrix<double, 5, 1>& dist) const { dist = m_colorDist; return true; }
bool CustomRGBDStream::depthToColorT(mat4& T) const { T = m_depthToColorT; return true; }

Lifecycle Methods

Implement open/close/start/stop to manage device resources:

bool CustomRGBDStream::openImpl()
{
    // Open the sensor and retrieve calibration
    return true;
}
 
bool CustomRGBDStream::closeImpl()
{
    // Release resources
    return true;
}
 
bool CustomRGBDStream::startImpl()
{
    m_running = true;
    // Start frame acquisition
    return true;
}
 
bool CustomRGBDStream::stopImpl()
{
    m_running = false;
    // Stop frame acquisition
    return true;
}

Frame Acquisition

Implement the main frame acquisition loop in doWork():

std::optional<WorkContinuation> CustomRGBDStream::doWork()
{
    // Wait for or acquire color and depth images from your sensor
    // Fill m_colorImg and m_depthImg with new data
 
    RGBDFrame frame(this, m_colorImg.get(), m_depthImg.get(), nullptr, nullptr);
    frame.setTimestampArrival(...);
    frame.setTimestampColor(...);
    frame.setTimestampDepth(...);
    frame.components().add<RGBDDataComponent>(getRGBDDataComponentCopy());
 
    signalNewData.emitSignal(frame);
 
    m_frame++;
    return WorkContinuation{std::nullopt};
}

Default Resolutions

Implement default resolution queries:

bool CustomRGBDStream::defaultColorResolution(int& width, int& height)
{
    width = m_colorFormat.width;
    height = m_colorFormat.height;
    return true;
}
 
bool CustomRGBDStream::defaultDepthResolution(int& width, int& height)
{
    width = m_depthFormat.width;
    height = m_depthFormat.height;
    return true;
}

Tips

• Always provide accurate calibration and metadata for best reconstruction results.
• Use the RGBDDataComponent to store sensor-specific information.
• Handle errors and edge cases in all lifecycle methods.

See also

CustomRGBDStream Class

RGBDStream Base Class