Optical Flow

Guide to optical flow estimation from RGB image sequences.

Collaboration diagram for Optical Flow:

Guide to optical flow estimation from RGB image sequences.

Optical Flow Overview

Optical flow is a computer vision technique that estimates the motion of objects between consecutive frames in an image sequence. By analyzing pixel-level movement, the algorithm can compute motion vectors that represent how each pixel moves from one frame to the next.

The OpticalFlowAlgorithm class provides a high-level interface for optical flow computation, while the underlying OpticalFlow class handles the core flow estimation logic.

Optical Flow Usage

Basic Optical Flow Computation

The following example demonstrates basic usage of the OpticalFlowAlgorithm where we assume as input we have a sequence of RGB images with name imageSequence represented as std::unique_ptr<SharedImageSet>.

#include <ImFusion/Base/SharedImageSet.h>
#include <ImFusion/Vision/OpticalFlowAlgorithm.h>
#include <ImFusion/Base/DataList.h>
 
using namespace ImFusion;
 
 
// Create optical flow algorithm
OpticalFlowAlgorithm opticalFlowComputer(imageSequence.get());
 
// Configure algorithm parameters
opticalFlowComputer.p_opticalFlowName = "Optical Flow RAFT";  // Use learning based optical flow method (RAFT)
opticalFlowComputer.p_removeInvalidFlow = true;  // Remove invalid flow vectors
 
// Run optical flow computation
opticalFlowComputer.compute();
 
// Extract results
OwningDataList output = opticalFlowComputer.takeOutput();
 
// Extract optical flow from the algorithm
std::unique_ptr<SharedImageSet> flowOutput = output.extractFirstImage();

Optical Flow with uncertainty

When you want to compute optical flow along with per-pixel flow uncertainty

#include <ImFusion/Base/SharedImageSet.h>
#include <ImFusion/Vision/OpticalFlowAlgorithm.h>
#include <ImFusion/Base/DataList.h>
 
using namespace ImFusion;
 
// As above, assume we have a variable imageSequence of type std::unique_ptr<SharedImageSet>
 
// Create optical flow algorithm
OpticalFlowAlgorithm opticalFlowComputer(imageSequence.get());
 
// Configure algorithm
opticalFlowComputer.p_opticalFlowName = "Optical Flow SEARAFT";           // Use SEARAFT optical flow method
opticalFlowComputer.p_removeInvalidFlow = true;                           // Remove invalid flow vectors
 
// Run computation
opticalFlowComputer.compute();
 
// Extract results
OwningDataList output = opticalFlowComputer.takeOutput();
 
auto outputImages = output.extractAllImages();
std::unique_ptr<SharedImageSet> opticalFlowResult = nullptr;
std::unique_ptr<SharedImageSet> uncertaintyResult = nullptr;
for (auto& output : outputImages)
    if (output)
    {
        if (output->name() == "OpticalFlow")
            opticalFlowResult = std::move(output);
        else if (output->name() == "OpticalFlowUncertainty")
            uncertaintyResult = std::move(output);
    }
if (!opticalFlowResult)
    throw AlgorithmExecutionError("Could not extract optical flow shared image set.");
// opticalFlowResult contains the optical flow 
// uncertaintyResult contains the flow uncertainty

See also

OpticalFlowAlgorithm

Custom Models

OpticalFlow allow running your custom models through our machine learning engines.

This is done by tracing or scripting your Torch or exporting your ONNX model and linking it through a configuration file.

Here is an example configuration file in .yaml format for OpticalFlow:

Version: '8.0'
Type: NeuralNetwork
Name: CustomModel
Description: Optical Flow
Engine:
  Name: torch
  DisableJITRecompile: true
  ModelFile: <path_to_model_file>
  ForceCPU: false
  Verbose: true
  InputFields: [FirstImage, SecondImage]
  OutputFields: [Flow]
PredictionOutput: [Image]
PreprocessingInputFields: [FirstImage, SecondImage]
PreProcessing:
  - MakeFloat: {}
Sampling:
  - SkipUnpadding: true

We require InputFields and OutputFields to have exactly the same names as mentioned above. In our current release, we only support the Depth output to be exposed to the user, returned as the first element by the model. Any additional outputs should still be appended in OutputFields and PredictionOutput with its return type but they will be discarded.

For further information on the machine learning configuration files, please refer to the Machine Learning Model page in our user documentation.