Research map / 3D vision / Depth and stereo
Flow-Guided Online Stereo Rectification for Wide Baseline Stereo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a transformer- and correlation-volume-based network that predicts relative stereo rotation for online rectification, trained with a self-supervised vertical optical flow loss plus weak supervision from offline calibration, and releases two wide-baseline datasets.
Problem
Stereo rectification degrades over time for wide-baseline camera rigs on vehicles due to vibration and structural stress, and traditional methods are either too slow for online use or inaccurate, while offline calibration does not hold in the wild.
Approach
Predict the relative rotation directly from a stereo pair using a cross-attention transformer encoder and correlation volume, and train it with a differentiable rectification module plus a self-supervised loss that minimizes vertical optical flow (RAFT) of the rectified pair, with offline rotation as a weak supervised loss.
Key results
- On Semi-Truck Highway the proposed method reaches MAE 0.015 and SuperGlue offset 0.59 px, beating offline ground-truth calibration (SuperGlue offset 0.77 px) and the best baseline ViTPose (MAE 0.07).
- Downstream depth MAE improves by over 63% on average over 0-300 m on the Semi-Truck Highway dataset for both DLNR and HITNet relative to other rectification methods.
- Runs at 86 ms per pair on an NVIDIA A40, faster than SIFT+LO-RANSAC (338 ms) and comparable to ViTPose (88 ms).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add vertical optical-flow self-supervision (ablation) | Ours · Semi-Truck Highway | MAE: 0.023 -> 0.015 | Helped |
| add vertical optical-flow self-supervision (ablation) | Ours · Semi-Truck Highway | SIFT offset (pixels): 3.13 -> 2.67 | Helped |
| add vertical optical-flow self-supervision (ablation) | Ours · Semi-Truck Highway | SuperGlue offset (pixels): 0.97 -> 0.59 | Helped |
| add vertical optical-flow self-supervision (ablation) | Ours · Semi-Truck Highway | Vertical flow (pixels): 1.43 -> 1.05 | Helped |
| add vertical optical-flow self-supervision (ablation) | Ours · KITTI | MAE: 0.015 -> 0.011 | Helped |
| add vertical optical-flow self-supervision (ablation) | Ours · KITTI | SIFT offset (pixels): 1.73 -> 1.44 | Helped |
| add vertical optical-flow self-supervision (ablation) | Ours · KITTI | SuperGlue offset (pixels): 0.53 -> 0.36 | Helped |
| add vertical optical-flow self-supervision (ablation) | Ours · KITTI | Vertical flow (pixels): 0.51 -> 0.28 | Helped |
| add vertical optical-flow self-supervision (ablation) | Ours · Carla | MAE: 0.13 -> 0.05 | Helped |
| add vertical optical-flow self-supervision (ablation) | Ours · Carla | SIFT offset (pixels): 7.68 -> 3.55 | Helped |
| add vertical optical-flow self-supervision (ablation) | Ours · Carla | SuperGlue offset (pixels): 5.53 -> 0.64 | Helped |
| add vertical optical-flow self-supervision (ablation) | Ours · Carla | Vertical flow (pixels): 5.67 -> 1.003 | Helped |
Surprising findings
- The learned online method outperforms offline ground-truth calibration on keypoint-offset metrics for the Semi-Truck Highway and KITTI datasets.
Limitations
- Relies on offline calibration as a weak supervised rotation target, so it is not fully calibration-free.
- Only estimates rotation; translation is taken from ground truth and not predicted.
- Downstream gains on depth are reported mainly as MAE improvements and are concentrated on the proprietary semi-truck dataset.
Experimental setup
Datasets: Semi-Truck Highway (new) KITTI Carla (new)
Benchmarks: Semi-Truck Highway KITTI Carla
Metrics: MAE of rectified images vs ground-truth rectification SIFT keypoint y-offset SuperGlue keypoint y-offset vertical flow downstream depth MAE (DLNR, HITNet) inference time
Baselines: SIFT + LO-RANSAC SuperGlue + MAGSAC LoFTR + MAGSAC RPNet DirectionNet ViTPose
Builds on: RAFT DETR Swin transformer 6D rotation representation (DirectionNet)
Closest papers by meaning
- Rectification-specific Supervision and Constrained Estimator for Online Stereo Rectification (CVPR 2025 · similarity 0.95)
- Beyond Row Alignment: Virtual-Camera-Aware Online Stereo Rectification (NeurIPS 2026 · similarity 0.92)
- Any-Stereo: Arbitrary Scale Disparity Estimation for Iterative Stereo Matching (AAAI 2024 · similarity 0.84)
- RobIA: Robust Instance-aware Continual Test-time Adaptation for Deep Stereo (NeurIPS 2025 · similarity 0.84)
- RRT-MVS: Recurrent Regularization Transformer for Multi-View Stereo (AAAI 2025 · similarity 0.83)
- Efficient Depth Estimation for Unstable Stereo Camera Systems on AR Glasses (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.