Research map / Optimization / Optimizers
From Variance to Veracity: Unbundling and Mitigating Gradient Variance in Differentiable Bundle Adjustment Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why differentiable bundle adjustment layers in deep visual odometry (DPVO) train slowly and proposes weighting the flow loss by the network-predicted BA weights (stop-gradient), yielding 2-2.5x faster training and 24% better TartanAir test ATE.
Problem
Training tightly coupled learned correspondence and differentiable bundle adjustment pose estimators (e.g., DPVO, DROID-SLAM) is slow and unstable because gradients through the BA layer have high variance, driven by outliers.
Approach
Identify three sources of gradient variance (flow loss interference, linearization errors in the BA layer, and weight gradient dependence on BA residuals) and reduce them by weighting the flow loss with the network-predicted BA confidence weights (stop-gradient), plus adaptive balancing of flow and pose loss gradients.
Key results
- On the TartanAir test split the weighted-flow-loss DPVO achieves average ATE 0.16 m vs 0.21 m for DPVO (about 24% lower).
- The weighted flow loss reaches ~0.2 m validation ATE in 80K iterations versus 180K for vanilla DPVO, and converges to ~0.10 m vs. the base model's ~0.2 m.
- Training in the non-streaming 8-frame setting is faster (0.6 s vs 1.6 s per iteration on RTX A6000) and uses less memory (7.2 GB vs 19.2 GB), generalizing to streaming evaluation; on EuRoC and TUM-RGBD performance is similar to DPVO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the unweighted flow loss with a flow loss weighted by the network-predicted BA weights | DPVO · TartanAir test split | ATE [m]: 0.21 -> 0.16 | Helped |
| Replace the unweighted flow loss with a flow loss weighted by the network-predicted BA weights | DPVO · EuRoC | ATE [m]: 0.105 -> 0.117 | Hurt |
| Replace the unweighted flow loss with a flow loss weighted by the network-predicted BA weights | DPVO · TUM-RGBD freiburg1 | ATE [m]: 0.089 -> 0.094 | Hurt |
| Train using 8-frame batch optimization rather than 15-frame streaming sequences (ablation) | DPVO with weighted flow loss · TartanAir validation split evaluated in streaming setting | ATE | No clear effect |
| Replace random patch sampling with SIFT patch selection (ablation) | DPVO · TartanAir validation split | ATE | Hurt |
| Replace random patch sampling with ORB patch selection (ablation) | DPVO · TartanAir validation split | ATE | Hurt |
| Replace random patch sampling with SuperPoint patch selection (ablation) | DPVO · TartanAir validation split | ATE | Hurt |
| Replace random patch sampling with gradient-based patch sampling (ablation) | DPVO · TartanAir validation split | ATE | Hurt |
Surprising findings
- The modified model performs similarly to DPVO on EuRoC and TUM-RGBD, indicating the gains do not resolve domain-shift generalization.
- Weighted flow loss does not automatically improve generalization; depth accuracy for low-weight points deteriorates.
Limitations
- Gains do not transfer to domain shift (EuRoC, TUM-RGBD performance similar to DPVO).
- Depth accuracy for low-weight points deteriorates, which may matter for dense SLAM.
- Analysis focused mainly on DPVO; transfer to DROID-SLAM only tested with little tuning.
Experimental setup
Datasets: TartanAir
Benchmarks: TartanAir EuRoC TUM-RGBD
Metrics: ATE (m) SNR of gradients training speed
Baselines: DPVO DROID-SLAM DROID-VO ORB-SLAM3 COLMAP DSO TartanVO SVO
Builds on: DPVO DROID-SLAM differentiable bundle adjustment
Closest papers by meaning
- Adaptive VIO: Deep Visual-Inertial Odometry with Online Continual Learning (CVPR 2024 · similarity 0.85)
- EqNIO: Subequivariant Neural Inertial Odometry (ICLR 2025 · similarity 0.85)
- Tartan IMU: A Light Foundation Model for Inertial Positioning in Robotics (CVPR 2025 · similarity 0.84)
- Leveraging Consistent Spatio-Temporal Correspondence for Robust Visual Odometry (AAAI 2025 · similarity 0.84)
- Co-op: Correspondence-based Novel Object Pose Estimation (CVPR 2025 · similarity 0.84)
- MambaVO: Deep Visual Odometry Based on Sequential Matching Refinement and Training Smoothing (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.