Research map / Video / Video segmentation and tracking

Rethinking the State Update Gate for Long-Sequence Recurrent 3D Reconstruction

NeurIPS 2026 · Video › Video segmentation and tracking · approach: Stochastic video prediction · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Adaptive Frame Gating (AFG), a parameter-free, training-free scalar frame-level gate computed from frame-to-frame feature change that scales TTT3R's per-token state update in constant-memory streaming 3D reconstruction.

Problem

Long-sequence drift in constant-memory recurrent streaming 3D reconstruction (CUT3R/TTT3R-style), where per-token update gates are bounded and nearly frame-invariant, giving a short effective memory horizon of about 3 frames.

Approach

Add a frame-level graded write weight alpha_t = sigmoid(kappa(||feature change|| - tau)) that multiplies TTT3R's per-token gate, so novel frames write at full strength and redundant frames are down-weighted (never discarded), with no parameters, training, or extra forward pass.

Key results

• On long TUM-RGBD pose sequences (L>=600), AFG-Pose reduces average ATE by 51% over TTT3R.
• On KITTI long-sequence pose, AFG-Pose achieves average ATE 43.96 m vs. 51.90 for LongStream and 87.00 for Keyframe-VO, and 68.54 for TTT3R.
• AFG-Pose reduces Bonn AbsRel by 13.0% over TTT3R averaged over lengths.

Surprising findings

• Binarizing the frame gate into an admit/skip rule with non-keyframes at zero hurts TUM ATE rotation and ScanNet ATE relative to the graded gate.
• Raising the gate cap above 1 severely degrades KITTI performance (mean ATE 97.71 at cap 5 vs 43.96 at cap 1).

Limitations

• Benefit tied to accumulated drift; little or negative gain on short sequences (e.g., KITTI 04, 05, 07).
• Trades some rotation accuracy for lower drift at default setting (TUM RPE_r 0.520 vs 0.443).
• Cannot amplify novel frames (alpha capped at 1), since amplification destabilizes the frozen model outdoors.

Experimental setup

Datasets: TUM-RGBD ScanNet KITTI Bonn 7-Scenes NRGBD

Benchmarks: TUM-RGBD ScanNet KITTI Bonn 7-Scenes NRGBD

Metrics: ATE RPE AbsRel RMSE delta<1.25 accuracy completeness

Baselines: TTT3R TTSA3R MeMix CUT3R StreamVGGT FastVGGT InfiniteVGGT LongStream Keyframe-VO VGGT

Builds on: CUT3R TTT3R test-time training

Closest papers by meaning

• BAT3R: Robust Online 3D Reconstruction with Bayesian Adaptive State Updates (NeurIPS 2026 · similarity 0.90)
• CLoSeR: Closing the Loop for Long-Context Streaming Reconstruction (NeurIPS 2026 · similarity 0.88)
• FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry (NeurIPS 2026 · similarity 0.88)
• GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction (NeurIPS 2026 · similarity 0.87)
• Continuous 3D Perception Model with Persistent State (CVPR 2025 · similarity 0.86)
• Restage4D: Reanimating Deformable 3D Reconstruction from a Single Video (NeurIPS 2025 · similarity 0.86)