Research map / 3D vision / Depth and stereo

Déjà View: Looping Transformers for Multi-View 3D Reconstruction

NeurIPS 2026 · 3D vision › Depth and stereo · approach: Cost-volume multi-view stereo · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Déjà View applies a single shared looped transformer block recurrently to DINOv2 per-view features for K refinement steps (K sampled in [8,16]), yielding a 117M-parameter multi-view 3D reconstruction model that matches or beats much larger feed-forward baselines.

Problem

Feed-forward multi-view 3D reconstruction transformers scale to over a billion parameters via depth, yet contiguous layers often act like repeated similar operations; the paper asks whether explicit weight-shared iteration can replace parameter scaling.

Approach

Replace the L unique decoder layers with one shared looped transformer block (frame + global attention) applied K times to per-view DINOv2 tokens, conditioned on continuous time intervals via zero-initialized scale vectors, with K sampled per batch so that inference-time step count is a compute knob.

Key results

• Déjà View (117M params, 75.9 TFLOPs, 4.9 GiB peak memory at 24 views) achieves average IR 80.3 and AUC@30 91.8, the best among compared methods, vs. VGGT-Ω-1B at 74.8/90.7 with 1144M params.
• Weight sharing alone outperforms the decoupled 16-block variant (Rel. L2 0.045 vs 0.056; AUC@30 84.8 vs 82.0) despite 16x fewer parameters.
• Variable-K training (K in [8,16]) is within about 2% of fixed K=16 at K_inf=16 on all metrics.

Surprising findings

• Weight sharing across 16 steps beats an otherwise identical decoupled variant with 16x more parameters under matched data and compute.
• The recurrent state does not converge to a fixed point; its norm keeps growing, and performance collapses when iterating far beyond the trained step range.

Limitations

• Recurrence does not extrapolate beyond K_max; some channels diverge when K_inf exceeds the trained range.
• Variable-K training matches rather than exceeds fixed-K quality at the same inference budget.
• Does not explicitly handle dynamic scenes.

Experimental setup

Datasets: Aria Synthetic Environments WildRGB-D TRELLIS CO3D Taskonomy ScanNet DL3DV ScanNet++ Waymo BEDLAM MegaDepth Hypersim

Benchmarks: DTU ETH3D 7-Scenes ScanNet++ nuScenes

Metrics: Rel. L2 Inlier ratio AUC@3 AUC@30 parameter count TFLOPs peak memory

Baselines: VGGT Pi3 MapAnything Depth Anything 3 MASt3R MASt3R-SfM VGGT-Ω

Builds on: VGGT RAPTOR Universal Transformers RAFT DUSt3R

Closest papers by meaning

• Understanding Multi-View Transformers (NeurIPS 2026 · similarity 0.89)
• DMV3D: Denoising Multi-view Diffusion Using 3D Large Reconstruction Model (ICLR 2024 · similarity 0.87)
• TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention (NeurIPS 2026 · similarity 0.86)
• Slice3D: Multi-Slice Occlusion-Revealing Single View 3D Reconstruction (CVPR 2024 · similarity 0.86)
• Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass (CVPR 2025 · similarity 0.86)
• View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selection (AAAI 2025 · similarity 0.86)