Research map / 3D vision / Depth and stereo

Understanding Multi-View Transformers

NeurIPS 2026 · 3D vision › Depth and stereo · approach: Cost-volume multi-view stereo

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Probes the skip-connection residual state of DUSt3R's multi-view decoder with layer-wise pointmap probes to visualize how 3D geometry, pose and correspondences form across layers.

Problem

Multi-view transformers such as DUSt3R solve 3D reconstruction feed-forward, but their internal mechanisms are poorly understood, limiting interpretability and reliable use.

Approach

Treat the residual skip-connection stream as a progressively updated latent state and train per-patch, non-contextual MLP probes after each skip connection to decode intermediate tokens into 3D pointmaps, enabling layer-by-layer geometric visualization and analysis.

Key results

• Self-attention layers reduce the Procrustes-aligned second-view error at decoder input by 94% on average, while cross-attention increases it by 11% and MLP by 7%.
• Zero-shot correspondences extracted from cross-attention heads improve from about 40% correct at decoder input to over 60% after the first six decoder blocks (16 px threshold).
• Per-view pointmap probes outperform metric depth probes on depth accuracy (e.g., Block 11 absrel 0.038 vs 0.149).

Surprising findings

• Knocking out the putative global-pose 'register' tokens and heads leaves pointmap estimation with the correct relative pose unchanged, suggesting global pose information is not crucial for self-attention.

Limitations

• Analysis limited to a single DUSt3R variant (224 linear checkpoint) and synthetic rendered data for probe training
• Probe-based analysis may still be influenced by probe capacity despite restricted receptive field
• Correspondence evaluation is coarse at patch level (14x14 attention maps)

Experimental setup

Datasets: Habitat Matterport 3D

Benchmarks: Habitat Matterport 3D held-out pairs

Metrics: scale-shift invariant pointmap error delta1 absrel correspondence percentage within 16 px

Baselines: metric depth probe

Builds on: DUSt3R attention knockout Procrustes alignment

Closest papers by meaning

• Déjà View: Looping Transformers for Multi-View 3D Reconstruction (NeurIPS 2026 · similarity 0.89)
• TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention (NeurIPS 2026 · similarity 0.87)
• GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers (ICLR 2024 · similarity 0.87)
• Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs (AAAI 2026 · similarity 0.87)
• Cameras as Relative Positional Encoding (NeurIPS 2025 · similarity 0.87)
• Cross-View Completion Models are Zero-shot Correspondence Estimators (CVPR 2025 · similarity 0.87)