Research map / 3D vision / 3D generation
SPA: 3D Spatial-Awareness Enables Effective Embodied Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a vanilla ViT with differentiable neural rendering (SDF, SH color, semantic features distilled from AM-RADIO) on multi-view images to add 3D spatial awareness, and evaluates it across 268 embodied tasks in 8 simulators plus real-world tasks.
Problem
Existing visual representations for embodied AI rely on 2D contrastive or MAE objectives and lack the 3D spatial understanding needed for manipulation and navigation; prior embodied evaluations are also too small to draw reliable conclusions.
Approach
Use multi-view differentiable neural rendering as the pre-text task on a vanilla 2D ViT: features are lifted into a dynamically scaled 3D feature volume via deformable attention, and rendered RGB, depth and semantic maps supervise the ViT without labels, endowing it with 3D spatial awareness.
Key results
- SPA-L reaches a mean success rate of 88.63 across the 13 benchmarks with mean rank 3.20, best or second-best in 11 of 13 benchmarks, outperforming MAE (85.13) and VC-1 (84.69).
- In zero-shot camera pose estimation on NAVI, SPA reduces translation error to 1.65e-2 (about 18.3% better than the second-best) and rotation error to 0.61e-1 (about 15.3% better).
- On three real-world tasks with a Low-Cost Robot Arm, SPA achieves 65.33% mean success rate versus 53.33% for the next best (MAE).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SPA rather than the strongest reported baseline | embodied representation learning across the evaluated benchmarks · All embodied benchmarks | Mean S.R.: 85.13 -> 88.63 | Helped |
| increase SPA model size from ViT-B to ViT-L | SPA · VC-1 | Mean S.R.: 73.66 -> 77.82 | Helped |
| use 3D-aware rendering pre-training instead of continued single-image MAE pre-training on the same data (ablation) | SPA-B · VC-1 | Mean S.R.: 73.11 -> 73.66 | Helped |
| set mask ratio to 0.00 instead of 0.50 (ablation) | SPA pre-training · VC-1 | Mean S.R.: 71.18 -> 70.36 | Hurt |
| set mask ratio to 0.25 instead of 0.50 (ablation) | SPA pre-training · VC-1 | Mean S.R.: 71.18 -> 70.17 | Hurt |
| set mask ratio to 0.75 instead of 0.50 (ablation) | SPA pre-training · VC-1 | Mean S.R.: 71.18 -> 71.01 | Hurt |
| set mask ratio to 0.95 instead of 0.50 (ablation) | SPA pre-training · VC-1 | Mean S.R.: 71.18 -> 70.67 | Hurt |
| add the depth rendering loss (ablation) | SPA pre-training · VC-1 | Mean S.R.: 71.01 -> 71.18 | No clear effect |
| add the color rendering loss (ablation) | SPA pre-training · VC-1 | Mean S.R.: 68.71 -> 71.18 | Helped |
| add the semantic rendering loss (ablation) | SPA pre-training · VC-1 | Mean S.R.: 71.16 -> 71.18 | No clear effect |
| train on Hypersim plus RH20T instead of ScanNet (ablation) | SPA dataset pre-training · VC-1 | Mean S.R.: 73.68 -> 67.35 | Hurt |
| use SPA instead of MAE | real-world frozen representations · Real-world tasks | Mean S.R.: 53.33 -> 65.33 | Helped |
Surprising findings
- Multi-modal models such as CLIP and InternViT-6B perform poorly in embodied evaluation, and InternViT-6B does not outperform InternViT-300M on some benchmarks.
- DINOv2, despite using about 10x more data, performs worse than MoCoV3 and MAE on embodied tasks.
Limitations
- Evaluation restricted to imitation learning (behavior cloning); reinforcement learning not evaluated.
- Focuses on static multi-view scenes; dynamic temporal scenes not handled.
- Some datasets (e.g., Droid) have noisy camera poses and need depth estimation or single-view treatment.
Experimental setup
Datasets: ScanNet ScanNet++ Hypersim ADT S3DIS Droid Arkitscenes Structured3D RH20T ASE VC-1 benchmark NAVI
Benchmarks: VC-1 Franka Kitchen Meta-World RLBench LIBERO NAVI camera pose estimation
Metrics: Mean success rate Mean rank translation error rotation error
Baselines: MoCoV3 MAE DINOv2 CLIP EVA InternViT-300M InternViT-6B MVP VC-1 R3M Voltron Theia
Builds on: NeuS Ponder MAE AM-RADIO VC-1
Closest papers by meaning
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.91)
- BIP3D: Bridging 2D Images and 3D Perception for Embodied Intelligence (CVPR 2025 · similarity 0.87)
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.87)
- Abstract 3D Perception for Spatial Intelligence in Vision-Language Models (CVPR 2026 · similarity 0.87)
- g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks (CVPR 2025 · similarity 0.87)
- 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.