Research map / 3D vision / 3D generation
Towards In-the-wild 3D Plane Reconstruction from a Single Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ZeroPlane, a Transformer-based model for zero-shot single-image 3D plane detection and reconstruction trained on a mixed indoor/outdoor planar benchmark of 560K high-resolution annotations.
Problem
Single-image 3D plane reconstruction has been trained and evaluated on single indoor or outdoor datasets, limiting generalization to diverse and in-the-wild scenes.
Approach
Disentangles plane normal and offset and learns both with an exemplar-guided classification-then-regression scheme, plus a pixel-geometry-enhanced plane embedding that injects predicted depth and normal cues into plane queries, all trained on a large mixed-domain dataset.
Key results
- On zero-shot NYUv2, ZeroPlane (DINO-B, mixed) reaches 37.29 plane recall at 5° normal vs 24.97 for PlaneRecTR (mixed).
- On zero-shot ParallelDomain, DINO-B mixed reaches 53.23 recall at 5° normal vs 24.17 for PlaneRecTR.
- Mixed-dataset training with the full 560K set outperforms the 110K starter mix on NYUv2 (depth recall @0.6m 55.08 vs 46.26).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZeroPlane instead of PlaneRecTR with mixed-dataset training | NYUv2 · NYUv2 | Plane recall (depth) @0.1m: 14.29 -> 17.86 | Helped |
| Use ZeroPlane instead of PlaneRecTR with mixed-dataset training | 7-Scenes · 7-Scenes | Plane recall (depth) @0.1m: 10.97 -> 17.19 | Helped |
| Use ZeroPlane instead of PlaneRecTR with mixed-dataset training | ParallelDomain · ParallelDomain | Plane recall (normal) @10°: 35.11 -> 65.74 | Helped |
| Use ZeroPlane instead of PlaneRecTR with mixed-dataset training | ApolloScape · ApolloScape | Plane recall (depth) @3m: 43.06 -> 40.44 | Hurt |
| Add DINO-Base encoder and DPT decoder (ablation) | ZeroPlane, mixed-dataset training · NYUv2 | Plane recall (depth) @0.05m: 3.9 -> 6.1 | Helped |
| Add classification-then-regression geometry learning (ablation) | ZeroPlane with DPT, mixed-dataset training · NYUv2 | Plane recall (depth) @0.05m: 6.1 -> 7.61 | Helped |
| Add geometry-enhanced attention (ablation) | ZeroPlane with DPT and Cls-Reg, mixed-dataset training · NYUv2 | Plane recall (depth) @0.05m: 7.61 -> 8.54 | Helped |
| Train using indoor datasets only rather than the full mixed dataset (ablation) | ZeroPlane · ParallelDomain | Plane recall (depth) @3m: 44.51 -> 1.19 | Hurt |
| Train using outdoor datasets only rather than the full mixed dataset (ablation) | ZeroPlane · NYUv2 | Plane recall (depth) @0.1m: 17.86 -> 1.71 | Hurt |
| Expand training from the starter mix to the full mixed dataset (ablation) | ZeroPlane · ParallelDomain | Plane recall (depth) @3m: 35.45 -> 44.51 | Helped |
| Train using high-resolution annotations rather than low-resolution annotations (ablation) | ZeroPlane · ParallelDomain | Plane recall (depth) @3m: 44.43 -> 50.0 | Helped |
Surprising findings
- Indoor-only trained model fails badly on ParallelDomain (0.81 recall at 1m), and outdoor-only fails on NYUv2 (0.43 at 0.05m), indicating large domain gaps.
- Mixed-domain model matches or outperforms the outdoor-only model on ParallelDomain in depth recall at 10m range.
Limitations
- Lacks real outdoor training data because dense ground-truth depth is unavailable, limiting generalization.
- Plane labels for many datasets are pseudo-labels generated from depth fitting and Mask2Former segmentation rather than manual annotation.
Experimental setup
Datasets: ScanNetv1 ScanNetv2 Matterport3D Replica HM3D DIODE Taskonomy Synthia Virtual KITTI Sanpo NYUv2 7-Scenes
Benchmarks: NYUv2 7-Scenes ParallelDomain ApolloScape
Metrics: RI VOI SC plane recall (depth) plane recall (normal)
Baselines: PlaneRecTR PlaneRCNN PlaneTR PlaneAE
Builds on: Mask2Former PlaneRecTR DINOv2 DPT PlaneRCNN
Closest papers by meaning
- AirPlanes: Accurate Plane Estimation via 3D-Consistent Embeddings (CVPR 2024 · similarity 0.89)
- AlphaTablets: A Generic Plane Representation for 3D Planar Reconstruction from Monocular Videos (NeurIPS 2024 · similarity 0.88)
- PLANA3R: Zero-shot Metric Planar 3D Reconstruction via Feed-forward Planar Splatting (NeurIPS 2025 · similarity 0.88)
- Robust 3D Shape Reconstruction in Zero-Shot from a Single Image in the Wild (CVPR 2025 · similarity 0.87)
- LRM-Zero: Training Large Reconstruction Models with Synthesized Data (NeurIPS 2024 · similarity 0.86)
- ZeroShape: Regression-based Zero-shot Shape Reconstruction (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.