Research map / 3D vision / Depth and stereo
AirPlanes: Accurate Plane Estimation via 3D-Consistent Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Estimates 3D planes from posed RGB sequences by distilling per-pixel planar embeddings into a per-scene test-time-optimized MLP, then clustering mesh vertices with RANSAC or mean-shift.
Problem
Extracting planar surfaces from posed multi-view images in a 3D-consistent, online-capable way; purely geometric plane extraction ignores semantics needed to separate distinct planes, and single-image plane methods are not multi-view consistent.
Approach
Train a per-scene MLP at test time to map 3D mesh points to multi-view-consistent plane embeddings distilled from per-pixel single-image embeddings plus normals, then cluster these embeddings jointly with geometry (sequential RANSAC or mean-shift) to obtain planes.
Key results
- Ours achieves chamfer 5.30, f1 64.92, VOI 2.268, RI 0.957, SC 0.568 on testplanes, the best segmentation scores, outperforming PlanarRecon (VOI 3.201) and SR+RANSAC (VOI 2.507).
- Sequential RANSAC on SimpleRecon geometry is a surprisingly strong baseline (chamfer 5.40, f1 65.45), beating PlanarRecon on most metrics.
- Adding the 3D embeddings to other geometry methods (Atlas, NeuralRecon, FineRecon, SimpleRecon) improves planar chamfer for all of them, e.g., FineRecon 9.72 to 7.76.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 3D-consistent plane embeddings to geometry-based grouping | SimpleRecon + RANSAC · ScanNetV2 testplanes | planar chamfer: 9.78 -> 8.37 | Helped |
| add 3D-consistent plane embeddings to geometry-based grouping | SR + RANSAC · ScanNetV2 testplanes | VOI: 2.507 -> 2.268 | Helped |
| add 3D-consistent embeddings to geometry from multiple estimators | Atlas + RANSAC, NeuralRecon + RANSAC, FineRecon + RANSAC, and SR + RANSAC · ScanNetV2 testplanes | planar chamfer: 20.16 -> 18.92; 14.92 -> 13.42; 9.72 -> 7.76; 9.78 -> 8.37 | Helped |
| use fused per-pixel embeddings without test-time optimization (ablation) | AirPlanes · ScanNetV2 testplanes | planar chamfer: 8.37 -> 9.97 | Hurt |
| use training-time multi-view consistency instead of test-time MLP optimization (ablation) | AirPlanes · ScanNetV2 testplanes | planar chamfer: 8.37 -> 9.96 | Hurt |
| remove planar probability from the TSDF (ablation) | AirPlanes · ScanNetV2 testplanes | chamfer: 5.30 -> 5.39 | Hurt |
| replace RANSAC grouping with mean-shift (ablation) | AirPlanes · ScanNetV2 testplanes | planar chamfer: 8.37 -> 8.88 | Hurt |
| replace 3D embeddings with predicted semantic labels (ablation) | SR + RANSAC · ScanNetV2 testplanes | planar chamfer: 8.37 -> 9.01 | Hurt |
| replace 3D embeddings with ground-truth semantic labels (ablation) | SR + RANSAC · ScanNetV2 testplanes | planar chamfer: 8.37 -> 8.04 | Helped |
| replace 3D embeddings with ground-truth instance labels (ablation) | SR + RANSAC · ScanNetV2 testplanes | planar chamfer: 8.37 -> 7.42 | Helped |
Surprising findings
- A geometry-only sequential RANSAC baseline on SimpleRecon meshes outperforms PlanarRecon on several metrics, contrary to the original PlanarRecon paper's results.
- Feed-forward per-frame embedding networks trained with multi-view consistency did not yield consistent 3D embeddings, motivating test-time per-scene optimization.
Limitations
- Errors in the geometry from the MVS system can severely affect extracted planes.
- Plane fitting is greedy rather than globally optimized.
- Only visible geometry is estimated; unobserved regions are not completed.
Experimental setup
Datasets: ScanNetV2 ScanNetV2 plane annotations from PlaneNet (Liu et al.)
Benchmarks: ScanNetV2 testplanes split
Metrics: chamfer distance f1 score variation of information rand index segmentation covering planar fidelity planar accuracy planar chamfer
Baselines: PlanarRecon PlaneRecTR Atlas + RANSAC NeuralRecon + RANSAC FineRecon + RANSAC SimpleRecon + RANSAC
Builds on: SimpleRecon PlanarRecon-style single-image plane embeddings (Yu et al.) iLabel iMap sequential RANSAC
Closest papers by meaning
- Towards In-the-wild 3D Plane Reconstruction from a Single Image (CVPR 2025 · similarity 0.89)
- PlaneMVS: 3D Plane Reconstruction From Multi-View Stereo (CVPR 2022 · similarity 0.89)
- AlphaTablets: A Generic Plane Representation for 3D Planar Reconstruction from Monocular Videos (NeurIPS 2024 · similarity 0.87)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.85)
- HyPlaneHead: Rethinking Tri-plane-like Representations in Full-Head Image Synthesis (NeurIPS 2025 · similarity 0.85)
- GeoDepth: From Point-to-Depth to Plane-to-Depth Modeling for Self-Supervised Monocular Depth Estimation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.