Research map / 3D vision / 3D generation

Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations

NeurIPS 2025 · 3D vision › 3D generation · approach: Semantic scene completion · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Concerto jointly trains a 3D point transformer (PTv3) with Sonata-style intra-modal self-distillation and a cross-modal cosine-similarity objective that predicts frozen DINOv2 image patch features from point features, yielding stronger 3D scene representations.

Problem

Self-supervised image and point cloud representations are learned separately and only partially overlap; the paper asks whether joint 2D-3D self-supervised learning can yield spatial representations that surpass single-modality models and their feature concatenation.

Approach

Couple intra-modal 3D self-distillation (Sonata) with cross-modal joint embedding prediction in which point features, pooled per image patch via camera projections, are regressed onto frozen DINOv2 patch features with a cosine loss, so 2D knowledge is internalized into the point encoder and yields features usable from point clouds alone at inference.

Key results

• Linear probing on ScanNet semantic segmentation reaches 77.3% mIoU, versus 72.5% for Sonata and 75.9% for Sonata×DINOv2 concatenation.
• With full fine-tuning Concerto reaches 80.7% mIoU on ScanNet, 39.2% on ScanNet200 and 50.7% on ScanNet++.
• Scaling to a 207M large model with video-lifted data gives 77.5% linear-probing mIoU on ScanNet and 38.6% on ScanNet200.

Measured effects reported in the paper

Surprising findings

• Linear probing outperforms full fine-tuning in extreme data-scarce settings (1%, 5% scenes, 20-point annotation), which the authors attribute to out-of-distribution adaptation.
• Strong DINOv2-style image augmentations hurt performance (linear probing 75.27% vs 75.6% default), so weaker augmentations were used.

Limitations

• Image encoder is frozen, so no joint native multi-modal training was explored.
• Language alignment is only shallow linear probing; zero-shot ScanNet mIoU of 44.56% lags supervised linear probing.
• Cross-modal training requires paired images with camera parameters and depth/mesh for correspondence.

Experimental setup

Datasets: ScanNet ScanNet++ Structured3D S3DIS ARKitScenes HM3D RealEstate10K ScanNet200 ScanNet Data Efficient

Benchmarks: ScanNet ScanNet200 ScanNet++ S3DIS Area 5 ScanNet Data Efficient

Metrics: mIoU mAcc allAcc mAP25 mAP50 mAP

Baselines: Sonata PTv3 supervised MSC CSC PPT DINOv2 Sonata×DINOv2 concatenation PTv2 SparseUNet Locate3D

Builds on: Sonata Point Transformer V3 DINOv2 JEPA VGGT

Closest papers by meaning

• Sonata: Self-Supervised Learning of Reliable Point Representations (CVPR 2025 · similarity 0.88)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.88)
• Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models (NeurIPS 2026 · similarity 0.88)
• HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models (CVPR 2026 · similarity 0.87)
• GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.87)
• UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding (ICLR 2026 · similarity 0.87)