Research map / 3D vision / 3D generation
DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces heterogeneous-teacher distillation (co-distillation) to train DUNE, a ViT-Base encoder distilled from DINO-v2, MASt3R, and Multi-HMR (ViT-Large teachers) on 19 mixed 2D/3D/human datasets, using transformer projectors.
Problem
Multi-teacher distillation has only combined teachers trained on similar generic web data; it is unclear how to distill teachers that differ both in task objectives and training data domains (2D generic, 3D, human perception) into one encoder.
Approach
Co-distillation from heterogeneous teachers using a shared ViT-Base student with teacher-specific transformer projectors (instead of per-patch MLPs or ladder projectors), trained on the union of teacher datasets with full data sharing, and then discarding projectors and fine-tuning task heads on the frozen encoder.
Key results
- DUNE (ViT-Base, 448px) reaches 94.7 AUC on Map-free VCRE<90px, surpassing MASt3R ViT-Large (91.2 in Table 3; 93.3 AUC on the leaderboard in Table 4) with a smaller encoder.
- Transformer projector improves over ladder projectors on all tasks; with all 19 datasets it reaches 44.9 mIoU on ADE20K and 0.377 RMSE on NYUd at 336px.
- On ADE20K semantic segmentation DUNE (44.9 mIoU) trails AM-RADIO-v2.5 (50.0) and DINO-v2 ViT-Base (47.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MASt3R’s ViT-Large encoder with DUNE ViT-Base and fine-tune the MASt3R decoder | MASt3R on Map-free Visual Relocalization · Map-free Visual Relocalization (VCRE<90px) | AUC: 93.3 -> 94.3 | Helped |
| Use all 19 teacher datasets instead of ImageNet-19K alone, with LP projectors (ablation) | DUNE distillation · BEDLAM | PA-PVE: 83.9 -> 78.2 | Helped |
| Use all 19 teacher datasets instead of ImageNet-19K alone, with LP projectors (ablation) | DUNE distillation · ADE20K | mIoU: 42.4 -> 44.7 | Helped |
| Use Transformer Projector instead of Ladder of Projectors with all datasets (ablation) | DUNE distillation · NYUv2 | RMSE: 0.384 -> 0.377 | Helped |
| Use Transformer Projector instead of Ladder of Projectors with all datasets (ablation) | DUNE distillation · BEDLAM | PA-PVE: 78.2 -> 68.3 | Helped |
| Use Transformer Projector instead of Simple Projector with all datasets (ablation) | DUNE distillation · ADE20K | mIoU: 42.3 -> 44.9 | Helped |
| Share all data among teacher projectors instead of no data sharing (ablation) | DUNE distillation · ADE20K | mIoU: 41.6 -> 44.9 | Helped |
| Share all data among teacher projectors instead of generic data only (ablation) | DUNE distillation · ADE20K | mIoU: 44.9 -> 40.1 | Hurt |
| Share all data among teacher projectors instead of generic data only (ablation) | DUNE distillation · NYUv2 | RMSE: 0.416 -> 0.377 | Helped |
| Use DUNE ViT-Base instead of AM-RADIO-v2.5 ViT-Base | Universal encoder evaluation · ADE20K | mIoU: 50.0 -> 44.9 | Hurt |
Surprising findings
- Sharing only generic ImageNet data across teachers gives the best semantic segmentation result, while full data sharing is best overall.
- Using all 19 specialized datasets improves performance on all tasks, contrary to the common finding that ImageNet-1K suffices for distillation.
Limitations
- Semantic segmentation remains well below AM-RADIO-v2.5 and DINO-v2 teacher-level generalization in some settings.
- Evaluation focuses on a limited set of tasks tied to three teachers.
Experimental setup
Datasets: ImageNet-19K Mapillary Google Landmarks v2 AGORA BEDLAM UBody CUFFS Habitat ARKitScenes BlendedMVS MegaDepth ScanNet++
Benchmarks: ADE20K NYUd BEDLAM Map-free Visual Relocalization
Metrics: mIoU RMSE F1-score PA-PVE AUC
Baselines: DINO-v2 ViT-Base AM-RADIO-v2.5 MASt3R LoFTR DUSt3R MicKey
Builds on: AM-RADIO UNIC DINO-v2 MASt3R Multi-HMR
Closest papers by meaning
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.87)
- DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features (NeurIPS 2024 · similarity 0.87)
- Masking Teacher and Reinforcing Student for Distilling Vision-Language Models (CVPR 2026 · similarity 0.86)
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.86)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.86)
- MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.