Research map / 3D vision / 3D generation
ImageNet3D: Towards General-Purpose Object-Level 3D Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ImageNet3D, a dataset extending 200 ImageNet categories with 6D pose, 2D boxes, cross-category 3D alignment and GPT-4V-generated captions interleaved with 3D info, and benchmarks linear probing, open-vocabulary pose estimation, and joint classification with pose estimation.
Problem
Existing 3D-annotated datasets cover few categories or have low-quality annotations, so models become category specialists and fail to generalize to unseen rigid object categories for object-level 3D understanding.
Approach
Scale object-level 6D pose annotation to 200 ImageNet rigid categories, align canonical poses across categories using semantic parts and shapes so that a shared viewpoint parameterization transfers to novel categories, and add captions interleaving 3D annotations for MLLM integration.
Key results
- DINO v2 achieves the best linear-probe object-level 3D awareness with 56.3% Pose Acc@pi/6 average, ahead of MAE (46.6%) and DINO (42.0%).
- On open-vocabulary pose estimation to novel categories, ResNet50-General reaches 37.1% Pose Acc@pi/6 versus 29.5% for NMM-Sphere, far below the 53.6% of the same model trained on novel categories.
- On joint classification and pose estimation over 189 categories, NMM-Sphere achieves 57.4% 3D-aware accuracy at pi/6, the best among baselines.
- Human evaluators favored ImageNet3D annotations over ObjectNet3D for 73.25% of samples on average across 16 shared categories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DINO v2 features instead of other tested visual foundation model features | Linear probing for object-level 3D awareness · ImageNet3D | Pose Acc@π/6: 56.3 vs 46.6 | Helped |
| Train with category-agnostic features on common categories and test on unseen categories | Open-vocabulary pose estimation · ImageNet3D novel categories | Pose Acc@π/6: 37.1 vs 53.6 | Mixed |
| Scale category-level 3D models to 200 categories | Joint image classification and category-level pose estimation · ImageNet3D | 3D-aware classification accuracy@π/6 | Hurt |
| Use ImageNet3D annotations instead of ObjectNet3D annotations | 3D pose annotation quality · 16-category human evaluation | ImageNet3D annotations favored: 73.25\% | Helped |
| Increase self-supervised backbone size for DINO v2 and DeiT | Linear probing for object-level 3D awareness · ImageNet3D | Pose accuracy | Hurt |
| Use MAE rather than DINO v2 at large and huge model sizes | Linear probing for object-level 3D awareness · ImageNet3D | Pose accuracy | Helped |
| Use Swin Transformer rather than ResNet at comparable model sizes | Joint image classification and category-level pose estimation · ImageNet3D | 3D-aware classification accuracy | Helped |
| Scale ResNet rather than Swin Transformer to larger model sizes | Joint image classification and category-level pose estimation · ImageNet3D | 3D-aware classification accuracy | Helped |
Surprising findings
- Scaling up backbones does not consistently help: DINO v2 and DeiT do not scale well with model size, and MAE outperforms DINO v2 at large and huge sizes.
- Open-vocabulary pose estimation generalizes only partially; models trained without novel-category annotations lag far behind models trained on those categories.
Limitations
- Images are object-centric with one or two instances, so the dataset is less suited to 3D object detection or multi-object co-occurrence tasks.
- Baseline performance degrades noticeably as the number of categories scales up compared with prior smaller-category benchmarks.
- Baselines use only random horizontal flip augmentation, leaving augmentation benefits unexplored.
Experimental setup
Datasets: ImageNet21k ObjectNet3D Objaverse ImageNet1k LVD-142M
Benchmarks: ImageNet3D
Metrics: pose estimation accuracy at pi/6 pose estimation accuracy at pi/18 median pose error 3D-aware classification accuracy
Baselines: DeIT III MAE DINO DINO v2 CLIP MiDaS ResNet50-General SwinTrans-T-General NMM-Sphere NOVUM LLaVA-pose
Builds on: ObjectNet3D Grounding DINO Objaverse NOVUM LLaVA
Closest papers by meaning
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.90)
- Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature Space (CVPR 2025 · similarity 0.89)
- 3DCoMPaT200: Language Grounded Large-Scale 3D Vision Dataset for Compositional Recognition (NeurIPS 2024 · similarity 0.89)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.89)
- 3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation (ICLR 2024 · similarity 0.89)
- LabelAny3D: Label Any Object 3D in the Wild (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.