Research map / 3D vision / 3D generation
2D-to-3D pretrained representations
Transfers pretrained 2D models to 3D, medical and molecular representation learning.
94 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in 3D generation, keyword cluster “semantic · occupancy · driving”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
29 papers in 2023–24 and 42 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 6.4% to 4.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Consistency of Silhouettes and Their Duals (CVPR 2016 · 3 citations)
- 2017 · Volumetric ConvNets with Mixed Residual Connections for Automated Prostate Segmentation from 3D MR Images (AAAI 2017 · 357 citations)
- 2018 · Memory Fusion Network for Multi-view Sequential Learning (AAAI 2018 · 824 citations)
- 2019 · Dissecting Person Re-Identification From the Viewpoint of Viewpoint (CVPR 2019 · 279 citations)
- 2020 · Decoupled Representation Learning for Skeleton-Based Gesture Recognition (CVPR 2020 · 103 citations)
- 2021 · Tangent Space Backpropagation for 3D Transformation Groups (CVPR 2021 · 38 citations)
- 2022 · M3T: Three-Dimensional Medical Image Classifier Using Multi-Plane and Multi-Slice Transformer (CVPR 2022 · 131 citations)
- 2023 · Uni-Mol: A Universal 3D Molecular Representation Learning Framework (ICLR 2023 · 333 citations)
- 2024 · Reliable Conflictive Multi-View Learning (AAAI 2024 · 126 citations)
- 2025 · VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging (CVPR 2025 · 69 citations)
- 2026 · Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots (ICLR 2026 · 0 citations)
Earliest papers here
- Consistency of Silhouettes and Their Duals (CVPR 2016 · 3 citations)
- Volumetric ConvNets with Mixed Residual Connections for Automated Prostate Segmentation from 3D MR Images (AAAI 2017 · 357 citations)
- Memory Fusion Network for Multi-view Sequential Learning (AAAI 2018 · 824 citations)
What has been tried
70 papers here report 752 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 4%, hurt 22%, mixed 1%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add SAM-derived 3D supervoxels for training | Zero-shot external datasets · zero-shot performance | 50% | Helped | VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging (CVPR 2025 · 69 citations) |
| add volume state estimation | R2R val unseen · SR | 72.71 -> 75.80 | Helped | Volumetric Environment Representation for Vision-Language Navigation (CVPR 2024 · 36 citations) |
| replace atom dropping with coordinate noise at 0.5 Å | Tox21 · ROC-AUC | 80.0 -> 78.6 | Hurt | Mol-AE: Auto-Encoder Based Molecular Representation Learning With 3D Cloze Test Objective (ICML 2024 · 10 citations) |
| Add static EEG features to dynamic EEG features | Object classification (72 categories) · top-1 accuracy (%) | 4.75 -> 5.91 | Helped | Neuro-3D: Towards 3D Visual Decoding from EEG Signals (CVPR 2025 · 8 citations) |
| Use adaptive λ₃ scheduling instead of training with L_seg only | 3 test blocks · Mean rank | 4.07 -> 1.41 | Helped | Learning Multimodal Volumetric Features for Large-Scale Neuron Tracing (AAAI 2024 · 8 citations) |
| Add multi-view inputs as separate batch samples | LucidAction · Spearman’s rank correlation (ρ) | 0.701 -> 0.730 | Helped | LucidAction: A Hierarchical and Multi-model Dataset for Comprehensive Action Quality Assessment (NeurIPS 2024 · 7 citations) |
| add cross-attention and GAT blocks | HER2 · PCC | 0.659 -> 0.693 | Helped | ASIGN: An Anatomy-aware Spatial Imputation Graphic Network for 3D Spatial Transcriptomics (CVPR 2025 · 6 citations) |
| Use range-null decomposition in the fidelity update | IXI, Gaussian mask, ×4 acceleration · PSNR | Helped | Null Space Matters: Range-Null Decomposition for Consistent Multi-Contrast MRI Reconstruction (AAAI 2024 · 6 citations) | |
| Add CAD to the 2D network | AC3/4, Waterz · VOI | 1.0317 -> 0.9259 | Helped | Cross-Dimension Affinity Distillation for 3D EM Neuron Segmentation (CVPR 2024 · 6 citations) |
| Add contrastive-inspired semantic alignment (CSA) | OCTA-6M · MAE | 0.0765 -> 0.0741 | Helped | MuTri: Multi-view Tri-alignment for OCT to OCTA 3D Image Translation (CVPR 2025 · 5 citations) |
| add PI to the pre-trained LFE representation | COIL-20 · ACCclu | 54.33 -> 75.67 | Helped | SeqMvRL: A Sequential Fusion Framework for Multi-view Representation Learning (CVPR 2025 · 5 citations) |
| add lesion prompts | MFIDDR test · accuracy | 81.64 -> 84.01 | Helped | Like an Ophthalmologist: Dynamic Selection Driven Multi-View Learning for Diabetic Retinopathy Grading (AAAI 2025 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 29 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- TotalSegmentator · 2 of 29 papers
- ScanNet · 2 of 29 papers
Baselines
- MedSAM · 2 of 29 papers
- SegVol · 2 of 29 papers
- nnU-Net · 2 of 29 papers
- SAM3D · 2 of 29 papers
- SuPreM · 2 of 29 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 71 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 6 papers
- Xiamen University · 4 papers
- Hong Kong University of Science and Technology · 4 papers
- Peking University · 4 papers
- The Chinese University of Hong Kong · 3 papers
- Hunan University · 2 papers
- University of Illinois, Urbana Champaign · 2 papers
- Tongji University · 2 papers
- Xidian University · 2 papers
- University of Chinese Academy of Sciences · 2 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Learning 3D reconstruction from images (196 papers)
- Semantic scene completion (153 papers)
- 3D perception for autonomous driving (73 papers)
- Vision-based 3D occupancy prediction (50 papers)
Elsewhere in the topic
- Diffusion-based novel view synthesis (156 papers)
- Generative indoor scene synthesis (131 papers)
- Hand-object interaction generation (116 papers)
- Feed-forward 3D asset generation and editing (116 papers)
- Score distillation for text-to-3D (106 papers)
- Open-vocabulary 3D scene understanding (95 papers)
- Monocular 3D object reconstruction (94 papers)
- Scene-aware human motion synthesis (82 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).