Research map / 3D vision / 3D generation
Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?
Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, Kaisheng Ma
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Transfer latent features from a pretrained-image-transformer 3D autoencoder rather than Point-BERT token targets (ablation) | ACT masked point modeling on ScanObjectNN PB_T50_RS · ScanObjectNN PB_T50_RS | OA (%): 82.51 -> 85.81 | Improved |
| Add latent-feature distillation targets from ACT instead of Point-BERT targets (ablation) | Point-BERT teacher on ScanObjectNN PB_T50_RS · ScanObjectNN PB_T50_RS | OA (%): 83.07 -> 83.69 | Improved |
| Use Point-BERT discrete-token targets with the ACT teacher (ablation) | ACT on ScanObjectNN PB_T50_RS · ScanObjectNN PB_T50_RS | OA (%): 85.81 -> 82.51 | Hurt |
| Add ACT feature distillation as auxiliary supervision (ablation) | Point-MAE on ScanObjectNN PB_T50_RS · ScanObjectNN PB_T50_RS | OA (%): 85.18 -> 86.05 | Improved |
| Use 3D positional coordinates rather than no positional embedding (ablation) | ViT-B dVAE and ACT · ScanObjectNN PB_T50_RS | OA (%): 84.21±0.45 -> 85.33±0.27 | Improved |
| Use 2D xy positional coordinates rather than no positional embedding (ablation) | ViT-B dVAE and ACT · ScanObjectNN PB_T50_RS | OA (%): 84.21±0.45 -> 85.10±0.45 | Improved |
| Freeze the pretrained DeiT-B Transformer rather than fully fine-tuning it (ablation) | DeiT-B dVAE · 3D autoencoder reconstruction | F-Score: 0.175 -> 0.180 | Improved |
| Add prompts to the frozen DeiT-B Transformer (ablation) | DeiT-B dVAE · 3D autoencoder reconstruction | F-Score: 0.180 -> 0.188 | Improved |
| Increase deep prompt count from 32 to 64 (ablation) | DeiT-B dVAE · 3D autoencoder reconstruction | CD-ℓ1: 23.873 -> 23.229 | Improved |
| Use a pretrained image Transformer rather than Point-BERT dVAE (ablation) | 3D autoencoder · 3D autoencoder reconstruction | F-Score: 0.166 -> 0.193 | Improved |
| Use a larger cross-modal teacher (ablation) | ACT masked point modeling · ScanObjectNN | OA (%) | Improved |
| Use BERT-B as the cross-modal teacher (ablation) | ACT masked point modeling · ScanObjectNN | OA (%): 85.12±0.54% | Improved |
Closest papers by meaning
- Learning 3D Representations From 2D Pre-Trained Models via Image-to-Point Masked Autoencoders (CVPR 2023 · similarity 0.90)
- Towards Compact 3D Representations via Point Feature Enhancement Masked Autoencoders (AAAI 2024 · similarity 0.89)
- Point Cloud Self-supervised Learning via 3D to Multi-view Masked Learner (ICCV 2025 · similarity 0.89)
- Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (NeurIPS 2022 · similarity 0.88)
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.87)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
