AI Paper Atlas

Research map / 3D vision / 3D generation

Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?

Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, Kaisheng Ma

ICLR 2023 · 3D vision › 3D generation · approach: 2D-to-3D pretrained representations · 20 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Transfer latent features from a pretrained-image-transformer 3D autoencoder rather than Point-BERT token targets (ablation)ACT masked point modeling on ScanObjectNN PB_T50_RS · ScanObjectNN PB_T50_RSOA (%): 82.51 -> 85.81Improved
Add latent-feature distillation targets from ACT instead of Point-BERT targets (ablation)Point-BERT teacher on ScanObjectNN PB_T50_RS · ScanObjectNN PB_T50_RSOA (%): 83.07 -> 83.69Improved
Use Point-BERT discrete-token targets with the ACT teacher (ablation)ACT on ScanObjectNN PB_T50_RS · ScanObjectNN PB_T50_RSOA (%): 85.81 -> 82.51Hurt
Add ACT feature distillation as auxiliary supervision (ablation)Point-MAE on ScanObjectNN PB_T50_RS · ScanObjectNN PB_T50_RSOA (%): 85.18 -> 86.05Improved
Use 3D positional coordinates rather than no positional embedding (ablation)ViT-B dVAE and ACT · ScanObjectNN PB_T50_RSOA (%): 84.21±0.45 -> 85.33±0.27Improved
Use 2D xy positional coordinates rather than no positional embedding (ablation)ViT-B dVAE and ACT · ScanObjectNN PB_T50_RSOA (%): 84.21±0.45 -> 85.10±0.45Improved
Freeze the pretrained DeiT-B Transformer rather than fully fine-tuning it (ablation)DeiT-B dVAE · 3D autoencoder reconstructionF-Score: 0.175 -> 0.180Improved
Add prompts to the frozen DeiT-B Transformer (ablation)DeiT-B dVAE · 3D autoencoder reconstructionF-Score: 0.180 -> 0.188Improved
Increase deep prompt count from 32 to 64 (ablation)DeiT-B dVAE · 3D autoencoder reconstructionCD-ℓ1: 23.873 -> 23.229Improved
Use a pretrained image Transformer rather than Point-BERT dVAE (ablation)3D autoencoder · 3D autoencoder reconstructionF-Score: 0.166 -> 0.193Improved
Use a larger cross-modal teacher (ablation)ACT masked point modeling · ScanObjectNNOA (%)Improved
Use BERT-B as the cross-modal teacher (ablation)ACT masked point modeling · ScanObjectNNOA (%): 85.12±0.54%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy