Research map / 3D vision / 3D generation
3D-LFM: Lifting Foundation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3D-LFM, an object-agnostic transformer that lifts single-frame 2D landmarks to canonical 3D shapes across 30+ categories without semantic correspondence, using token positional encoding, hybrid graph/global attention, and Procrustean alignment.
Problem
Single-frame 2D-to-3D lifting of landmarks typically requires object-specific models and semantic correspondence between 2D/3D training data, which limits scaling across many object categories and rigs.
Approach
A permutation-equivariant graph-transformer with analytical random Fourier token positional encoding (replacing correspondence encodings) and a Procrustean alignment step, trained as one object-agnostic model across many categories, lifting landmarks into a canonical deformable shape.
Key results
- On H3WB hidden test set, 3D-LFM reaches 64.13 mm whole-body MPJPE versus 81.5 mm for Jointformer (extra data) and 88.3 mm for Jointformer.
- On PASCAL3D+ with no object info, 3D-LFM achieves 3.97 MPJPE versus 41.08 for C3DPO without object info.
- Combined training reduces average MPJPE on the animal supercategory from 21.22 mm to 12.5 mm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train a single model without object-specific information instead of using object-specific C3DPO | 3D-LFM vs. C3DPO · PASCAL3D+ | MPJPE: 41.08 -> 3.27 | Helped |
| train 3D-LFM on a merged multi-category dataset rather than separate datasets | 3D-LFM · Animal supercategory validation | MPJPE: 21.22 mm -> 12.5 mm | Helped |
| train on a merged multi-category dataset rather than isolated datasets | 3D-LFM · PASCAL3D+ | MPJPE: 4.31 mm -> 1.1 mm | Helped |
| train on a merged multi-category dataset rather than isolated datasets | 3D-LFM · OpenMonkey | MPJPE: 19.45 mm -> 9.59 mm | Helped |
| train on a merged multi-category dataset rather than isolated datasets | 3D-LFM · AMASS | MPJPE: 1.67 mm -> 1.66 mm | No clear effect |
| add Procrustean alignment to focus on deformable shape rather than rigid transformations (ablation) | 3D-LFM · Human3.6M validation | MPJPE | Helped |
| combine local graph attention and global self-attention rather than using either alone (ablation) | 3D-LFM · Human3.6M validation | MPJPE | Helped |
| add analytical TPE instead of learnable MLP positional encoding (ablation) | 3D-LFM · Hippo (underrepresented category), model size 128 | MPJPE improvement: 3.27% | Helped |
| add analytical TPE instead of learnable MLP positional encoding (ablation) | 3D-LFM · Hippo (underrepresented category), model size 512 | MPJPE improvement: 12.28% | Helped |
| add analytical TPE instead of learnable MLP positional encoding (ablation) | 3D-LFM · Hippo (underrepresented category), model size 1024 | MPJPE improvement: 22.02% | Helped |
| add analytical TPE instead of learnable MLP positional encoding (ablation) | 3D-LFM · Rig transfer: 17-joint Human3.6M to 15-joint Panoptic | Improvement: 12% | Helped |
| add analytical TPE instead of learnable MLP positional encoding (ablation) | 3D-LFM · Rig transfer: 15-joint Panoptic to 17-joint Human3.6M | Improvement: 23.29% | Helped |
Surprising findings
- Procrustean alignment enables faster convergence and lower error even for transformers, most notably on imbalanced datasets.
- Gains from combined multi-category training are larger for small, unbalanced datasets than for large balanced ones (AMASS: 1.67 to 1.66 mm).
Limitations
- Cross-category knowledge transfer requires further investigation.
- Handling inputs with different perspectives is a potential limitation.
- OOD cheetah and chimpanzee errors are notably higher than in-distribution errors.
Experimental setup
Datasets: PASCAL3D+ AMASS Human3.6M PanOptic Hands BP4D+ OpenMonkey Animals3D H3WB Acinoset MBW
Benchmarks: H3WB PASCAL3D+ Human3.6M Acinoset MBW Panoptic Studio
Metrics: MPJPE PA-MPJPE
Baselines: C3DPO SimpleBaseline Large SimpleBaseline CanonPose Jointformer PAUL
Builds on: C3DPO PAUL Deep NRSfM++ Jointformer random Fourier features
Closest papers by meaning
- Lift3D Policy: Lifting 2D Foundation Models for Robust 3D Robotic Manipulation (CVPR 2025 · similarity 0.87)
- VGGSfM: Visual Geometry Grounded Deep Structure From Motion (CVPR 2024 · similarity 0.86)
- Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D (CVPR 2024 · similarity 0.86)
- PandaPose: 3D Human Pose Lifting from a Single Image via Propagating 2D Pose Prior to 3D Anchor Space (NeurIPS 2025 · similarity 0.85)
- Lifting by Image – Leveraging Image Cues for Accurate 3D Human Pose Estimation (AAAI 2024 · similarity 0.85)
- PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.