Research map / 3D vision / Human pose estimation
FusionFormer: A Concise Unified Feature Fusion Transformer for 3D Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FusionFormer, a camera-parameter-free transformer that jointly fuses multi-view and multi-frame 2D pose features in one encoder-decoder scheme for 3D human pose estimation, reporting state-of-the-art results on Human3.6M and TotalCapture.
Problem
3D human pose estimation suffers from depth uncertainty, especially without camera parameters; existing multi-view/multi-frame fusion schemes are handcrafted, fuse views and frames separately, and underperform camera-parameter-required methods.
Approach
Flatten all V views times T frames of pose features into one token sequence, fuse them with a vanilla Transformer encoder so every token pair has its own attention weight, then decode per-view poses by attending from view-specific features to the global feature, with minimal handcrafted design.
Key results
- On Human3.6M with CPN 2D poses, FusionFormer (T=27) reaches 25.3mm MPJPE vs 26.2mm for MTF-Transformer+ and 31.7mm for FLEX, the best camera-parameter-free result in the table; with ViTPose it reaches 15.1mm vs 18.6mm for MvP.
- With 2D ground-truth input, FusionFormer achieves 7.91mm MPJPE on Human3.6M versus 15.3mm for MTF-Transformer.
- FusionFormer uses 1.89M parameters, versus about 10M for the best MTF-Transformer configuration, while outperforming it at equal parameters/FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use unified multi-view and multi-frame feature fusion | FusionFormer · Human3.6M | MPJPE: 26.2 -> 25.4 | Helped |
| use unified multi-view and multi-frame feature fusion | FusionFormer · TotalCapture | MPJPE: 26.1 -> 21.7 | Helped |
| use unified multi-view and multi-frame feature fusion | FusionFormer · HumanEva | MPJPE: 22.8 -> 15.4 | Helped |
| use unified multi-view and multi-frame feature fusion | FusionFormer · MPI-INF-3DHP | MPJPE: 14.6 -> 5.4 | Helped |
| increase the number of frames from T=1 to T=9 (ablation) | FusionFormer · Human3.6M (2DGT) | MPJPE: 8.87 -> 7.32 | Helped |
| increase the number of frames from T=1 to T=27 (ablation) | FusionFormer · Human3.6M (CPN) | MPJPE: 27.3 -> 25.3 | Helped |
| use more views, from V=1 to V=4 (ablation) | FusionFormer · Human3.6M (2DGT) | MPJPE: 42.5 -> 7.91 | Helped |
| use more views, from V=1 to V=4 (ablation) | FusionFormer · Human3.6M (CPN) | MPJPE: 39.9 -> 25.3 | Helped |
| replace the EF C feature extractor with EP (ablation) | FusionFormer · Human3.6M (2DGT) | MPJPE: 9.00 -> 7.91 | Helped |
| replace the ET feature extractor with EP (ablation) | FusionFormer · Human3.6M (2DGT) | MPJPE: 8.43 -> 7.91 | Helped |
Surprising findings
- A 2-layer or 3-layer FCN feature extractor still gives competitive results (9.00mm with 2DGT for the 3-layer FCN), suggesting the fusion scheme rather than the extractor drives gains.
- FusionFormer does not beat the camera-parameter-required MTF-Transformer+ on TotalCapture overall, though the authors argue the comparison is unfair to the camera-free method.
Limitations
- Accuracy with 2D pose estimator inputs is bottlenecked by the 2D estimator (CPN results ~25mm vs 7.9mm with ground truth).
- Does not outperform camera-parameter-required MTF-Transformer+ on TotalCapture overall.
- Evaluated on few-view datasets (V up to 4), so scaling to many views is untested.
Experimental setup
Datasets: Human3.6M TotalCapture HumanEva MPI-INF-3DHP
Benchmarks: Human3.6M TotalCapture HumanEva MPI-INF-3DHP
Metrics: MPJPE P-MPJPE
Baselines: MTF-Transformer MTF-Transformer+ FLEX Multi-PPT MvP PoseFormer MHFormer AdaFuse LToHP CanonFusion
Builds on: PoseFormer MTF-Transformer ViT Vanilla Transformer
Closest papers by meaning
- SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation (AAAI 2025 · similarity 0.87)
- 3D Human Pose Perception from Egocentric Stereo Videos (CVPR 2024 · similarity 0.87)
- PoseIRM: Enhance 3D Human Pose Estimation on Unseen Camera Settings via Invariant Risk Minimization (CVPR 2024 · similarity 0.87)
- Multiple View Geometry Transformers for 3D Human Pose Estimation (CVPR 2024 · similarity 0.86)
- Adaptive Fusion of Single-View and Multi-View Depth for Autonomous Driving (CVPR 2024 · similarity 0.86)
- Lifting by Image – Leveraging Image Cues for Accurate 3D Human Pose Estimation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.