Research map / 3D vision / Human pose estimation
Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a semantics-aware motion prior (MotionCLIP motion-text alignment with VLM-generated action labels) and text-guided 2D keypoint fill-in to test-time adaptation for 3D human pose estimation, improving over CycleAdapt on 3DPW, 3DHP and EgoBody.
Problem
Test-time adaptation for 3D human pose estimation suffers from depth ambiguity, over-smoothing toward average poses, and unguided predictions when 2D keypoints are occluded or truncated.
Approach
Align predicted motion segments with VLM-derived action text in MotionCLIP's shared motion-text space as a regularizer during TTA, and fill missing 2D keypoints from the 2D projection of text-aligned motion predictions when motion-text similarity exceeds a threshold, updated via EMA across epochs.
Key results
- On 3DPW with HMR, MPJPE 76.4 and PA-MPJPE 47.2 vs CycleAdapt 87.7 and 53.8 (about 12.9% and 12.3% improvement).
- On 3DHP, MPJPE 101.3 and PA-MPJPE 65.1 vs CycleAdapt 110.3 and 74.4.
- On EgoBody with HMR2.0, MPJPE 104.6 and PA-MPJPE 56.1 vs CycleAdapt 108.7 and 60.4; runtime 107.6 ms per frame vs CycleAdapt 74.1 ms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add semantics-aware motion-text alignment (ablation) | CycleAdapt on HMR · 3DPW | MPJPE: 87.7 to 79.3 | Helped |
| add 2D EMA updates (ablation) | motion-text alignment method on HMR · 3DPW | MPJPE: 79.3 to 78.0 | Helped |
| add 2D keypoint fill-in (ablation) | alignment plus 2D EMA on HMR · 3DPW | MPJPE: 78.0 to 76.4 | Helped |
| replace motion-text alignment with a motion discriminator (ablation) | CycleAdapt · 3DPW, all videos | MPJPE: 87.7 to 115.0 | Hurt |
| replace motion-text alignment with unpaired local-pose supervision (ablation) | CycleAdapt · 3DPW, all videos | MPJPE: 87.7 to 88.2 | No clear effect |
| set 2D fill-in threshold to 0.60 instead of 0.75 (ablation) | Full method on HMR · 3DPW | MPJPE: 76.4 to 76.9 | Hurt |
| set 2D fill-in threshold to 0.55 instead of 0.75 (ablation) | Full method on HMR · 3DPW | MPJPE: 76.4 to 78.1 | Hurt |
| set 2D fill-in threshold to 0.65 instead of 0.75 (ablation) | Full method on HMR · 3DPW | MPJPE: 76.4 to 77.2 | Hurt |
| set 2D fill-in threshold to 0.70 instead of 0.75 (ablation) | Full method on HMR · 3DPW | MPJPE: 76.4 to 77.0 | Hurt |
| set 2D fill-in threshold to 0.80 instead of 0.75 (ablation) | Full method on HMR · 3DPW | MPJPE: 76.4 to 77.2 | Hurt |
| set EMA α to 0.95 instead of 0.90 (ablation) | Full method on HMR · 3DPW | MPJPE: 76.4 to 77.5 | Hurt |
| use the incorrect VLM action label instead of the correct oracle label (ablation) | Full method · 3DPW VLM failure cases | MPJPE: 83.7 to 92.2 | Hurt |
Surprising findings
- Adapting with an incorrect VLM label still reduces MPJPE (213.8 to 92.2), though less than the oracle label (83.7).
- Replacing motion-text alignment with unpaired local-pose supervision achieves overall performance comparable to CycleAdapt, and a motion discriminator performs worse than CycleAdapt overall.
Limitations
- Refined motion under occlusion may not capture the actual step frequency or duration.
- Reliance on MotionCLIP limits adaptability to new motion descriptions; VLM labels can hallucinate from background context (3.6% failure rate on 3DPW frames).
Experimental setup
Datasets: Human3.6M 3DPW 3DHP EgoBody AMASS
Benchmarks: 3DPW 3DHP EgoBody
Metrics: MPJPE PA-MPJPE MPVPE
Baselines: Pre-trained HMR BOA DynaBOA DAPA CycleAdapt HMR2.0 motion discriminator unpaired local poses
Builds on: CycleAdapt MotionCLIP BOA SMPL
Closest papers by meaning
- Robust Long-Term Test-Time Adaptation for 3D Human Pose Estimation Through Motion Discretization (AAAI 2026 · similarity 0.90)
- $\text{Di}^2\text{Pose}$: Discrete Diffusion Model for Occluded 3D Human Pose Estimation (NeurIPS 2024 · similarity 0.86)
- ManiPose: Manifold-Constrained Multi-Hypothesis 3D Human Pose Estimation (NeurIPS 2024 · similarity 0.86)
- Tri-Modal Motion Retrieval by Learning a Joint Embedding Space (CVPR 2024 · similarity 0.86)
- VideoMDM: Towards 3D Human Motion Generation From 2D Supervision (NeurIPS 2026 · similarity 0.86)
- TCPFormer: Learning Temporal Correlation with Implicit Pose Proxy for 3D Human Pose Estimation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.