Research map / Learning theory and algorithms / Prediction and decision losses
Incorporating Test-Time Optimization into Training with Dual Networks for Human Mesh Recovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Integrates test-time optimization into HMR training via meta-learning and uses a dual-network (main + auxiliary) design to unify training and test-time objectives, yielding a meta-model that improves per-sample test-time fitting.
Problem
Human mesh recovery from a single image: pretrained regression models are not ideal starting points for per-sample test-time optimization, and test-time objectives (2D only, no GT) differ from training objectives (with GT 3D meshes).
Approach
Perform a per-sample test-time optimization step inside each training iteration (MAML-style, first-order) so the learned meta-parameters suit test-time fitting, and add an auxiliary network that produces pseudo-GT SMPL meshes so the test-time loss matches the training loss form.
Key results
- Ours_CLIFF (OpenPose 2D joints, HR-W48) reaches 62.9 MPJPE / 39.7 PA-MPJPE / 80.1 PVE on 3DPW, versus CLIFF's 69.0 MPJPE, 43.0 PA-MPJPE, 81.2 PVE.
- With RSN joints, Ours_CLIFF achieves 62.4 MPJPE and 39.5 PA-MPJPE on 3DPW and 42.0 MPJPE and 29.1 PA-MPJPE on Human3.6M.
- On COCO-trained ablation tested on 3DPW, the full method reaches 76.7 MPJPE / 49.5 PA-MPJPE versus EFT_CLIFF's 84.6 / 54.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| integrate test-time optimization into training (meta-learning) (ablation) | CLIFF HMR · 3DPW test, COCO training | MPJPE: 84.6 -> 78.5 | Helped |
| add an auxiliary network to provide pseudo SMPL supervision (ablation) | Meta-HMR with CLIFF · 3DPW test, COCO training | MPJPE: 78.5 -> 76.7 | Helped |
| use ground-truth 2D joints instead of OpenPose joints (ablation) | Ours CLIFF · 3DPW test | MPJPE: 62.9 -> 57.8 | Helped |
| use ground-truth 2D joints instead of OpenPose joints (ablation) | Ours CLIFF · Human3.6M test | MPJPE: 43.9 -> 39.4 | Helped |
| use CLIFF instead of HMR as the regression model (ablation) | Meta-HMR · 3DPW test, OpenPose joints | MPJPE: 73.3 -> 62.9 | Helped |
| use RSN rather than OpenPose detections for test-time 2D joints (ablation) | Ours CLIFF · 3DPW test | MPJPE: 62.9 -> 62.4 | Helped |
| increase test-time optimization steps during inference (ablation) | Meta-HMR · 3DPW test | MPJPE | Helped |
| use 3 test-time optimization steps rather than 1 during training (ablation) | Meta-HMR · 3DPW test, COCO training | MPJPE: 76.6 -> 76.3 | Helped |
| use 3 test-time optimization steps rather than 1 during training (ablation) | Meta-HMR · 3DPW test, COCO training | Training time: 5.8mins -> 7.4mins | Hurt |
| set test-time learning rate to 1e-6 with training learning rate 1e-4 (ablation) | Meta-HMR · 3DPW test, COCO training | Training stability: - | Hurt |
| set test-time learning rate to 1e-3 with training learning rate 1e-4 (ablation) | Meta-HMR · 3DPW test, COCO training | Training stability: - | Hurt |
| set training learning rate to 1e-3 with test-time learning rate 1e-4 (ablation) | Meta-HMR · 3DPW test, COCO training | Training stability: - | Hurt |
Surprising findings
- Test-time optimization with EFT deteriorates after several steps, whereas the proposed method keeps improving with more steps.
- Two averaged CLIFF models with EFT (ensembling) gave only a modest gain, suggesting the gain is not merely ensembling.
Limitations
- Requires 2D joints at test time from an external detector, and accuracy depends on their quality.
- Inference is slower than regression-only and most optimization baselines when running many test-time steps (about 1.1s for 14 steps).
- Failure cases include 3D misalignment from monocular ambiguity, occlusion, and SMPL's limited flexibility for feet.
Experimental setup
Datasets: Human3.6M MPI-INF-3DHP COCO MPII 3DPW LSP-Extended
Benchmarks: 3DPW Human3.6M LSP-Extended
Metrics: MPJPE PA-MPJPE PVE 2D loss
Baselines: HMR CLIFF EFT PyMAF METRO FastMETRO ProPose ReFit NIKI PLIKS HybrIK SPIN
Builds on: MAML EFT CLIFF SPIN HMR
Closest papers by meaning
- ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization (ICML 2025 · similarity 0.90)
- GenHMR: Generative Human Mesh Recovery (AAAI 2025 · similarity 0.87)
- FactorizedHMR: A Hybrid Framework for Video Human Mesh Recovery (NeurIPS 2026 · similarity 0.86)
- Score-Guided Diffusion for 3D Human Recovery (CVPR 2024 · similarity 0.86)
- DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery (CVPR 2024 · similarity 0.86)
- MEGA: Masked Generative Autoencoder for Human Mesh Recovery (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.