Research map / Image generation / Autoregressive and latent generation
DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
DPMesh uses a pre-trained Stable Diffusion U-Net with ControlNet-style 2D heatmap conditioning as a single-step feature backbone for occluded human mesh recovery, plus a noisy key-point distillation loss.
Problem
Recovering 3D human meshes under severe occlusion and in crowded scenes, where image features and 2D keypoint detectors are unreliable.
Approach
Replace conventional image backbones with the pre-trained denoising U-Net of a text-to-image diffusion model, run it for a single denoising step, inject 2D keypoint heatmaps via ControlNet-style conditioning and visible joint coordinates as cross-attention prompts, and use cross-attention maps as occlusion-aware features; a noisy key-point reasoning distillation (SimCLR + MSE) improves robustness
Key results
- Achieves 70.9 MPJPE on 3DPW-OC, 82.2 on 3DPW-PC, 97.1 on 3DOH, and 79.9 on 3DPW-Crowd, outperforming JOTR (75.7, 86.5, 98.7, 82.4).
- Reaches 73.6 MPJPE and 47.4 PA-MPJPE on the 3DPW test split without fine-tuning on 3DPW train, beating JOTR (76.4/48.7).
- On CMU-Panoptic, mean MPJPE of 110.4 vs. 114.7 for JOTR; in backbone ablation the diffusion backbone beats ResNet50, HRNet-W48, ViT-L-16 and Swin-V2-L (73.6 vs. 75.2 best other on 3DPW).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DPMesh instead of the strongest baseline JOTR | 3DPW-OC · 3DPW-OC | MPJPE: 75.7 -> 70.9 | Helped |
| use DPMesh instead of the strongest baseline JOTR | 3DPW-PC · 3DPW-PC | MPJPE: 86.5 -> 82.2 | Helped |
| use DPMesh instead of the strongest baseline JOTR | 3DOH · 3DOH | MPJPE: 98.7 -> 97.1 | Helped |
| use DPMesh instead of the strongest baseline JOTR | 3DPW-Crowd · 3DPW-Crowd | MPJPE: 82.4 -> 79.9 | Helped |
| use DPMesh instead of the strongest baseline JOTR | 3DPW test split · 3DPW test split | MPJPE: 76.4 -> 73.6 | Helped |
| use DPMesh instead of the strongest baseline JOTR | CMU-Panoptic · CMU-Panoptic | MPJPE (Mean): 114.7 -> 110.4 | Helped |
| add heatmap condition cj by concatenating it with image latent z0 (ablation) | DPMesh · 3DPW | MPJPE: 87.6 -> 75.1 | Helped |
| add heatmap condition cj by concatenating it with image latent z0 (ablation) | DPMesh · 3DPW-OC | MPJPE: 89.6 -> 73.7 | Helped |
| add heatmap condition cj by element-wise addition (ablation) | DPMesh · 3DPW | MPJPE: 87.6 -> 100.3 | Hurt |
| add heatmap condition cj by element-wise addition (ablation) | DPMesh · 3DPW-OC | MPJPE: 89.6 -> 109.0 | Hurt |
| add joint-coordinate prompt ct to the heatmap condition (ablation) | DPMesh · 3DPW | MPJPE: 75.1 -> 73.6 | Helped |
| add joint-coordinate prompt ct to the heatmap condition (ablation) | DPMesh · 3DPW-OC | MPJPE: 73.7 -> 70.9 | Helped |
Surprising findings
- Naively adding the heatmap to the latent z0 (ControlNet-style element-wise add) makes results worse than having no heatmap condition at all.
- Noisy key-point reasoning gives only a slight improvement for the diffusion backbone (e.g., 74.9 to 73.6 MPJPE on 3DPW).
Limitations
- Relies on an off-the-shelf 2D keypoint detector whose noise and occlusion errors affect results.
- NKR yields only a slight gain for the diffusion backbone, so robustness gains rest mostly on the backbone and conditions.
Experimental setup
Datasets: Human3.6M MuCo-3DHP MSCOCO CrowdPose AMASS
Benchmarks: 3DPW-OC 3DPW-PC 3DOH 3DPW-Crowd 3DPW CMU-Panoptic
Metrics: MPJPE PA-MPJPE MPVE
Baselines: SPIN PyMAF ROMP OCHMR PARE 3DCrowdNet JOTR
Builds on: ControlNet Stable Diffusion SMPL SimCLR LoRA
Closest papers by meaning
- ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization (ICML 2025 · similarity 0.89)
- FactorizedHMR: A Hybrid Framework for Video Human Mesh Recovery (NeurIPS 2026 · similarity 0.87)
- $\text{Di}^2\text{Pose}$: Discrete Diffusion Model for Occluded 3D Human Pose Estimation (NeurIPS 2024 · similarity 0.87)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.87)
- Instance-aware Contrastive Learning for Occluded Human Mesh Reconstruction (CVPR 2024 · similarity 0.87)
- DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.