Research map / 3D vision / Human pose estimation
Object Pose Estimation via the Aggregation of Diffusion Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses frozen Stable Diffusion v1-5 UNet features at timestep 0, aggregated by three trainable networks (vanilla, nonlinear, context-aware weighted), to estimate unseen-object 3D pose via template matching.
Problem
Template-based object pose estimation generalizes poorly from seen to unseen objects because image features are not discriminative enough for novel objects.
Approach
Use intermediate features from a frozen text-to-image diffusion model (Stable Diffusion) at a small timestep, and learn aggregation networks that fuse multi-layer features with increasingly expressive extractors and context-aware layer weighting.
Key results
- Unseen LM accuracy 98.2% vs 93.5% for template-pose (CWA, SD v1-5).
- Unseen O-LM average accuracy 85.9% vs 76.3% for template-pose.
- T-LESS recall VSD 71.03% vs 58.87% for template-pose using 21K templates vs 92K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Template-pose features with aggregated SD-V1-5 diffusion features | object pose estimation · LM, unseen objects | Acc15: 93.5 -> 98.2 | Helped |
| replace Template-pose features with aggregated SD-V1-5 diffusion features | object pose estimation · O-LM, unseen objects | Acc15: 76.3 -> 85.9 | Helped |
| replace Template-pose with Ours (CWA) using 21 K templates | T-LESS pose estimation · T-LESS, Avg. | Recall VSD: 58.25 -> 71.03 | Helped |
| use nonlinear aggregation instead of vanilla aggregation (ablation) | diffusion feature aggregation · LM, unseen objects | Acc15: 98.3 -> 98.2 | No clear effect |
| use nonlinear aggregation instead of vanilla aggregation (ablation) | diffusion feature aggregation · O-LM, unseen objects | Acc15: 84.6 -> 85.5 | Helped |
| use context-aware weight aggregation instead of nonlinear aggregation (ablation) | diffusion feature aggregation · LM, unseen objects | Acc15: 98.2 -> 98.2 | No clear effect |
| use context-aware weight aggregation instead of nonlinear aggregation (ablation) | diffusion feature aggregation · O-LM, unseen objects | Acc15: 85.5 -> 85.9 | Helped |
| replace SD-V1-5 with SD-V2-0 (ablation) | context-aware diffusion feature aggregation · LM, unseen objects | Acc15: 98.2 -> 98.4 | No clear effect |
| replace SD-V1-5 with SD-V2-0 (ablation) | context-aware diffusion feature aggregation · O-LM, unseen objects | Acc15: 85.9 -> 85.2 | Hurt |
| increase the diffusion timestep from 0 (ablation) | vanilla aggregation network · LM and O-LM, seen split #1 | accuracy | Hurt |
| replace SD-V1-5 features with OpenCLIP features (ablation) | context-aware feature aggregation · LM and O-LM | accuracy | Mixed |
| replace SD-V1-5 features with DINOv2 features (ablation) | context-aware feature aggregation · LM and O-LM | accuracy | Mixed |
Surprising findings
- Stable Diffusion features outperform discriminative DINOv2 and OpenCLIP backbones on most benchmarks for pose estimation.
Limitations
- Evaluation restricted to template-based pose estimation with ground-truth 2D localization/cropping.
- Larger aggregation networks risk overfitting given limited training data, motivating zero-initialization.
Experimental setup
Datasets: LINEMOD Occlusion-LINEMOD T-LESS
Benchmarks: LINEMOD Occlusion-LINEMOD T-LESS
Metrics: Acc15 (accuracy at 15 degrees geodesic error) Recall VSD
Baselines: Template-pose MPL Zhao et al. Wohlhart et al. Balntas et al. Implicit DINOv2 OpenCLIP
Builds on: template-pose Stable Diffusion InfoNCE ControlNet zero-initialization
Closest papers by meaning
- Confronting Ambiguity in 6D Object Pose Estimation via Score-Based Diffusion on SE(3) (CVPR 2024 · similarity 0.89)
- Cameras as Rays: Pose Estimation via Ray Diffusion (ICLR 2024 · similarity 0.89)
- 6D-Diff: A Keypoint Diffusion Framework for 6D Object Pose Estimation (CVPR 2024 · similarity 0.88)
- DiffusionRegPose: Enhancing Multi-Person Pose Estimation using a Diffusion-Based End-to-End Regression Approach (CVPR 2024 · similarity 0.88)
- $\text{Di}^2\text{Pose}$: Discrete Diffusion Model for Occluded 3D Human Pose Estimation (NeurIPS 2024 · similarity 0.88)
- MonoDiff: Monocular 3D Object Detection and Pose Estimation with Diffusion Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.