Research map / 3D vision / Human pose estimation
AniMer: Animal Pose and Shape Estimation Using Family Aware Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AniMer, a ViT-based transformer for SMAL animal pose and shape estimation with an animal-family supervised contrastive loss, trained on an aggregated 41.3k-image quadruped dataset that includes a new ControlNet-generated synthetic set, CtrlAni3D.
Problem
Animal pose and shape estimation across multiple quadrupedal species is underexplored due to small backbone capacity in prior methods and scarce multi-species SMAL-labeled data.
Approach
Scale up to a ViT backbone with a Transformer decoder predicting SMAL parameters directly (not residuals), add a learnable class token trained with supervised contrastive loss over animal families, use two-stage training (3D data first), and generate pixel-aligned SMAL-labeled synthetic images with ControlNet conditioned on rendered masks and depth.
Key results
- On Animal3D, AniMer reaches AUC 88.9 and PCK@HTH 89.5 versus 86.7/84.6 for HMR2.0 and 78.2/68.7 for WLDO.
- On the out-of-distribution Animal Kingdom dataset, AniMer achieves AUC 82.9 and PCK@0.1 34.9, above HMR2.0 (77.3 / 22.7).
- Pretraining the HMR baseline on CtrlAni3D improves Animal3D PCK@HTH from 60.5 (HMR) to 64.0, beating CG synthetic pretraining at 63.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AniMer instead of HMR2.0 | animal pose and shape estimation · Animal3D | PA-MPJPE: 94.1 -> 80.4 | Helped |
| Use AniMer instead of HMR2.0 | animal pose and shape estimation · CtrlAni3D | PA-MPVPE: 66.4 -> 47.6 | Helped |
| Use AniMer instead of HMR2.0 | animal pose and shape estimation · Animal Kingdom | PCK@0.1: 22.7 -> 34.9 | Helped |
| Add CtrlAni3D pretraining (ablation) | HMR · Animal3D | PA-MPJPE: 127.8 -> 121.9 | Helped |
| Add animal family supervised contrastive learning (ablation) | AniMer · Animal3D | PA-MPJPE: 82.5 -> 80.8 | Helped |
| Add animal family supervised contrastive learning (ablation) | AniMer · CtrlAni3D | PA-MPVPE: 48.4 -> 47.6 | Helped |
| Add animal family supervised contrastive learning (ablation) | AniMer · Animal Kingdom | AUC: 82.7 -> 82.9 | Helped |
| Add CtrlAni3D to training data (ablation) | AniMer trained with other datasets · Animal Kingdom | PCK@0.1: 33.7 -> 34.9 | Helped |
Surprising findings
- Without ViT pretraining (AniMer-b), performance drops sharply on Animal3D (AUC 60.6), showing backbone pretraining is critical.
- AniMer-a with a ResNet152 backbone underperforms the baseline WLDO on Animal3D AUC (75.2 vs 78.2).
Limitations
- Solution space restricted to the SMAL shape and pose space, so species like birds or goats are poorly represented.
- Motion blur and severe occlusions degrade reconstruction.
- CtrlAni3D generation requires manual filtering of generated images.
Experimental setup
Datasets: Animal Pose APT-36K AwA-Pose Stanford Extra Zebra synthetic Animal3D CtrlAni3D COCO
Benchmarks: Animal3D CtrlAni3D Animal Kingdom
Metrics: PA-MPJPE PA-MPVPE PCK@HTH PCK@0.1 PCK@0.15 AUC
Baselines: HMR WLDO HMR2.0 AniMer-a (ResNet152) AniMer-b (no ViT pretraining)
Builds on: HMR2.0 HMR Animal3D ControlNet SMAL
Closest papers by meaning
- SAOR: Single-View Articulated Object Reconstruction (CVPR 2024 · similarity 0.82)
- 3D Menagerie: Modeling the 3D Shape and Pose of Animals (CVPR 2017 · similarity 0.81)
- Implicit Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes (NeurIPS 2024 · similarity 0.78)
- CatFormer: Category-Level 6D Object Pose Estimation with Transformer (AAAI 2024 · similarity 0.78)
- Pose Splatter: A 3D Gaussian Splatting Model for Quantifying Animal Pose and Appearance (NeurIPS 2025 · similarity 0.78)
- Multiple View Geometry Transformers for 3D Human Pose Estimation (CVPR 2024 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.