Research map / 3D vision / Gaussian splatting
Pose Splatter: A 3D Gaussian Splatting Model for Quantifying Animal Pose and Appearance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pose Splatter is a feed-forward shape-carving plus stacked 3D U-Net and Gaussian splatting model that reconstructs laboratory animal geometry from 4–6 calibrated views without annotations, and derives a rotation-invariant visual embedding for behavior analysis.
Problem
Quantifying 3D animal pose and appearance for behavior studies is limited by sparse keypoint representations needing manual labels and by mesh-based methods requiring per-frame optimization and species templates.
Approach
Combine shape carving from sparse calibrated masked views with a stacked 3D U-Net and voxel-to-Gaussian MLP, trained end-to-end with image losses (IoU + L1) for feed-forward 3DGS reconstruction; then embed pose via spherical-harmonic power spectra of ResNet-18 features of renders from a sphere of virtual cameras, with adversarial PCA to remove azimuth information.
Key results
- Pose Splatter with 5 cameras reaches IoU 0.760 and PSNR 29.0 on mouse, versus 0.732 IoU / 28.8 PSNR for GaussianObject (best per-scene baseline) while running ~30 ms per forward pass.
- In a 102-person nearest-neighbor study, visual embedding neighbors were preferred 54.0% of the time over 3D keypoint neighbors (p=1.5e-5).
- Visual embedding outperformed egocentric 3D keypoint features on six of eight annotated behavior classification tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add stacked U-Net refinement (ablation) | Pose Splatter · Mouse, Finch, and Rat (6-camera evaluation) | IoU: Mouse: 0.825 -> 0.868; Finch: 0.876 -> 0.913; Rat: 0.664 -> 0.797 | Helped |
| add stacked U-Net refinement (ablation) | Pose Splatter · Mouse, Finch, and Rat (5-camera evaluation) | IoU: Mouse: 0.748 -> 0.760; Finch: 0.838 -> 0.848; Rat: 0.688 -> 0.794 | Helped |
| add stacked U-Net refinement (ablation) | Pose Splatter · Mouse and Finch (4-camera evaluation) | IoU: Mouse: 0.701 -> 0.721; Finch: 0.675 -> 0.731 | Helped |
| reduce input camera views from 6 to 5 or 4 (ablation) | Pose Splatter · Mouse, Finch, and Rat | IoU: Mouse: 0.868 -> 0.760 (5 cam), 0.721 (4 cam); Finch: 0.913 -> 0.848 (5 cam), 0.731 (4 cam); Rat: 0.797 -> 0.794 (5 cam), | Hurt |
| fine-tune AnySplat rather than use it zero-shot | AnySplat · Mouse and Finch (4- and 5-camera evaluation) | IoU: Mouse 5 cam: 0.403 -> 0.431; Mouse 4 cam: 0.392 -> 0.415; Finch 5 cam: 0.365 -> 0.412; Finch 4 cam: 0.421 -> 0.459 | Helped |
| increase PixelSplat and MVSplat context-view count | PixelSplat and MVSplat · Sparse-view animal reconstruction | performance | No clear effect |
Surprising findings
- Feed-forward sparse-view baselines PixelSplat and MVSplat failed to reconstruct the animal (IoU about 0.42), attributed to minimal inter-view overlap.
- Per-scene 3DGS and FSGS achieve moderate PSNR/SSIM despite collapsed geometry because the white background dominates pixel metrics.
Limitations
- Shape carving assumes minimal occlusion and precise calibration, so it does not directly apply to multi-animal recordings.
- Requires roughly 4 or more calibrated cameras; performance degrades with fewer views and center/rotation estimation is less reliable with fewer than 6 cameras.
- Visual embedding is less interpretable and hard to compare across species.
Experimental setup
Datasets: mouse multi-view video (released) zebra finch multi-view video Rat7M 3D-POP (pigeon subset)
Benchmarks: mouse multi-view video zebra finch multi-view video Rat7M
Metrics: IoU L1 PSNR SSIM R^2 balanced accuracy human preference rate
Baselines: 3DGS FSGS GaussianObject PixelSplat MVSplat MagicPony 3D Fauna BANMo AnySplat VGGT SLEAP 3D keypoints
Builds on: 3D Gaussian splatting SAM2 gsplat adversarial PCA ResNet-18
Closest papers by meaning
- BEAST3D: animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting (NeurIPS 2026 · similarity 0.92)
- The PAIR-R24M Dataset for Multi-animal 3D Pose Estimation (NeurIPS 2021 · similarity 0.89)
- AP-10K: A Benchmark for Animal Pose Estimation in the Wild (NeurIPS 2021 · similarity 0.88)
- BITE: Beyond Priors for Improved Three-D Dog Pose Estimation (CVPR 2023 · similarity 0.88)
- Deformation-Aware Unpaired Image Translation for Pose Estimation on Laboratory Animals (CVPR 2020 · similarity 0.88)
- Probabilistic Prompt Distribution Learning for Animal Pose Estimation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.