Research map / Video / Video segmentation and tracking
Web-Scale Collection of Video Data for 4D Animal Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a fully automated YouTube-mining pipeline that produces 30K object-centric animal video clips (2M frames), releases the Animal-in-Motion benchmark (230 curated sequences), and proposes 4D-Fauna, a sequence-optimized extension of 3D-Fauna for 4D quadruped reconstruction.
Problem
Existing animal video datasets are small (about 2.4K 15-frame clips) and not processed for 4D animal reconstruction, which needs masks, keypoints, and object-centric crops; there is also no benchmark or established baseline for 4D quadruped reconstruction.
Approach
Automatically mine and process YouTube videos through a multi-stage engine (scene splitting, CLIPScore filtering, Grounded-SAM-2 tracking with heuristic filters, and auxiliary feature extraction), then adapt the pretrained model-free 3D-Fauna with per-frame test-time optimization using keypoint loss and temporal smoothness losses.
Key results
- SMALify achieves the best 2D metrics (IoU 0.867, PCK@0.1 0.954) on Animal-in-Motion but shows unrealistic 3D shapes in failure cases.
- 4D-Fauna improves over 3D-Fauna on all reported metrics: IoU 0.670 to 0.814, PCK@0.1 0.470 to 0.664, KT-PCK@0.1 0.329 to 0.418, MPJVE 0.058 to 0.044.
- The automated pipeline yields 29,979 video clips totaling 2,046,414 frames, with 230 benchmark sequences (11,061 frames) after manual curation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add per-sequence optimization, keypoint supervision, and smoothness losses | 3D-Fauna · Animal-in-Motion | IoU: 0.670 -> 0.814 | Helped |
| Add per-sequence optimization, keypoint supervision, and smoothness losses | 3D-Fauna · Animal-in-Motion | PCK@0.1: 0.470 -> 0.664 | Helped |
| Add per-sequence optimization, keypoint supervision, and smoothness losses | 3D-Fauna · Animal-in-Motion | PCK@0.05: 0.177 -> 0.317 | Helped |
| Add per-sequence optimization, keypoint supervision, and smoothness losses | 3D-Fauna · Animal-in-Motion | KT-PCK@0.1: 0.329 -> 0.418 | Helped |
| Add per-sequence optimization, keypoint supervision, and smoothness losses | 3D-Fauna · Animal-in-Motion | KT-PCK@0.05: 0.130 -> 0.193 | Helped |
| Add per-sequence optimization, keypoint supervision, and smoothness losses | 3D-Fauna · Animal-in-Motion | MPJVE: 0.058 -> 0.044 | Helped |
Surprising findings
- 2D metrics favor the model-based SMALify even though it produces unrealistic 3D shapes, so the benchmark's 2D metrics do not reflect 3D naturalness.
- The model-free 4D-Fauna remains below SMALify on all metrics, including IoU and PCK, despite being more natural visually.
Limitations
- Automatically processed data is not perfectly clean and still needs manual validation for reliable benchmarking.
- Benchmark relies on 2D projection-based metrics that cannot fully capture 3D reconstruction quality due to view ambiguity.
- 4D-Fauna shows limited temporal coherence understanding since it is a per-frame model with added smoothness losses; raw RGB frames are withheld for copyright reasons.
Experimental setup
Datasets: YouTube videos (collected) AnimalKingdom (motion type labels) APT-36K (prior comparison only)
Benchmarks: Animal-in-Motion
Metrics: silhouette IoU PCK@0.1 PCK@0.05 KT-PCK@0.1 KT-PCK@0.05 MPJVE
Baselines: SMALify AniMer 3D-Fauna
Builds on: 3D-Fauna MagicPony Grounded-SAM-2 SAM-2 DINOv2
Closest papers by meaning
- SAM 3D Animal: Promptable Animal 3D Reconstruction from Images in the Wild (NeurIPS 2026 · similarity 0.89)
- Animal-Bench: Benchmarking Multimodal Video Models for Animal-centric Video Understanding (NeurIPS 2024 · similarity 0.89)
- Reconstructing Animals and the Wild (CVPR 2025 · similarity 0.88)
- BEAST3D: animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting (NeurIPS 2026 · similarity 0.88)
- Learning the 3D Fauna of the Web (CVPR 2024 · similarity 0.86)
- WildBox: A Dataset and Benchmark for Aerial Monocular 3D Detection of African Savanna Wildlife (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.