Research map / 3D vision / 3D generation
Reconstructing Animals and the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an LLM (Vicuna-7B with frozen CLIP encoder) to decode a CLIP embedding of a natural image into a compositional graphics-code scene of animals and environment, using a 1M-image synthetic Infinigen-based dataset with a CLIP-projection head for asset appearance.
Problem
Reconstructing 3D natural scenes containing animals and their environment (trees, bushes, boulders) from a single monocular image, where prior work only reconstructs isolated animals and asset names as discrete tokens generalize poorly out of distribution.
Approach
Replace discrete asset-name tokens with a special [CLIP] token whose hidden state is projected into a continuous CLIP embedding supervised by cosine similarity, enabling semantically meaningful asset estimation; combined with a [ROT] rotation head, scene-level attributes, value fuzzing and pixel-count conditioning.
Key results
- CLIP-projection head improves ID LPIPS from 0.720 (IG-LLM) to 0.654 and RAW-RW LPIPS from 0.772 to 0.762.
- Adding pixel-count conditioning and value fuzzing further reduces ID LPIPS to 0.598 and RAW-RW LPIPS to 0.724.
- BioCLIP as target embedding performs worst across metrics, while DINOv2 is best in-distribution but generalizes poorly to real images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace discrete asset names with CLIP appearance embeddings (ablation) | RAW · ID and RAW-RW | LPIPS: 0.720 -> 0.654; 0.772 -> 0.762 | Helped |
| Add ±0.5% uniform noise to scene-level training attributes (ablation) | RAW with CLIP appearance embeddings · ID | LPIPS: 0.654 -> 0.612 | Helped |
| Add ±0.5% uniform noise to scene-level training attributes (ablation) | RAW with CLIP appearance embeddings · RAW-RW | LPIPS: 0.762 -> 0.762 | No clear effect |
| Add visible pixel count for each object to the sequence (ablation) | RAW with CLIP appearance embeddings and value fuzzing · ID | LPIPS: 0.612 -> 0.598 | Helped |
| Add visible pixel count for each object to the sequence (ablation) | RAW with CLIP appearance embeddings and value fuzzing · RAW-RW | LPIPS: 0.762 -> 0.724 | Helped |
| Use BioCLIP rather than CLIP for appearance embeddings (ablation) | RAW · ID and RAW-RW | LPIPS: 0.598 -> 0.676; 0.724 -> 0.730 | Hurt |
| Use DINOv2 rather than CLIP for appearance embeddings (ablation) | RAW · ID and RAW-RW | LPIPS: 0.598 -> 0.597; 0.724 -> 0.743 | Mixed |
| Use DINOv2 rather than CLIP for appearance embeddings (ablation) | RAW · ID and RAW-RW | DINOv2 cosine similarity: 0.858 -> 0.865; 0.631 -> 0.611 | Mixed |
Surprising findings
- BioCLIP, despite being domain-specialized for taxonomic species, performed worst as the target embedding across metrics.
- DINOv2 embeddings performed best in-distribution but did not generalize effectively to real-world images.
Limitations
- Infinigen creature articulation is non-functional, so animals are static in pose.
- Struggles with highly out-of-distribution layouts or camera poses and lacks diverse assets (e.g., vehicles).
- Non-invertible, seeded asset generation makes parameters non-smooth and hard to reconstruct as code.
Experimental setup
Datasets: CC3M RAW synthetic dataset
Benchmarks: RAW synthetic in-distribution test set RAW-RW
Metrics: LPIPS CLIP cosine similarity BioCLIP cosine similarity DINOv2 cosine similarity
Baselines: IG-LLM YOLOX-6D-Pose
Builds on: IG-LLM Infinigen LLaMA CLIP
Closest papers by meaning
- SAM 3D Animal: Promptable Animal 3D Reconstruction from Images in the Wild (NeurIPS 2026 · similarity 0.90)
- Web-Scale Collection of Video Data for 4D Animal Reconstruction (NeurIPS 2025 · similarity 0.88)
- Learning the 3D Fauna of the Web (CVPR 2024 · similarity 0.88)
- BEAST3D: animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting (NeurIPS 2026 · similarity 0.87)
- WildBox: A Dataset and Benchmark for Aerial Monocular 3D Detection of African Savanna Wildlife (NeurIPS 2026 · similarity 0.87)
- Generating the Wild: Individual-Consistent Image-to-Video Generation for Wildlife (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.