Research map / Vision-language and multimodal / CLIP and zero-shot
UniAP: Towards Universal Animal Perception in Vision via Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniAP, a few-shot universal animal perception model that uses support image-label prompts, a BEiT ViT encoder with task-specific bias tuning, and a matching module to perform pose estimation, segmentation, and classification across species.
Problem
Building a single animal perception model that adapts across species (including rare or unseen ones with few labels) and across tasks like pose estimation, segmentation, and classification, despite pose variation and semantic inconsistency between tasks.
Approach
A hierarchical encoder-decoder with a shared BEiT image encoder (task-specific bias parameters), a lightweight shared label encoder, and a multi-head attention matching module that transfers prompt labels to query images, trained episodically with homoscedastic uncertainty multi-task weighting and adapted at test time by bias tuning on the prompt set.
Key results
- On pose estimation, UniAP with 30/35/40 shots reaches PCK@0.05 of 98.59 on Animal Kingdom versus 62.04 for fully supervised HRNet-w48 and 30.65 for one-shot POMNet.
- On Oxford-IIIT Pet segmentation, UniAP one-shot achieves 97.08 Acc / 93.38 mIoU, exceeding SAM (92.06 / 88.99) and Painter one-shot (86.47 / 77.72).
- On Animal Kingdom classification, UniAP reaches 93.13 Acc with 5 shots versus 90.37 for zero-shot CLIP ViT-Large.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the prompt set from 1 shot to the selected multi-shot setting | UniAP · Animal Kingdom pose estimation | PCK@0.2: 64.44 -> 99.65 | Helped |
| Increase the prompt set from 1 shot to the selected multi-shot setting | UniAP · Animal Pose pose estimation | PCK@0.2: 76.67 -> 90.10 | Helped |
| Increase the prompt set from 1 shot to the selected multi-shot setting | UniAP · APT-36K pose estimation | PCK@0.2: 85.31 -> 96.47 | Helped |
| Use UniAP instead of Painter for one-shot segmentation | UniAP · Oxford-IIIT Pet | mIoU: 77.72 -> 93.38 | Helped |
| Use UniAP instead of SAM-user | UniAP · Oxford-IIIT Pet | mIoU: 88.99 -> 93.38 | Helped |
| Use UniAP instead of CLIP-ViT-Large | Animal Kingdom · Animal Kingdom classification | Acc.: 90.37 -> 93.13 | Helped |
| Add adaptive weighted loss and task-specific bias tuning to the raw model (ablation) | UniAP · Animal Kingdom pose estimation, 30 shots | PCK@0.2: 81.90 -> 99.76 | Helped |
| Add adaptive weighted loss and task-specific bias tuning to the raw model (ablation) | UniAP · Animal Pose pose estimation, 20 shots | PCK@0.2: 77.12 -> 90.10 | Helped |
| Add adaptive weighted loss and task-specific bias tuning to the raw model (ablation) | UniAP · APT-36K pose estimation, 25 shots | PCK@0.05: 43.52 -> 86.18 | Helped |
| Add adaptive weighted loss and task-specific bias tuning to the raw model (ablation) | UniAP · Oxford-IIIT Pet segmentation, 10 shots | mIoU: 91.41 -> 94.27 | Helped |
| Add adaptive weighted loss to the raw model (ablation) | UniAP · Animal Kingdom pose estimation, 30 shots | PCK@0.2: 81.90 -> 99.65 | Helped |
| Add task-specific bias tuning to the raw model (ablation) | UniAP · Animal Kingdom pose estimation, 30 shots | PCK@0.2: 81.90 -> 87.32 | Helped |
Surprising findings
- More shots do not monotonically help: performance drops at the largest shot counts, which the authors attribute to added noise from excess prompts.
- The baseline with naive multi-task training without adaptation performs worst at 5 shots, and awl+bt tuning of only bias parameters matches or nearly matches full fine-tuning.
Limitations
- Only pose estimation, segmentation, and classification are covered; object detection, behavior recognition, and video tasks are left to future work.
- Animal perception datasets are small and few fully annotated datasets exist, limiting training.
- Test-time adaptation relies on performance that varies non-monotonically with the number of shots.
Experimental setup
Datasets: Animal Kingdom Animal Pose APT-36K Oxford-IIIT Pet ImageNet-22k
Benchmarks: Animal Kingdom Animal Pose APT-36K Oxford-IIIT Pet
Metrics: PCK@0.2 PCK@0.05 Acc. mIoU
Baselines: HRNet-w48 POMNet Painter SAM CLIP ViT-Base CLIP ViT-Large
Builds on: Painter BEiT Dense Prediction Transformer bias tuning (BitFit) homoscedastic uncertainty multi-task loss
Closest papers by meaning
- Probabilistic Prompt Distribution Learning for Animal Pose Estimation (CVPR 2025 · similarity 0.86)
- Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis (CVPR 2025 · similarity 0.85)
- Animal-Bench: Benchmarking Multimodal Video Models for Animal-centric Video Understanding (NeurIPS 2024 · similarity 0.85)
- Meow-Omni 1: A Multimodal Large Language Model for Feline Ethology (NeurIPS 2026 · similarity 0.84)
- BEAST3D: animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting (NeurIPS 2026 · similarity 0.84)
- Pose Splatter: A 3D Gaussian Splatting Model for Quantifying Animal Pose and Appearance (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.