Research map / Vision-language and multimodal / Human and embodied modeling
BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales hierarchical taxonomic contrastive vision-language training (BioCLIP 2) on the 214M-image TreeOfLife-200M dataset and reports emergent inter- and intra-species structure in the embedding space.
Problem
Whether representations learned from species-level hierarchical contrastive supervision at very large scale generalize beyond species classification, and what emergent structure (ecological alignment, preserved intra-species variation) appears in the embedding space.
Approach
Scale BioCLIP's hierarchical taxonomic contrastive training to a 214M-image, 952K-taxa dataset with a ViT-L/14 initialized from CLIP, add experience replay of LAION-2B, and analyze emergent properties with a theorem showing intra-species variation can lie orthogonal to inter-species prototypes under contrastive loss.
Key results
- BioCLIP 2 reaches a mean zero-shot species classification accuracy of 55.6% across 10 tasks, versus 37.6% for BioCLIP and 39.5% for SigLIP.
- On five non-species biological tasks, BioCLIP 2 achieves a mean of 57.5%, versus 49.0% for BioCLIP and 48.6% for DINOv3.
- Scaling training data from 1M to 214M samples yields steadily improving non-species performance; the 1M model underperforms CLIP on AwA2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale hierarchical contrastive training from BioCLIP to BioCLIP 2 | biological species classification · Species classification, zero-shot (10-task mean) | accuracy: 37.6 -> 55.6 | Helped |
| Scale hierarchical contrastive training from BioCLIP to BioCLIP 2 | biological species classification · Species classification, one-shot (10-task mean) | accuracy: 50.0 -> 64.1 | Helped |
| Scale hierarchical contrastive training from BioCLIP to BioCLIP 2 | biological species classification · Species classification, five-shot (10-task mean) | accuracy: 68.5 -> 78.3 | Helped |
| Train on larger-scale hierarchical biological data | BioCLIP 2 · Biological visual tasks beyond species classification (5-task mean) | accuracy: 49.0 -> 57.5 | Helped |
| Remove taxonomic hierarchy and train with scientific names only (ablation) | hierarchical contrastive training · FishNet | accuracy: -1.3% | Hurt |
| Replace hierarchical contrastive supervision with one-hot cross-entropy (ablation) | hierarchical contrastive training · All ablation benchmarks | accuracy | Hurt |
| Increase visual encoder capacity from ViT-B/16 to ViT-L/14 (ablation) | BioCLIP · TreeOfLife-10M ablation benchmarks | accuracy | Helped |
| Add LAION-2B replay using the shared visual projector (ablation) | biological contrastive training · Herb. 19 and PlantDoc | accuracy: >2% improvement | Helped |
| Add LAION-2B replay using the shared visual projector (ablation) | biological contrastive training · Rare Species | accuracy: 1.3% | Hurt |
| Add LAION-2B replay using a separate visual projector (ablation) | biological contrastive training · TreeOfLife-10M replay ablation, mean | accuracy: 49.0 -> 50.2 | Helped |
| Add LAION-2B replay using a separate visual projector (ablation) | biological contrastive training · INQUIRE-Rerank | accuracy: +2.1% | Helped |
| Add LAION-2B replay using a separate visual projector (ablation) | biological contrastive training · AwA2 | accuracy | No clear effect |
Surprising findings
- Intra-species variation (life stage, sex) is preserved and orthogonal to inter-species differences after species-only contrastive training, rather than collapsing.
- Freshwater and saltwater fish separate in embeddings, and Darwin's finch embeddings order by beak size, despite no trait supervision.
Limitations
- Theory covers only the near-orthogonal prototype regime; the separation of intra-species variation is not formally proved.
- TreeOfLife-200M is long-tailed and imbalanced across taxa and image types, with citizen science dominating.
- Replay with a shared projector hurt some benchmarks, requiring a separate projector design.
Experimental setup
Datasets: TreeOfLife-200M LAION-2B TreeOfLife-10M NABirds Rare Species NeWT AwA2 FishNet Herbarium 2019 PlantDoc IDLE-OO Camera Traps
Benchmarks: NABirds Rare Species Plankton Insects Insects 2 Camera Trap PlantNet Fungi PlantVillage Medicinal Leaf FishNet NeWT
Metrics: top-1 accuracy explained-variance ratio Fisher Discriminant Ratio
Baselines: CLIP (ViT-L/14) SigLIP DINOv3 BioTrove-CLIP BioCLIP Supervised-IN21K
Builds on: BioCLIP CLIP SigLIP
Closest papers by meaning
- BioCLIP: A Vision Foundation Model for the Tree of Life (CVPR 2024 · similarity 0.91)
- VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images (NeurIPS 2024 · similarity 0.87)
- BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity (NeurIPS 2024 · similarity 0.86)
- Convolution Goes Higher-Order: A Biologically Inspired Mechanism Empowers Image Classification (NeurIPS 2025 · similarity 0.85)
- CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale (ICLR 2025 · similarity 0.84)
- What Do You See in Common? Learning Hierarchical Prototypes over Tree-of-Life to Discover Evolutionary Traits (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.