Research map / Vision-language and multimodal / Human and embodied modeling
BioCLIP: A Vision Foundation Model for the Tree of Life
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Curates TREEOFLIFE-10M (10M+ taxonomically labeled biology images) and trains BIOCLIP by continually pre-training CLIP with a contrastive objective over flattened taxonomic names, evaluated zero/few-shot on 10 fine-grained biology classification tasks.
Problem
General-domain vision-language models lack fine-grained, taxonomy-aware representations and generalization to unseen taxa for biology image tasks, and no sufficiently large ML-ready biology image dataset existed for training a tree-of-life foundation model.
Approach
Repurpose CLIP's contrastive objective by flattening each organism's taxonomy (kingdom to species) into a text string, so the autoregressive text encoder encodes hierarchical structure, combined with mixed text-type training (taxonomic, scientific, common names).
Key results
- BIOCLIP zero-shot mean accuracy 39.4% vs CLIP 21.9% (+17.5) across 10 tasks.
- BIOCLIP five-shot mean accuracy 68.8% vs OpenCLIP 55.9% (+17.3 over CLIP baseline).
- On RARE SPECIES (unseen taxa), BIOCLIP zero-shot reaches 38.0% vs CLIP 15.9%; mixed text training ablation shows TREEOFLIFE-10M (38.0) beats iNat21-2.7M (21.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on TREEOFLIFE-10M instead of general-domain CLIP data | BIOCLIP versus CLIP/OpenCLIP · 10 biology classification datasets, zero-shot mean | Top-1 accuracy: +17.5 | Helped |
| Train on TREEOFLIFE-10M instead of general-domain CLIP data | BIOCLIP versus CLIP/OpenCLIP · 10 biology classification datasets, five-shot mean | Top-1 accuracy: +17.3 | Helped |
| Use taxonomic names rather than scientific names (ablation) | BIOCLIP trained on TREEOFLIFE-1M · Rare Species | Zero-shot accuracy: 22.3→26.6; 28.0→30.4 | Helped |
| Train with mixed text types instead of a single fixed text type (ablation) | BIOCLIP trained on TREEOFLIFE-1M · Rare Species | Zero-shot accuracy: 30.9 | Helped |
| Train on 1M examples from TREEOFLIFE-10M rather than 2.7M iNat21 examples (ablation) | BIOCLIP · Rare Species | Zero-shot accuracy: 30.9 vs 21.3 | Helped |
| Use CLIP contrastive pre-training rather than hierarchical cross-entropy (ablation) | BIOCLIP trained on TREEOFLIFE-1M · 10-task mean | One-shot accuracy: 19.3→44.7 | Helped |
| Use CLIP contrastive pre-training rather than hierarchical cross-entropy (ablation) | BIOCLIP trained on TREEOFLIFE-1M · 10-task mean | Five-shot accuracy: 30.5→63.8 | Helped |
| Use hierarchical cross-entropy rather than ordinary cross-entropy (ablation) | BIOCLIP objective ablation · 10-task mean | One-shot accuracy: 16.5→19.3 | Helped |
| Use hierarchical cross-entropy rather than ordinary cross-entropy (ablation) | BIOCLIP objective ablation · 10-task mean | Five-shot accuracy: 26.2→30.5 | Helped |
| Increase training data from iNat21 to TREEOFLIFE-10M (ablation) | BIOCLIP · 10 biology classification datasets, zero-shot mean | Top-1 accuracy: +9.8 | Helped |
Surprising findings
- Hierarchical cross-entropy pre-training is far worse than CLIP-style contrastive pre-training in few-shot settings (30.5 vs 63.8 five-shot on TREEOFLIFE-1M).
- OpenCLIP zero-shot is slightly worse than original CLIP (-1.5 mean).
Limitations
- BIOCLIP is fundamentally trained for classification, limiting trait-level representations.
- Taxonomic hierarchies are noisy and labels in BIOSCAN-1M are often only at family level.
Experimental setup
Datasets: TREEOFLIFE-10M iNat21 BIOSCAN-1M EOL Birds 525 Plankton Insects Insects 2 PlantNet Fungi PlantVillage Medicinal Leaf
Benchmarks: Birds 525 Plankton Insects Insects 2 PlantNet Fungi PlantVillage Medicinal Leaf PlantDoc RARE SPECIES
Metrics: top-1 accuracy mean accuracy
Baselines: CLIP OpenCLIP Supervised-IN21K DINO iNat21-only BIOCLIP ablation
Builds on: CLIP OpenCLIP SimpleShot
Closest papers by meaning
- BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning (NeurIPS 2025 · similarity 0.91)
- BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity (NeurIPS 2024 · similarity 0.89)
- VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images (NeurIPS 2024 · similarity 0.88)
- What Do You See in Common? Learning Hierarchical Prototypes over Tree-of-Life to Discover Evolutionary Traits (ICLR 2025 · similarity 0.85)
- Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images (CVPR 2025 · similarity 0.85)
- BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.