Research map / Vision-language and multimodal / CLIP and zero-shot
CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIBD, a CLIP-style trimodal contrastive model aligning insect images, DNA barcodes, and taxonomic label text in a shared space, evaluated on BIOSCAN-1M and INSECT for zero-shot species classification and image-to-DNA retrieval.
Problem
Taxonomic classification of insects from images is hard for unseen and fine-grained species, and existing image-text models (BioCLIP) depend on taxonomic labels that are often missing at species level; DNA barcodes are available at scale but underused with images.
Approach
Use pairwise NT-Xent contrastive alignment between image, DNA barcode, and text encoders (summed into a trimodal loss) so DNA barcodes serve as the alignment target for image embeddings, enabling zero-shot retrieval over unseen species and image-to-DNA queries without task-specific fine-tuning.
Key results
- Trimodal CLIBD (I+D+T) reaches 51.2% macro H.M. species-level image-to-image accuracy on BIOSCAN-1M test vs. 5.9% for unaligned image encoder.
- Image-to-DNA species-level macro H.M. of 14.7% for CLIBD (I+D+T), versus near 0% without alignment.
- On INSECT BZSL, CLIBD-D + CLIBD-I with fine-tuning reaches 57.9% seen and 25.1% unseen species accuracy, H.M. 35.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| align image representations with DNA rather than text (ablation) | CLIBD on BIOSCAN-1M · BIOSCAN-1M test, species, image-to-image | macro accuracy H.M.: 41.5 -> 51.2 | Helped |
| add text alignment to image-and-DNA alignment (ablation) | CLIBD on BIOSCAN-1M · BIOSCAN-1M test, species, DNA-to-DNA | macro accuracy H.M.: 90.5 -> 92.9 | Helped |
| add DNA alignment to image-and-text alignment (ablation) | CLIBD on BIOSCAN-1M · BIOSCAN-1M test, species, image-to-image | macro accuracy H.M.: 41.5 -> 51.2 | Helped |
| fine-tune the image encoder with SimCLR instead of multimodal alignment (ablation) | Image encoder on BIOSCAN-1M · BIOSCAN-1M test, species, image-to-image | macro accuracy H.M.: 10.9 -> 51.2 | Hurt |
| add multimodal CLIBD alignment rather than leave the image encoder unaligned (ablation) | Image encoder on BIOSCAN-1M · BIOSCAN-1M test, species, image-to-image | macro accuracy H.M.: 5.9 -> 51.2 | Helped |
| use CLIBD encoders instead of BarcodeBERT and ViT-B encoders | BZSL on INSECT · INSECT species classification | macro accuracy H.M.: 29.3 -> 35.0 | Helped |
| add CLIBD image and DNA embeddings to BZSL | BZSL on INSECT · INSECT species classification, seen | macro accuracy: 38.4 -> 57.9 | Helped |
| add CLIBD image and DNA embeddings to BZSL | BZSL on INSECT · INSECT species classification, unseen | macro accuracy: 20.8 -> 25.1 | Helped |
| use image-to-DNA alignment | CLIBD on BIOSCAN-1M · BIOSCAN-1M test, species, image-to-DNA | macro accuracy H.M.: {0.0} -> 13.6 | Helped |
Surprising findings
- Aligning image with DNA outperforms aligning image with taxonomic text labels even at finer levels, since only 3.36% of pretraining data has species labels.
- Image-to-DNA retrieval is near chance without alignment, yet becomes feasible after contrastive alignment, while SimCLR fine-tuning alone substantially underperforms.
Limitations
- Evaluated mainly on insects from BIOSCAN-1M and INSECT; generalization to other taxa is not shown.
- Cross-modal image-to-DNA retrieval remains low in absolute accuracy, especially at species level.
- DNA barcodes are limited in availability and require specialized equipment.
Experimental setup
Datasets: BIOSCAN-1M INSECT BarcodeBERT pretraining DNA barcodes (893k)
Benchmarks: BIOSCAN-1M INSECT
Metrics: top-1 macro accuracy top-1 micro accuracy harmonic mean zero-shot accuracy
Baselines: BioCLIP BZSL SimCLR DNABERT-2 CNN DNA encoder
Builds on: CLIP BioCLIP BarcodeBERT NT-Xent Bayesian zero-shot learning
Closest papers by meaning
- BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity (NeurIPS 2024 · similarity 0.89)
- Open-Insect: Benchmarking Open-Set Recognition of Novel Species in Biodiversity Monitoring (NeurIPS 2025 · similarity 0.85)
- BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning (NeurIPS 2025 · similarity 0.84)
- BioCLIP: A Vision Foundation Model for the Tree of Life (CVPR 2024 · similarity 0.84)
- BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity (NeurIPS 2024 · similarity 0.84)
- Insect-Foundation: A Foundation Model and Large-scale 1M Dataset for Visual Insect Understanding (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.