Research map / Vision-language and multimodal / Human and embodied modeling
Insect-Foundation: A Foundation Model and Large-scale 1M Dataset for Visual Insect Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Insect-1M, a 1M-image hierarchical insect dataset with taxonomy and descriptions, and Insect-Foundation, a ViT-B self-supervised model with patch-wise relevant attention and description losses that reaches SOTA on IP102 classification and detection.
Problem
Existing insect datasets are small and lack species diversity and textual descriptions, so vision models fail to capture the fine-grained micro-features needed for insect recognition in precision agriculture.
Approach
A self-supervised pre-text task where a contextual token from a partially visible insect image scores relevance against shuffled patches from a pool of other images (Patch-wise Relevant Attention), combined with image-text contrastive and autoregressive description-decoding losses using insect taxonomic descriptions.
Key results
- Insect-Foundation (ViT-B/16) reaches 75.8% Acc@1 on IP102 classification with descriptions, versus 72.0% for MAE pre-trained on Insect-1M and 71.6% for ImageNet-1K ViT.
- Zero-shot IP102 classification accuracy of 49.9%, above CoCa (45.3%), LiT (43.6%) and CLIP (41.1%).
- IP102 detection with FPN: 36.6 AP and 59.1 AP.50, above MAE pre-trained on Insect-1M (34.7 AP) and ImageNet ViT (32.8 AP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train Insect-Foundation on 1M-Insect with insect descriptions | Insect-Foundation · IP102 Classification | Acc@1: 73.3 -> 75.8 | Helped |
| Use Insect-Foundation pre-training instead of MAE pre-training | FPN with ViT backbone · IP102 Detection | AP: 34.7 -> 36.6 | Helped |
| Use Insect-Foundation image and text encoders instead of CoCa | Zero-shot insect classification · IP102 Zero-shot Classification | accuracy: 45.3 -> 49.9 | Helped |
| Add attention pooling (ablation) | Insect-Foundation with ViT-small/16 · IP102 Classification | Acc@1: 68.9% to 69.5% | Helped |
| Add image-text contrastive loss (ablation) | Insect-Foundation with ViT-small/16 · IP102 Classification | Acc@1: 69.5 -> 70.7 | Helped |
| Add description loss (ablation) | Insect-Foundation with ViT-small/16 · IP102 Classification | Acc@1: 70.7% to 71.5% | Helped |
| Increase backbone size from ViT-small/16 to ViT-base/16 (ablation) | Insect-Foundation · IP102 Classification | Acc@1: 71.5 -> 75.8 | Helped |
| Increase backbone size from ViT-base/16 to ViT-large/16 (ablation) | Insect-Foundation · IP102 Classification | Acc@1: 75.8 -> 76.9 | Helped |
Limitations
- Patch-wise Relevant Attention treats background and foreground patches equally, which may hinder learning insect-specific features.
- Evaluation limited to IP102 classification and detection; the dataset provides only labels and links rather than images, per the paper's rights statement.
Experimental setup
Datasets: Insect-1M IP102 ImageNet-1K
Benchmarks: IP102 Classification IP102 Detection
Metrics: Acc@1 Acc@5 AP AP.50 AP.75
Baselines: MoCo DINO MAE CoCa CLIP LiT Micron-BERT Jigsaw-ViT ViT (ImageNet-1K)
Builds on: MAE CoCa BERT ViT Faster R-CNN
Closest papers by meaning
- Open-Insect: Benchmarking Open-Set Recognition of Novel Species in Biodiversity Monitoring (NeurIPS 2025 · similarity 0.88)
- BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity (NeurIPS 2024 · similarity 0.88)
- AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmark (NeurIPS 2025 · similarity 0.87)
- BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity (NeurIPS 2024 · similarity 0.87)
- BioCLIP: A Vision Foundation Model for the Tree of Life (CVPR 2024 · similarity 0.83)
- CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.