Research map / Vision-language and multimodal / CLIP and zero-shot
Automatic Image-Level Morphological Trait Annotation for Organismal Images
Vardaan Pahuja, Samuel Stevens, Alyson East, Sydne Record, Yu Su
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SAE-localized patches to MLLM prompt (ablation) | MLLM trait annotation (3 images) · trait description quality | avg rating: 3.15 -> 3.91 | Improved |
| use 3 images per latent instead of 1 (ablation) | MLLM + SAE · trait description quality | avg rating: 3.84 -> 3.91 | Improved |
| use 3 images instead of 1 for MLLM-only (ablation) | MLLM only · trait description quality | avg rating: 3.00 -> 3.15 | Improved |
| reduce SAE sparsity (alpha 8e-4 to 4e-4) (ablation) | MLLM + SAE · trait description quality, t_freq=3e-3 | avg rating: 3.87 -> 3.91 | Improved |
| reduce SAE sparsity (alpha 4e-4 to 2e-4) (ablation) | MLLM + SAE · trait description quality, t_freq=1e-2 | avg rating: 3.58 -> 3.84 | Improved |
| raise normalized frequency threshold from 3e-3 to 1e-2 (ablation) | MLLM + SAE · trait count | number of traits: 7897 -> 20 | Hurt |
| use Qwen2.5-VL-72B instead of 7B (ablation) | MLLM + SAE · human evaluation | rating | Improved |
| fine-tune BioCLIP on trait-level descriptions | BioCLIP · Insects zero-shot | accuracy: 34.8 -> 39.9 | Improved |
| trait-level vs species-level fine-tuning on Bioscan-Traits (ablation) | BioCLIP · Insects zero-shot | accuracy: 39.6 -> 39.9 | No clear effect |
| fine-tune BioCLIP 2 on trait-level descriptions | BioCLIP 2 · Insects zero-shot | accuracy: 55.3 -> 56.23 | Improved |
Closest papers by meaning
- BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity (NeurIPS 2024 · similarity 0.89)
- Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images (CVPR 2025 · similarity 0.87)
- BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning (NeurIPS 2025 · similarity 0.86)
- BioCLIP: A Vision Foundation Model for the Tree of Life (CVPR 2024 · similarity 0.86)
- BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity (NeurIPS 2024 · similarity 0.85)
- Insect-Foundation: A Foundation Model and Large-scale 1M Dataset for Visual Insect Understanding (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
