Research map / Vision-language and multimodal / CLIP and zero-shot
Enhancing Vision-Language Models with Morphological and Taxonomic Knowledge: Towards Coral Recognition for Ocean Health
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the HSCR16K hierarchical stony coral dataset with morphology text prompts and CORAL-Adapter, a parameter-efficient CLIP adapter with morphological and taxonomic branches for coral recognition.
Problem
Pretrained vision-language models like CLIP perform poorly on fine-grained coral species recognition because they lack coral-specific taxonomic and morphological knowledge, and existing coral datasets lack species-level taxonomy and text descriptions.
Approach
A CLIP adapter with two complementary branches: a morphological adapter that fuses last-two-layer visual features (outer product plus non-local structure block) and aligns them with morphology text embeddings, and a taxonomic adapter with a bottleneck residual; the branches are coupled by a reversed JS divergence loss and cross-entropy.
Key results
- On 16-shot coral species recognition, CORAL-Adapter reaches 77.3% accuracy versus 63.8% for CoOp and 60.7% for LP++; zero-shot CLIP gets 13.5%.
- Generalization to unseen bleaching-prone species (novel classes) reaches 60.5% vs 49.7% for LP++.
- Generalization to unexplored Atlantic ocean (novel classes) reaches 50.4% vs 30.6% for LP++.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CORAL-Adapter instead of the strongest listed baseline | CLIP ResNet50 · HSCR16K coral species recognition, 1–16 shots | accuracy: 41.4 vs 29.5 (1-shot); 50.1 vs 31.6 (2-shot); 64.8 vs 48.9 (4-shot); 66.2 vs 56.6 (8-shot); 77.3 vs 63.8 (16-shot) | Helped |
| add CORAL-Adapter instead of the strongest listed baseline | CLIP ResNet50 · HSCR16K coral genera recognition, 1–16 shots | accuracy: 37.1 vs 25.2 (1-shot); 58.9 vs 46.9 (2-shot); 70.0 vs 46.4 (4-shot); 70.0 vs 57.9 (8-shot); 75.7 vs 66.1 (16-shot) | Helped |
| use CORAL-Adapter rather than LP++ | CLIP ResNet50 · Unseen bleaching-prone coral species, novel classes | accuracy: 60.5 vs 49.7 | Helped |
| use CORAL-Adapter rather than LP++ | CLIP ResNet50 · Unexplored ocean, novel classes (Atlantic) | accuracy: 50.4 vs 30.6 | Helped |
| use coral-customized prompts instead of the original zero-shot prompts | Zero-shot CLIP · HSCR16K coral species recognition | accuracy: 13.46 -> 19.14; +5.68% | Helped |
| use coral-customized prompts instead of the original zero-shot prompts | Zero-shot CLIP · HSCR16K coral genera recognition | accuracy: 15.14 -> 41.72; +26.58% | Helped |
| add the morphological adapter (ablation) | CORAL-Adapter · HSCR16K coral species recognition, 16-shot | accuracy: 28.88 -> 77.27 | Helped |
| add the morphological adapter (ablation) | CORAL-Adapter · Unseen bleaching-prone coral species, novel classes | accuracy: 48.55 -> 60.50 | Helped |
| add the taxonomic adapter (ablation) | CORAL-Adapter · HSCR16K coral species recognition, 16-shot | accuracy: 46.50 -> 77.27 | Helped |
| add the taxonomic adapter (ablation) | CORAL-Adapter · Unseen bleaching-prone coral species, novel classes | accuracy: 54.45 -> 60.50 | Helped |
Surprising findings
- Zero-shot CLIP with coral-customized prompts improves genera recognition from 15.14% to 41.72%, showing text prompts alone carry much domain value.
Limitations
- Only 16 stony coral species from one source (XL Catlin Seaview Survey), limiting coverage.
- Few-shot accuracy remains modest (77.3% at 16 shots) and genera recognition is weaker at low shots than some baselines.
- Morphological prompts require marine biologist expertise to construct.
Experimental setup
Datasets: HSCR16K
Benchmarks: HSCR16K
Metrics: accuracy Micro Accuracy Macro Accuracy Macro F1
Baselines: Zero-shot CLIP CoOp CLIP-Adapter TaskRes LP++
Builds on: CLIP CLIP-Adapter CoOp TaskRes LP++
Closest papers by meaning
- ReefNet: A Large-Scale Dataset and Benchmark for Fine-Grained Coral Reef Recognition (NeurIPS 2026 · similarity 0.93)
- CoralSCOP: Segment any COral Image on this Planet (CVPR 2024 · similarity 0.91)
- CoralVQA: A Large-Scale Visual Question Answering Dataset for Coral Reef Image Understanding (NeurIPS 2025 · similarity 0.90)
- Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images (CVPR 2025 · similarity 0.85)
- SeafloorAI: A Large-scale Vision-Language Dataset for Seafloor Geological Survey (NeurIPS 2024 · similarity 0.84)
- BioCLIP: A Vision Foundation Model for the Tree of Life (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.