Research map / Vision-language and multimodal / Human and embodied modeling

Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images

CVPR 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Scientific domain vision datasets · 12 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Fish-Vista, a museum-image dataset of 69,269 fish images across 4,316 species with pipeline-processed images and annotations for species classification, trait identification, and trait segmentation, benchmarked with standard CV baselines.

Problem

Existing biodiversity image datasets lack trait-level annotations (image, species, or pixel level) and mostly use natural-habitat images with cluttered backgrounds, hindering analysis of visual traits of fishes.

Approach

Curate museum fish specimen images with uniform backgrounds (Grounding DINO cropping plus SAM background removal), and pair them with species-level trait labels from knowledge bases, expert image-level labels, and pixel-level trait masks to define three tasks of increasing granularity.

Key results

• On species classification, MaxViT-T achieves the best F1 of 57.8%, while Swin-B-22k reaches 55.1% F1; zero-shot CLIP and BioCLIP are near random.
• Trait identification mAP drops substantially from in-species test sets to leave-out-species and manual-annotation test sets, showing poor generalization to unseen species.
• Trait segmentation mIoU ranges from 73.8% to 83.1% across seven segmentation models, with adipose fin and barbel IoUs very low (e.g., Mask2Former barbel IoU 0.0%).

Measured effects reported in the paper

Surprising findings

• Query2Label, despite high classification accuracy, produces extremely low IoU and scattered attention maps for traits, failing to localize them.
• Class-balanced re-weighting combined with focal loss yields only marginal gains on the long-tailed species classification task.

Limitations

• Some images remain noisy and do not clearly show traits, such as deformed fins.
• Species-level trait labels may not hold for every individual image, especially when traits are occluded.
• Segmentation set is small (6,132 images) due to labor-intensive pixel-level annotation.

Experimental setup

Datasets: GLIN museum fish images IDigBio MorphBank Phenoscape KnowledgeBase FishBase

Benchmarks: Fish-Vista FV-Classification Fish-Vista FV-Id Fish-Vista FV-Segmentation

Metrics: macro F1 accuracy mAP IoU mIoU

Baselines: VGG-19 ResNeXt-50 RegNetY-4G ViT-B16 Swin-B-22k CvT-13 MaxViT-T PVT-v2-b0 BioCLIP CLIP DINOv2 INTR

Builds on: Grounding DINO Segment Anything Model BioCLIP Query2Label

Closest papers by meaning

• BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity (NeurIPS 2024 · similarity 0.85)
• BirdCollect: A Comprehensive Benchmark for Analyzing Dense Bird Flock Attributes (AAAI 2024 · similarity 0.85)
• Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis (CVPR 2025 · similarity 0.85)
• BioCLIP: A Vision Foundation Model for the Tree of Life (CVPR 2024 · similarity 0.85)
• Enhancing Vision-Language Models with Morphological and Taxonomic Knowledge: Towards Coral Recognition for Ocean Health (AAAI 2025 · similarity 0.85)
• VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images (NeurIPS 2024 · similarity 0.84)