Research map / Vision-language and multimodal / Human and embodied modeling
Taxonomy-Aware Evaluation of Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes taxonomy-aware evaluation of unconstrained VLM text on fine-grained classification by mapping generated text onto taxonomy nodes (Wikidata and iNaturalist21 taxonomies) and scoring with hierarchical precision/recall/F1.
Problem
Evaluating unconstrained text generated by vision-language models for fine-grained visual categorization is hard: generated text must be mapped to label spaces, and binary exact-match accuracy gives no partial credit for less specific but correct answers.
Approach
Places VLM free-text predictions onto a taxonomy using a CLIP-based string-matching and shared-ancestor heuristic (Alg. 1), then applies hierarchical precision and recall (hP, hR, hF) to measure correctness and specificity, showing existing text similarity measures poorly capture taxonomic structure.
Key results
- Surface-level measures such as exact match and containment correlate weakly with taxonomic hP/hR (e.g., exact match ω-hP 0.01 on iNat21), while CLIP image-to-text correlates best (ω-hP 0.35 on iNat21).
- On a human-annotated set of 416 outputs, CLIP-t2t + Alg. 1 reaches hF 0.80 and 47.1% accuracy, versus 30.3% for plain CLIP-t2t and 17.5% for exact match.
- GPT-4 ranks highest on most measures but makes more specific yet erroneous predictions on iNaturalist21 than QVLChat; LLaVA has high hP but low hR, showing rankings change under taxonomy-aware measures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP image-to-text similarity to assess taxonomic similarity | iNat21 synthetic pairs · iNat21 synthetic pairs | ω-hR correlation: 0.49 | Helped |
| Use SentenceBERT representations to assess taxonomic similarity | OVEN synthetic pairs · OVEN synthetic pairs | ω-hP correlation: 0.34 | Helped |
| Use CLIP text-to-text similarity to assess taxonomic similarity | iNat21 synthetic pairs · iNat21 synthetic pairs | ω-hP correlation: 0.14 | Helped |
| Use NLI similarity to assess taxonomic similarity | OVEN synthetic pairs · OVEN synthetic pairs | ω-hP correlation: -0.17 | Hurt |
| Use CLIP text-to-text similarity with Alg. 1 for taxonomy mapping | iNaturalist21 mapping · iNaturalist21 mapping subset | hF: 0.75 -> 0.80 | Helped |
| Use CLIP image-to-text similarity with Alg. 1 for taxonomy mapping | iNaturalist21 mapping · iNaturalist21 mapping subset | hF: 0.23 -> 0.80 | Helped |
| Use CLIP text-to-text similarity with Alg. 1 rather than direct CLIP text-to-text mapping | iNaturalist21 mapping · iNaturalist21 mapping subset | accuracy: 30.3% -> 47.1% | Helped |
| Use CLIP image-to-text similarity with Alg. 1 rather than direct CLIP image-to-text mapping | iNaturalist21 mapping · iNaturalist21 mapping subset | accuracy: 5.3% -> 44.0% | Helped |
| Use exact string matching rather than the proposed taxonomy mapping | iNaturalist21 mapping · iNaturalist21 mapping subset | hF: 0.80 -> 0.39 | Hurt |
Surprising findings
- Text similarity measures such as NLI show negative correlation with taxonomic metrics in most cases.
- LLaVA ranks lowest under exact match on iNaturalist21 yet achieves high hP and strong hF.
Limitations
- Mapping unconstrained text onto taxonomy is a low-resource problem with no large dataset to train or evaluate a dedicated mapper; the heuristic introduces errors.
- Wikidata-derived taxonomies are noisy and iNaturalist21 subtrees have uneven granularity, complicating averaged measures.
- Prompt-tuning case study is a toy example with 20 tuning and 200 evaluation images.
Experimental setup
Datasets: iNaturalist21 OVEN Wikidata
Benchmarks: iNaturalist21 OVEN
Metrics: hierarchical precision hierarchical recall hierarchical F1 exact match accuracy Kendall's tau-like correlation (Kendall's omega)
Baselines: Exact Match Contained match BLEU METEOR ROUGE BERTScore SentenceBERT NLI CLIP-t2t CLIP-i2t
Builds on: hierarchical precision and recall CLIP OVEN benchmark BERTScore
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.86)
- Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy (ICLR 2024 · similarity 0.86)
- Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.86)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.86)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.86)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.