Research map / Vision-language and multimodal / Human and embodied modeling
ContextRef: Evaluating Referenceless Metrics for Image Description Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ContextRef, a benchmark pairing human ratings of contextualized image descriptions with ten robustness augmentations to test whether referenceless metrics (CLIPScore variants, likelihood-based VLMs) align with human judgments, and shows fine-tuning on it improves CLIP's robustness.
Problem
Referenceless image description metrics built on pretrained vision-language models may not truly align with human preferences, and existing evaluations ignore the context in which images appear and do not test sensitivity to fundamental error types.
Approach
A two-part benchmark: human ratings of contextualized descriptions along five quality questions (collected from BLV-validated scheme) plus ten context and content augmentations expected to lower scores, used to evaluate similarity-based and likelihood-based referenceless metrics and to guide fine-tuning.
Key results
- Post-image overall ratings correlate with most referenceless metrics at r≈0.4, with InstructBLIP lower at r=0.2.
- No metric passes all ten augmentations; likelihood-based Frozen and BLIP-2 raise scores for over 25% and over 50% of incompatible descriptions, respectively.
- Joint fine-tuning CLIP raises shuffled-words accuracy from 67.6% to 91.9% and exact repetition from 65.9% to 100%, while overall correlation drops from 0.36 to 0.30.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly fine-tune CLIP on augmentations and human judgment ratings (ablation) | CLIP · Overall post-image human ratings | Pearson correlation: 0.36 to 0.30 | Hurt |
| Fine-tune CLIP solely on data augmentations (ablation) | CLIP · Overall post-image human ratings | Pearson correlation: 0.36 to 0.27 | Hurt |
| Jointly fine-tune CLIP on augmentations and human judgments (ablation) | CLIP · Shuffled words | accuracy: 67.6 -> 91.9 | Helped |
| Jointly fine-tune CLIP on augmentations and human judgments (ablation) | CLIP · GPT-2 continuation long | accuracy: 65.9 -> 100.0 | Helped |
| Jointly fine-tune CLIP on augmentations and human judgments (ablation) | CLIP · Irrelevant final sentence | accuracy: 80.5 -> 100.0 | Helped |
| Jointly fine-tune CLIP on augmentations and human judgments (ablation) | CLIP · Exact repetition | accuracy: 65.9 -> 100.0 | Helped |
| Jointly fine-tune CLIP on augmentations and human judgments (ablation) | CLIP · Shuffled contexts | accuracy: 43.9 -> 48.8 | Helped |
| Jointly fine-tune Frozen on augmentations and human judgments (ablation) | Frozen · Shuffled contexts | accuracy: 58.5 -> 65.9 | Helped |
| Jointly fine-tune Frozen on augmentations and human judgments (ablation) | Frozen · Exact repetition | accuracy: 0.0 -> 0.0 | No clear effect |
| Jointly fine-tune Frozen on augmentations and human judgments (ablation) | Frozen · Irrelevant final sentence | accuracy: 2.4 -> 19.5 | Helped |
| Use similarity-based scoring rather than likelihood-based scoring | Referenceless metrics · Shuffled descriptions | sensitivity to image-description mismatch | Mixed |
Surprising findings
- Likelihood-based metrics assign higher scores to exact repetitions and irrelevant appended sentences, with Frozen preferring repeated descriptions on average.
- Fine-tuned CLIP still gives higher scores to about 40% of shuffled-context augmentations, so context sensitivity remains unsolved.
Limitations
- Small dataset of 204 data points from a single source (English Wikipedia WIT).
- Fine-tuning does not yield sufficient context sensitivity; context dependence remains challenging.
- Fine-tuning on augmentations trades off against human-correlation performance.
Experimental setup
Datasets: WIT
Benchmarks: ContextRef
Metrics: Pearson correlation augmentation accuracy
Baselines: CLIPScore BLEU CIDEr SPICE METEOR InstructBLIP
Builds on: CLIPScore OpenFlamingo BLIP-2 Frozen Flamingo
Closest papers by meaning
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.87)
- PoSh: Using Scene Graphs to Guide LLMs-as-a-Judge for Detailed Image Descriptions (ICLR 2026 · similarity 0.87)
- ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.87)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.86)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.86)
- MMLongCite: A Benchmark for Evaluating Faithfulness of Long-Context Vision-Language Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.