Research map / Vision-language and multimodal / Human and embodied modeling
Polos: Multimodal Metric Learning from Human Feedback for Image Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Polos, a supervised regression metric for image captioning that fuses CLIP and SimCSE-RoBERTa features, trained on the new 131K-judgment Polaris dataset via the M2LHF framework.
Problem
Existing automatic metrics for image captioning correlate weakly with human judgments, struggle with hallucinations, and generalize poorly across diverse images and captions.
Approach
A supervised regression metric that takes candidate, multiple references, and image as input, using a parallel feature extraction mechanism (difference and Hadamard products of CLIP and SimCSE-RoBERTa features) with max aggregation over references, trained on human Likert judgments from many evaluators.
Key results
- Polos reaches Kendall correlation 57.6 on Composite, 56.4 on Flickr8K-Expert, 37.8 on Flickr8K-CF, and 57.8 on Polaris, beating RefPAC-S by 0.3, 0.5, 0.2, and 1.8 points.
- On PASCAL-50S, Polos achieves 86.5% mean accuracy, the highest among compared metrics.
- On FOIL 4-ref setting, Polos attains 95.4% accuracy versus 94.9% for RefPAC-S.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Polos instead of RefPAC-S | image-caption evaluation metrics · Composite | Kendall's τc: 57.3 -> 57.6 | Helped |
| use Polos instead of RefPAC-S | image-caption evaluation metrics · Flickr8K-Expert | Kendall's τc: 55.9 -> 56.4 | Helped |
| use Polos instead of RefPAC-S | image-caption evaluation metrics · Flickr8K-CF | Kendall's τb: 37.6 -> 37.8 | Helped |
| use Polos instead of RefPAC-S | image-caption evaluation metrics · Polaris | Kendall's τc: 56.0 -> 57.8 | Helped |
| use Polos instead of MID | image-caption evaluation metrics · PASCAL-50S Mean | accuracy: 85.2 -> 86.5 | Helped |
| use Polos instead of RefPAC-S | image-caption evaluation metrics · FOIL 1-ref | pairwise detection accuracy: 93.7 -> 93.3 | Hurt |
| use Polos instead of RefPAC-S | image-caption evaluation metrics · FOIL 4-ref | pairwise detection accuracy: 94.9 -> 95.4 | Helped |
| add difference and Hadamard-product features (ablation) | Polos · Composite | Kendall's τc: 39.3 -> 57.6 | Helped |
| add image features (ablation) | Polos · Composite | Kendall's τc: 56.8 -> 57.6 | Helped |
| add RoBERTa features (ablation) | Polos · Composite | Kendall's τc: 55.0 -> 57.6 | Helped |
| add CLIP features (ablation) | Polos · Composite | Kendall's τc: 56.0 -> 57.6 | Helped |
| use max instead of mean aggregation (ablation) | Polos · Composite | Kendall's τc: 55.1 -> 57.6 | Helped |
Surprising findings
- Ablation shows CLIP features are the most influential feature extractor, with removing CLIP causing larger drops than removing RoBERTa.
- Polos overestimates captions lacking intricate details and fails when the image contains multiple objects beyond the main subject, as in the teddy-bear example.
Limitations
- Tendency to overestimate captions lacking fine-grained details due to CLIP's coarse region-text alignment.
- Reference-free setting was deliberately not adopted, so the metric requires human reference captions.
- Polaris uses a five-point Likert scale with crowdsourced evaluators, which may introduce noise.
Experimental setup
Datasets: Polaris Composite Flickr8K-Expert Flickr8K-CF PASCAL-50S FOIL
Benchmarks: Composite Flickr8K-Expert Flickr8K-CF PASCAL-50S FOIL Polaris
Metrics: Kendall tau-c Kendall tau-b pairwise accuracy hallucination detection accuracy
Baselines: BLEU ROUGE METEOR CIDEr SPICE CLIP-S RefCLIP-S MID PAC-S RefPAC-S UMIC BERTScore
Builds on: COMET RUSE CLIPScore PAC-S BLEURT
Closest papers by meaning
- II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.84)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.84)
- Learning Multi-Dimensional Human Preference for Text-to-Image Generation (CVPR 2024 · similarity 0.83)
- Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models (ICLR 2026 · similarity 0.83)
- Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2) (NeurIPS 2024 · similarity 0.83)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.