Research map / Vision-language and multimodal / Multimodal LLMs
Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Q-Eval-100K, a 100K-instance (60K images, 40K videos) MOS dataset for text-to-vision visual quality and alignment, and Q-Eval-Score, a Qwen2-VL-7B-based evaluator trained with CE+MSE loss, context prompts, and a Vague-to-Specific alignment strategy for long prompts.
Problem
Automated evaluation of text-to-image and text-to-video content needs decoupled visual quality and alignment scores, but existing human-annotated datasets are small, often merge the two dimensions, or use SBS labels, limiting LMM-based evaluators.
Approach
Scale up MOS annotations to 100K instances with separate quality and alignment scores, convert scores into adjective ratings inside context-prompt SFT data for an LMM, compute scores as probability-weighted rating averages, and split long prompts into vague and specific sub-prompts for alignment scoring.
Key results
- Q-Eval-Score reaches 0.732 instance-level SRCC on Q-Eval-Image visual quality vs 0.587 for Q-Align.
- On alignment, Q-Eval-Score achieves 0.822 image instance-level SRCC vs 0.768 for CLIPScore and 0.607 video instance-level SRCC vs 0.483 for BLIP2Score.
- On GenAI-Bench image alignment, Q-Eval-Score gets 0.757 SRCC vs 0.556 for VQAScore (default weights).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Q-Eval-Score instead of the strongest listed baseline | Visual-quality evaluation · Q-Eval-Image, instance level | SRCC: 0.587 -> 0.732 | Helped |
| Use Q-Eval-Score instead of the strongest listed baseline | Visual-quality evaluation · Q-Eval-Video, instance level | SRCC: 0.500 -> 0.601 | Helped |
| Use Q-Eval-Score instead of the strongest listed baseline | Alignment evaluation · Q-Eval-Image, instance level | SRCC: 0.768 -> 0.822 | Helped |
| Use Q-Eval-Score instead of the strongest listed baseline | Alignment evaluation · Q-Eval-Video, instance level | SRCC: 0.483 -> 0.607 | Helped |
| Train Q-Eval-Score on Q-Eval-100K | Alignment evaluation · GenAI-Bench Image | SRCC: 0.687 -> 0.757 | Helped |
| Train Q-Eval-Score on Q-Eval-100K | Alignment evaluation · GenAI-Bench Video | SRCC: 0.679 -> 0.717 | Helped |
| Add the Vague-to-Specific prompt strategy (ablation) | Q-Eval-Score alignment evaluation · Q-Eval-Image long prompts | SRCC: 0.591 -> 0.620 | Helped |
| Add the Vague-to-Specific prompt strategy (ablation) | Q-Eval-Score alignment evaluation · Q-Eval-Video long prompts | SRCC: 0.480 -> 0.517 | Helped |
| Add SFT training (ablation) | Q-Eval-Score visual-quality evaluation · Q-Eval-Image, instance level | SRCC: 0.071 -> 0.732 | Helped |
| Add context prompts (ablation) | Q-Eval-Score visual-quality evaluation · Q-Eval-Image, instance level | SRCC: 0.504 -> 0.732 | Helped |
| Add CE loss (ablation) | Q-Eval-Score visual-quality evaluation · Q-Eval-Image, instance level | SRCC: 0.652 -> 0.732 | Helped |
| Add MSE loss (ablation) | Q-Eval-Score visual-quality evaluation · Q-Eval-Image, instance level | SRCC: 0.665 -> 0.732 | Helped |
Surprising findings
- Visual quality scores are generally lower than alignment scores across generation models, and visual quality prediction is harder for the evaluator than alignment prediction.
- Direct alignment assessment with long prompts yields low scores due to LMM over-strictness; decomposition helps.
Limitations
- Visual quality evaluation is subjective and may vary across individuals and cultures.
- Heavy reliance on human annotations raises scalability and bias concerns.
- Video evaluation uses 1 fps frame sampling, losing temporal information.
Experimental setup
Datasets: Q-Eval-100K GenAI-Bench DOCCI AGIQA T2VQA TIFA160
Benchmarks: Q-Eval-100K GenAI-Bench AGIQA T2VQA TIFA160
Metrics: SRCC PLCC
Baselines: NIQE CLIP-IQA Q-Align IPCE CLIPScore BLIP2Score ImageReward VQAScore
Builds on: Q-Align VQAScore Qwen2-VL chain-of-thought
Closest papers by meaning
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.89)
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.89)
- Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels (ICML 2024 · similarity 0.88)
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.87)
- Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (CVPR 2025 · similarity 0.87)
- UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.