Research map / Vision-language and multimodal / Human and embodied modeling
Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Q-Align teaches LMMs to score visual content by training on five text-defined rating levels converted from MOS and inferring scores via softmax-weighted level expectations, and unifies IQA, IAA and VQA in one model (OneAlign).
Problem
Visual scoring (IQA, IAA, VQA) with LMMs: LMMs fail to output scores aligned with human opinions, and regression-based visual assessors generalize poorly out of distribution and struggle to be trained jointly across datasets.
Approach
Mimic human MOS collection by converting training scores into five text-defined rating levels (excellent/good/fair/poor/bad) for instruction tuning, then at inference take a close-set softmax over level-token logits and compute a weighted average (1-5) as the predicted score.
Key results
- Q-Align reaches SRCC 0.940 on KonIQ test (trained on KonIQ) and 0.887 on SPAQ cross-set, outperforming prior visual-only and CLIP-based IQA methods.
- Q-Align on AVA achieves SRCC 0.822 / PLCC 0.817, surpassing VILA (0.774) and LIQE (0.776).
- Level-based training yields about 10% average cross-dataset gain over score-based training, with 40%+ gain on SPAQ/KonIQ to KADID transfer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with discrete text-defined levels instead of scores (ablation) | Q-Align · KonIQ, SPAQ, LIVE Challenge, AGIQA, and KADID | average SRCC/PLCC: +2.2%; +3.4%; +8.1%; +3.1%; +29.6% | Helped |
| train with discrete text-defined levels instead of scores (ablation) | Q-Align · SPAQ-trained cross-dataset IQA, including KADID | average SRCC/PLCC: +1.5%; +6.2%; +6.9%; 14.9%; +52.8% | Helped |
| add SPAQ training data to KonIQ training (ablation) | Q-Align · LIVE Challenge IQA | SRCC: 0.860 -> 0.879 | Helped |
| add KADID training data to KonIQ + SPAQ (ablation) | Q-Align · CSIQ IQA | SRCC: 0.767 -> 0.845 | Helped |
| use 10% of AVA training data | Q-Align · AVA test | SRCC: 0.776 | Helped |
| unify IQA, IAA, and VQA training in one model (ablation) | OneAlign · AGIQA | SRCC: +6.8% SRCC | Helped |
| unify IQA, IAA, and VQA training in one model (ablation) | OneAlign · CSIQ | SRCC: +3.6% SRCC | Helped |
| add FAST-VQA to Q-Align (ablation) | Q-Align · LSVQ test, LSVQ 1080p, KoNViD-1k, and MaxWell test | SRCC: 0.883 -> 0.899; 0.797 -> 0.818; 0.865 -> 0.895; 0.780 -> 0.779 | Mixed |
Surprising findings
- Off-the-shelf LMMs respond with qualitative adjectives rather than numerical ratings in 96-100% of cases on LIVE Challenge images.
- Q-Align converges in 2 epochs (about 9 minutes on 4 A100s), whereas prior IQA methods need around 10 epochs.
Limitations
- Video input is sparsely sampled at 1fps, leaving room for improvement as shown by ensemble gains with FAST-VQA.
- Conversion from scores to levels is many-to-one and loses some precision (PLCC around 0.95 on training sets).
- Limited to 30 images context under 2048-token LLaMA2 window, so video is fed as sparse frame sequences.
Experimental setup
Datasets: KonIQ-10k SPAQ KADID-10k AVA LSVQ
Benchmarks: KonIQ-10k SPAQ LIVE Challenge AGIQA-3K KADID-10k LIVE CSIQ AVA LSVQ test LSVQ 1080p KoNViD-1k MaxWell
Metrics: SRCC PLCC
Baselines: NIMA DBCNN HyperIQA MUSIQ CLIP-IQA+ LIQE VILA FAST-VQA DOVER Aesthetic Predictor Training with Scores variant
Builds on: mPLUG-Owl2 Q-Bench binary softmax ITU rating levels LLaMA2
Closest papers by meaning
- Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content (CVPR 2025 · similarity 0.88)
- Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare (NeurIPS 2024 · similarity 0.87)
- Teaching Large Language Models to Regress Accurate Image Quality Scores Using Score Distribution (CVPR 2025 · similarity 0.86)
- Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models (CVPR 2024 · similarity 0.86)
- Q-Bench-Video: Benchmark the Video Quality Understanding of LMMs (CVPR 2025 · similarity 0.86)
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.