Research map / Vision-language and multimodal / Multimodal LLMs

LL-Bench: Rethinking Low-Level Vision Evaluation in the Era of Large-Scale Generative Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LL-Bench, a human-preference benchmark of 16 low-level vision tasks evaluating 10 large-scale generative models and 21 restoration models, and LL-Score, a Qwen3-VL-based one-for-all quality and hallucination evaluator that also serves as an RL reward.

Problem

Large-scale generative models are increasingly applied to low-level vision tasks requiring pixel-level fidelity, but their capabilities and failure modes are understudied, and existing FR/NR IQA metrics correlate poorly with human preferences on restored images.

Approach

A large expert-annotated pairwise preference and hallucination benchmark for restoration outputs from generative and conventional models, plus an MLLM evaluator trained with a rank-aware margin Bradley-Terry loss and a hallucination head, which is then reused as a GRPO reward.

Key results

• LL-Score achieves overall SRCC of 0.6600 and pairwise accuracy of 0.8178 on LL-Bench, the best among 17 IQA/reward baselines (next best DeQA-Score-style MLLM and EditReward at ~0.35 SRCC).
• All evaluated large-scale generative models show higher hallucination rates than specialist (4.09%) and all-in-one (3.98%) baselines; NanoBanana has the lowest LGM rate at 7.9%.
• GRPO fine-tuning of Qwen-Image-Edit with LL-Score as reward yields sharper edges and more faithful details on super-resolution (qualitative results).

Surprising findings

• Full-reference metrics such as LPIPS and DISTS outperform many no-reference metrics on several restoration tasks, while NR metrics can beat FR on general restoration.
• Even the strongest commercial generative models exhibit substantially higher hallucination rates than conventional specialist restoration methods.

Limitations

• Reward-model experiment is limited to a single task (super-resolution) and a single editing backbone.
• Hallucination detection is three-type taxonomy derived from qualitative inspection of failure cases.

Experimental setup

Datasets: LL-Bench RealBlur_J RealSR DRealSR LoLv2 RealDOF

Benchmarks: LL-Bench

Metrics: SRCC PLCC KRCC pairwise accuracy accuracy precision recall F1 AUROC

Baselines: PSNR SSIM LPIPS DISTS NIQE BRISQUE MANIQA HyperIQA TOPIQ CLIP-IQA MUSIQ DeQA-Score

Builds on: Qwen3-VL Flow-GRPO Q-Align DeQA-Score

Closest papers by meaning

• Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.90)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
• UARE: A Unified Vision-Language Model for Image Quality Assessment, Restoration, and Enhancement (CVPR 2026 · similarity 0.89)
• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.89)
• Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMs (ICLR 2025 · similarity 0.88)
• MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs (ICML 2026 · similarity 0.88)