Research map / Vision-language and multimodal / Human and embodied modeling

Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Human-like visual cognition in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

A position paper arguing that evaluation of visual processing (especially image restoration) should be human-centered, multi-dimensional, and semantic-aware rather than driven by single-metric IQA scores, supported by analyses of PSNR/SSIM/LPIPS and NR-IQA metric behavior on the SRIQA benchmark.

Problem

Single-metric image quality assessment (PSNR, SSIM, LPIPS, MUSIQ, MANIQA, CLIP-IQA) has drifted from human perception and user preferences for generative and perception-oriented image restoration, and the field optimizes for these scores.

Approach

Argues that metrics should not be the sole arbiter of progress; instead evaluation should be human-centered, scenario-aware, and multi-dimensional, and IQA itself must scale and become semantic-aware. Illustrates failures with metric-trend analyses on SRIQA, exploitable high-frequency bias of NR-IQA metrics, and a scenario-wise user study of SUPIR vs HAT.

Key results

• In a scenario-wise user study, SUPIR is preferred over HAT overall (83.5%), but HAT is preferred for hand-drawn scenes (77.7%) and cartoon/comic scenes (66.6%).
• Across the SRIQA benchmark, PSNR and SSIM favor PSNR-oriented models while LPIPS ranks GAN-based results above diffusion-based ones, contradicting visual judgments.
• NR-IQA metrics (MUSIQ, MANIQA, CLIP-IQA) increase steadily from PSNR-oriented to GAN to diffusion methods, yet can be inflated by over-sharpening and USM.

Surprising findings

• LPIPS, designed for perceptual quality, ranks GAN-based methods above diffusion-based ones despite visually better diffusion outputs.
• Simple processing tricks such as unsharp masking can inflate NR-IQA scores, and some no-reference metrics assign higher scores than GT images.

Limitations

• Position paper without a new method or systematic benchmark; evidence is largely illustrative case studies.
• Human evaluation is expensive and noisy; the paper acknowledges the difficulty of standardizing it.

Experimental setup

Datasets: SRIQA PIPAL KonIQ SPAQ KADID AVA LIVE CSIQ TID2013 PaQ-2-PiQ LSVQ

Benchmarks: SRIQA

Metrics: PSNR SSIM LPIPS MUSIQ MANIQA CLIP-IQA user preference rate

Baselines: PSNR SSIM LPIPS MUSIQ MANIQA CLIP-IQA

Builds on: LPIPS Q-ALIGN DeQA DepictQA

Closest papers by meaning

• UARE: A Unified Vision-Language Model for Image Quality Assessment, Restoration, and Enhancement (CVPR 2026 · similarity 0.88)
• LL-Bench: Rethinking Low-Level Vision Evaluation in the Era of Large-Scale Generative Models (NeurIPS 2026 · similarity 0.87)
• Augmenting Perceptual Super-Resolution via Image Quality Predictors (CVPR 2025 · similarity 0.87)
• FoMo: Forking Moment in Generative Trajectory as a Perceptual Distance (NeurIPS 2026 · similarity 0.87)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception (ICML 2026 · similarity 0.86)
• Multidimensional Observer Model and Perceptual Dimensions of Human Image Quality Assessment (NeurIPS 2026 · similarity 0.86)