Research map / Vision-language and multimodal / Multimodal LLMs

VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VisJudge-Bench, a 3,090-sample expert-rated benchmark for MLLM judgment of visualization fidelity, expressiveness, and aesthetics, and fine-tunes VisJudge (Qwen2.5-VL-7B with GRPO and LoRA) that narrows the gap to human ratings relative to GPT-5.

Problem

Existing MLLM benchmarks do not measure how well models judge visualization quality and aesthetics (data fidelity, expressiveness, design), and general MLLMs diverge substantially from human expert ratings on such judgments.

Approach

A benchmark grounded in a Fidelity-Expressiveness-Aesthetics framework with six sub-dimensions and adaptive, chart-specific scoring questions, plus a domain-fine-tuned judge (VisJudge) trained with GRPO on human-annotated scores to correct score inflation and improve alignment.

Key results

• GPT-5 reaches MAE 0.553 and correlation 0.428 with human ratings on VisJudge-Bench, showing a significant gap to experts.
• VisJudge (Qwen2.5-VL-7B fine-tuned) reduces MAE to 0.421 (23.9% reduction vs GPT-5) and raises correlation to 0.687 (60.5% improvement vs GPT-5).
• Fine-tuning reduces MAE by 31.5-40.2% across Qwen2.5-VL-3B, InternVL3-8B, and Llava-v1.6-mistral-7B backbones.

Surprising findings

• Gemini-2.5-Pro is overly conservative (mean score 3.02 vs human 3.13), while most other models inflate scores; Qwen2.5-VL-7B inflates to 3.89.
• Base MLLMs as feedback models often degrade MatPlotAgent generation performance (7B average -2.39 points), while VisJudge improves all generators.

Limitations

• Focuses on static visualizations with limited coverage of dynamic and interactive content.
• Web-collected samples lack raw data, so data fidelity is judged only at the visual level.
• Point-estimate scoring may not capture diversity of human preferences.

Experimental setup

Datasets: VisJudge-Bench (web-crawled and expert-annotated visualizations) VizML dataset

Benchmarks: VisJudge-Bench

Metrics: MAE MSE Pearson correlation

Baselines: GPT-5 GPT-4o Claude-4-Sonnet Claude-3.5-Sonnet Gemini-2.5-Pro Gemini-2.0-Flash Qwen2.5-VL-72B-Instruct InternVL3-8B Llava-v1.6-mistral-7B

Builds on: GRPO LoRA MatPlotAgent HAIChart

Closest papers by meaning

• Can Vision–Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective. (ICLR 2026 · similarity 0.89)
• SpatialViz-Bench: A Cognitively-Grounded Benchmark for Diagnosing Spatial Visualization in MLLMs (ICLR 2026 · similarity 0.87)
• ViStoryBench: Comprehensive Benchmark Suite for Story Visualization (CVPR 2026 · similarity 0.86)
• OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.86)
• Bridging Cognitive Gap: Hierarchical Description Learning for Artistic Image Aesthetics Assessment (AAAI 2026 · similarity 0.86)
• Factuality Matters: When Image Generation and Editing Meet Structured Visuals (ICLR 2026 · similarity 0.86)