Research map / Vision-language and multimodal / Multimodal LLMs

FRABench and UFEval: Unified Fine-grained Evaluation with Task and Aspect Generalization

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a 112-aspect taxonomy and the FRABench pairwise dataset (60.4k pairs, 325k labels), then fine-tunes Qwen2-VL-7B into UFEval, a unified multi-task, multi-aspect MLLM evaluator that generalizes to unseen tasks and aspects.

Problem

Automated MLLM-as-a-judge evaluators are tied to specific tasks and evaluation aspects and lack large-scale, multimodal, aspect-level training data, so they do not generalize to unseen aspects or tasks.

Approach

Organize 112 aspects into a universal/task-specific aspect tree, build a large aspect-level pairwise dataset (FRABench) mixing human and GPT-4o labels, and jointly train one evaluator across NLG, IU, IG and ITIG so that aspects and tasks transfer to unseen ones.

Key results

• UFEval reaches about 85% (FRA-OOD) and 83% (FRA-OOD-H) average accuracy on task generalization and 86.2% / 83.2% on aspect generalization, outperforming Themis and LLaVA-Critic.
• On MANS, UFEval achieves 69.3% accuracy, the best among the reported baselines in the paper.
• DPO with UFEval-generated preference data improves LLaVA-Next-7B on LLaVABench.Wild from 52.9 to 61.4 and SDXL-Turbo HPSv2 from 28.1 to 29.9.

Surprising findings

• Claude-3.5 outperforms UFEval on several NLG benchmarks (SummEval, MT-Bench), which the authors note as the sole exception.
• On Winoground and GenAI-Bench, UFEval trails VisionReward only slightly (0.9 and 0.3 points), so it does not clearly beat the specialized image evaluator.

Limitations

• Evaluation is limited to four tasks spanning text and image modalities; video is left for future work.
• Most aspect labels are GPT-4o-generated rather than human-annotated; only IG labels draw on human data from ImageRewardDB.
• Unseen-aspect evaluation is partly validated by ROUGE-L overlap analysis, and the authors note that the lack of a large-scale aspect-level benchmark forced an internal split.

Experimental setup

Datasets: FRABench ImageRewardDB Pick-a-Pic RLHF-V LLaVA-RLHF HPDv2 SummEval MANS MT-Bench WildVision MLLM-as-a-Judge VLRewardBench

Benchmarks: FRA-OOD FRA-OOD-H SummEval MANS MT-Bench WildVision MLLM-as-a-Judge VLRewardBench GenAI-Bench Winoground Pick-a-Pic MMHal

Metrics: accuracy kendall tau position consistency HPSv2 ImageReward VisionReward

Baselines: GPT-4o Claude-3.5 Qwen2-VL-72B Qwen2-VL-7B Themis Auto-J Prometheus 2 LLaVA-Critic ImageReward VisionReward

Builds on: Auto-J Themis LLaVA-Critic ImageReward VisionReward X-Eval Direct Preference Optimization

Closest papers by meaning

• Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.88)
• MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.88)
• UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation (NeurIPS 2025 · similarity 0.88)
• Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception (ICML 2026 · similarity 0.87)
• Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.87)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.87)