Research map / Vision-language and multimodal / Multimodal LLMs
SpecV: Specification Verification for Robust Unified Multimodal Evaluation
Weihao Yu, Rongyao Fang, Yuxuan Cai, Linjiang Huang, Yuhuan Yang, Xianwei Zhuang, Junyang Lin, Yixuan Yuan, Shuai Bai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace holistic judge scoring with specification verification (SVP) | VLM judges · Cross-family judge ranking stability | Kendall tau (avg): 0.856 -> 0.902 | Improved |
| replace holistic scoring with SVP | VLM judges · Intra-family judge ranking stability | Kendall tau (avg): 0.805 -> 0.867 | Improved |
| replace holistic scoring with SVP | VLM judges · Cross-family stability, Editing | Kendall tau: 0.758 -> 0.903 | Improved |
| replace holistic scoring with SVP | VLM judges · Cross-family stability, Many-to-One/Interleave | Kendall tau: 0.807 -> 0.830; intra Many-to-One 0.749 -> 0.758 | Mixed |
| replace holistic scoring with SVP | VLM judges · Agreement with human ratings (avg) | Kendall tau: 0.459 -> 0.543 | Improved |
| use multi-model SER instead of Only Claude (ablation) | SER · Human correlation (avg) | Kendall tau: 0.471 -> 0.493 | Improved |
| add cross-model quality filtering to merged specs (ablation) | SER · Human correlation (avg) | Kendall tau: 0.444 -> 0.493 | Improved |
| merge specs from three models without filtering vs best single model (ablation) | SER · Human correlation (avg) | Kendall tau: 0.471 -> 0.444 | Hurt |
| use SER instead of single-model specs on Understanding/T2I (ablation) | SER · Human correlation, T2I | Kendall tau: 0.561 (Only Claude) -> 0.516 | Hurt |
| use SER on Many-to-One (ablation) | SER · Human correlation, Many-to-One | Kendall tau: 0.312 (Only Claude) -> 0.442 | Improved |
Closest papers by meaning
- MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? (NeurIPS 2025 · similarity 0.89)
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.88)
- MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.88)
- ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.87)
- Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following (CVPR 2026 · similarity 0.87)
- PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
