Research map / Vision-language and multimodal / Multimodal LLMs
Information Density Principle for MLLM Benchmarks
Chunyi Li, Xiaozhe Li, Zicheng Zhang, Yuan Tian, Ziheng Jia, Xiaohong Liu, Xiongkuo Min, Jia Wang, Haodong Duan, Kai Chen, Guangtao Zhai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| average over time: newer benchmarks raise difficulty and cut fallacy | MLLM benchmarks · 19 MLLM benchmarks | Difficulty and Fallacy | Improved |
| newer benchmarks over time on diversity | MLLM benchmarks · 19 MLLM benchmarks | Diversity | No clear effect |
| making a benchmark harder (MME-Realworld) raises fallacy and lowers diversity | MME-Realworld · MME-Realworld | Fallacy / Diversity: Difficulty 0.666, Fallacy 0.480, Diversity 0.701 | Mixed |
| Model Eval versus Data Eval for estimating human judgments (ablation) | Information density pipeline · 19 MLLM benchmarks | correlation (SRCC/PLCC) | Mixed |
| automated evaluation agreement with human annotation (ablation) | Information density pipeline · 19 MLLM benchmarks | correlation (SRCC/PLCC): above 0.7 | Improved |
Closest papers by meaning
- Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach (ICML 2025 · similarity 0.89)
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (NeurIPS 2025 · similarity 0.87)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.87)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.87)
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.87)
- SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
