Research map / Vision-language and multimodal / Cross-modal learning
MixEval-X: Any-to-any Evaluations from Real-world Data Mixture
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MixEval-X, an any-to-any real-world benchmark built via benchmark mixture (MMU) and adaptation-rectification pipelines (MMG and agent tasks) to reduce query, grading, and generalization biases.
Problem
Current multimodal evaluations are fragmented across communities and suffer from query, grading, and generalization biases, so their results do not reliably reflect real-world performance across input-output modalities.
Approach
Reconstructs benchmark distributions by matching web-mined real-world user queries to similar ground-truth benchmark items (benchmark mixture for MMU), and uses an LLM-driven adaptation-rectification pipeline to create MMG and agent tasks aligned with real-world queries; grades MMU with model parsers, MMG with crowd-sourced Bradley-Terry Elo, and agent tasks with frontier LLM judges.
Key results
- Image2Text rankings correlate with Vision Arena at 98.1% and Arena (Vision) at 96.3% Spearman correlation; Image2Text-Hard reaches 94.5% and 95%.
- Model judges show low average correlation with crowd-sourced human preferences on MMG tasks (about 78% on average).
- Gemini 1.5 Pro ranks first on the Audio2Text general split at 62.7% accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MixEval-X Image2Text evaluation | Image2Text model rankings · Vision Arena | Spearman's model ranking correlation: 98.1% | Helped |
| Use MixEval-X Image2Text evaluation | Image2Text model rankings · Arena (Vision) | Spearman's model ranking correlation: 96.3% | Helped |
| Use the Image2Text-Hard split | Image2Text model rankings · Vision Arena | Spearman's model ranking correlation: 94.5% | Helped |
| Use the Image2Text-Hard split | Image2Text model rankings · Arena (Vision) | Spearman's model ranking correlation: 95% | Helped |
| Use multi-modal model judges instead of crowd-sourced human preferences | MMG task evaluation · MMG tasks | correlation with human preferences: 78% on average | Hurt |
| Train Vicuna 33B on GPT-4-labeled query examples | Vicuna 33B web query detector · Web query detection benchmarks | recall: >99% | Helped |
| Train Vicuna 33B on GPT-4-labeled query examples | Vicuna 33B web query detector · Web query detection benchmarks | precision: >98% | Helped |
Surprising findings
- Model judges correlate poorly with crowd-sourced human preferences on MMG tasks (about 78% on average), suggesting VLM judges evaluate generation differently from humans.
- Text2Action model rankings differ from Text2Text rankings, indicating strong text understanding does not guarantee textual agent proficiency.
Limitations
- Correlation with real-world leaderboards can only be verified for Image2Text; other modalities lack stable real-world evaluations.
- MMG grading relies on crowd-sourced human preferences because model judges correlate poorly with humans, making evaluation costly.
- Open-ended MMU and interleaved tasks are not evaluated due to lack of capable judges for video and audio.
Experimental setup
Datasets: Common Crawl Wikipedia MMU benchmark pool from existing community benchmarks
Benchmarks: MixEval-X Image2Text MixEval-X Image2Text-Hard MixEval-X Video2Text MixEval-X Video2Text-Hard MixEval-X Audio2Text MixEval-X Audio2Text-Hard MixEval-X Text2Action MixEval-X Image2Action MixEval-X Text2Image MixEval-X Text2Video MixEval-X Text2Audio
Metrics: accuracy Elo rating (Bradley-Terry) win rate Spearman rank correlation C-Dist cluster distance
Baselines: GPT-4o Claude 3.5 Sonnet Gemini-1.5-Pro Flux DALL·E 3 HD Qwen2-Audio-7B
Builds on: MixEval Chatbot Arena MT-Bench
Closest papers by meaning
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.87)
- SKILL-MIX: a Flexible and Expandable Family of Evaluations for AI Models (ICLR 2024 · similarity 0.87)
- Scaling Laws for Multimodal Data Mixtures (NeurIPS 2026 · similarity 0.85)
- Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees (NeurIPS 2025 · similarity 0.85)
- SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation (NeurIPS 2024 · similarity 0.85)
- AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.