Research map / Vision-language and multimodal / Multimodal LLMs
On Path to Multimodal Generalist: General-Level and General-Bench
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes General-Level, a 5-level synergy-based taxonomy for evaluating multimodal generalists, and General-Bench, a 702-task, 325.8K-instance multimodal benchmark spanning many modalities and comprehension/generation tasks, used to evaluate 100+ LLMs/MLLMs.
Problem
Existing MLLM benchmarks rank models by task performance and multiple-choice QA, which fails to measure whether a multimodal generalist shows genuine cross-task, cross-modal synergy toward AGI, and they under-cover non-image modalities and generation.
Approach
Classify generalists into five levels using synergy as the criterion, operationalized as outperforming SoTA specialists on tasks (with mask/harmonic-mean/averaging formulas per level), and pair it with a large benchmark that keeps native free-form output formats across modalities and comprehension/generation tasks.
Key results
- General-Bench contains 702 tasks, 145 skills, and about 325.8K instances across image, video, audio, 3D, and language modalities.
- Evaluation covers over 100 generalists (172+102 tested models) against 172 SoTA specialists.
- Per the paper, most MLLMs lack cross-task or cross-modal synergy, and even GPT-4V and GPT-4o do not achieve top ranks; no model reaches Level-5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Level-5 total synergy: no model has yet achieved it; the paper reports none found | multimodal generalists · General-Level evaluation | Level-5 score | No clear effect |
| Moving from Level-3 to Level-4 requires unified comprehension and generation with synergy between them | multimodal generalists · General-Level evaluation | Level-4 score | No clear effect |
Surprising findings
- Even GPT-4V and GPT-4o do not achieve top ranks on the General-Level leaderboard.
- No evaluated model demonstrates total synergy (Level-5), i.e., improving language intelligence via non-language modalities.
Limitations
- Level-5 is defined but no model qualifies; NLP-specialist scoring is left largely unfilled.
- Specialist reference scores are tied to SoTA choices that change over time, making rankings dynamic.
- Closed-source model results depend on API access and may not be reproducible.
Experimental setup
Datasets: General-Bench test sets from existing benchmarks (rephrased/replaced)
Benchmarks: General-Bench
Metrics: Level-2 score Level-3 score Level-4 score task completion win-over-specialist count accuracy BLEU ROUGE CIDEr FID
Baselines: GPT-4o Gemini-1.5-Pro Claude-3.5-Sonnet Qwen2-VL-72B InternVL-2.5-26B LLaVA-One-Vision-72B Unified-io-2-XXL NExT-GPT-V1.5 AnyGPT Emu2-37B
Builds on: LLaVA NExT-GPT AnyGPT Vitron HuggingGPT
Closest papers by meaning
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.93)
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.93)
- NExT-GPT: Any-to-Any Multimodal LLM (ICML 2024 · similarity 0.91)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
- MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? (ICLR 2025 · similarity 0.90)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.