Research map / Vision-language and multimodal / Multimodal LLMs
LiveXiv - A Multi-Modal live benchmark based on Arxiv papers content
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LiveXiv, a fully automated, evolving multimodal VQA/TQA benchmark generated from recent arXiv papers, plus an IRT-based method that predicts LMM performance from re-evaluating only a few models.
Problem
Static LMM benchmarks suffer from test-set contamination from web-scraped training data, and re-evaluating many models on continuously updated benchmarks is computationally costly.
Approach
Automatically generate multiple-choice VQA and table QA pairs from newly published arXiv papers using GPT-4o, filter them with a blind LLM test and cross-model agreement with Claude-Sonnet, and estimate old models' scores on new versions via Rasch/IRT models while re-evaluating only a small core set of models chosen by Fisher information.
Key results
- On LiveXiv v1 (16,328 questions), Claude-Sonnet achieves the top mean VQA accuracy (~83.5% TQA, the top performer), while GPT-4o ranks lower than expected.
- Manually verified 500-sample VQA subset changes average model accuracy by only 2.336% absolute (TQA 2.105%).
- Re-evaluating 3–5 of up to 19 models with IRT predicts the remaining models' performance with the paper reporting ~75–85% fewer evaluations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use manually verified examples rather than the full automatically filtered benchmark (ablation) | LiveXiv v1 VQA · LiveXiv v1 VQA | average accuracy across models: 46.734 -> 47.273; 2.336 | No clear effect |
| Use manually verified examples rather than the full automatically filtered benchmark (ablation) | LiveXiv v1 TQA · LiveXiv v1 TQA | average accuracy across models: 45.101 -> 46.028; 2.105 | No clear effect |
| Switch QA generation from GPT-4o to Claude-Sonnet and filtering from Claude-Sonnet to GPT-4o (ablation) | LiveXiv v1 VQA · LiveXiv v1 VQA | average accuracy change: -3.10 | Mixed |
| Switch QA generation from GPT-4o to Claude-Sonnet and filtering from Claude-Sonnet to GPT-4o (ablation) | LiveXiv v1 TQA · LiveXiv v1 TQA | average accuracy change: +8.70 | Mixed |
| Generate questions with three models rather than the standardized GPT-4o and Claude-Sonnet setup (ablation) | LiveXiv v3 · LiveXiv v3 overall and domains | prediction accuracy of efficient evaluation | Hurt |
| Re-evaluate only 3 or 5 models per benchmark version and use IRT to estimate the rest (ablation) | LiveXiv efficient evaluation · LiveXiv versions v1-v4 | performance prediction accuracy: 3 or 5 models | Helped |
Surprising findings
- GPT-4o performs unexpectedly low on LiveXiv relative to its static-benchmark rank, despite hyperparameter sweeps.
- Claude-Sonnet, which also filters questions, dominates all tasks, possibly reflecting filter bias.
Limitations
- Relies on proprietary LMMs (GPT-4o, Claude-Sonnet) whose behavior can change over time.
- Claude-Sonnet participates in filtering, which may bias results in its favor.
- Only multiple-choice questions; no free-form or reasoning-focused evaluation in v1.
Experimental setup
Datasets: arXiv papers (cs, eess, q-bio, physics categories)
Benchmarks: LiveXiv
Metrics: accuracy mean absolute error spearman rank correlation
Baselines: GPT-4o Claude-Sonnet Qwen2-VL InternVL2-8B LLaVA-1.6-34B Idefics3 Phi3v LLaVA-OneVision-7B
Builds on: ConMe tinyBenchmarks PromptEval Rasch model
Closest papers by meaning
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.86)
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.86)
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark (ICLR 2025 · similarity 0.85)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.85)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.85)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.