Research map / Vision-language and multimodal / Multimodal LLMs
MMReD: a Cross-Modal Benchmark for Dense Context Reasoning
Maxim Kurkin, Boris Shirokikh, Irina Abdullaeva, Viktoriia A. Chekalina, Andrey Kuznetsov
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMReD, a cross-modal benchmark that tests multi-modal reasoning over dense, information-rich long contexts where simple retrieval is insufficient.
Problem
LLMs and LVLMs with extended context windows still have limited ability to perform complex multi-modal reasoning over long, information-dense contexts.
Approach
A benchmark designed so that answering requires reasoning over dense context rather than simple retrieval, spanning multiple modalities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 5 solved in-context examples (from N<=16) (ablation) | LLMs 4B-32B · MMReD full dataset | relative improvement: +13.8% | Improved |
| SFT on N<=16 sequences (ablation) | 7B LLMs · MMReD full dataset | relative improvement: +57.7% | Improved |
| GRPO on N<=16 sequences (ablation) | DeepSeek-R1-Distill-Qwen-7B · MMReD full dataset | relative improvement: +27.8% | Improved |
| use GRPO instead of SFT (ablation) | 7B models · MMReD full dataset | relative improvement: 57.7 -> 27.8 | Hurt |
| fine-tune on short sequences (SFT or GRPO) and test on longer contexts (ablation) | Qwen2.5-7B, Falcon3-Mamba-7B, R1-Distill-Qwen-7B · MMReD long contexts | accuracy | No clear effect |
| use text input instead of images (ablation) | Qwen2.5-3B-Instruct · MMReD N=128 | relative gap (%): 6.7 -> -37.5 | Mixed |
| use text input instead of images (ablation) | GPT-4o-mini · MMReD N=64 | relative gap (%): 32.5 | Improved |
| add 5% perception noise to environment (ablation) | LLMs · MMReD NIAH and DC | accuracy | Hurt |
| replace visual rooms and characters with symbolic locations and entities (ablation) | LLMs · MMReD | Pearson correlation | No clear effect |
| increase model parameter count (ablation) | Qwen2.5-VL, InternVL2.5, R1-distilled · MMReD | accuracy | Improved |
| use reasoning-specialized LLM (R1-distilled 70B/32B) instead of GPT-4o (ablation) | LLMs · MMReD | accuracy: +2-4% initially | Improved |
| use code-specialized Qwen2.5-Coder instead of Qwen2.5 (ablation) | LLMs · MMReD | accuracy | Hurt |
Experimental setup
Benchmarks: MMReD
Closest papers by meaning
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
- MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.89)
- Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.88)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.88)
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.88)
- MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
