Research map / Large language models / Agents, code and math
ReMI: A Dataset for Reasoning with Multiple Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ReMI, a 13-task benchmark for reasoning over multiple images, and evaluates Gemini, Claude 3 and GPT-4 Turbo against naive and human baselines, finding a large gap to humans.
Problem
Existing vision-language benchmarks mostly use a single image and do not test complex step-by-step reasoning that integrates information across multiple images.
Approach
A holistic multi-image reasoning benchmark of 13 tasks spanning math, physics, charts, maps, code, and logic, with controlled properties (sequential vs set, same vs different concept, interleaved vs not, number of images), plus analysis showing separate image inputs can beat a single combined image.
Key results
- Gemini 1.5 achieves the best overall ReMI accuracy at 50.5%, versus 95.8% for humans and 13.1% for the naive baseline.
- GPT-4 Turbo scores 45.8% overall and Gemini Flash 44.5%; Claude 3 Sonnet reaches 35.2%.
- Feeding images separately rather than as one combined image gives positive error reduction for most tasks; interleaved tasks gain 19.8% on average versus 4.9% for non-interleaved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide images separately rather than combining them into one image | Gemini 1.5 Pro · Interleaved ReMI tasks | ERP: 19.8% | Helped |
| Provide images separately rather than combining them into one image | Gemini 1.5 Pro · Non-interleaved ReMI tasks | ERP: 4.9% | Helped |
| Add two demonstrations with manually written chain-of-thought solutions to the prompt | Gemini 1.5 Pro · ReMI | accuracy: 51.5% to 57.9% | Helped |
| Use two demonstrations with manually written chain-of-thought solutions instead of zero-shot prompting | Gemini 1.5 Pro · ReMI | relative improvement: 12.5% | Helped |
| Evaluate frontier models on multi-image reasoning | State-of-the-art models · ReMI | accuracy: 50.5 vs 95.8 | Hurt |
| Use Gemini 1.5 Pro rather than GPT-4 Turbo | ReMI · EmojiAlgebra | accuracy: 44.5 vs 57.5 | Hurt |
| Use Gemini 1.5 Pro rather than GPT-4 Turbo | ReMI · IQ | accuracy: 62.5 -> 76.0 | Helped |
Surprising findings
- Models often perform better when images are fed separately than stitched into one image, especially when interleaved with question text.
- Reasoning errors with correctly read images were the most frequent failure type (37.6%), and Gemini 1.5 often reached correct answers despite misreading images.
Limitations
- Covers only a subset of possible multi-image domains (e.g., chemistry and music sheets are excluded).
- Performance-by-property analysis relies on proxy normalization for random chance and task difficulty.
- Human evaluation uses only 20 examples per task and failure analysis uses 20 examples per task.
Experimental setup
Datasets: GeomVerse refCOCO NetworkX-generated graphs Google Maps API images
Benchmarks: ReMI
Metrics: accuracy relaxed accuracy error reduction percentage
Baselines: Claude 3 Sonnet Gemini Ultra Gemini Flash GPT-4 Turbo Naive Baseline Human
Builds on: GeomVerse NLVR2 BLINK NetworkX RefCOCO
Closest papers by meaning
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency (NeurIPS 2026 · similarity 0.88)
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.88)
- Will Multimodal Models Be Dazzled by Multi-Image Visual Puzzles? (CVPR 2026 · similarity 0.87)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.87)
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.