Research map / Vision-language and multimodal / Multimodal LLMs

Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Zebra-CoT, a 182K-sample dataset of interleaved text-image chain-of-thought traces across scientific, 2D, 3D, and visual logic/game tasks, and fine-tunes Anole-7B and Bagel-7B on it to show gains on visual reasoning benchmarks.

Problem

Visual chain-of-thought (reasoning with intermediate generated images) is hard to train because off-the-shelf models perform poorly on it and high-quality interleaved text-image reasoning data is scarce.

Approach

Curate a large, diverse dataset of logically coherent interleaved text-image reasoning traces from real sources and templated synthetic generation (with VLMs filling templates and enhancing traces), then fine-tune unified multimodal models to natively generate visual thoughts.

Key results

• Frontier models score poorly zero-shot on Zebra-CoT (average 31.51%; GPT-5 41.98%), rising to 47.99% with one multimodal CoT step and 56.70% with two.
• Fine-tuning Anole-7B on Zebra-CoT raises in-distribution accuracy from 4.2% to 16.9%, and improves visual reasoning benchmarks by an average of 4.9 points, with up to 13.1 points on a visual logic benchmark.
• Fine-tuned Bagel-7B learns to natively generate interleaved visual CoT images, a capability absent in the base model.

Surprising findings

• Anole-Zebra-CoT declines on EMMA physics (21.79 to 10.90) and on MathVista textbook QA despite overall gains.

Limitations

• Anole-7B absolute accuracy remains low (16.9% in-distribution), so the model is far from reliable visual CoT.
• Interleaved generation is slow, so the paper evaluates on mini versions of MathVision and MathVista.
• Many traces are templated or synthetic and generated by VLMs, which may limit realism and diversity.

Experimental setup

Datasets: MATH ALFRED RoboMIND ImageNet USPTO-50K Lichess games maze-dataset ARC-AGI RPM

Benchmarks: Zebra-CoT test set MathVision MathVista VisuLogic EMMA MMVP Blink Vstar

Metrics: accuracy

Baselines: GPT-5 Claude-4 Sonnet Gemini-2.5 Pro Anole with CoT prompting Visual-CoT

Builds on: Anole Bagel Visual Sketchpad Chameleon

Closest papers by meaning

• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.92)
• TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.92)
• Uni-CoT: Towards Unified Chain-of-Thought Reasoning Across Text and Vision (ICLR 2026 · similarity 0.92)
• MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning (NeurIPS 2025 · similarity 0.92)
• Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalization (CVPR 2026 · similarity 0.91)
• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.91)