Research map / Vision-language and multimodal / Multimodal LLMs

Multimodal DeepResearcher: Generating Text-Chart Interleaved Reports from Scratch with Agentic Framework

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Formal Description of Visualization (FDV), a structured textual chart representation, and the Multimodal DeepResearcher agentic framework that generates text-chart interleaved reports from scratch, evaluated on a 100-topic MultimodalReportBench against a DataNarrative-adapted baseline.

Problem

Existing deep research frameworks generate text-only reports, and automated generation of interleaved text and visualizations (charts embedded in reports) remains underexplored.

Approach

Represent charts as structured textual descriptions (FDV, extending the grammar of graphics across layout, scale, data, and marks) so LLMs can learn from human exemplar reports via in-context learning and generate charts by coding them in D3.js, with an actor-critic loop using a multimodal critic on rendered screenshots.

Key results

• With Claude 3.7 Sonnet, Multimodal DeepResearcher achieves an 82% overall win rate over DataNarrative under GPT-4.1 judging (90% with gemini-2.5-pro).
• With open-source Qwen3-235B-A22B and Qwen2.5-VL-72B, the overall win rate is 55% under GPT-4.1 (93% under gemini-2.5-pro).
• Human evaluation on 20 topics gives a 100% overall win rate over DataNarrative with Claude 3.7 Sonnet.

Surprising findings

• Under GPT-4.1 judging with open-source models, the overall advantage is modest (55% win vs 40% loss), and Coherence and Organization is slightly lost (41% vs 51%).
• Average charts per report are comparable (9.3 vs 9.4), yet the framework produces far more diverse chart types, with 280 'Others' charts versus 96 for DataNarrative.

Limitations

• Overlapping elements in charts is the most common error, since the coding process lacks real-time visual feedback.
• Hallucination occurs when data is insufficient, e.g., coloring regions with missing data.
• Generation takes 767 seconds per report versus 373 for the baseline, largely due to the refinement loop.

Experimental setup

Datasets: Pew Research reports Our World in Data Open Knowledge Foundation reports

Benchmarks: MultimodalReportBench

Metrics: win rate informativeness and depth coherence and organization verifiability visualization quality visualization consistency readability layout aesthetics data faithfulness

Baselines: DataNarrative

Builds on: DataNarrative grammar of graphics DeepResearcher WebThinker

Closest papers by meaning

• VisInteract: Towards Dynamic Interactive Text-to-Visualization under Imperfect Queries (NeurIPS 2026 · similarity 0.86)
• CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs (NeurIPS 2024 · similarity 0.86)
• ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding (CVPR 2026 · similarity 0.86)
• nvBench 2.0: Resolving Ambiguity in Text-to-Visualization through Stepwise Reasoning (NeurIPS 2025 · similarity 0.86)
• BizGen: Advancing Article-level Visual Text Rendering for Infographics Generation (CVPR 2025 · similarity 0.86)
• OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.85)