Research map / Vision-language and multimodal / Multimodal LLMs

PixelCraft: A Multi-Agent system for High-Fidelity Visual Reasoning on Structured Images

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PixelCraft, a multi-agent system that pairs a fine-tuned Qwen2.5-VL-3B grounding model with CV-based visual tools and a planner-managed image memory to enable high-fidelity, non-linear visual reasoning on charts and geometry diagrams.

Problem

Multimodal LLMs struggle with structured images (charts, geometric diagrams), where small perceptual errors cascade into wrong answers; existing visual-CoT methods rely on low-fidelity processing or linear, rigid reasoning chains.

Approach

A multi-agent workflow (dispatcher, planner, reasoner, planning and visual critics, tool agents) where a fine-tuned grounding MLLM supplies pixel coordinates to classical CV operators for precise image edits, and a planner-managed image memory lets the system recall, branch, and backtrack over intermediate visual states instead of streaming all historical images.

Key results

• On CharXiv, ChartQAPro and EvoChart, PixelCraft improves GPT-4o by +5.6, +6.32 and +7.60 points over CoT (e.g., CharXiv 55.2 vs 51.1).
• Fine-tuned Qwen2.5-VL-3B grounding reaches 0.93 overall versus 0.26 for the base model on a 500-sample test set.
• Ablation on CharXiv/ChartQAPro: tool agents give the largest average gain; full system reaches 68.1 and 65.56 with GPT-4o-style backbone evaluation on the reported split.

Surprising findings

• Multi-agent debate and ReConcile without specialized visual tools show little or negative benefit over CoT on ChartQAPro (e.g., Debate 49.38 vs Direct 52.51 for GPT-4o).
• Refocus underperforms even the CoT baseline on several benchmarks (e.g., 46.30 vs 56.52 on ChartQAPro for GPT-4o).

Limitations

• Purely LLM-generated visual tools were often invalid or inaccurate, requiring manual refinement and validation.
• Framework depends on strong backbone MLLMs for planning and tool orchestration; weaker models degrade performance.
• Higher inference latency than CoT (about 4x on CharXiv).

Experimental setup

Datasets: CharXiv ChartQAPro EvoChart Geometry3K ArxivQA Inter-GPS

Benchmarks: CharXiv ChartQAPro EvoChart Geometry3K (auxiliary-line subset)

Metrics: accuracy IoU PCK@0.01 average response time

Baselines: Direct answer CoT Debate ReConcile Refocus Visual CoT

Builds on: Refocus Visual Sketchpad Qwen2.5-VL multi-agent debate ReConcile

Closest papers by meaning

• AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoning (ICLR 2026 · similarity 0.91)
• Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
• ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both (NeurIPS 2026 · similarity 0.90)
• Pixel Reasoner: Incentivizing Pixel Space Reasoning via Curiosity-Driven Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Latent Implicit Visual Reasoning (CVPR 2026 · similarity 0.89)
• SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.89)