Research map / Vision-language and multimodal / Multimodal LLMs

Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AGILE, an agentic jigsaw environment where a VLM solves shuffled image puzzles via executable code actions (swap, observe, crop, zoom), trained with cold-start SFT on Gemini trajectories followed by GRPO, improving jigsaw accuracy and generalization to general vision benchmarks.

Problem

Existing VLMs perform near random even on simple 2x2 jigsaw tasks, and high-quality vision-language RL data is scarce and costly to create.

Approach

Model jigsaw solving as a multi-turn interaction in which the VLM writes Python code calling swap/observe/crop/zoom APIs and receives visual feedback, with programmatically generated jigsaw puzzles giving scalable, controllable-difficulty verifiable RL data, plus a step-count reward.

Key results

• On the 2x2 jigsaw, Qwen2.5-VL-7B accuracy rises from 9.5% to 82.8% after cold-start SFT and RL, versus 46.4% for Gemini-2.5-Pro.
• Across 9 general vision benchmarks, the AGILE-trained Qwen2.5-VL-7B averages 65.2 vs 62.1 for the base model (+3.1).
• Gains include +4.2 on HRBench4K, +5.2 on HRBench8K and +4.2 on VStarBench; 3x3 accuracy rises from 0.4% to 20.8%.

Surprising findings

• RL without any Gemini expert trajectories still improves average benchmark score by about 1.8 points over the base model.
• Under equal 20K budgets, pure jigsaw RL (64.9 avg) roughly matches high-resolution general QA RL (64.7 avg).

Limitations

• Multi-turn interaction increases context length and computational overhead.
• 3x3 RL training exceeds the model's maximum context window, so main RL is restricted to 2x2 puzzles.
• Cold-start depends on expensive Gemini 2.5 Pro trajectories filtered and manually verified.

Experimental setup

Datasets: VStar DeepEyes high-resolution samples HRBench TextVQA InfoVQA ChartQA-style structured charts COCO2017 RealWorldQA MMEureka

Benchmarks: MME-RealWorld-Lite RealWorldQA HRBench4K HRBench8K VStarBench MMVP BLINK HallusionBench MMMU

Metrics: accuracy score (fraction of correctly placed patches) average accuracy

Baselines: GPT-4o Gemini-2.5-Pro Qwen2.5-VL-72B InternVL3-8B InternVL3-78B MiMo-VL-7B-RL LLaVA-OV-7B InternVL2.5-8B InternVL2.5-78B

Builds on: GRPO DeepSeek-R1 DeepEyes Jigsaw-R1 Logic-RL

Closest papers by meaning

• Visual Jigsaw Post-Training Improves MLLMs (ICLR 2026 · similarity 0.92)
• SenseSearch: Empowering Vision-Language Models with High-Resolution Agentic Search-Reasoning via Reinforcement Learning (CVPR 2026 · similarity 0.91)
• DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.90)
• Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs (CVPR 2026 · similarity 0.90)
• VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments (CVPR 2026 · similarity 0.90)
• GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning (NeurIPS 2025 · similarity 0.90)