Research map / Large language models / Reasoning and chain of thought

R-4B: Incentivizing General-Purpose Auto-Thinking in MLLMs via Bi-Mode Annealing and Reinforce Learning

CVPR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes R-4B, a 4B multimodal LLM that decides adaptively whether to think, trained by bi-mode annealing on curated thinking/non-thinking data followed by Bi-mode Policy Optimization (a GRPO variant with paired thinking and non-thinking rollouts).

Problem

Always-thinking MLLMs waste computation on simple queries, while existing auto-thinking methods need complex reward engineering, manual complexity labels, or are text-only and prone to mode collapse.

Approach

Bi-mode annealing gives one model both thinking and non-thinking modes using heuristically labeled data; BPO then runs GRPO with deterministic equal-size groups of thinking and non-thinking rollouts per query, with a simple rule-based math reward, which transfers to auto-thinking on other topics.

Key results

• R-4B-RL achieves 68.1% on MMMU val, the top score among compared models, and 58.9 on HallusionBench, per the paper's reported tables.
• BPO raises average auto-thinking accuracy on reasoning benchmarks from 43.2 (R-4B-Base A-T) to 57.0 (R-4B-RL A-T) across MathVista, MathVision, MathVerse, DynaMath, WeMath, LogicVista.
• On OCRBench, R-4B-RL auto-thinking uses 66 tokens on average versus 394 for full thinking, while reaching 83.6% accuracy.

Surprising findings

• Vanilla GRPO develops a thinking preference that causes non-thinking correctness to drop during RL training, which BPO avoids.
• A rule-based reward using only math data yields auto-thinking behavior that generalizes to non-math topics such as OCR and counting.

Limitations

• Relies on a 32B MLLM annotator and Qwen3-32B judge, which adds dependence on external models.
• Auto-thinking still underperforms full thinking on several reasoning benchmarks (e.g., MathVerse, WeMath) in the reported tables.

Experimental setup

Datasets: bi-mode annealing mixture (General, Math/K12, Code, Chart, OCR, Grounding, Caption, Knowledge, Text-Only) LAION image-caption data

Benchmarks: MMMU MMMU-Pro MMStar MMBench MMVet HallusionBench VLMs are Blind MMVP VisuLogic RealWorldQA AI2D CharXiv

Metrics: accuracy average output tokens

Baselines: Qwen2.5-VL-7B InternVL3-8B InternVL3.5-4B Kimi-VL-A3B-Thinking-2506 Keye-VL-8B vanilla GRPO

Builds on: GRPO Qwen3 SigLIP2 Qwen2.5-VL

Closest papers by meaning

• Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL (NeurIPS 2025 · similarity 0.91)
• SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward (ICLR 2026 · similarity 0.90)
• Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.89)
• RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.89)
• MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization (ICLR 2026 · similarity 0.89)
• Hybrid Latent Reasoning via Reinforcement Learning (NeurIPS 2025 · similarity 0.89)