Research map / Vision-language and multimodal / Multimodal LLMs
NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NoisyRollout, a GRPO-based data augmentation that mixes rollouts from clean and Gaussian-noised images in one advantage group, with a noise annealing schedule, improving VLM visual reasoning at no extra training cost.
Problem
RL for VLM visual reasoning suffers from limited policy exploration and imperfect visual perception, and standard tricks like raising rollout temperature add superficial diversity rather than targeted exploration.
Approach
Generate half of each GRPO rollout group from a moderately distorted version of the input image, compute advantages over the combined group, but update the policy only on the clean image-query pair; anneal the distortion strength over training with a sigmoid schedule.
Key results
- NoisyRollout on Qwen2.5-VL-7B trained with 2.1K Geometry3K samples reaches 53.2% MathVerse, 28.5% MathVision, 72.1% HallusionBench, vs 50.8/27.3/69.8 for vanilla GRPO.
- With MMK12 (6.4K) and Qwen2.5-VL-7B, NoisyRollout reaches 74.5% MathVista and 30.6% MathVision, best among compared open-source RL-tuned 7B models.
- On Qwen2.5-VL-32B (Geometry3K), NoisyRollout improves OOD average to 59.2 for 7B; 32B results show gains of about 0.4 to 1.6 points over GRPO on MMK12 setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hybrid clean and noisy rollouts with noise annealing | Qwen2.5-VL-7B-Instruct trained with Geometry3K · MathVerse | accuracy (%): 50.8 -> 53.2 | Helped |
| add hybrid clean and noisy rollouts with noise annealing | Qwen2.5-VL-7B-Instruct trained with Geometry3K · MathVision | accuracy (%): 27.3 -> 28.5 | Helped |
| add hybrid clean and noisy rollouts with noise annealing | Qwen2.5-VL-7B-Instruct trained with Geometry3K · HallusionBench | accuracy (%): 69.8 -> 72.1 | Helped |
| add hybrid clean and noisy rollouts with noise annealing | Qwen2.5-VL-32B-Instruct trained with Geometry3K · MathVerse | accuracy (%): 58.9 -> 58.9 | No clear effect |
| add hybrid clean and noisy rollouts with noise annealing | Qwen2.5-VL-32B-Instruct trained with MMK12 · MathVision | accuracy (%): 40.0 -> 41.6 | Helped |
| remove noise annealing (ablation) | NoisyRollout on Qwen2.5-VL-7B-Instruct · Geometry3K test set | accuracy (%): 54.9 -> 43.9 | Hurt |
| remove noise annealing (ablation) | NoisyRollout on Qwen2.5-VL-7B-Instruct · OOD benchmarks, average | accuracy (%): 59.2 -> 58.0 | Hurt |
| use power noise decay instead of sigmoid decay (ablation) | NoisyRollout on Qwen2.5-VL-7B-Instruct · OOD benchmarks, average | accuracy (%): 58.5 -> 57.1 | Hurt |
| use exponential noise decay instead of sigmoid decay (ablation) | NoisyRollout on Qwen2.5-VL-7B-Instruct · OOD benchmarks, average | accuracy (%): 58.5 -> 57.0 | Hurt |
| replace Gaussian noise with rotation augmentation (ablation) | NoisyRollout on Qwen2.5-VL-7B-Instruct · OOD benchmarks, average | accuracy (%): 59.2 -> 58.2 | Hurt |
| increase vanilla GRPO rollout count from n=8 to n=16 (ablation) | Vanilla GRPO on Qwen2.5-VL-7B-Instruct · Geometry3K test set | accuracy (%): 49.6% to 54.7% | Helped |
| increase vanilla GRPO rollout count from n=8 to n=16 (ablation) | Vanilla GRPO on Qwen2.5-VL-7B-Instruct · OOD benchmarks, average | accuracy (%): 56.8% to 57.5% | Helped |
Surprising findings
- Decoupling clean and noisy rollouts for separate policy updates (sample-level augmentation) did not help over GRPO.
- Cropping and rotation without expansion caused critical information loss and near-zero rewards, leading to divergence.
Limitations
- Experiments limited to models up to 32B and datasets of a few thousand samples.
- Applied only at RL fine-tuning stage; not explored in pretraining.
- No formal theoretical analysis of exploration-exploitation or convergence effects.
Experimental setup
Datasets: Geometry3K MMK12
Benchmarks: MathVerse MathVision MathVista WeMath HallusionBench Geometry3K test
Metrics: accuracy average accuracy Bradley-Terry win rate rollout diversity (cosine distance)
Baselines: Vanilla GRPO OpenVLThinker-7B MM-Eureka-Qwen-7B ADORA-7B ThinkLite-7B-VL VLAA-Thinker-7B MM-Eureka-Qwen-32B R1-OneVision-7B Vision-R1-7B
Builds on: GRPO PPO EasyR1 DAPO
Closest papers by meaning
- NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation (NeurIPS 2025 · similarity 0.91)
- More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
- Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.90)
- PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning (NeurIPS 2025 · similarity 0.90)
- ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs (NeurIPS 2025 · similarity 0.90)
- Reinforcing Multimodal Reasoning Against Visual Degradation (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.