Research map / Vision-language and multimodal / Multimodal LLMs
NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NoisyGRPO, which injects diffusion-style Gaussian noise into visual inputs during GRPO rollouts and fuses the noise-level prior with SBERT-based reward observations via a Bayesian posterior advantage, improving general multimodal CoT reasoning for Qwen2.5-VL 3B/7B.
Problem
GRPO-based RL for multimodal CoT reasoning in MLLMs generalizes poorly beyond the training distribution, due to limited exploration (near-identical rollouts) and outcome-only rewards that leave intermediate reasoning unsupervised.
Approach
Inject controllable noise into images during rollout sampling for exploration (training updates use clean inputs), and compute advantages via a Bayesian Gaussian posterior that treats the noise level as a prior and the semantic reward as an observation, with prior variance tied to group reward standard deviation.
Key results
- On Qwen2.5-VL 3B, NoisyGRPO reaches MMStar average 58.2 vs 54.5 for GRPO (+3.7) and MME-CoT F1 40.6 vs 36.2 (+4.4).
- On Qwen2.5-VL 7B, NoisyGRPO improves MMStar average to 65.8 from 62.6 for GRPO and lowers AMBER generative hallucination to 23.9 from 28.9.
- NoisyGRPO adds negligible overhead: 3B wall-clock 6h40min vs 6h12min for GRPO on 8 A100s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GRPO with NoisyGRPO | Qwen2.5-VL 3B · MME-CoT | CoT quality F1: 36.2 -> 40.6 | Helped |
| replace GRPO with NoisyGRPO | Qwen2.5-VL 7B · MME-CoT | CoT Efficiency Avg.: 66.4 -> 59.7 | Hurt |
| replace GRPO with NoisyGRPO | Qwen2.5-VL 3B · MMStar | Avg.: 54.5 -> 58.2 | Helped |
| replace GRPO with NoisyGRPO | Qwen2.5-VL 3B · MMStar Coarse Perception | CP.: 68.0 -> 69.5 | Hurt |
| replace GRPO with NoisyGRPO | Qwen2.5-VL 3B · AMBER Gen. | Cov.: 68.5 -> 67.7 | Hurt |
| replace GRPO with NoisyGRPO | Qwen2.5-VL 3B · AMBER Gen. | Hal.: 44.6 -> 44.3 | Helped |
| remove Bayesian adaptive prior-observation fusion (naive NoisyGRPO) (ablation) | NoisyGRPO · MMStar | reasoning performance | Hurt |
| add dynamic prior-observation fusion to noise-injected GRPO (ablation) | GRPO with Noise Injection · MMStar | reasoning performance | Helped |
| vary reward threshold τ from 0.6 to 0.5 (ablation) | NoisyGRPO · MMStar | Avg.: 58.2 -> 56.5 | Hurt |
| vary reward threshold τ from 0.6 to 0.7 (ablation) | NoisyGRPO · MMStar | Avg.: 58.2 -> 56.4 | Hurt |
| replace SBERT reward with ROUGE reward (ablation) | NoisyGRPO · MMStar | Avg.: 58.2 -> 53.5 | Hurt |
| replace uniform noise with Gaussian or fixed noise (ablation) | NoisyGRPO · MMStar | average performance | Hurt |
Surprising findings
- Naively injecting noise into rollouts without the Bayesian fusion degrades performance relative to vanilla GRPO.
- SFT on the same 13k data gives no consistent gains over the base model, suggesting RL drives the improvements.
Limitations
- Assumes injected noise perturbs generation and correlates with answer correctness; fails for coarse-grained perception where noise has little effect.
- Coarse Perception and AMBER coverage metrics drop relative to baselines.
- Accuracy reward uses SBERT embedding similarity, which may introduce embedding bias; only 4 rollouts per prompt due to compute.
Experimental setup
Datasets: MM-RLHF VQA subset
Benchmarks: MME-CoT MMStar AMBER MME-RealWorld-Lite
Metrics: F1 precision recall CoT robustness avg CoT efficiency accuracy per category AMBER CHAIR-style scores accuracy
Baselines: GRPO SFT LLaVA-OV-7B GPT-4o Kimi k1.5 Qwen2.5-VL
Builds on: GRPO VLM-R1 DAPO DeepSeek-R1
Closest papers by meaning
- NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation (NeurIPS 2025 · similarity 0.91)
- PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping (NeurIPS 2026 · similarity 0.89)
- Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Reinforcing Multimodal Reasoning Against Visual Degradation (NeurIPS 2026 · similarity 0.89)
- Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.89)
- Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.