Research map / Reinforcement learning / RL from feedback
ReDit: Reward Dithering for Improved LLM Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds zero-mean random noise (Gaussian or uniform) to discrete GRPO rewards ("reward dithering") to improve gradient stability and convergence speed of LLM policy optimization.
Problem
Discrete rule-based rewards used in GRPO-style RL for LLMs cause gradient vanishing and explosion, unstable optimization, and slow convergence.
Approach
Inject independent zero-mean noise into each sampled discrete reward before computing group-normalized advantages, so that advantages vary continuously within batches while the expected policy gradient is unchanged.
Key results
- On GSM8K, ReDit (Gauss) with GRPO reaches 89.02 at 1000 steps versus GRPO's 89.07 at 9000 steps on Qwen2.5-7B-Instruct; final GSM8K 90.76 (Uniform) and 90.46 (Gauss) vs GRPO 89.07.
- On MATH, final accuracy rises from 48.01 (GRPO) to 52.55 (Gaussian ReDit) on Qwen2.5-7B-Instruct, with backbone at 39.
- Applying the same smoothing to DAPO, Dr.GRPO, and REINFORCE++ on GSM8K improves final accuracy by +1.82, +1.56, and +1.71 points respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Gaussian reward dithering to GRPO | GRPO · GSM8K test | accuracy: +1.69 | Helped |
| Add Gaussian reward dithering to GRPO | GRPO · MATH test | accuracy: +4.54 | Helped |
| Add Gaussian reward dithering to GRPO | GRPO · Geometry3K test | accuracy: +1.57 | Helped |
| Add Gaussian reward dithering to DAPO | DAPO · GSM8K, 9000 steps | accuracy: +1.82 | Helped |
| Add Gaussian reward dithering to Dr.GRPO | Dr.GRPO · GSM8K, 9000 steps | accuracy: +1.56 | Helped |
| Add Gaussian reward dithering to REINFORCE++ | REINFORCE++ · GSM8K, 9000 steps | accuracy: +1.71 | Helped |
| Set perturbation amplitude to a=0.05 rather than no smoothing (ablation) | ReDit with GRPO · GSM8K | accuracy | Helped |
| Use a very small perturbation amplitude (a=0.01) (ablation) | ReDit with GRPO · GSM8K | accuracy | Mixed |
| Use excessive perturbation amplitude (a=0.5) (ablation) | ReDit with GRPO · GSM8K | accuracy | Hurt |
| Apply reward dithering to continuous reward model scores rather than discrete rewards (ablation) | GRPO with a pretrained reward model · Reward-model training experiment | accuracy | No clear effect |
| Use cosine-reverse perturbation scheduling (ablation) | ReDit with GRPO · Main benchmarks | accuracy | Helped |
| Use Gaussian reward dithering rather than uniform reward dithering (ablation) | ReDit with GRPO · MATH test | accuracy | Helped |
Surprising findings
- Adding noise to continuous reward-model scores produced no discernible effect, suggesting the benefit is specific to discrete rewards.
- Random noise on rewards can improve performance even though the expected gradient is unchanged, matching prior findings that random rewards help Qwen models.
Limitations
- Requires careful tuning of perturbation variance; the optimal amplitude is found via small-dataset search.
- Theoretical analysis is limited to a simplified binary-reward setting and does not fully establish convergence benefits.
- Experiments mostly use LoRA on 7-8B instruct models with a single GPU and limited seeds.
Experimental setup
Datasets: GSM8K MATH Geometry3K APPS HumanEval CodeContests
Benchmarks: GSM8K MATH Geometry3K APPS HumanEval CodeContests
Metrics: test accuracy pass@1 gradient norm convergence steps
Baselines: GRPO DAPO Dr.GRPO REINFORCE++ Gradient Clipping Dynamic Sampling
Builds on: GRPO DeepSeek-R1 DAPO Dr.GRPO REINFORCE++
Closest papers by meaning
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.88)
- WARM: On the Benefits of Weight Averaged Reward Models (ICML 2024 · similarity 0.88)
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.88)
- ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.88)
- Discriminative Policy Optimization for Token-Level Reward Models (ICML 2025 · similarity 0.88)
- Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.