Research map / Large language models / Reasoning and chain of thought
Angles Don’t Lie: Unlocking Training‑Efficient RL Through the Model’s Own Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GAIN-RL, which reorders and samples RL training data per epoch using a model-intrinsic "angle concentration" signal from a single prefill pass, achieving about 2.5x training speedup for GRPO on math and code.
Problem
RFT for LLMs is sample-inefficient because uniform sampling repeatedly exposes the model to the same queries, and existing curriculum or selection methods use fixed, model-agnostic difficulty signals with high preprocessing costs.
Approach
Uses the cosine similarity between token hidden states at the final layer (angle concentration, intra-question and question-to-prompt) from a cheap prefill pass as a model-intrinsic signal; it is linked theoretically to gradient norm and used to sort data and drive a Gaussian-based, accuracy- and angle-adaptive sampling schedule.
Key results
- On Qwen2.5-Math-1.5B-Instruct, GAIN-RL(GRPO) reaches vanilla GRPO's 200-epoch accuracy at epoch 80 (2.5x speedup) and averages 42.63 vs 39.95 across six math benchmarks.
- On LLaMA-3.2-3B-Instruct, GAIN-RL(GRPO) reaches 28.5 average vs 26.8 for GRPO with a 2.5x speedup, versus 1.43x for ADARFT(GRPO).
- On Qwen2.5-Coder-3B-Instruct, GAIN-RL(GRPO) reaches 28.8 average vs 26.7 for GRPO with a 1.81x speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GAIN-RL(GRPO) instead of ADARFT(GRPO) | Qwen 2.5 Math 1.5B Instruct · Math benchmarks (six-dataset average) | Avg: 41.09 -> 42.63 | Helped |
| Use GAIN-RL(GRPO) instead of ADARFT(GRPO) | LLaMA 3.2 3B Instruct · Math benchmarks (six-dataset average) | Avg: 27.2 -> 28.5 | Helped |
| Use GAIN-RL(GRPO) instead of ADARFT(GRPO) | Qwen 2.5 Math 7B Instruct · Math benchmarks (six-dataset average) | Avg: 43.61 -> 46.33 | Helped |
| Use GAIN-RL(GRPO) instead of GRPO | Qwen 2.5 Coder 3B Instruct · Code benchmarks (average) | Avg: 26.7 -> 28.8 | Helped |
| Use high angular concentration-biased sampling rather than uniform sampling | GAIN-RL(GRPO), Qwen2.5-0.5B-Instruct · Math dataset, half of training data | performance | Helped |
| Use uniform sampling of half the data instead of full data | GAIN-RL(GRPO), Qwen2.5-0.5B-Instruct · Math dataset | performance | Mixed |
| Use low angular concentration-biased sampling | GAIN-RL(GRPO), Qwen2.5-0.5B-Instruct · Math dataset, half of training data | performance | Hurt |
| Use data ordering with accuracy-based probability updates instead of full GAIN-RL (ablation) | GAIN-RL(GRPO), Qwen2.5-0.5B-Instruct · Math test set | performance | Hurt |
| Use data ordering with angle-based probability updates instead of full GAIN-RL (ablation) | GAIN-RL(GRPO), Qwen2.5-0.5B-Instruct · Math test set | performance | Hurt |
| Remove data ordering and use accuracy-only updates, discarding fully correct data (ablation) | GAIN-RL(GRPO), Qwen2.5-0.5B-Instruct · Math test set | performance | Hurt |
| Reduce the training batch size to 512 (ablation) | GAIN-RL(GRPO), Qwen2.5-0.5B-Instruct · Math dataset | performance | No clear effect |
| Weight angle signal as C_intra + k·C_inter rather than using the unweighted signal (ablation) | GAIN-RL(GRPO) · Math benchmarks | performance | Helped |
Surprising findings
- Training on only half of the math data selected by high angle concentration outperforms training on the full dataset with GRPO.
- Biasing sampling toward low angle concentration leads to unstable and poor results.
Limitations
- Theoretical justification relies on empirical assumptions (approximate orthogonality of weight matrices) that are only checked visually.
- Speedup on code tasks (1.81x) is smaller than on math tasks, and some benchmark entries do not improve over ADARFT (e.g., LLaMA GSM8K).
- Evaluation is on relatively small models and the main results are single-run with no variance reported.
Experimental setup
Datasets: DeepScaleR DeepCoder GSM8K MATH AMC 23
Benchmarks: GSM8K MATH AMC 23 AIME 24 OlympiadBench Minerva Math LiveCodeBench Codeforces HumanEval+
Metrics: Pass@1 Pass@8 epochs to match vanilla GRPO accuracy speedup
Baselines: GRPO ADARFT(GRPO) PPO
Builds on: GRPO ADARFT PPO
Closest papers by meaning
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.89)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.88)
- UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection (NeurIPS 2025 · similarity 0.88)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.88)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.88)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.