Research map / Reinforcement learning / RL from feedback
Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
Luke J. Huang, Zhuoyang Zhang, Qinghao Hu, Shang Yang, Song Han
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ESS-guided step-size scaling and off-policy optimal baseline to TIS (ablation) | sequence-level TIS · MATH-500, Qwen2.5-7B, 10 steps off-policy | stability / accuracy | Improved |
| add ESS-guided learning-rate scaling to TIS (ablation) | TIS · MATH-500, Qwen2.5-7B | stability / accuracy | Improved |
| add off-policy optimal baseline (OPOB) to TIS (ablation) | TIS · MATH-500, Qwen2.5-7B | stability / accuracy | Improved |
| use VCPO instead of sequence-level TIS | TIS · GSM8K, Qwen2-1.5B, 12 steps off-policy | training stability | Improved |
| use VCPO with async k=10 instead of synchronous | Sync RL · Countdown, Qwen2.5-7B | GPU hours: 143.2 -> 89.6 | Improved |
| use VCPO with async k=2 instead of synchronous | Sync RL · AIME 2025 tool-use, Qwen2.5-7B | GPU hours: 420.2 -> 168.9 | Improved |
| increase policy lag up to 128 steps off-policy with VCPO (ablation) | VCPO · Qwen2.5-7B math | stability | No clear effect |
| lower the learning rate instead of ESS scaling (ablation) | TIS · MATH-500, high asynchrony | accuracy | Hurt |
| raise TIS threshold from c=2.0 to c=8.0 (ablation) | sequence-level TIS · GSM8K, Qwen2-1.5B | collapse delay | Improved |
| use sequence-level MIS masking (ablation) | MIS · GSM8K, Qwen2-1.5B | stability | Hurt |
| use M2PO-style masking (max 0.04) (ablation) | M2PO · GSM8K, Qwen2-1.5B, k=12 | stability | Hurt |
| add KL-in-reward (beta=0.001) (ablation) | Seq-TIS · GSM8K, Qwen2-1.5B, k=12 | validation accuracy | Hurt |
Closest papers by meaning
- Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training (NeurIPS 2025 · similarity 0.93)
- AsyncOPD: How Stale Can On-Policy Distillation Be? (NeurIPS 2026 · similarity 0.91)
- DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training (NeurIPS 2026 · similarity 0.90)
- Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models (ICLR 2025 · similarity 0.89)
- VESPO: Variational Sequence-level Soft Policy Optimization for Off-Policy LLM Training (NeurIPS 2026 · similarity 0.89)
- Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
