Research map / Reinforcement learning / Policy gradient and actor-critic

Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies the missing-old-logit problem in asynchronous agentic RL that breaks decoupled discrepancy/staleness PPO correction, compares three exact old-logit acquisition routes, and proposes a revised PPO-EWMA proximal reference with auto-reset as a low-cost approximation.

Problem

In asynchronous LLM agentic RL with delayed updates and partial rollouts, the training-side old logits needed for decoupled importance correction (discrepancy repair vs. policy staleness) are often unavailable, so the decomposition becomes semantically invalid and clipping/masking interfere.

Approach

Decomposes the total importance ratio into a training-inference discrepancy ratio and a staleness ratio; proposes exact old-logit acquisition routes (snapshots, dedicated old-logit model, partial-rollout interruption) and a revised PPO-EWMA proximal reference with staleness-aware decay (beta ~ W/(W+2)) and an automatic reset when the train-infer mask falls below a threshold.

Key results

• On Qwen3-30B-A3B, PPO-EWMA reaches 67.82 avg@4 on tau2-Bench retail versus 65.43 for Decoupled PPO and 69.70 for the Snapshot idealized reference.
• PPO-EWMA cuts CPU storage for old-logit handling from 76.4 GB to 15.2 GB and extra time from 150 s to 34 s on the 30B model compared with Snapshot.
• On Qwen3-30B-A3B, PPO-EWMA attains 60 avg@4 on airline, exceeding Snapshot's 59 avg@4 and Decoupled PPO's 57.

Surprising findings

• PPO-EWMA beats the exact-old-logit Snapshot reference on some splits (e.g., 30B airline avg@4 60 vs 59; 4B airline avg@4 54 vs 56 is not beaten, so mixed), suggesting exact correction is not uniformly superior.
• Interpolation-based proxies (linear_prox, log-linear) are shown analytically and empirically to merely reparameterize clipping boundaries of the total ratio rather than restoring the missing reference.

Limitations

• PPO-EWMA is an approximate reference, not true recovery of the missing old policy logits.
• May become unreliable under highly non-stationary staleness or extreme version gaps.
• Snapshot-based overhead measurements come from short runs (77 and 11 steps), limiting conclusions.

Experimental setup

Datasets: Agentic RL corpus (TU_Pipeline, multiple environments)

Benchmarks: tau2-Bench (retail) tau2-Bench (airline) tau2-Bench (telecom) VitaBench (in-store) VitaBench (delivery)

Metrics: avg@k pass@k system overhead (storage, extra time)

Baselines: Decoupled PPO Linear_prox Snapshot

Builds on: PPO Decoupled PPO IcePop GRPO A-3PO

Closest papers by meaning

• ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING (NeurIPS 2026 · similarity 0.90)
• AsyncOPD: How Stale Can On-Policy Distillation Be? (NeurIPS 2026 · similarity 0.90)
• Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models (ICLR 2025 · similarity 0.89)
• DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training (NeurIPS 2026 · similarity 0.89)
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training (NeurIPS 2025 · similarity 0.88)
• AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning (NeurIPS 2025 · similarity 0.88)