Research map / Reinforcement learning / Policy gradient and actor-critic

Cost-Aware Learning

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Alternative policy update rules · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Cost-Aware SGD, sampling finite-sum components with probability proportional to gradient norm over sqrt(cost), and applies it to GRPO by using advantage magnitude as a gradient-norm proxy, reducing policy-optimization tokens on Qwen models.

Problem

Finite-sum optimization where sampling components incurs different costs; the goal is to reach target error at minimum total cost. In LLM RL, sequence-level policy gradient cost scales with length.

Approach

Derive the cost-optimal importance sampling distribution p_i ∝ G_i/sqrt(c_i) (minimizing the product of second moment and per-iteration cost), and use |advantage| as a cheap proxy for gradient norm inside GRPO with length as cost.

Key results

• Qwen3-8B: Cost-Aware GRPO reaches baseline's peak AIME accuracy with 51% fewer policy-optimization tokens.
• Qwen2.5-Math-1.5B-Instruct under GRPO: p* sampling matches baseline peak AIME accuracy with 28% fewer tokens and surpasses it by 5pp.
• Average accuracy across AIME, AMC, MATH, GSM8K for Math-1.5B-IT: 74.0 with p* vs 71.2 for GRPO baseline.

Surprising findings

• Using constant proxy G_i=1 (cost-only sampling) performs worse than using advantage magnitude, showing gradient-norm estimation matters.
• Under GRPO+ZVF, unsmoothed p* sampling slightly underperforms the no-sampling baseline on average (69.8 vs 70.6), while smoothed p* helps.

Limitations

• Proxy |A_i| for gradient norm is heuristic; theory assumes convexity while LLM setting is non-convex.
• Gap between upper and lower bounds remains open.
• Reported gains vary across models and settings; some configurations show no improvement or slight drops.

Experimental setup

Datasets: DAPO

Benchmarks: AIME 1983-2024 AMC MATH500 GSM8K

Metrics: pass@1 (mean@32) cumulative policy optimization tokens cost-biased chi-squared divergence Pearson correlation

Baselines: GRPO GRPO+ZVF uniform sampling variance-reduction sampling CISPO

Builds on: GRPO SGD importance sampling CISPO zero-variance filtering

Closest papers by meaning

• Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning (ICLR 2026 · similarity 0.87)
• Conformal Constrained Policy Optimization for Cost-Effective LLM Agents (AAAI 2026 · similarity 0.86)
• Cost-Aware Near-Optimal Policy Learning (AAAI 2025 · similarity 0.86)
• Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.86)
• TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback (NeurIPS 2026 · similarity 0.86)