Research map / Reinforcement learning / RL from feedback

AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AstraFlow, a dataflow-oriented RL system for agentic LLMs that decouples rollout (RaaS), dataflow coordination, and trainers to natively support multi-policy training, elastic/cross-region rollout, and composable data algorithms.

Problem

Existing LLM RL systems are trainer-centered and rigid, making multi-policy collaborative agentic RL, elastic and heterogeneous/cross-region rollout, and modular data algorithms require ad hoc, feature-specific engineering.

Approach

Replace trainer-centered control with a dataflow layer that mediates prompts, trajectories, and batches between substitutable Rollout-as-a-Service nodes and independent trainers, each running autonomous control loops and interacting only through data and weight interfaces.

Key results

• Math multi-policy solver+verifier in AstraFlow raises average accuracy from 51.1 to 56.5 and cuts time per iteration from 212.64s (verl) to 77.65s (2.7x speedup).
• Auto-scaling rollout pool reduces total GPU-hours to 312.0 vs 358.0 for fixed 6-GPU and 359.2 for fixed 11-GPU, with accuracy 67.9 vs 68.0 for the 11-GPU baseline.
• On Qwen3-8B math vs AReaL, AstraFlow matches accuracy (64.8 vs 65.4 avg) and per-iteration time (139.6s vs 137.0s).

Surprising findings

• Under bf16, 98.9%+ of parameters are bit-exactly unchanged per iteration, enabling delta weight transfer with about 30x compression bound.
• Dynamic sampling raises accuracy but increases generation cost about 3.5x, while GRESO and buffer replay reach baseline accuracy with fewer rollouts.

Limitations

• Focuses on systems abstractions rather than new RL algorithms; evaluation limited to math, code, and a few agentic tasks.
• Some cross-region experiments are simulated (tc/netem shaping, power-capped GPUs) rather than true multi-region deployment.
• Speedup comparison against verl uses re-run baseline numbers, and multi-policy comparisons use matched single-policy baselines rather than specialized agent baselines.

Experimental setup

Datasets: DAPO math set DeepCoder-Preview-Dataset DeepScaler RL math set AlfWorld WebShop ASearcher search QA

Benchmarks: AIME24 AIME25 AMC MATH500 Minerva Math LiveCodeBench v5 LiveCodeBench v6 Codeforces AlfWorld WebShop Bamboogle HotpotQA

Metrics: pass@1 (avg@4) time per iteration (s) GPU-hours wall-clock time weight delta sparsity rollouts generated

Baselines: AReaL Dr. MAS (verl) verl single-policy Solver

Builds on: M2PO GRESO dynamic sampling verl AReaL

Closest papers by meaning

• AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning (NeurIPS 2025 · similarity 0.89)
• Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.89)
• Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.89)
• Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.89)
• Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs (ICLR 2026 · similarity 0.89)
• SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.89)