Research map / Reinforcement learning / Visual and representation RL

Learning CLI Agents with Structured Action Credit under Selective Observation

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: Probabilistic and autonomous RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes A^3, a turn-level advantage RL method using AST-based shell action similarity and tree-level credit, plus sigma-Reveal token-budgeted workspace context selection, evaluated on new ShellOps CLI benchmarks with Qwen3-14B.

Problem

Training CLI agents with RL under partial workspace observation and sparse terminal rewards, where per-action credit is hard to assign and repeated states for state-grouped normalization are rare.

Approach

Builds per-turn advantages by fusing episode-level median/MAD normalization, AST-signature Levenshtein-based sub-chain residuals clustered per turn, and gated tree-level margins over bucket-abstracted histories; paired with a relevance-scored, token-budgeted workspace view at inference.

Key results

• On ShellOps hybrid tasks, A^3 with sigma-Reveal reaches 24.6 exact match versus 11.3 for the strongest baseline (GSPO).
• On ShellOps-Pro at H_max=6, A^3 with sigma-Reveal reaches 29.9 macro EM, above Qwen3-235B-A22B (26.7) but below Kimi-K2.6 (45.9).
• A^3 advantage computation costs 0.023 s per step, comparable to GSPO and RetroAgent, versus 31.8 s for GiGPO with LLM judge.

Surprising findings

• Tree-level and episode-level advantage variants are reported to be weaker than the full combination, with episode-only the weakest variant.
• Plain ReACT prompting with Qwen3-14B outperforms several GRPO-family RL baselines on AB-OS and DataBench string tasks.

Limitations

• Structurally similar shell commands can differ in effect due to paths, options, or workspace state, so AST similarity is a proxy.
• sigma-Reveal cannot replace exhaustive evidence discovery when relevance is weakly reflected in the file hierarchy.
• Empirical scope limited to shell-driven filesystem interaction; GUI, web, embodied, or networked settings untested.

Experimental setup

Datasets: AgentBench DataBench EHRCon TableBench ShellOps ShellOps-Pro

Benchmarks: AgentBench DataBench EHRCon TableBench ShellOps ShellOps-Pro

Metrics: exact match Pass@3 Pass@5 LLM Judge accuracy line-level file-change recall macro exact match advantage cost (s)

Baselines: ReACT LATS rStar GSPO GiGPO HGPO RetroAgent Kimi-K2.6 GLM-5.1 Qwen3-235B-A22B

Builds on: GRPO GiGPO HGPO GSPO PPO

Closest papers by meaning

• ECHO: Terminal Agents Learn World Models for Free (NeurIPS 2026 · similarity 0.88)
• Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents? (NeurIPS 2026 · similarity 0.86)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (ICLR 2026 · similarity 0.86)
• Reinforcing LLM Agents via Policy Optimization with Action Decomposition (NeurIPS 2024 · similarity 0.86)
• CoAct-1: Computer-using Multi-agent System with Coding Actions (ICLR 2026 · similarity 0.86)
• What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces? (ICML 2026 · similarity 0.86)