Research map / Multi-agent systems and games / LLM agents

Curriculum Design for Trajectory-Constrained Agent: Compressing Chain-of-Thought Tokens in LLMs

NeurIPS 2025 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CuRLTraC, a curriculum that adaptively tightens the training-time cost budget per task so the agent keeps a threshold reward, enabling RL and LLM agents to satisfy strict trajectory-level constraints such as CoT token budgets.

Problem

Agents deployed under strict trajectory-level constraints (e.g., token budgets, safety limits) receive sparse reward when trained with the full deployment constraint from the start, and contextual curricula are too costly for large context spaces like LLMs.

Approach

At each step, the teacher picks the training cost budget closest to the deployment target such that the current policy's expected shaped reward stays above a threshold beta (found by binary search over recent rollouts), giving a non-contextual, per-task adaptive curriculum for strict constraints.

Key results

• On the M1 configuration, CuRLTraC reduces SVAMP-Qwen response time from 7.5s to 0.6s and length from 138 to 3 tokens while keeping accuracy at 74% vs 77% for the base model.
• Target training yields 61% constrained accuracy on SVAMP-Qwen, while CuRLTraC reaches 74% under the same constraint.
• Theoretical analysis on a binary-tree MDP gives O(H^3) rollouts for the curriculum versus O(2^H) for the target baseline.

Measured effects reported in the paper

Surprising findings

• Fine-tuned models learn to skip explicit reasoning steps and output the answer directly for some SVAMP questions, with no CoT tokens.

Limitations

• Assumes deployment-time constraints are predefined; no single model adapts to varying budgets.
• Only tested on token-budget constraints in math reasoning; generality to other constraint types is limited to the presented domains.
• Theoretical analysis is limited to a simple binary-tree tabular setting.

Experimental setup

Datasets: SVAMP GSM8K

Benchmarks: SVAMP GSM8K BinaryTree PuddleGrid-Single PuddleGrid-Multi

Metrics: constrained accuracy mean return response time response length

Baselines: Target Unconstraint iid ExpSchedule Soft-RL AnsOnlyPrompt ProCuRL-Target

Builds on: REINFORCE RLOO CURROT ProCuRL-Target ExpSchedule

Closest papers by meaning

• Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.89)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.88)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.88)
• Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning (ICML 2025 · similarity 0.88)
• Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.88)
• Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.87)