Research map / Large language models / Reasoning and chain of thought

Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Process reward models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CARL, a segment-level RL method for LLM tool use that trains a learned value critic on the model's own rollouts to assign per-segment advantages (invoke/assimilate/commit) from a single binary outcome reward, improving EM and reducing unnecessary tool calls.

Problem

Trajectory-level RL with a single outcome reward cannot isolate which tool call helped or penalize unnecessary tool calls, so models fail to learn when to use tools versus rely on parametric knowledge.

Approach

Decompose tool-use rollouts at structural boundaries (code fences, context blocks) into segments and use a learned critic V(s) over boundary states as a potential, so each segment's advantage is the change in predicted success probability; the critic learns the model's competence boundary from binary outcomes plus curated warm-up data.

Key results

• At 7B, CARL reaches 47.9 EM on HotpotQA, 48.3 on 2WikiMQA and 30.4 on Musique, versus 42.8, 41.7 and 22.1 for Search-R1 PPO.
• CARL invokes tools on 38.6% of Tier 2 questions versus 82.6% for Search-R1 PPO at 7B, with Tier 2 EM 87.0 vs 77.2.
• The critic separates Tier 1 from Tier 2 questions with AUC 0.93 at 7B and 0.85 at 3B.

Surprising findings

• Warm-up on base-model rollouts is essential; cold-start critic gives near-chance AUC (about 0.54) and trails the full method by about 9 EM on HotpotQA at 7B.
• The lambda=1 limit of the segment advantage reduces to trajectory-level credit, and the paper reports CARL's HotpotQA EM falls to 42.5 as lambda goes to 1.

Limitations

• Requires a calibrated critic via a warm-up phase; the benefit shrinks as warm-up quality degrades.
• All tokens in a segment share one advantage, so intra-segment variation is not captured.
• Segment boundaries require per-segment round-trips in current serving stacks, adding computational overhead.

Experimental setup

Datasets: HotpotQA 2WikiMultihopQA GSM8K FinQA Musique

Benchmarks: GSM8K HotpotQA 2WikiMultihopQA FinQA Musique

Metrics: exact match accuracy tokens per episode tool-use rate on Tier 1/Tier 2 questions AUC ECE

Baselines: Search-R1 PPO Search-R1 GRPO CoT+Tools (ReAct-style) SFT (rejection-sampled)

Builds on: PPO Search-R1 Options framework GAE RUDDER

Closest papers by meaning

• Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning (AAAI 2025 · similarity 0.88)
• Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models (NeurIPS 2025 · similarity 0.88)
• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.87)
• S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF (NeurIPS 2026 · similarity 0.87)
• Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.87)
• ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.86)