Research map / Large language models / Reasoning and chain of thought

$\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Textual Space

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ∇-Reasoner, which runs gradient descent on token logits (DTO) with a reward model and LLM likelihood inside the decoding loop, plus rejection sampling and caching accelerations, to improve math reasoning at test time.

Problem

Existing inference-time scaling methods for LLM reasoning rely on zeroth-order discrete search (BoN, ToT, RAP) or trial-and-error sampling, which is sample-inefficient under sparse rewards.

Approach

Treat the sequence's per-token logits as continuous variables and perform gradient descent on a loss combining the reward model score and LLM log-likelihood (straight-through Gumbel-softmax), then resample the next token from the refined logits and accept it via rejection sampling; theoretically relates this inference-time gradient flow to KL-regularized RL (PPO).

Key results

• On Qwen-2.5-7B-Instruct, ∇-Reasoner reaches 80.4% on MATH-500 and 26.6% on AIME24 with N_max=8, outperforming BoN (77.8%, 22.5%).
• Reduces model calls by up to 40.2% versus BoN/SC on instruct models while achieving higher accuracy.
• On Qwen-2.5-7B base, ∇-Reasoner scores 71.0% on MATH-500, comparable to GRPO (70.8%), a training-based method.

Surprising findings

• Rejection sampling without DTO matches the theoretical BoN rejection rate, while DTO lowers rejection rate substantially (e.g., 65.9% to 32.8% for Qwen-2.5).

Limitations

• Performance bounded by base model and reward model capability, especially under limited compute.
• Base and reward models must share a vocabulary for end-to-end logit optimization.
• Integrating with efficient LLM serving pipelines requires further system co-design.

Experimental setup

Datasets: MATH-500 AMC AIME24 AIME25 Open-thoughts (10k subset for SFT baseline) Numina-Math (35k subset for GRPO baseline)

Benchmarks: MATH-500 AMC AIME24 AIME25

Metrics: accuracy number of model calls rejection rate wall-clock time

Baselines: Greedy decoding Self-Consistency Best-of-N Tree-of-Thoughts Reasoning-as-Planning TPO SFT GRPO

Builds on: Best-of-N Reasoning-as-Planning Tree-of-Thoughts Gumbel-softmax straight-through PPO

Closest papers by meaning

• General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.90)
• Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space (NeurIPS 2026 · similarity 0.89)
• Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning (ICLR 2026 · similarity 0.89)
• Bag of Tricks for Inference-time Computation of LLM Reasoning (NeurIPS 2025 · similarity 0.89)
• SABER: Switchable and Balanced Training for Efficient LLM Reasoning (AAAI 2026 · similarity 0.89)
• Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.89)