Research map / Large language models / Reasoning and chain of thought

In-Token Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-Feedback

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InTRO, a token-level self-feedback training method that reweights sampled next-token candidates by the ratio of answer-conditioned to unconditioned token probabilities, evaluated on math reasoning and OOD coding/knowledge tasks.

Problem

SFT on a single golden rationale penalizes valid alternative reasoning paths, while RL with verifiable rewards suffers from sparse sequence-level feedback, credit assignment difficulty, and high cost; process supervision needs external verifiers.

Approach

Estimate a posterior by conditioning the model on question plus ground-truth answer, and use token-wise ratios of answer-conditioned to unconditioned probabilities (correction factors) to weight n sampled alternative next tokens at each position of correct rationales, giving dense self-generated token-level feedback in one forward pass.

Key results

• On Qwen3-4B, InTRO averages 39.6 across six math benchmarks vs 32.9 base (+20.4% relative) and 36.2 for GRPO.
• On Qwen2.5-7B, InTRO averages 37.6 vs GRPO 36.6 and base 33.5.
• On OOD tasks with Qwen3-4B, InTRO averages 47.2 vs GRPO 36.0 and base 36.2 (LiveCodeBench 22.6 vs 6.2).

Surprising findings

• Answer-conditioned reasoning slightly hurts on easy benchmarks like MATH500 while helping on hard sets.
• Llama-3.1-8B-Instruct results show InTRO is not best on OlympiadBench (14.1 vs GRPO 16.9) and averages only slightly above GRPO (22.6 vs 22.3).

Limitations

• Relies on ground-truth answers being available for correct-answer conditioning, limiting to verifiable tasks.
• Weaker models (Llama) show smaller gains and mixed results, as the self-generated correction factors depend on model quality.
• Theoretical equivalence relies on deterministic answer assumption and approximation of the posterior by answer-conditioned policy; the sanity check is a toy setup.

Experimental setup

Datasets: MATH

Benchmarks: MATH500 Minerva Math OlympiadBench College Math AMC23 AIME25 LiveCodeBench BigCodeBench GPQA HumanEval IFEval

Metrics: pass@1 avg@32 average response length

Baselines: SFT LaTRO RAFT++ GPG GRPO

Builds on: LaTRO GRPO RAFT++ GPG OpenRLHF

Closest papers by meaning

• Thinkless: LLM Learns When to Think (NeurIPS 2025 · similarity 0.91)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.91)
• SABER: Switchable and Balanced Training for Efficient LLM Reasoning (AAAI 2026 · similarity 0.90)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
• Efficient Reasoning via Constrained Optimization in Latent Space (NeurIPS 2026 · similarity 0.90)
• Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization (ICML 2025 · similarity 0.90)