Research map / Large language models / Reasoning and chain of thought

Native Reasoning Models: Training Language Models to Reason on Unverifiable Data

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Native Reasoning Training (NRT), an RL framework that treats reasoning as a latent variable and rewards sampled traces by the model's own likelihood of the reference answer, trained on question-answer pairs without reasoning demonstrations or verifiers.

Problem

Training reasoning models without expert-written reasoning traces or external verifiers, especially for unverifiable tasks where RLVR cannot be applied.

Approach

Treat the chain-of-thought as a latent variable and use differentiable aggregation functions (geometric mean, weighted sum over baseline-uncertain tokens) of the reference-answer token probabilities as intrinsic trace rewards, optimized with GRPO-style off-policy RL plus a format loss.

Key results

• NRT-WS (-log p) on Llama-3.1-8B averages 56.2 vs 46.0 for SFT and 50.8 for RLPR.
• On Llama-3.2-3B, NRT-WS (-log p) averages 39.9 vs 36.4 for SFT, while JLB, Verifree and RLPR fail to beat SFT.
• On Llama-3.1-8B, GSM8K rises from 29.0 (SFT) to 76.0 with NRT-WS (-log p).

Surprising findings

• On the 3B model, the verifier-free baselines JLB, Verifree and RLPR fail to surpass the SFT baseline.
• The RLPR baseline collapses to short, low-entropy reasoning during training while NRT variants keep long, diverse traces.

Limitations

• Reward aggregation functions are handcrafted; adaptive or learned schemes left to future work.
• Sampling-based RL is computationally expensive compared to SFT.
• Evaluated only via fine-tuning of base models, not pretraining.

Experimental setup

Datasets: tulu-3-sft-mixture (200K subset)

Benchmarks: BBH MMLU DROP PopQA TruthfulQA GSM8K MATH HumanEval IFEval

Metrics: primary metric per benchmark average score reasoning entropy reasoning length LLM-judged reasoning quality

Baselines: SFT JLB Verifree RLPR

Builds on: GRPO Dr.GRPO RLPR Verifree JLB

Closest papers by meaning

• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
• Hybrid Latent Reasoning via Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.90)
• Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization (ICLR 2026 · similarity 0.90)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.90)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.90)