Research map / Large language models / Reasoning and chain of thought

NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Negative-aware Fine-Tuning (NFT), a supervised online method that uses an implicit negative policy parameterized by the same LLM to learn from wrong answers, showing it matches RL methods like GRPO and DAPO on math reasoning.

Problem

Verification-driven self-improvement for LLM math reasoning is dominated by RL; supervised learning (e.g., rejection fine-tuning) discards negative answers and cannot learn from mistakes.

Approach

Model the negative answer distribution implicitly as a reparameterization of the same positive policy (pi_old minus r*pi+ over 1-r), so a maximum-likelihood loss on negative samples directly optimizes the positive policy; the resulting loss is shown equivalent to GRPO gradients under strictly on-policy training.

Key results

• On Qwen2.5-Math-7B, NFT reaches 51.7 average across six benchmarks, versus 49.5 for GRPO, 51.2 for DAPO and 48.3 for RFT.
• On Qwen2.5-32B, NFT reaches 59.2 average, close to DAPO's 59.9 and well above RFT's 52.8.
• The paper proves NFT and GRPO produce identical gradients under strictly on-policy training (R=1, epsilon<=1).

Surprising findings

• Pure supervised NFT matches or surpasses RL methods such as GRPO and DAPO on math benchmarks.
• Negative-sample learning is the main source of the gap between RFT and RL; RFT alone already contributes about 80% of the gain in the 32B setting.

Limitations

• Gains from negative data are modest in the 32B setting, where positive data accounts for about 80% of gains.
• Equivalence to GRPO holds only under strict on-policy training; off-policy behavior differs via clipping strategy.
• Negative-ratio clipping epsilon must be tuned; very small values degrade performance.

Experimental setup

Datasets: DAPO-Math-17k

Benchmarks: AIME24 AIME25 AMC23 MATH500 OlympiadBench Minerva Math

Metrics: avg@32 accuracy avg@1 accuracy

Baselines: GRPO DAPO Dr. GRPO RFT Iterative DPO

Builds on: GRPO DAPO rejection fine-tuning DPO InfoNCA

Closest papers by meaning

• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.91)
• AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.90)
• MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model (ICLR 2026 · similarity 0.90)
• RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold (NeurIPS 2024 · similarity 0.90)
• Native Reasoning Models: Training Language Models to Reason on Unverifiable Data (ICLR 2026 · similarity 0.89)