Research map / Large language models / Reasoning and chain of thought

AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies the synergy of SFT and RL for math and code reasoning, scaling SFT prompts/responses/epochs and analyzing RL temperature, overlong filtering, and stage-1 length, yielding AceReason-Nemotron-1.1-7B.

Problem

Understanding how SFT scaling and RL training choices interact to build strong long-CoT math and code reasoning models, which has been under-studied in the community.

Approach

Scales SFT data along unique prompts and responses per prompt (seven datasets v1–v7, up to 2.2M samples), then applies the stage-wise math-only/code-only GRPO curriculum from AceReason-Nemotron-1.0 to a much stronger Qwen2.5-Math-7B SFT model, with systematic analyses of RL temperature (entropy ~0.3 rule of thumb), overlong filtering across stages, and stage-1 (8K) compression.

Key results

• AceReason-Nemotron-1.1-7B reaches 72.6 on AIME24 and 64.8 on AIME25 (avg@64) at 32K output length, outperforming AceReason-Nemotron-1.0-7B (69.0 and 53.6).
• Math-only RL from the stronger SFT model improves LiveCodeBench v5 from 48.8 to 57.2 and v6 from 43.8 to 52.1 (the RL recipe applied to the SFT starting point).
• SFT scaling from v1 to v7 raises AIME25 by 8 points in the last step (41.3 to 49.3); regression finds prompt count coefficient a=4.831 vs responses-per-prompt b=2.635.

Surprising findings

• Overlong filtering helps at 8K but at 32K removing it (negative reward for truncated samples) gives better results, contradicting DAPO-style advice.
• Stage-1 (8K) RL causes an initial drop in AIME25 but skipping it leads to a lower final plateau (56.7 vs 51.8 is reported with stage-1 variants; skipping yields lower ceiling).

Limitations

• Results depend on a single 7B Qwen2.5-Math base and mostly on a fixed RL data curriculum from AceReason-Nemotron-1.0.
• Claims of contamination risk on earlier benchmark versions; decontamination relies on 9-gram overlap filtering.
• Several findings (e.g., temperature rule of thumb) are based on empirical trials without theoretical grounding.

Experimental setup

Datasets: AceMath NuminaMath OpenMathReasoning TACO APPs OpenCoder-Stage2 OpenCodeReasoning AceReason-Nemotron-1.0 RL data

Benchmarks: AIME24 AIME25 HMMT 2025 BRUMO 2025 MATH500 LiveCodeBench v5 LiveCodeBench v6 EvalPlus

Metrics: avg@n pass@1 pass@k

Baselines: DeepSeek-R1-Distill-Qwen-7B AceReason-Nemotron-1.0-7B Light-R1-7B Llama-Nemotron-Nano-8B-v1 OpenMath-Nemotron-7B OpenCodeReasoning-Nemotron-7B MiMo-7B-RL Skywork-OR1-7B

Builds on: AceReason-Nemotron DeepSeek-R1-Distill-Qwen GRPO DAPO DeepScaleR

Closest papers by meaning

• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning (NeurIPS 2025 · similarity 0.94)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.93)
• Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead (ICLR 2026 · similarity 0.91)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.91)
• NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning (ICLR 2026 · similarity 0.90)