Research map / Large language models / Reasoning and chain of thought

The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that reward-model-preferred Oracle-refined math traces underperform the SLM's own rejection-sampled traces when distilling small models, and proposes Style-Aligned Refinement to restore utility by reducing distributional drift.

Problem

Distilling stronger reasoning models into small language models for math is assumed to benefit from traces with higher reward-model scores, but higher-reward Oracle-refined data can underperform the SLM's own rejection-sampled traces.

Approach

Oracle refinement couples logical repair with distributional drift from the learner's native reasoning style, raising adaptation cost (perplexity) and lowering downstream accuracy. Style-Aligned Refinement asks the Oracle to fix logic while mimicking the SLM's native style, lowering perplexity and restoring utility.

Key results

• Under DFT on Qwen2.5-Math-1.5B, SLM-RFT reaches 37.06 Avg@16 versus 34.06 for Oracle-Refined and 30.02 for Oracle-Synthesized, despite having the lowest mean reward (1.47 vs 1.70 and 1.88).
• Style-Aligned (Qwen) achieves 39.12 Avg@16 and global perplexity 1.46, below the SLM-RFT value of 1.52, while receiving the lowest reward score (1.37).
• Style-Aligned (GPT-5.2) reaches 38.21 Avg@16, improving over Oracle-Refined (34.06).

Surprising findings

• The highest-utility training data (SLM-RFT, Style-Aligned Qwen) receives among the lowest reward-model scores across three independent reward models.
• Style-Aligned Qwen data has lower perplexity than the model's own raw generations, yet it is rated lowest by the reward model.

Limitations

• Evidence is confined to small language models (1.5B-7B) and math-centric training data; generalization to larger models or other domains is untested.
• Style-Aligned Refinement is a prompt-based intervention, and more systematic style transfer methods are not explored.
• Mechanism is supported by perplexity and an LLM judge, with a single GPT-5.2 judge for semantic preservation and no human evaluation.

Experimental setup

Datasets: NuminaMath CoT

Benchmarks: MATH500 AIME 2024 AMC 2023 Minerva Math OlympiadBench

Metrics: Avg@16 accuracy reward score global perplexity semantic preservation score

Baselines: SLM-RFT Oracle-Refined Oracle-Synthesized NuminaMath Subset

Builds on: rejection sampling fine-tuning dynamic fine-tuning NuminaMath

Closest papers by meaning

• Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data (AAAI 2024 · similarity 0.89)
• Measuring Weak-to-Strong Legibility of Reasoning Models (NeurIPS 2026 · similarity 0.89)
• Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.88)
• RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold (NeurIPS 2024 · similarity 0.88)
• Evaluating Mathematical Reasoning Beyond Accuracy (AAAI 2025 · similarity 0.88)
• Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.88)