Research map / Large language models / Reasoning and chain of thought
Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a three-step model specialization framework (negative assistant training, negative calibrated enhancement, adaptive self-consistency) that distills LLM chain-of-thought reasoning into LLaMA-7B using both positive and negative synthesized samples on the MATH dataset.
Problem
Distilling complex mathematical reasoning from LLMs into small models usually discards rationales with incorrect final answers, wasting knowledge in negative samples, especially on hard problems like MATH.
Approach
Train a frozen negative LoRA on incorrect rationales and fuse it into a positive LoRA via a corrected attention mechanism with signed weights in [-0.5, 0.5]; use the negative model as a calibration baseline for self-distillation via KL-based weighting; and train a ranking model on positive vs negative rationales to reweight self-consistency votes.
Key results
- On MATH with gpt-3.5-turbo CoT data, NAT reaches 6.81% average accuracy versus 3.88% for fine-tuning on original data (+75.5%) and 5.29% for CoT KD.
- NCE improves over KD by about 10% relative and NAT-derived NCE reaches 10.23% average accuracy with adaptive self-consistency on MATH.
- Adaptive self-consistency raises NAT from 9.25% (standard SC) to 9.84% average accuracy across the reported subjects.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add negative data via negative assistant training (dual LoRA) vs. positive-only CoT KD | LLaMA 7B distilled from GPT-3.5 Turbo · MATH test set | average accuracy: 6.81 vs 5.29 | Helped |
| add negative data via negative assistant training vs. positive-only CoT KD | LLaMA 7B distilled from GPT-4 · MATH test set | average accuracy: 6.83 vs 5.59 | Helped |
| directly mixing negative and positive data in training | LLaMA 7B · MATH test set | average accuracy: 3.03 vs 5.29 CoT KD | Hurt |
| remove negative data (dual LoRA on positive only) (ablation) | NAT · MATH test set | average accuracy: 6.58 vs 6.81 | Hurt |
| remove negative LoRA module (ablation) | NAT · MATH test set | average accuracy: 6.05 vs 6.81 | Hurt |
| replace corrected attention with gated integration (ablation) | NAT · MATH test set | average accuracy: 5.22 vs 6.81 | Hurt |
| restrict correction range to [0,1] (no negative direction) (ablation) | NAT · MATH test set | average accuracy: 6.30 vs 6.81 | Hurt |
| adaptive self-consistency (ranking model) instead of standard SC | NAT and CoT KD models · MATH test set | average accuracy: NAT 9.84 vs SC 9.25; CoT KD 8.25 vs 7.85 | Helped |
| SC with weighted sum (WS) instead of base SC | NAT, CoT KD, NCE · MATH test set | average accuracy: e.g. NAT 9.21 vs 9.25; NCE 9.64 vs 9.69 | No clear effect |
| negative calibrated enhancement (KL-weighted self-distillation) vs. KD | LLaMA 7B with GPT-3.5 data · MATH test set | average progress: avg progress 10% (0.66) over KD | Helped |
| negative calibrated enhancement vs. NAT (with fewer parameters) | LLaMA 7B · MATH test set | average progress: 6.5% (0.44) over NAT | Helped |
| NCE vs KD with data augmentation on OOD generalization | LLaMA 7B trained on GSM8K · GSM8K, ASDiv, MultiArith, SVAMP | accuracy: GSM8K 41.93 vs 41.55; MultiArith 88.67 vs 88.05; SVAMP 51.50 vs 50.70 | Mixed |
Surprising findings
- Models trained only on positive versus only on negative samples have very low overlap of correct answers on MATH (IoU 0.074), suggesting negative data holds complementary knowledge.
- Naively mixing negative samples into training (MIX) substantially hurts accuracy (3.03% vs 5.29% for CoT KD with gpt-3.5-turbo data).
Limitations
- Ranking model accuracy is only about 60%, limiting ASC gains.
- Evaluation restricted to LLaMA-7B as student and arithmetic/math reasoning tasks.
- Absolute accuracies on MATH remain low (below 10%).
Experimental setup
Datasets: MATH GSM8K ASDiv MultiArith SVAMP StrategyQA
Benchmarks: MATH GSM8K ASDiv MultiArith SVAMP StrategyQA
Metrics: accuracy
Baselines: Fine-tune CoT KD MIX CL NT UL Self-Consistency with weighted sum PaLM GPT3
Builds on: chain-of-thought distillation self-consistency LoRA unlikelihood training self-distillation
Closest papers by meaning
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.90)
- RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold (NeurIPS 2024 · similarity 0.90)
- Evaluating Mathematical Reasoning Beyond Accuracy (AAAI 2025 · similarity 0.90)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.90)
- Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.