Research map / Large language models / Reasoning and chain of thought

Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes On-Policy Harness Self-Distillation (OPHSD), which uses the model run inside an inference-time harness as a self-teacher to distill harness-induced reasoning back into the unassisted model via on-policy reverse-KL.

Problem

Inference-time harnesses improve LLM reasoning, but the gains come from external workflows and are lost when the harness is removed; the paper asks whether harness procedures can be internalized into model weights.

Approach

Generalize self-distillation's privileged context from a static variable (reference solution, instruction) to a harness-driven workflow: the same model, wrapped in a draft-verify or plan-solve harness with privileged input, serves as a stop-gradient teacher for on-policy reverse-KL training of the bare student.

Key results

• On HMMT25, OPHSD reaches 53.33 pass@8, exceeding OPSD by 10.83 points and GRPO by 8.33 points.
• On text classification, OPHSD scores 69.51 F1 on LawBench and 90.81% accuracy on USPTO, above GRPO (62.44 / 90.01) and OPSD (64.25 / 88.01).
• Average pass@8 across four math benchmarks is 69.50% for OPHSD versus 66.68% for OPSD and 66.57% for GRPO; re-attaching the harness at inference gives no gain (69.13%).

Surprising findings

• Re-attaching the draft-verify harness to the OPHSD-trained model lowers LawBench F1 and USPTO accuracy (68.41 and 83.62 vs 69.51 and 90.81 standalone), suggesting the harness interferes once internalized.
• OPSD's math response length collapses after about 40 training steps and its benchmark performance declines, whereas OPHSD stays stable.

Limitations

• Validated on only two harnesses (draft-verify and plan-solve) rather than a broad sweep of harness designs.
• Real-time external interaction such as search or code execution is argued to be non-distillable and is not empirically tested.
• Math results use pass@8 with a small number of problems (AIME, HMMT), so differences of a few points carry high variance.

Experimental setup

Datasets: CAIL-2018 USPTO-50k DeepMath LawBench AIME24 AIME25 OlympiadBench HMMT25

Benchmarks: LawBench USPTO-50k AIME24 AIME25 OlympiadBench HMMT25

Metrics: F1 accuracy pass@8 pass@1

Baselines: GRPO OPSD CRISP Base Qwen3-8B with harness

Builds on: on-policy distillation OPSD SDFT GRPO CRISP learning using privileged information

Closest papers by meaning

• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (ICML 2026 · similarity 0.91)
• KL for a KL: On-Policy Distillation with Control Variate Baseline (NeurIPS 2026 · similarity 0.89)
• Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation (NeurIPS 2026 · similarity 0.89)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.89)
• Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation (NeurIPS 2026 · similarity 0.88)
• The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.88)