Research map / Large language models / Reasoning and chain of thought
Antidistillation Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes antidistillation sampling, which perturbs a teacher reasoning model's next-token distribution using finite-difference gradients from a proxy student so that generated traces stay high-utility for the teacher but poison distillation.
Problem
Frontier reasoning models' extended reasoning traces can be used to distill their capabilities into student models, creating IP and safety risks for model owners; existing defenses are static or limit access.
Approach
Adds a distillation-aware penalty to the teacher's sampling logits, equal to the finite-difference approximation of how sampling each token would change a proxy student's downstream loss after one gradient step (computed as the directional derivative via two forward passes with weights θ_P ± ε∇ℓ). The method is dynamic, per-token, and needs only the proxy model and one precomputed gradient.
Key results
- On MMLU, a ~4% teacher accuracy drop with temperature sampling leaves the student at ~52%, whereas antidistillation sampling lowers the student to as low as 40%.
- Antidistillation sampling keeps teacher accuracy near 70% while pushing student accuracy below the undistilled level; temperature sampling needs teacher accuracy to fall to ~20% to achieve the same effect.
- On high-teacher-accuracy settings, going from 90% to 89% teacher accuracy drops the poisoned student from 65% to 56%, while temperature sampling does not reduce the student.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply antidistillation sampling instead of temperature sampling, at the same 4% reduction in teacher accuracy | DeepSeek-R1-Distill-Qwen-7B teacher and Llama-3.2-3B student · MMLU | student accuracy: 52% -> 40% | Hurt |
| Increase teacher sampling temperature | DeepSeek-R1-Distill-Qwen-7B teacher and Llama-3.2-3B student · MMLU | student accuracy: 52% -> 52% | No clear effect |
| Apply antidistillation sampling while reducing teacher accuracy by 1% | Antidistillation sampling · Reasoning benchmarks | student accuracy: 65% -> 56% | Hurt |
| Reduce teacher accuracy by 1% using temperature sampling instead | Temperature sampling · Reasoning benchmarks | student accuracy | No clear effect |
| Use finite differences instead of JVP to compute the antidistillation penalty (ablation) | Antidistillation sampling · GSM8K | antidistillation performance | No clear effect |
| Use JVP instead of finite differences (ablation) | Antidistillation sampling · GSM8K | sampling speed: around eight times | Hurt |
| Sweep the sampling temperature between 0 and 1 (ablation) | Antidistillation sampling · Reasoning benchmarks | antidistillation performance | No clear effect |
| Replace the antidistillation perturbation with randomly permuted, sign-flipped perturbations (ablation) | Permutation sampling · Reasoning benchmarks | student accuracy | Mixed |
| Use a smaller Qwen2.5-1.5B proxy or a larger Qwen2.5-7B proxy (ablation) | Antidistillation sampling · GSM8K | student accuracy | No clear effect |
Surprising findings
- Permutation sampling, which keeps the statistics of the perturbation but scrambles gradient information, is a strong baseline that is not effective, indicating the gradient information itself is necessary.
- The finite-difference approximation performed better than the exact JVP implementation in bfloat16 and even in some float32 settings.
Limitations
- Trade-off persists: teacher accuracy still drops with stronger poisoning, so utility is reduced to some degree.
- Proof-of-concept only; evaluated on a limited set of models (3B proxies and students) and datasets (GSM8K, MATH, MMLU).
- Security analysis against model extraction and other attacks is out of scope; roughly 8x slowdown with JVP implementation.
Experimental setup
Datasets: GSM8K GSM8K Platinum MATH MMLU
Benchmarks: GSM8K GSM8K Platinum MATH MMLU
Metrics: accuracy teacher accuracy student accuracy holdout loss
Baselines: temperature sampling permutation sampling noisy sampling greedy sampling
Builds on: LoRA temperature sampling Jacobian-vector product
Closest papers by meaning
- Reasoning with Sampling: Your Base Model is Smarter Than You Think (ICLR 2026 · similarity 0.76)
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens (NeurIPS 2026 · similarity 0.75)
- Learnable Sampler Distillation for Discrete Diffusion Models (NeurIPS 2025 · similarity 0.75)
- Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation (NeurIPS 2026 · similarity 0.74)
- Agentic Abstention: Do Agents Know When to Stop Instead of Act? (NeurIPS 2026 · similarity 0.74)
- Mark, Don't Erase: Token Inoculation for Dual-Use Knowledge in LLMs (NeurIPS 2026 · similarity 0.74)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.