Research map / Label-efficient and robust learning / Knowledge distillation

AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AVSD, an on-policy self-distillation method that combines multiple privileged-information views by using a geometric consensus target plus a gated arithmetic residual, improving math and code reasoning over OPSD and GRPO.

Problem

Single-view self-distillation fixes one type of privileged information (solution, answer, feedback) as teacher context, but the best view is task-dependent and the teacher can rely on information the student cannot access at inference.

Approach

Treats multiple privileged views as a teacher family; separates a geometric-mean consensus target from an arithmetic-marginal target, and adds the cross-view residual to the consensus advantage only through a gate that requires sign agreement and magnitude proportionality, so the residual cannot flip the consensus direction.

Key results

• On Qwen3-4B, AVSD reaches 59.9 average vs 58.2 for OPSD (best baseline), a 2.2-point gain over the strongest baseline.
• On Qwen3-8B, AVSD averages 62.1 vs 59.2 for OPSD, and AVSD achieves 69.6 on AIME25 and 47.1 on HMMT25.
• The fraction of high-impact tokens with incorrect-sign credit drops from 12.3% (OPSD) to 2.9% (AVSD); GRPO gives no signal in 29.6% of cases.

Surprising findings

• SFT underperforms the base model on Qwen3-4B and Qwen3-8B (47.6 and 56.0 vs 55.0 and 57.3 averages).

Limitations

• Focused on math and code reasoning; not validated on tool-use agents or broader settings.
• Performance depends on quality of constructed views and underlying (possibly noisy) solution data.
• Theory and advantage interpretation are specific to reverse-KL; other distillation losses not validated.

Experimental setup

Datasets: OpenThoughts (math subset) Codeforces

Benchmarks: AIME24 AIME25 HMMT25 Codeforces LiveCodeBench v6

Metrics: Avg@8

Baselines: SFT GRPO OPSD

Builds on: On-Policy Self-Distillation (OPSD) GRPO reverse KL distillation RLSD

Closest papers by meaning

• Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.90)
• Entropy-Aware On-Policy Distillation of Language Models (ICML 2026 · similarity 0.89)
• Learning from Language Feedback via Variational Policy Distillation (NeurIPS 2026 · similarity 0.89)
• Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information (NeurIPS 2026 · similarity 0.89)
• TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.89)
• Visual-Advantage On-Policy Distillation for Vision-Language Models (NeurIPS 2026 · similarity 0.89)