Research map / Label-efficient and robust learning / Knowledge distillation

SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SimCT, a cross-tokenizer on-policy distillation method that expands the supervision space with minimal aligned multi-token units shared by teacher and student tokenizers, keeping the reverse-KL OPD loss unchanged.

Problem

On-policy distillation assumes teacher and student predict over the same token units; with heterogeneous tokenizers, exact shared-token matching discards much of the teacher signal.

Approach

Instead of changing the OPD objective, SimCT builds a common supervision space from shared tokens plus minimal aligned units (finest text continuations jointly tokenizable by both tokenizers), scoring multi-token units by length-normalized log-likelihood and normalizing over the candidate set before applying reverse KL.

Key results

• SimCT achieves average 32.80 vs 31.12 for SimpleOPD and 32.01 for DSKD on Qwen2.5-7B-Instruct to Gemma-2-2B-it.
• SimCT reaches average 55.06 on Qwen2.5-7B-Instruct to Phi-4-mini-instruct, above GOLD (53.90) and DSKD (53.88).
• The paper reports an average relative gain of 6.0% over SFT.

Surprising findings

• Shared-vocabulary OPD (SimpleOPD) improves only marginally over the SFT warm start across the three settings.

Limitations

• Evaluated only on math and code with three teacher-student pairs.
• Relies on BPE-style tokenizers; may need adaptation for tokenizers with different segmentation.
• Uses 1/k length normalization heuristic without exhaustive study of alternatives; white-box teacher required.

Experimental setup

Datasets: GSM8K Orca-Math 200K OpenMathInstruct-1 MATH OpenCodeInstruct KodCode TACO CodeContests

Benchmarks: GSM8K MATH-500 MBPP LiveCodeBench-v6

Metrics: pass@1

Baselines: SimpleOPD DSKD ALM GOLD SFT

Builds on: on-policy distillation reverse KL universal logit distillation

Closest papers by meaning

• Your Teacher Can’t Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation (NeurIPS 2026 · similarity 0.90)
• SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (NeurIPS 2026 · similarity 0.90)
• When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning (NeurIPS 2026 · similarity 0.89)
• The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.89)
• Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation (NeurIPS 2026 · similarity 0.88)
• Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching (NeurIPS 2025 · similarity 0.88)