Research map / Label-efficient and robust learning / Knowledge distillation

Understanding Layer Patching in Model Size Interpolation

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Few-shot and metric distillation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematically studies which student layers to patch when interpolating model size via boomerang distillation, and proposes KLPatch, a greedy KL-divergence-based patching order algorithm.

Problem

Choosing which student layers to patch with teacher blocks for zero-shot model size interpolation is a key but underexplored design choice; naive patching can degrade performance.

Approach

Casts patching-order selection as a shortest-path problem on a KL-weighted Boolean lattice of partially patched models, and uses a greedy KL-minimizing construction (KLPatch) that approximates the optimum in O(N^2) evaluations.

Key results

• On DistilBERT, last-to-first patching attains the globally optimal AUPIC (0.397) among all 720 patching orders, and KLPatch matches it.
• On Qwen3-4B-Base, KLPatch achieves maximum AUIC of 0.533 among sampled orders (vs 0.462 for last-to-first), beating all 200 random orderings.
• KLPatch reduces search complexity from O(2^N) or O(N!) to O(N^2) KL evaluations, with N=6 layers in the small-model experiments.

Surprising findings

• Last-to-first patching has the lowest Wikitext AUPIC on Qwen models yet does not achieve the highest downstream AUIC.
• Naive patching orders can degrade performance, making intermediate models worse than the student in some orderings.

Limitations

• KLPatch does not work out-of-the-box for Llama models and needs a modification (always patch layer 1 first).
• Gains from KLPatch vary substantially across model families.
• Evaluation relies on distilled student models from prior work, inheriting their limitations.

Experimental setup

Datasets: The Pile (64-sample calibration set) Wikitext

Benchmarks: ARC-easy ARC-challenge BoolQ HellaSwag OpenBookQA PIQA WinoGrande RACE MMLU RTE GSM8K IFEval

Metrics: accuracy exact match perplexity pseudo-perplexity AUIC AUPIC AUPIC percentile

Baselines: first-to-last patching last-to-first patching random patching orders

Builds on: boomerang distillation knowledge distillation iterative layer pruning

Closest papers by meaning

• Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.87)
• Scalable Model Merging with Progressive Layer-wise Distillation (ICML 2025 · similarity 0.86)
• Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.86)
• A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.86)
• Efficient stagewise pretraining via progressive subnetworks (ICLR 2025 · similarity 0.86)
• Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models (ICLR 2025 · similarity 0.86)