Research map / Architectures and efficiency / RNNs and normalization
Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OPDF, which over-parameterizes a small student model during distillation by MPO (matrix product operator) decomposition of its weight matrices into higher-order tensors, plus an auxiliary tensor alignment loss against the teacher, then merges back to the original shape for inference.
Problem
Small student models have limited capacity, leaving a performance gap to large teachers in knowledge distillation; scaling student parameters during training usually raises inference cost.
Approach
Decompose each student weight matrix via MPO into higher-dimensional local tensors to increase trainable parameters during distillation, align the auxiliary tensors with the teacher's decomposed tensors via an MSE loss, then contract back to the original matrix so inference cost is unchanged.
Key results
- On GLUE, OPDF improves BERT-of-Theseus from 79.2 to 80.5 average, LGTM from 78.8 to 79.8, DBKD from 74.4 to 77.6, and AD-KD from 82.4 to 83.4 (matching BERT-base at 83.4).
- On ImageNet-1k linear probe, TinyViT-5M with OPDF reaches 80.0 top-1 vs 77.4 without over-parameterization; TinyViT-11M with OPDF (82.5) exceeds TinyViT-21M without it (82.3).
- MPO over-parameterization consistently outperforms SVD-based over-parameterization across GLUE tasks and TinyViT variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OPDF to BERT-of-Theseus | BERT-of-Theseus · GLUE average | average score: 79.2 -> 80.5 | Helped |
| Add OPDF to LGTM | LGTM · GLUE average | average score: 78.8 -> 79.8 | Helped |
| Add OPDF to DBKD | DBKD · GLUE average | average score: 74.4 -> 77.6 | Helped |
| Add OPDF to AD-KD | AD-KD · GLUE average | average score: 82.4 -> 83.4 | Helped |
| Use MPO instead of SVD for over-parameterization | TinyViT-5M distillation · ImageNet-1k | top-1 accuracy: 77.9 -> 80.0 | Helped |
| Use MPO instead of SVD for over-parameterization | TinyViT-11M distillation · ImageNet-1k | top-1 accuracy: 82.0 -> 82.5 | Helped |
| Use MPO instead of SVD for over-parameterization | TinyViT-21M distillation · ImageNet-1k | top-1 accuracy: 82.9 -> 84.0 | Helped |
| Add OPDF to TinyViT-5M | TinyViT-5M · ImageNet-1k | top-1 accuracy: 77.4* -> 80.0 | Helped |
| Add auxiliary tensor alignment loss (ablation) | OPDF with MPO over-parameterization · GLUE | task performance | Helped |
| Add MPO over-parameterization (ablation) | BERT-of-Theseus distillation · GLUE | task performance | Helped |
Surprising findings
- OPDF-enhanced students match or exceed the teacher on some GLUE tasks (e.g., MRPC and RTE), which the authors attribute to over-parameterization.
- Optimal learning rate decreases as the over-parameterization scale increases.
Limitations
- Gains plateau beyond certain over-parameterization thresholds (e.g., 1.6x for DBKD, 2.5x for LGTM).
- Training cost rises with over-parameterization (extra training time and memory), and the learning rate must be tuned as the scale grows.
- Evaluated only on GLUE and ImageNet-based ViT distillation; multimodal and other backbones are left to future work.
Experimental setup
Datasets: GLUE (RTE, MRPC, STS-B, CoLA, SST-2, QNLI, QQP, MNLI) ImageNet-21k
Benchmarks: GLUE ImageNet-1k ImageNet Real ImageNet V2
Metrics: accuracy F1 Matthews correlation Pearson/Spearman correlation top-1 accuracy top-5 accuracy
Baselines: BERT-of-Theseus LGTM DBKD AD-KD SVD over-parameterization TinyViT-5M TinyViT-11M TinyViT-21M
Builds on: BERT-of-Theseus LGTM DBKD AD-KD TinyViT
Closest papers by meaning
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.88)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.88)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.88)
- Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher (ICLR 2025 · similarity 0.88)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.