Research map / Architectures and efficiency / RNNs and normalization
How a Student becomes a Teacher: learning and forgetting through Spectral methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use spectral parametrization with regularization instead of conventional parametrization with weight decay | Student networks with h > 20 in the Gaussian teacher-student task · Gaussian teacher-student test | MSE: 9 ± 1 × 10^{-3} (S) and 9 ± 3 × 10^{-3} (Sλ) | No clear effect |
| Add spectral parametrization and regularization, then prune nodes ranked by the spectral relevance indicator | Sλ(h) · Gaussian teacher-student task | MSE after pruning | Mixed |
| Replace spectral parametrization with conventional parametrization and L2 weight regularization (ablation) | Student networks · Gaussian teacher-student task | Teacher-like path distribution | Hurt |
| Remove nodes with low spectral relevance while retaining the teacher-sized core (ablation) | Sλ(h) · Gaussian teacher-student task | Generalization performance | No clear effect |
Closest papers by meaning
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.90)
- On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting (ICML 2021 · similarity 0.89)
- Student-Teacher Learning From Clean Inputs to Noisy Inputs (CVPR 2021 · similarity 0.89)
- Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks (NeurIPS 2026 · similarity 0.87)
- Scaling Multi-Teacher Distillation for Digital Pathology (NeurIPS 2026 · similarity 0.87)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.