Research map / Learning theory and algorithms / Sample complexity
Should Under-parameterized Student Networks Copy or Average Teacher Weights?
Berfin Simsek, Amire Bendjeddou, Wulfram Gerstner, Johanni Brea
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| student width n below 0.46 k | gradient flow, erf student · unit-orthonormal teacher, erf | convergence to optimal copy-average point: n < gamma_1 k, gamma_1 near 0.46 | Improved |
| student width n above 0.6 k | gradient flow, erf student · unit-orthonormal teacher, erf | convergence to optimal copy-average point: n > gamma_2 k, gamma_2 near 0.6 | Hurt |
| student width between 0.46 k and 0.6 k | gradient flow, erf student · unit-orthonormal teacher, erf | convergence to optimal copy-average point: gamma_1 k < n < gamma_2 k | Mixed |
| increase n with fixed k-n (n=8,12,16) | optimal CA critical point · erf, k=n+h with h=1,2,3 | minimum Hessian eigenvalue | Hurt |
| use k much greater than n (k=n,2n,3n) | optimal CA critical point · erf | minimum Hessian eigenvalue | Improved |
| low compression (student width close to teacher width) | perturbed n-copy point vs optimal CA · erf | loss | Mixed |
| one-neuron student with ReLU | gradient flow · unit-orthonormal teacher, k=2..10 | norm times outgoing weight | Improved |
| ReLU activation instead of erf for multi-neuron student | gradient flow · unit-orthonormal teacher | optimal configuration | No clear effect |
| ReLU, k slightly larger than n (n=4,12) | gradient flow · unit-orthonormal teacher | configuration found | Mixed |
Closest papers by meaning
- How a Student becomes a Teacher: learning and forgetting through Spectral methods (NeurIPS 2023 · similarity 0.87)
- Student Specialization in Deep Rectified Networks With Finite Width and Input Dimension (ICML 2020 · similarity 0.86)
- On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting (ICML 2021 · similarity 0.86)
- ReLU Network with Width $d+\mathcal{O}(1)$ Can Achieve Optimal Approximation Rate (ICML 2024 · similarity 0.84)
- Expand-and-Cluster: Parameter Recovery of Neural Networks (ICML 2024 · similarity 0.84)
- $C^2M^3$: Cycle-Consistent Multi-Model Merging (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
