Research map / Architectures and efficiency / RNNs and normalization
Expand-and-Cluster: Parameter Recovery of Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Expand-and-Cluster, which recovers the weights and hidden-layer sizes of teacher networks by training overparameterised students to imitate teacher input-output behaviour and clustering neuron weight vectors shared across students.
Problem
Recovering the exact parameters (weights and layer widths) of a neural network from its input-output mapping, which is non-convex and affected by permutation, overparameterisation and activation-function symmetries.
Approach
Lift the parameter space by overparameterising student networks so that near-zero-loss solutions are easy to find, then cluster the input weight vectors across N students; clusters that are consistent across students correspond to teacher neurons, up to sign or scale depending on the activation's symmetries.
Key results
- On 150 synthetic teachers, Expand-and-Cluster with overparameterisation rho=4 correctly identified 118 (~80%).
- On MNIST, FashionMNIST and CIFAR10 one-hidden-layer teachers, reconstructed networks reached RMSE on the order of 1e-3 with size ratio m/r close to 1 (e.g. 1.008 for g on MNIST).
- In the shallow MNIST comparison, pruning baselines (magnitude pruning, LTH, OTOv2) could not simultaneously achieve low RMSE and zero excess neurons, while Expand-and-Cluster did.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Expand the student network and cluster neurons across multiple trained students instead of directly fitting a student of teacher width | Synthetic teacher networks with r=4 and d_in=4 · Synthetic regression | successful zero-loss recovery: None of the 20 students with 4 hidden neurons reached zero loss; ... if an overparameterisation of ρ ≥ 2 is used | Helped |
| Use overparameterisation factor ρ=4 | Expand-and-Cluster · 150 synthetic teacher problems | correct identification count: 118 ($ \sim 80 \% $) | Helped |
| Use FashionMNIST teacher queries rather than the MNIST training dataset | Expand-and-Cluster on an MNIST-trained teacher · MNIST teacher queried with FashionMNIST | RMSE: 1.14 \cdot 10^{-3} | No clear effect |
| Use overparameterised students with ρ=8 instead of same-size students with ρ=1 | Synthetic teacher configuration d_in=16, r=4 · Synthetic teacher #7 | successful identical input-output mapping runs: 2 times out of 1024 runs; ... 72 out of 128 runs | Helped |
| Train the next layer's students again after each layer reconstruction (ablation) | Expand-and-Cluster for deep networks · Deep MNIST teacher | reconstruction stability | Helped |
| Omit the retraining step at each layer reconstruction (ablation) | Expand-and-Cluster for deep synthetic networks · Deep synthetic networks | number of superfluous neurons: one or two superfluous neurons | Hurt |
| Use ρ=8 overparameterised students instead of direct training with the correct architecture | Synthetic deep teacher networks · Deep synthetic networks | loss: below 10^{-10}; ... never reached a loss below 10^{-2} | Helped |
| Increase the overparameterisation factor | Students trained on synthetic teachers · Synthetic regression | training loss | Mixed |
| Use γ=0.75 and β=π/180 for clustering (ablation) | Expand-and-Cluster · Shallow MNIST teacher recovery | successful reconstruction | Hurt |
Surprising findings
- Direct training of 20 same-size students (rho=1) for a teacher with r=4 and d_in=4 never reached zero loss, whereas overparameterised students did.
- Only 2 of 1024 same-size runs recovered the teacher mapping, versus 72 of 128 runs with rho=8 at equal compute.
Limitations
- Results limited to small-scale networks due to the compute needed to reach low loss in larger networks.
- Recovery of convolutional and residual architectures is only preliminary or not addressed.
- Identification is only up to sign, scale or affine transformation for some activation functions.
Experimental setup
Datasets: MNIST FashionMNIST CIFAR10 synthetic teacher-student tasks
Benchmarks: synthetic teacher-student tasks MNIST FashionMNIST CIFAR10
Metrics: RMSE loss hidden size ratio m/r average cosine distance between reconstructed and teacher we
Baselines: magnitude pruning lottery ticket hypothesis OTOv2 direct training of same-size student
Builds on: Simsek et al. 2021 global minima manifold of overparameterised networks gradient flow hierarchical clustering
Closest papers by meaning
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.86)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.86)
- The Effect of Weight Precision on the Neuron Count in Deep ReLU Networks (ICML 2024 · similarity 0.86)
- Towards Scalable and Versatile Weight Space Learning (ICML 2024 · similarity 0.85)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.85)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.