Research map / Architectures and efficiency / RNNs and normalization

Till the Layers Collapse: Compressing a Deep Neural Network Through the Lenses of Batch Normalization Layers.

AAAI 2025 · Architectures and efficiency › RNNs and normalization · approach: Normalization layer theory · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Till the Layers Collapse (TLC), which uses batch-normalization (or pre-activation statistics) parameters to rank layers, linearize or remove neurons in the OFF/ON state, and iteratively remove layers with retraining to reduce network depth.

Problem

Deep neural networks are overparameterized and their inference latency is dominated by the sequential depth of the forward pass, yet most compression methods reduce width or parameters rather than depth.

Approach

Use batch-norm statistics (beta and gamma) to decide per neuron whether a rectifier is mostly ON or OFF, remove OFF neurons and linearize ON neurons into the next layer, then rank layers by the accuracy drop from removing each one and iteratively collapse the least important layers with retraining.

Key results

• On CIFAR-10 with ResNet-18, TLC removes 12 of 17 layers with top-1 90.91 ± 0.57 vs. dense 92.00 (table 1 reports 91.36 in the timing table).
• On Swin-T CIFAR-10, TLC removes 6 of 12 layers with top-1 91.98 ± 0.07 vs. dense 91.63.
• On ImageNet with MobileNet-V2, TLC removes 2 of 35 layers with top-1 59.43 vs. dense 71.87, far above EASIER's 41.14 at same removal count.

Measured effects reported in the paper

Surprising findings

• Removing some layers sometimes increases validation accuracy, which the authors attribute to overfitting of the full model.
• Removing the layer with smallest weights/gradients fails on MobileNet-V2 and EGP collapses VGG-16bn to 10% accuracy on CIFAR-10.

Limitations

• TLC requires long training times and high computational requirements due to full retraining per iteration.
• Layer removal is ineffective on RTE, where few layers can be removed and the dense model underperforms.
• TLC-finetuning removes fewer layers and is only tested on CIFAR-10.

Experimental setup

Datasets: CIFAR-10 Tiny-ImageNet ImageNet PACS VLCS SST-2 QNLI RTE

Benchmarks: CIFAR-10 Tiny-ImageNet ImageNet PACS VLCS SST-2 QNLI RTE

Metrics: top-1 accuracy number of removed layers MFLOPs memory usage training time KL divergence

Baselines: Smallest weights Smallest gradients EGP Layer folding (LF) EASIER

Builds on: Layer Folding EASIER EGP Network slimming

Closest papers by meaning

• Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.85)
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.85)
• Neural collapse vs. low-rank bias: Is deep neural collapse really optimal? (NeurIPS 2024 · similarity 0.84)
• Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.84)
• Layer Compression of Deep Networks with Straight Flows (AAAI 2024 · similarity 0.84)
• The Unreasonable Ineffectiveness of the Deeper Layers (ICLR 2025 · similarity 0.84)