Research map / Architectures and efficiency / RNNs and normalization

Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Theory of depth and width · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes StoMPP, a layerwise progressive freezing scheme with stochastic masks and soft refresh that binarizes weights and activations from input to output, with an STE-free variant and a StoMPP+STE variant, improving deep binary networks over vanilla STE.

Problem

Training binary neural networks from scratch degrades sharply with depth under the straight-through estimator; naive global progressive binarization collapses for full BNNs because frozen binary activations block gradients.

Approach

Progressively freeze weights and activations layer by layer from input to output using stochastic partial masks with soft refresh, so an unfrozen suffix always carries gradient; progression order is shown to be decisive for BNNs, and frozen-entry gradient can be zero (STE-free) or STE.

Key results

• STE-free StoMPP on ResNet-50 BNN improves over vanilla STE by +18.0 (CIFAR-10), +13.5 (CIFAR-100), +3.8 (ImageNet) points.
• StoMPP+STE on ResNet-50 BNN reaches +27.1 / +19.8 / +17.7 over vanilla STE on CIFAR-10 / CIFAR-100 / ImageNet.
• Reverse layerwise ordering collapses to near-chance on CIFAR-100 BNNs (ResNet-18 28.4%, ResNet-50 8.6%), while forward layerwise reaches 53.8% and 40.0%.

Surprising findings

• Binary-weight networks are largely insensitive to progression order, whereas binary-activation networks are highly sensitive.
• Vanilla STE on SST-2 BERT collapses to 50.9%, near random for binary classification.

Limitations

• Gradient-blockade mechanism is supported by experiments but not isolated by a direct intervention.
• Adds hyperparameters (schedule shape, refresh rate, per-layer step allocation) absent from vanilla STE.
• Matched minimal recipe yields absolute accuracies below SOTA; all runs are single-seed.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 ImageNet SST-2

Benchmarks: CIFAR-10 CIFAR-100 ImageNet SST-2

Metrics: top-1 accuracy

Baselines: STE (BinaryConnect/BinaryNet-style) Global progressive masking (INQ-style) BiTAT Bi-Real Net

Builds on: straight-through estimator INQ BinaryConnect Bi-Real Net

Closest papers by meaning

• Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers (NeurIPS 2023 · similarity 0.86)
• Layer Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant (NeurIPS 2026 · similarity 0.86)
• Deep Layers as Stochastic Solvers (ICLR 2019 · similarity 0.86)
• Training Binary Neural Networks via Gaussian Variational Inference and Low-Rank Semidefinite Programming (NeurIPS 2024 · similarity 0.85)
• Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.84)
• On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.84)