Research map / Continual learning / Online and replay-based
Auto-Compressing Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Auto-Compressing Networks, which replace short residual connections with long feedforward connections from every layer to the output so that training itself compresses information into early layers, yielding auto-compression, parameter savings, and gains in noise robustness, low-data, continual learning and transfer.
Problem
Residual networks often contain redundant depth and parameters, and their shortcut pathways can leave components underused or overfit; the goal is an architecture that adapts effective depth to task difficulty while keeping accuracy and generalization.
Approach
Replace short residual connections with additive long connections from each layer's output directly to the network output (y = sum of all layer outputs), so gradients get a strong direct path and training pushes information into early layers, leaving deeper layers redundant ("auto-compression") without any auxiliary losses or compression regularizers.
Key results
- AC-ViT matches residual ViT accuracy on ImageNet-1K (70.76% vs 70.74%) with 51M vs 86M parameters, and reaches top performance at 6 of 12 layers.
- In split CIFAR-100 continual learning with SI, AC-Mixer reaches 46.2% average accuracy at L=15 with 32% forgetting, versus 32.1% accuracy and 50% forgetting for the residual Mixer.
- On C-100 to C-10 transfer, AC-Mixer achieves 85.38% vs 82.9% for Aligned and 79% for LayerSkip.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace residual connections with long connections to the output | ViT · ImageNet-1K under input noise | average accuracy: 67.68 -> 69.50; 62.80 -> 64.54; 45.46 -> 51.89; 56.80 -> 59.80; 27.48 -> 36.35; 10.34 -> 19.98 | Helped |
| Replace residual connections with long connections to the output | BERT · SST-2 | accuracy: 86.63 ± 0.09 -> 86.68 ± 0.06 | Helped |
| Replace residual connections with long connections to the output | BERT · QNLI | accuracy: 83.14 ± 0.07 -> 83.07 ± 0.10 | Hurt |
| Replace residual connections with long connections to the output | BERT · QQP | accuracy: 87.20 ± 0.09 -> 87.30 ± 0.07 | Helped |
| Replace residual connections with long connections to the output | MLP-Mixer · Split CIFAR-100 continual learning, naive fine-tuning | average forgetting: 52.76±2.3 -> 46.55±2.2; 54.89±1.6 -> 45.46±5.8; 54.49±2.2 -> 46.91±2.4 | Helped |
| Replace residual connections with long connections to the output | MLP-Mixer with Synaptic Intelligence · Split CIFAR-100 continual learning | average forgetting: 42.4±4.1 -> 35.7±2.1; 44.6±3.7 -> 33.8±0.4; 50±2.1 -> 32 ± 1.8 | Helped |
| Replace residual connections with long connections to the output | MLP-Mixer · CIFAR-100 to CIFAR-10 transfer | accuracy: 82.9 ± 0.9 -> 85.38 ± 0.7; 79 ± 1.2 -> 85.38 ± 0.7 | Helped |
| Replace residual connections with long connections to the output | MLP-Mixer · CIFAR-10 | accuracy: 90.3 ± 0.09 -> 92.0 ± 0.08 | Helped |
| Use AC-Mixer instead of Res-Mixer | MLP-Mixer with shared early-exit head · CIFAR-10 | inference speedup (FLOPs): 2.4× -> 3.3× | Helped |
| Use only the direct-gradient component (DG) in ACN training (ablation) | AC-Mixer · CIFAR-10 | layer-wise compression | Hurt |
| Increase the classification subset from 2 to 5 to 10 classes (ablation) | AC-Mixer · CIFAR-10 class subsets | effective depth: 8, 10 and 12 layers | Mixed |
| Increase AC-Mixer depth while using Synaptic Intelligence (ablation) | AC-Mixer · Split CIFAR-100 continual learning | average forgetting: 35.7±2.1 -> 33.8±0.4 -> 32 ± 1.8 | Helped |
Surprising findings
- ACN training converges considerably slower (e.g., 700 epochs for AC-ViT vs 300 for residual ViT), a noted cost of the approach.
- Residual networks under Synaptic Intelligence show increasing forgetting with depth, while ACNs show decreasing forgetting.
Limitations
- Evaluation limited to relatively small-scale tasks due to resource constraints; no large-scale language or multimodal validation.
- ACNs require substantially longer training time than residual architectures.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K BooksCorpus English Wikipedia GLUE (SST-2, QQP, QNLI)
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K GLUE SST-2 GLUE QQP GLUE QNLI
Metrics: accuracy average accuracy average forgetting train and test loss FLOPs speedup parameter count
Baselines: Residual ViT Residual MLP-Mixer DenseNet-Mixer DenseFormer-Mixer LayerSkip Aligned Residual BERT Magnitude pruning Movement pruning BranchyNet
Builds on: ResNet DenseNet DenseFormer Deeply supervised nets Synaptic Intelligence
Closest papers by meaning
- Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition (NeurIPS 2021 · similarity 0.89)
- Compressing Neural Networks using the Variational Information Bottleneck (ICML 2018 · similarity 0.88)
- Model Preserving Compression for Neural Networks (NeurIPS 2022 · similarity 0.88)
- Compression-aware Training of Deep Networks (NeurIPS 2017 · similarity 0.88)
- Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth (ICML 2024 · similarity 0.86)
- Layer Compression of Deep Networks with Straight Flows (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.