ICML 2021 accepted papers on Architectures and efficiency
108 of the 1,184 papers accepted at ICML 2021 (9.1%) are on Architectures and efficiency, compared with 8.4% at ICML 2020. They fall into 8 topics.
See ICML 2021 in the interactive map
Submitting to ICML? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 41 papers
- High-Performance Large-Scale Image Recognition Without Normalization (252 citations)
- Memory-Efficient Pipeline-Parallel DNN Training (59 citations)
- Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers (33 citations)
- Consensus Control for Decentralized Deep Learning (23 citations)
- Massively Parallel and Asynchronous Tsetlin Machine Architecture Supporting Almost Constant-Time Scaling (23 citations)
- Selfish Sparse RNN Training (16 citations)
- Learning Neural Network Subspaces (14 citations)
- Training Recurrent Neural Networks via Forward Propagation Through Time (14 citations)
- Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances (13 citations)
- Revealing the Structure of Deep Neural Networks via Convex Duality (13 citations)
- Align, then memorise: the dynamics of learning with feedback alignment (11 citations)
- Deep Learning for Functional Data Analysis with Adaptive Basis Layers (11 citations)
and 29 more in the interactive map.
Attention and transformers · 21 papers
- Training data-efficient image transformers & distillation through attention (1,042 citations)
- ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases (788 citations)
- MSA Transformer (368 citations)
- Perceiver: General Perception with Iterative Attention (125 citations)
- Attention is not all you need: pure attention loses rank doubly exponentially with depth (69 citations)
- TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models (33 citations)
- The Lipschitz Constant of Self-Attention (31 citations)
- Linear Transformers Are Secretly Fast Weight Programmers (23 citations)
- Poolingformer: Long Document Modeling with Pooling Attention (22 citations)
- Relative Positional Encoding for Transformers with Linear Complexity (20 citations)
- Synthesizer: Rethinking Self-Attention for Transformer Models (19 citations)
- SparseBERT: Rethinking the Importance Analysis in Self-attention (13 citations)
and 9 more in the interactive map.
Neural architecture search · 15 papers
- EfficientNetV2: Smaller Models and Faster Training (1,141 citations)
- Dataset Condensation with Differentiable Siamese Augmentation (51 citations)
- Neural Architecture Search without Training (43 citations)
- Path Planning using Neural A* Search (25 citations)
- HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search (17 citations)
- iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (16 citations)
- AutoAttend: Automated Attention Representation Search (11 citations)
- CATE: Computation-aware Neural Architecture Encoding with Transformers (10 citations)
- Few-Shot Neural Architecture Search (9 citations)
- K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets (6 citations)
- AlphaNet: Improved Training of Supernets with Alpha-Divergence (4 citations)
- KNAS: Green Neural Architecture Search (4 citations)
and 3 more in the interactive map.
CNN design · 11 papers
- SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks (652 citations)
- LieTransformer: Equivariant Self-Attention for Lie Groups (33 citations)
- Boosting the Throughput and Accelerator Utilization of Specialized CNN Inference Beyond Increasing Batch Size (7 citations)
- On-Off Center-Surround Receptive Fields for Accurate and Robust Image Classification (7 citations)
- Equivariant Networks for Pixelized Spheres (5 citations)
- Grid-Functioned Neural Networks (1 citation)
- Connecting Sphere Manifolds Hierarchically for Regularization
- FILTRA: Rethinking Steerable CNN by Filter Transform
- Skew Orthogonal Convolutions
- Winograd Algorithm for AdderNet
- Deep Continuous Networks
Pruning and efficient tuning · 10 papers
- Group Fisher Pruning for Practical Network Compression (24 citations)
- Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework (21 citations)
- Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (18 citations)
- On the Predictability of Pruning Across Scales (16 citations)
- Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not? (11 citations)
- Efficient Lottery Ticket Finding: Less Data is More (7 citations)
- A Probabilistic Approach to Neural Network Pruning (4 citations)
- Zoo-Tuning: Adaptive Transfer from A Zoo of Models (3 citations)
- Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset
- Learn-to-Share: A Hardware-friendly Transfer Learning Framework Exploiting Computation and Parameter Sharing
Quantization · 7 papers
- HAWQ-V3: Dyadic Neural Network Quantization (38 citations)
- I-BERT: Integer-only BERT Quantization (16 citations)
- ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training (10 citations)
- Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution (7 citations)
- Soft then Hard: Rethinking the Quantization in Neural Image Compression (7 citations)
- Training Quantized Neural Networks to Global Optimality via Semidefinite Programming (1 citation)
- Accurate Post Training Quantization With Small Calibration Sets
Spiking networks · 2 papers
Mixture of experts · 1 paper
- BASE Layers: Simplifying Training of Large, Sparse Models (63 citations)
