ICML 2020 accepted papers on Architectures and efficiency
91 of the 1,087 papers accepted at ICML 2020 (8.4%) are on Architectures and efficiency, compared with 7.2% at ICML 2019. They fall into 6 topics.
See ICML 2020 in the interactive map
Submitting to ICML? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 39 papers
- Which Tasks Should Be Learned Together in Multi-task Learning? (222 citations)
- Voice Separation with an Unknown Number of Multiple Speakers (111 citations)
- On Layer Normalization in the Transformer Architecture (105 citations)
- When Explanations Lie: Why Many Modified BP Attributions Fail (58 citations)
- Improving the Gating Mechanism of Recurrent Neural Networks (41 citations)
- Learning to Branch for Multi-Task Learning (41 citations)
- Do RNN and LSTM have Long Memory? (40 citations)
- Inducing and Exploiting Activation Sparsity for Fast Inference on Deep Neural Networks (36 citations)
- Radioactive data: tracing through training (34 citations)
- Reverse-engineering deep ReLU networks (28 citations)
- Non-Autoregressive Neural Text-to-Speech (26 citations)
- Two Routes to Scalable Credit Assignment without Weight Symmetry (25 citations)
and 27 more in the interactive map.
Attention and transformers · 16 papers
- Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (301 citations)
- Stabilizing Transformers for Reinforcement Learning (130 citations)
- Sparse Sinkhorn Attention (114 citations)
- Non-autoregressive Machine Translation with Disentangled Context Transformer (65 citations)
- Improving Transformer Optimization Through Better Initialization (49 citations)
- Self-Attentive Hawkes Process (42 citations)
- PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination (32 citations)
- Infinite attention: NNGP and NTK for deep attention networks (27 citations)
- Learning to Encode Position for Transformer with Continuous Dynamical Model (23 citations)
- Time-aware Large Kernel Convolutions (21 citations)
- Transformer Hawkes Process (21 citations)
- Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules (15 citations)
and 4 more in the interactive map.
CNN design · 12 papers
- Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (69 citations)
- Attentive Group Equivariant Convolutional Networks (38 citations)
- On the Number of Linear Regions of Convolutional Neural Networks (36 citations)
- Decoupled Greedy Learning of CNNs (32 citations)
- Neural Kernels Without Tangents (27 citations)
- Deep Isometric Learning for Visual Recognition (23 citations)
- Disentangling Trainability and Generalization in Deep Neural Networks (14 citations)
- PDO-eConvs: Partial Differential Operator Based Equivariant Convolutions (12 citations)
- Universal Equivariant Multilayer Perceptrons (10 citations)
- Revisiting Spatial Invariance with Low-Rank Local Connectivity (9 citations)
- PENNI: Pruned Kernel Sharing for Efficient CNN Inference (8 citations)
- Channel Equilibrium Networks for Learning Deep Representation (5 citations)
Pruning and efficient tuning · 12 papers
- Rigging the Lottery: Making All Tickets Winners (117 citations)
- Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (105 citations)
- Linear Mode Connectivity and the Lottery Ticket Hypothesis (98 citations)
- Soft Threshold Weight Reparameterization for Learnable Sparsity (96 citations)
- Proving the Lottery Ticket Hypothesis: Pruning is All You Need (69 citations)
- Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection (48 citations)
- Operation-Aware Soft Channel Pruning using Differentiable Masks (32 citations)
- DropNet: Reducing Neural Network Complexity via Iterative Pruning (19 citations)
- RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr (9 citations)
- Go Wide, Then Narrow: Efficient Training of Deep Thin Networks (5 citations)
- Finding trainable sparse networks through Neural Tangent Transfer (4 citations)
- DessiLBI: Exploring Structural Sparsity of Deep Networks via Differential Inclusion Paths
Quantization · 6 papers
- Accelerating Large-Scale Inference with Anisotropic Vector Quantization (109 citations)
- Up or Down? Adaptive Rounding for Post-Training Quantization (51 citations)
- Multi-Precision Policy Enforced Training (MuPPET) : A Precision-Switching Strategy for Quantised Fixed-Point Training of CNNs (9 citations)
- Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks (6 citations)
- Don’t Waste Your Bits! Squeeze Activations and Gradients for Deep Neural Networks via TinyScript
- Towards Accurate Post-training Network Quantization via Bit-Split and Stitching
Neural architecture search · 6 papers
- AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (124 citations)
- Stabilizing Differentiable Architecture Search via Perturbation-based Regularization (82 citations)
- Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data (46 citations)
- Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search (33 citations)
- Neural Architecture Search in A Proxy Validation Loss Landscape (20 citations)
- Learning to Rank Learning Curves (6 citations)
