ICLR 2020 accepted papers on Architectures and efficiency
121 of the 687 papers accepted at ICLR 2020 (17.6%) are on Architectures and efficiency, compared with 16.5% at ICLR 2019. They fall into 7 topics.
See ICLR 2020 in the interactive map
Submitting to ICLR? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 40 papers
- Selection via Proxy: Efficient Data Selection for Deep Learning (91 citations)
- BatchEnsemble: an Alternative Approach to Efficient Ensemble and Lifelong Learning (90 citations)
- DiffTaichi: Differentiable Programming for Physical Simulation (73 citations)
- Encoding word order in complex embeddings (63 citations)
- Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data (58 citations)
- Four Things Everyone Should Know to Improve Batch Normalization (43 citations)
- An Exponential Learning Rate Schedule for Deep Learning (41 citations)
- BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations (31 citations)
- Understanding Knowledge Distillation in Non-autoregressive Machine Translation (31 citations)
- Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks (30 citations)
- Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization (28 citations)
- Emergence of functional and structural properties of the head direction system by optimization of recurrent neural networks (26 citations)
and 28 more in the interactive map.
CNN design · 23 papers
- How much Position Information Do Convolutional Neural Networks Encode? (91 citations)
- On the Relationship between Self-Attention and Convolutional Layers (89 citations)
- Quantum Algorithms for Deep Convolutional Neural Networks (77 citations)
- Scale-Equivariant Steerable Networks (69 citations)
- Training binary neural networks with real-to-binary convolutions (66 citations)
- Batch-shaping for learning conditional channel gated networks (49 citations)
- Pay Attention to Features, Transfer Learn Faster CNNs (49 citations)
- Identity Crisis: Memorization and Generalization Under Extreme Overparameterization (47 citations)
- Generalization bounds for deep convolutional neural networks (38 citations)
- Provable Filter Pruning for Efficient Neural Networks (23 citations)
- Deep Audio Priors Emerge From Harmonic Convolutional Networks (22 citations)
- Capsules with Inverted Dot-Product Attention Routing (16 citations)
and 11 more in the interactive map.
Attention and transformers · 20 papers
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (535 citations)
- Reducing Transformer Depth on Demand with Structured Dropout (371 citations)
- Reformer: The Efficient Transformer (310 citations)
- Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring (167 citations)
- Lite Transformer with Long-Short Range Attention (163 citations)
- Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention (104 citations)
- Monotonic Multihead Attention (101 citations)
- Pre-training Tasks for Embedding-based Large-scale Retrieval (98 citations)
- On Identifiability in Transformers (76 citations)
- Depth-Adaptive Transformer (59 citations)
- Multiplicative Interactions and Where to Find Them (54 citations)
- Compressive Transformers for Long-Range Sequence Modelling (48 citations)
and 8 more in the interactive map.
Pruning and efficient tuning · 14 papers
- Comparing Rewinding and Fine-tuning in Neural Network Pruning (195 citations)
- Picking Winning Tickets Before Training by Preserving Gradient Flow (138 citations)
- Dynamic Model Pruning with Feedback (98 citations)
- Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP (97 citations)
- Drawing Early-Bird Tickets: Toward More Efficient Training of Deep Networks (72 citations)
- Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers (27 citations)
- A Signal Propagation Perspective for Pruning Neural Networks at Initialization (19 citations)
- Data-Independent Neural Pruning via Coresets (18 citations)
- Progressive Memory Banks for Incremental Domain Adaptation (18 citations)
- Lookahead: A Far-sighted Alternative of Magnitude-based Pruning (16 citations)
- One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation (14 citations)
- DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures (13 citations)
and 2 more in the interactive map.
Neural architecture search · 13 papers
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (420 citations)
- Understanding and Robustifying Differentiable Architecture Search (257 citations)
- Evaluating The Search Phase of Neural Architecture Search (164 citations)
- NAS evaluation is frustratingly hard (122 citations)
- Once-for-All: Train One Network and Specialize it for Efficient Deployment (99 citations)
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (81 citations)
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search (77 citations)
- Towards Fast Adaptation of Neural Architectures with Meta Learning (54 citations)
- AtomNAS: Fine-Grained End-to-End Neural Architecture Search (38 citations)
- How to 0wn the NAS in Your Spare Time (16 citations)
- BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget (13 citations)
- In Search for a SAT-friendly Binarized Neural Network Architecture (12 citations)
and 1 more in the interactive map.
Quantization · 9 papers
- LEARNED STEP SIZE QUANTIZATION (289 citations)
- Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks (118 citations)
- And the Bit Goes Down: Revisiting the Quantization of Neural Networks (65 citations)
- Mixed Precision DNNs: All you need is a good parametrization (49 citations)
- Linear Symmetric Quantization of Neural Networks for Low-precision Integer Hardware (35 citations)
- AutoQ: Automated Kernel-Wise Neural Network Quantization (20 citations)
- Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activations (16 citations)
- Gradient $\ell_1$ Regularization for Quantization Robustness (6 citations)
- Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (3 citations)
