NeurIPS 2019 accepted papers on Architectures and efficiency
127 of the 1,429 papers accepted at NeurIPS 2019 (8.9%) are on Architectures and efficiency, compared with 9.5% at NeurIPS 2018. They fall into 7 topics.
See NeurIPS 2019 in the interactive map
Submitting to NeurIPS? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 67 papers
- PyTorch: An Imperative Style, High-Performance Deep Learning Library (15,937 citations)
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (256 citations)
- Meta-Learning with Implicit Gradients (196 citations)
- Understanding and Improving Layer Normalization (175 citations)
- Volumetric Correspondence Networks for Optical Flow (161 citations)
- Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers (154 citations)
- Intrinsic dimension of data representations in deep neural networks (153 citations)
- Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs (107 citations)
- Root Mean Square Layer Normalization (106 citations)
- Deep ReLU Networks Have Surprisingly Few Activation Patterns (105 citations)
- Fast Structured Decoding for Sequence Models (94 citations)
- Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks (92 citations)
and 55 more in the interactive map.
CNN design · 24 papers
- On Exact Computation with an Infinitely Wide Neural Net (384 citations)
- CondConv: Conditionally Parameterized Convolutions for Efficient Inference (241 citations)
- Channel Gating Neural Networks (133 citations)
- Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks (108 citations)
- General E(2)-Equivariant Steerable CNNs (93 citations)
- Deep Scale-spaces: Equivariance Over Scale (86 citations)
- Deep Multimodal Multilinear Fusion with High-order Polynomial Pooling (66 citations)
- Convolution with even-sized kernels and symmetric padding (50 citations)
- Deep Signature Transforms (50 citations)
- STAR-Caps: Capsule Networks with Straight-Through Attentive Routing (44 citations)
- A General Theory of Equivariant CNNs on Homogeneous Spaces (35 citations)
- Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks (30 citations)
and 12 more in the interactive map.
Neural architecture search · 12 papers
- Network Pruning via Transformable Architecture Search (137 citations)
- SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers (114 citations)
- Fast AutoAugment (100 citations)
- NAT: Neural Architecture Transformer for Accurate and Compact Architectures (71 citations)
- Discovering Neural Wirings (61 citations)
- DATA: Differentiable ArchiTecture Approximation (42 citations)
- Deep Active Learning with a Neural Architecture Search (29 citations)
- Meta Architecture Search (25 citations)
- XNAS: Neural Architecture Search with Expert Advice (21 citations)
- Towards modular and programmable architecture search (10 citations)
- Efficient Forward Architecture Search (9 citations)
- Constrained deep neural network architecture search for IoT devices accounting for hardware calibration
Attention and transformers · 10 papers
- Deep Equilibrium Models (264 citations)
- Stand-Alone Self-Attention in Vision Models (216 citations)
- A Tensorized Transformer for Language Modeling (98 citations)
- Novel positional encodings to enable tree-based transformers (83 citations)
- Are Sixteen Heads Really Better than One? (45 citations)
- Saccader: Improving Accuracy of Hard Attention Models for Vision (37 citations)
- Self-attention with Functional Time Representation Learning (32 citations)
- Compositional De-Attention Networks (15 citations)
- Neural Shuffle-Exchange Networks - Sequence Processing in O(n log n) Time (15 citations)
- Ouroboros: On Accelerating Training of Transformer-Based Language Models
Quantization · 8 papers
- Post training 4-bit quantization of convolutional networks for rapid-deployment (261 citations)
- Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks (129 citations)
- Normalization Helps Training of Quantized LSTM (29 citations)
- MetaQuant: Learning to Quantize by Learning to Penetrate Non-differentiable Quantization (27 citations)
- Focused Quantization for Sparse CNNs (11 citations)
- Random Projections with Asymmetric Quantization (10 citations)
- The Synthesis of XNOR Recurrent Neural Networks with Stochastic Logic (10 citations)
- A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off (5 citations)
Pruning and efficient tuning · 5 papers
- Global Sparse Momentum SGD for Pruning Very Deep Neural Networks (136 citations)
- Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask (105 citations)
- One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers (105 citations)
- AutoPrune: Automatic Network Pruning by Regularizing Auxiliary Parameters (89 citations)
- Superposition of many models into one (75 citations)
