NeurIPS 2020 accepted papers on Architectures and efficiency
144 of the 1,901 papers accepted at NeurIPS 2020 (7.6%) are on Architectures and efficiency, compared with 8.9% at NeurIPS 2019. They fall into 8 topics.
See NeurIPS 2020 in the interactive map
Submitting to NeurIPS? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 61 papers
- AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (138 citations)
- DynaBERT: Dynamic BERT with Adaptive Width and Depth (118 citations)
- Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks (78 citations)
- Neuron Shapley: Discovering the Responsible Neurons (61 citations)
- Can the Brain Do Backpropagation? --- Exact Implementation of Backpropagation in Predictive Coding Networks (58 citations)
- Evolving Normalization-Activation Layers (58 citations)
- The phase diagram of approximation rates for deep neural networks (53 citations)
- Model Rubik’s Cube: Twisting Resolution, Depth and Width for TinyNets (47 citations)
- The interplay between randomness and structure during learning in RNNs (46 citations)
- BERT Loses Patience: Fast and Robust Inference with Early Exit (45 citations)
- Data Diversification: A Simple Strategy For Neural Machine Translation (43 citations)
- Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel (38 citations)
and 49 more in the interactive map.
Attention and transformers · 20 papers
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (633 citations)
- Big Bird: Transformers for Longer Sequences (267 citations)
- Investigating Gender Bias in Language Models Using Causal Mediation Analysis (133 citations)
- ConvBERT: Improving BERT with Span-based Dynamic Convolution (129 citations)
- Modern Hopfield Networks and Attention for Immune Repertoire Classification (113 citations)
- CogLTX: Applying BERT to Long Texts (77 citations)
- Incorporating BERT into Parallel Sequence Decoding with Adapters (48 citations)
- Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping (35 citations)
- Deep Transformers with Latent Depth (22 citations)
- Limits to Depth Efficiencies of Self-Attention (22 citations)
- SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection (14 citations)
- Language Through a Prism: A Spectral Approach for Multiscale Language Representations (13 citations)
and 8 more in the interactive map.
CNN design · 17 papers
- SCOP: Scientific Control for Reliable Neural Network Pruning (96 citations)
- Kernel Based Progressive Distillation for Adder Neural Networks (30 citations)
- Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification (29 citations)
- RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference (27 citations)
- Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency (23 citations)
- Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts (14 citations)
- Spin-Weighted Spherical CNNs (10 citations)
- Structured Convolutions for Efficient Neural Network Design (10 citations)
- Towards Learning Convolutions from Scratch (9 citations)
- Neural Anisotropy Directions (7 citations)
- How to Characterize The Landscape of Overparameterized Convolutional Neural Networks (5 citations)
- Curriculum By Smoothing (4 citations)
and 5 more in the interactive map.
Neural architecture search · 16 papers
- MCUNet: Tiny Deep Learning on IoT Devices (269 citations)
- Hierarchical Neural Architecture Search for Deep Stereo Matching (230 citations)
- BRP-NAS: Prediction-based NAS using GCNs (101 citations)
- Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS (66 citations)
- Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search (46 citations)
- Semi-Supervised Neural Architecture Search (44 citations)
- ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding (40 citations)
- Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? (33 citations)
- Improving Auto-Augment via Augmentation-Wise Weight Sharing (28 citations)
- Revisiting Parameter Sharing for Automatic Neural Channel Number Search (25 citations)
- Theory-Inspired Path-Regularized Differential Network Architecture Search (24 citations)
- Neural Architecture Generator Optimization (20 citations)
and 4 more in the interactive map.
Pruning and efficient tuning · 16 papers
- Pruning neural networks without any data by iteratively conserving synaptic flow (74 citations)
- Neuron-level Structured Pruning using Polarization Regularizer (64 citations)
- Winning the Lottery with Continuous Sparsification (48 citations)
- TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning (44 citations)
- The Lottery Ticket Hypothesis for Pre-trained BERT Networks (36 citations)
- Movement Pruning: Adaptive Sparsity by Fine-Tuning (33 citations)
- The Generalization-Stability Tradeoff In Neural Network Pruning (32 citations)
- What Do Neural Networks Learn When Trained With Random Labels? (32 citations)
- Stochastic Normalization (21 citations)
- Storage Efficient and Dynamic Flexible Runtime Channel Pruning via Deep Reinforcement Learning (14 citations)
- Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot (13 citations)
- Neuron Merging: Compensating for Pruned Neurons (11 citations)
and 4 more in the interactive map.
Quantization · 10 papers
- Ultra-Low Precision 4-bit Training of Deep Neural Networks (93 citations)
- Bayesian Bits: Unifying Quantization and Pruning (79 citations)
- Rotated Binary Neural Network (65 citations)
- HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks (54 citations)
- ShiftAddNet: A Hardware-Inspired Deep Network (38 citations)
- Robust Quantization: One Model to Rule Them All (33 citations)
- Searching for Low-Bit Weights in Quantized Neural Networks (33 citations)
- Position-based Scaled Gradient for Model Quantization and Pruning (17 citations)
- FleXOR: Trainable Fractional Quantization
- FracTrain: Fractionally Squeezing Bit Savings Both Temporally and Spatially for Efficient DNN Training
Spiking networks · 3 papers
- Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networks (116 citations)
- Unifying Activation- and Timing-based Learning Rules for Spiking Neural Networks (8 citations)
- Minimax Dynamics of Optimally Balanced Spiking Networks of Excitatory and Inhibitory Neurons (4 citations)
