NeurIPS 2021 accepted papers on Architectures and efficiency
225 of the 2,508 papers accepted at NeurIPS 2021 (9.0%) are on Architectures and efficiency, compared with 7.6% at NeurIPS 2020. They fall into 8 topics.
See NeurIPS 2021 in the interactive map
Submitting to NeurIPS? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 86 papers
- Revisiting ResNets: Improved Training and Scaling Strategies (205 citations)
- Revisiting Deep Learning Models for Tabular Data (115 citations)
- Revisiting the Calibration of Modern Neural Networks (69 citations)
- A Geometric Analysis of Neural Collapse with Unconstrained Features (43 citations)
- Zero Time Waste: Recycling Predictions in Early Exit Neural Networks (33 citations)
- Associative Memories via Predictive Coding (31 citations)
- Efficiently Identifying Task Groupings for Multi-Task Learning (29 citations)
- FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition (29 citations)
- Why Spectral Normalization Stabilizes GANs: Analysis and Improvements (29 citations)
- SketchGen: Generating Constrained CAD Sketches (26 citations)
- Deep Learning on a Data Diet: Finding Important Examples Early in Training (23 citations)
- Similarity and Matching of Neural Network Representations (23 citations)
and 74 more in the interactive map.
Attention and transformers · 58 papers
- Transformer in Transformer (1,022 citations)
- CoAtNet: Marrying Convolution and Attention for All Data Sizes (739 citations)
- Twins: Revisiting the Design of Spatial Attention in Vision Transformers (612 citations)
- DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification (313 citations)
- R-Drop: Regularized Dropout for Neural Networks (305 citations)
- Intriguing Properties of Vision Transformers (302 citations)
- TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (263 citations)
- HRFormer: High-Resolution Vision Transformer for Dense Predict (156 citations)
- ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias (153 citations)
- ResT: An Efficient Transformer for Visual Recognition (146 citations)
- All Tokens Matter: Token Labeling for Training Better Vision Transformers (143 citations)
- Chasing Sparsity in Vision Transformers: An End-to-End Exploration (86 citations)
and 46 more in the interactive map.
CNN design · 27 papers
- MLP-Mixer: An all-MLP Architecture for Vision (1,429 citations)
- Early Convolutions Help Transformers See Better (351 citations)
- Do Vision Transformers See Like Convolutional Neural Networks? (108 citations)
- Efficient Training of Visual Transformers with Small Datasets (84 citations)
- Rethinking the Pruning Criteria for Convolutional Neural Network (26 citations)
- Gaussian Kernel Mixture Network for Single Image Defocus Deblurring (23 citations)
- Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Network (22 citations)
- Dataset Distillation with Infinitely Wide Convolutional Networks (21 citations)
- Memory-efficient Patch-based Inference for Tiny Deep Learning (21 citations)
- Automatic Symmetry Discovery with Lie Algebra Convolutional Network (20 citations)
- Container: Context Aggregation Networks (18 citations)
- Efficient Equivariant Network (8 citations)
and 15 more in the interactive map.
Pruning and efficient tuning · 25 papers
- Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (82 citations)
- MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge (42 citations)
- Only Train Once: A One-Shot Neural Network Training And Pruning Framework (42 citations)
- Aligned Structured Sparsity Learning for Efficient Image Super-Resolution (27 citations)
- CHIP: CHannel Independence-based Pruning for Compact Neural Networks (25 citations)
- DRONE: Data-aware Low-rank Compression for Large NLP Models (18 citations)
- Sparse Training via Boosting Pruning Plasticity with Neuroregeneration (13 citations)
- Validating the Lottery Ticket Hypothesis with Inertial Manifold Theory (12 citations)
- Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (11 citations)
- AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks (10 citations)
- Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot? (9 citations)
- Hyperparameter Tuning is All You Need for LISTA (7 citations)
and 13 more in the interactive map.
Neural architecture search · 15 papers
- Evaluating Efficient Performance Estimators of Neural Architectures (29 citations)
- Neural Ensemble Search for Uncertainty Estimation and Dataset Shift (17 citations)
- How Powerful are Performance Predictors in Neural Architecture Search? (15 citations)
- BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer (14 citations)
- TNASP: A Transformer-based NAS Predictor with a Self-evolution Framework (14 citations)
- Speedy Performance Estimation for Neural Architecture Search (13 citations)
- Stronger NAS with Weaker Predictors (13 citations)
- Hardware-adaptive Efficient Latency Prediction for NAS via Meta-Learning (8 citations)
- NAS-Bench-x11 and the Power of Learning Curves (7 citations)
- Progressive Feature Interaction Search for Deep Sparse Network (6 citations)
- Task-Adaptive Neural Network Search with Meta-Contrastive Learning (5 citations)
- Generic Neural Architecture Search via Regression (3 citations)
and 3 more in the interactive map.
Spiking networks · 6 papers
- Deep Residual Learning in Spiking Neural Networks (220 citations)
- Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks (60 citations)
- Increasing Liquid State Machine Performance with Edge-of-Chaos Dynamics Organized by Astrocyte-modulated Plasticity (23 citations)
- Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State (7 citations)
- Sparse Spiking Gradient Descent (4 citations)
- Learning to Time-Decode in Spiking Neural Networks Through the Information Bottleneck (2 citations)
Quantization · 6 papers
- MQBench: Towards Reproducible and Deployable Model Quantization Benchmark (23 citations)
- Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples (5 citations)
- Post-Training Quantization for Vision Transformer (3 citations)
- Post-Training Sparsity-Aware Quantization (2 citations)
- CBP: backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method
- S$^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks
