NeurIPS 2022 accepted papers on Architectures and efficiency
265 of the 2,901 papers accepted at NeurIPS 2022 (9.1%) are on Architectures and efficiency, compared with 9.0% at NeurIPS 2021. They fall into 8 topics.
See NeurIPS 2022 in the interactive map
Submitting to NeurIPS? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 93 papers
- MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields (295 citations)
- Locating and Editing Factual Associations in GPT (173 citations)
- Beyond neural scaling laws: beating power law scaling via data pruning (84 citations)
- Transformer Memory as a Differentiable Search Index (80 citations)
- On the Parameterization and Initialization of Diagonal State Space Models (73 citations)
- On Embeddings for Numerical Features in Tabular Deep Learning (58 citations)
- A Neural Corpus Indexer for Document Retrieval (47 citations)
- Theseus: A Library for Differentiable Nonlinear Optimization (45 citations)
- Matryoshka Representation Learning (38 citations)
- Dataset Distillation using Neural Feature Regression (33 citations)
- Single-phase deep learning in cortico-cortical networks (24 citations)
- Revisiting Neural Scaling Laws in Language and Vision (22 citations)
and 81 more in the interactive map.
Attention and transformers · 47 papers
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (498 citations)
- ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation (348 citations)
- EfficientFormer: Vision Transformers at MobileNet Speed (256 citations)
- HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (184 citations)
- Cross Aggregation Transformer for Image Restoration (122 citations)
- Fast Vision Transformers with HiLo Attention (114 citations)
- Squeezeformer: An Efficient Transformer for Automatic Speech Recognition (72 citations)
- Diagonal State Spaces are as Effective as Structured State Spaces (64 citations)
- Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging (63 citations)
- What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (58 citations)
- Sequencer: Deep LSTM for Image Classification (53 citations)
- Data Distributional Properties Drive Emergent In-Context Learning in Transformers (50 citations)
and 35 more in the interactive map.
Pruning and efficient tuning · 44 papers
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (288 citations)
- AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (262 citations)
- LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (76 citations)
- Merging Models with Fisher-Weighted Averaging (43 citations)
- A Fast Post-Training Pruning Framework for Transformers (39 citations)
- LIFT: Language-Interfaced Fine-Tuning for Non-language Machine Learning Tasks (35 citations)
- Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation (21 citations)
- Advancing Model Pruning via Bi-level Optimization (17 citations)
- Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Tasks (14 citations)
- Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively (12 citations)
- Rare Gems: Finding Lottery Tickets at Initialization (10 citations)
- Structural Pruning via Latency-Saliency Knapsack (10 citations)
and 32 more in the interactive map.
CNN design · 25 papers
- GhostNetV2: Enhance Cheap Operation with Long-Range Attention (278 citations)
- SwinTrack: A Simple and Strong Baseline for Transformer Tracking (48 citations)
- Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets (39 citations)
- Revisiting Sparse Convolutional Model for Visual Recognition (21 citations)
- Benign Overfitting in Two-layer Convolutional Neural Networks (20 citations)
- SKFlow: Learning Optical Flow with Super Kernels (18 citations)
- Wavelet Feature Maps Compression for Image-to-Image CNNs (18 citations)
- Signal Processing for Implicit Neural Representations (11 citations)
- Relaxing Equivariance Constraints with Non-stationary Continuous Filters (6 citations)
- AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learning (5 citations)
- BiMLP: Compact Binary Architectures for Vision Multi-Layer Perceptrons (5 citations)
- On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels (5 citations)
and 13 more in the interactive map.
Quantization · 18 papers
- On-Device Training Under 256KB Memory (72 citations)
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (70 citations)
- Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (36 citations)
- Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer (31 citations)
- Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (28 citations)
- BiT: Robustly Binarized Multi-distilled Transformer (20 citations)
- Towards Efficient Post-training Quantization of Pre-trained Language Models (20 citations)
- Quantized Training of Gradient Boosting Decision Trees (13 citations)
- FP8 Quantization: The Power of the Exponent (12 citations)
- Is Integer Arithmetic Enough for Deep Learning Training? (7 citations)
- Hierarchical Channel-spatial Encoding for Communication-efficient Collaborative Learning (4 citations)
- Redistribution of Weights and Activations for AdderNet Quantization (3 citations)
and 6 more in the interactive map.
Neural architecture search · 17 papers
- HyperTree Proof Search for Neural Theorem Proving (25 citations)
- NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies (13 citations)
- NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks (10 citations)
- Tensor Program Optimization with Probabilistic Programs (10 citations)
- Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search (9 citations)
- BLOX: Macro Neural Architecture Search Benchmark and Algorithms (6 citations)
- Efficient Architecture Search for Diverse Tasks (5 citations)
- LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models (5 citations)
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring (4 citations)
- A Classification of $G$-invariant Shallow Neural Networks (3 citations)
- Bridge the Gap Between Architecture Spaces via A Cross-Domain Predictor (2 citations)
- Saliency-Aware Neural Architecture Search (2 citations)
and 5 more in the interactive map.
Spiking networks · 13 papers
- GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks (45 citations)
- Online Training Through Time for Spiking Neural Networks (28 citations)
- Biologically Inspired Dynamic Thresholds for Spiking Neural Networks (23 citations)
- Training Spiking Neural Networks with Local Tandem Learning (20 citations)
- Training Spiking Neural Networks with Event-driven Backpropagation (15 citations)
- Temporal Effective Batch Normalization in Spiking Neural Networks (10 citations)
- IM-Loss: Information Maximization Loss for Spiking Neural Networks (8 citations)
- Biologically plausible solutions for spiking networks with efficient coding (7 citations)
- Differentiable hierarchical and surrogate gradient search for spiking neural networks (7 citations)
- LTMD: Learning Improvement of Spiking Neural Networks with Learnable Thresholding Neurons and Moderate Dropout (6 citations)
- Theoretically Provable Spiking Neural Networks (2 citations)
- Dance of SNN and ANN: Solving binding problem by combining spike timing and reconstructive attention (1 citation)
and 1 more in the interactive map.
Mixture of experts · 8 papers
- Mixture-of-Experts with Expert Choice Routing (90 citations)
- Towards Understanding the Mixture-of-Experts Layer in Deep Learning (34 citations)
- Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs (28 citations)
- On the Representation Collapse of Sparse Mixture of Experts (20 citations)
- Spatial Mixture-of-Experts (5 citations)
- A Theoretical View on Sparsely Activated Networks (2 citations)
- TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training (2 citations)
- M³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
