ICML 2022 accepted papers on Architectures and efficiency
115 of the 1,233 papers accepted at ICML 2022 (9.3%) are on Architectures and efficiency, compared with 9.1% at ICML 2021. They fall into 8 topics.
See ICML 2022 in the interactive map
Submitting to ICML? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 39 papers
- Dataset Condensation via Efficient Synthetic-Data Parameterization (34 citations)
- Multi-Task Learning as a Bargaining Game (26 citations)
- Unified Scaling Laws for Routed Language Models (22 citations)
- Prioritized Training on Points that are Learnable, Worth Learning, and not yet Learnt (18 citations)
- A Modern Self-Referential Weight Matrix That Learns to Modify Itself (15 citations)
- Flashlight: Enabling Innovation in Tools for Machine Learning (14 citations)
- Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models (13 citations)
- The Neural Race Reduction: Dynamics of Abstraction in Gated Networks (12 citations)
- Data Scaling Laws in NMT: The Effect of Noise and Architecture (8 citations)
- Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Sets (8 citations)
- Towards Scaling Difference Target Propagation by Learning Backprop Targets (8 citations)
- DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks (7 citations)
and 27 more in the interactive map.
Attention and transformers · 32 papers
- Improving Language Models by Retrieving from Trillions of Tokens (298 citations)
- Transformer Quality in Linear Time (44 citations)
- Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding (39 citations)
- Particle Transformer for Jet Tagging (37 citations)
- General-purpose, long-context autoregressive modeling with Perceiver AR (34 citations)
- Flowformer: Linearizing Transformers with Conservation Flows (32 citations)
- Directed Acyclic Transformer for Non-Autoregressive Machine Translation (28 citations)
- HyperPrompt: Prompt-based Task-Conditioning of Transformers (28 citations)
- XAI for Transformers: Better Explanations through Conservative Propagation (25 citations)
- What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization? (22 citations)
- It’s Raw! Audio Generation with State-Space Models (20 citations)
- Revisiting End-to-End Speech-to-Text Translation From Scratch (16 citations)
and 20 more in the interactive map.
Pruning and efficient tuning · 15 papers
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (205 citations)
- Monarch: Expressive Structured Matrices for Efficient and Accurate Training (8 citations)
- POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging (6 citations)
- Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets (5 citations)
- The State of Sparse Training in Deep Reinforcement Learning (5 citations)
- PAC-Net: A Model Pruning Approach to Inductive Transfer Learning (4 citations)
- Sparse Double Descent: Where Network Pruning Aggravates Overfitting (4 citations)
- SPDY: Accurate Pruning with Speedup Guarantees (4 citations)
- The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks (4 citations)
- Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations (3 citations)
- Training Your Sparse Neural Network Better with Any Mask (3 citations)
- Winning the Lottery Ahead of Time: Efficient Early Network Pruning (2 citations)
and 3 more in the interactive map.
CNN design · 13 papers
- Secure Quantized Training for Deep Learning (11 citations)
- Blurs Behave Like Ensembles: Spatial Smoothings to Improve Accuracy, Uncertainty, and Robustness (4 citations)
- Equivariance versus Augmentation for Spherical Images (4 citations)
- Neural Tangent Kernel Analysis of Deep Narrow Neural Networks (3 citations)
- Scaling-up Diverse Orthogonal Convolutional Networks by a Paraunitary Framework (2 citations)
- A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks (1 citation)
- Deploying Convolutional Networks on Untrusted Platforms Using 2D Holographic Reduced Representations (1 citation)
- ButterflyFlow: Building Invertible Layers with Butterfly Matrices
- Exploiting Redundancy: Separable Group Convolutional Networks on Lie Groups
- Implicit Bias of Linear Equivariant Networks
- CerDEQ: Certifiable Deep Equilibrium Model
- Efficient Learning of CNNs using Patch Based Features
and 1 more in the interactive map.
Quantization · 6 papers
- Overcoming Oscillations in Quantization-Aware Training (17 citations)
- Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training (11 citations)
- SDQ: Stochastic Differentiable Quantization with Mixed Precision (7 citations)
- Finding the Task-Optimal Low-Bit Sub-Distribution in Deep Neural Networks (4 citations)
- Fast Lossless Neural Compression with Integer-Only Discrete Flows
- Be Like Water: Adaptive Floating Point for Machine Learning
Neural architecture search · 4 papers
- Analyzing and Mitigating Interference in Neural Architecture Search (11 citations)
- ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks (5 citations)
- AGNAS: Attention-Guided Micro and Macro-Architecture Search
- Searching for BurgerFormer with Micro-Meso-Macro Space Design
Mixture of experts · 4 papers
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (165 citations)
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale (54 citations)
- Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers (11 citations)
- Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs (7 citations)
