ICLR 2022 accepted papers on Architectures and efficiency
138 of the 1,095 papers accepted at ICLR 2022 (12.6%) are on Architectures and efficiency, compared with 14.7% at ICLR 2021. They fall into 8 topics.
See ICLR 2022 in the interactive map
Submitting to ICLR? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 36 papers
- Exploring the Limits of Large Scale Pre-training (33 citations)
- Scaling Laws for Neural Machine Translation (19 citations)
- The MultiBERTs: BERT Reproductions for Robustness Analysis (17 citations)
- GradMax: Growing Neural Networks using Gradient Information (9 citations)
- Deep ReLU Networks Preserve Expected Length (7 citations)
- SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit models (7 citations)
- DKM: Differentiable k-Means Clustering Layer for Neural Network Compression (5 citations)
- Fast Differentiable Matrix Square Root (4 citations)
- Churn Reduction via Distillation (2 citations)
- Concurrent Adversarial Learning for Large-Batch Training (2 citations)
- Data-Driven Offline Optimization for Architecting Hardware Accelerators (2 citations)
- Information Bottleneck: Exact Analysis of (Quantized) Neural Networks (2 citations)
and 24 more in the interactive map.
Attention and transformers · 35 papers
- Efficiently Modeling Long Sequences with Structured State Spaces (498 citations)
- How Do Vision Transformers Work? (202 citations)
- When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations (102 citations)
- RegionViT: Regional-to-Local Attention for Vision Transformers (93 citations)
- Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (81 citations)
- ViTGAN: Training GANs with Vision Transformers (79 citations)
- cosFormer: Rethinking Softmax In Attention (65 citations)
- Equivariant Transformers for Neural Network based Molecular Potentials (62 citations)
- Relating transformers to models and neural representations of the hippocampal formation (59 citations)
- Memorizing Transformers (40 citations)
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (31 citations)
- Unified Visual Transformer Compression (29 citations)
and 23 more in the interactive map.
Pruning and efficient tuning · 27 papers
- Towards a Unified View of Parameter-Efficient Transfer Learning (274 citations)
- Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (158 citations)
- The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training (34 citations)
- Language model compression with weighted low-rank factorization (20 citations)
- MetaMorph: Learning Universal Controllers with Transformers (12 citations)
- Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity (11 citations)
- Dual Lottery Ticket Hypothesis (11 citations)
- Exploring extreme parameter compression for pre-trained language models (10 citations)
- Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients (10 citations)
- No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models (8 citations)
- Sparsity Winning Twice: Better Robust Generalization from More Efficient Training (8 citations)
- Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (7 citations)
and 15 more in the interactive map.
CNN design · 17 papers
- Omni-Dimensional Dynamic Convolution (181 citations)
- MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer (151 citations)
- On the Connection between Local Attention and Dynamic Depth-wise Convolution (69 citations)
- FlexConv: Continuous Kernel Convolutions With Differentiable Kernel Sizes (25 citations)
- CKConv: Continuous Kernel Convolution For Sequential Data (13 citations)
- Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100 (11 citations)
- Proving the Lottery Ticket Hypothesis for Convolutional Neural Networks (8 citations)
- Scattering Networks on the Sphere for Scalable and Rotationally Equivariant Spherical CNNs (6 citations)
- Learning Features with Parameter-Free Layers (5 citations)
- Quantitative Performance Assessment of CNN Units via Topological Entropy Calculation (5 citations)
- Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views? (1 citation)
- Gaussian Mixture Convolution Networks (1 citation)
and 5 more in the interactive map.
Neural architecture search · 11 papers
- Learning Strides in Convolutional Neural Networks (42 citations)
- Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks (20 citations)
- Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models (20 citations)
- NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy (15 citations)
- Generalizing Few-Shot NAS with Gradient Matching (9 citations)
- Learning Versatile Neural Architectures by Propagating Network Codes (8 citations)
- NASI: Label- and Data-agnostic Neural Architecture Search at Initialization (8 citations)
- On Redundancy and Diversity in Cell-based Neural Architecture Search (7 citations)
- NASPY: Automated Extraction of Automated Machine Learning Models
- NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict aware Supernet Training
- SUMNAS: Supernet with Unbiased Meta-Features for Neural Architecture Search
Quantization · 7 papers
- QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization (43 citations)
- BiBERT: Accurate Fully Binarized BERT (33 citations)
- F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (21 citations)
- SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation (19 citations)
- 8-bit Optimizers via Block-wise Quantization (4 citations)
- How Low Can We Go: Trading Memory for Error in Low-Precision Training (2 citations)
- Toward Efficient Low-Precision Training: Data Format Optimization and Hysteresis Quantization
Spiking networks · 4 papers
- Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting (99 citations)
- Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks (86 citations)
- Spike-inspired rank coding for fast and accurate recurrent neural networks
- Sequence Approximation using Feedforward Spiking Neural Network for Spatiotemporal Learning: Theory and Optimization Methods
