ICLR 2021 accepted papers on Architectures and efficiency
126 of the 860 papers accepted at ICLR 2021 (14.7%) are on Architectures and efficiency, compared with 17.6% at ICLR 2020. They fall into 8 topics.
See ICLR 2021 in the interactive map
Submitting to ICLR? Checklist →
Working on something in this area? Describe your idea in scime atlas to see the closest papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 56 papers
- Autoregressive Entity Retrieval (197 citations)
- Hopfield Networks is All You Need (189 citations)
- Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth (104 citations)
- Dataset Condensation with Gradient Matching (102 citations)
- Recurrent Independent Mechanisms (71 citations)
- Large Associative Memory Problem in Neurobiology and Machine Learning (55 citations)
- Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU (52 citations)
- EVALUATION OF NEURAL ARCHITECTURES TRAINED WITH SQUARE LOSS VS CROSS-ENTROPY IN CLASSIFICATION TASKS (51 citations)
- Lipschitz Recurrent Neural Networks (47 citations)
- Minimum Width for Universal Approximation (44 citations)
- Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling (38 citations)
- Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $1/n$ Parameters (36 citations)
and 44 more in the interactive map.
Attention and transformers · 17 papers
- DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION (913 citations)
- Long Range Arena : A Benchmark for Efficient Transformers (191 citations)
- Rethinking Positional Encoding in Language Pre-training (171 citations)
- GraphCodeBERT: Pre-training Code Representations with Data Flow (161 citations)
- Random Feature Attention (116 citations)
- Rethinking Attention with Performers (116 citations)
- Is Attention Better Than Matrix Decomposition? (51 citations)
- On Position Embeddings in BERT (49 citations)
- Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (49 citations)
- LambdaNetworks: Modeling long-range Interactions without Attention (47 citations)
- DeLighT: Deep and Light-weight Transformer (46 citations)
- PMI-Masking: Principled masking of correlated spans (13 citations)
and 5 more in the interactive map.
Pruning and efficient tuning · 17 papers
- On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (207 citations)
- Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch (74 citations)
- Layer-adaptive Sparsity for the Magnitude-based Pruning (68 citations)
- Pruning Neural Networks at Initialization: Why Are We Missing the Mark? (47 citations)
- Progressive Skeletonization: Trimming more fat from a network at initialization (40 citations)
- ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations (31 citations)
- Learnable Embedding sizes for Recommender Systems (21 citations)
- Better Fine-Tuning by Reducing Representational Collapse (20 citations)
- A Gradient Flow Framework For Analyzing Network Pruning (18 citations)
- HyperGrid Transformers: Towards A Single Model for Multiple Tasks (17 citations)
- Network Pruning That Matters: A Case Study on Retraining Variants (17 citations)
- Robust Pruning at Initialization (13 citations)
and 5 more in the interactive map.
CNN design · 13 papers
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (21,812 citations)
- Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs (81 citations)
- Generalization in data-driven models of primary visual cortex (38 citations)
- Meta-learning Symmetries by Reparameterization (27 citations)
- Computational Separation Between Convolutional and Fully-Connected Networks (16 citations)
- Revisiting Dynamic Convolution via Matrix Decomposition (12 citations)
- Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization (5 citations)
- Separation and Concentration in Deep Networks (4 citations)
- Hyperbolic Neural Networks++ (2 citations)
- The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods (2 citations)
- Efficient Generalized Spherical CNNs (1 citation)
- A Discriminative Gaussian Mixture Model with Sparsity
and 1 more in the interactive map.
Neural architecture search · 13 papers
- DARTS-: Robustly Stepping out of Performance Collapse Without Indicators (89 citations)
- Zero-Cost Proxies for Lightweight NAS (65 citations)
- Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (50 citations)
- Rethinking Architecture Selection in Differentiable NAS (32 citations)
- Locally Free Weight Sharing for Network Width Search (23 citations)
- ECONOMIC HYPERPARAMETER OPTIMIZATION WITH BLENDED SEARCH STRATEGY (8 citations)
- DrNAS: Dirichlet Neural Architecture Search (7 citations)
- Geometry-Aware Gradient Algorithms for Neural Architecture Search (5 citations)
- Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets (5 citations)
- CompOFA – Compound Once-For-All Networks for Faster Multi-Platform Deployment (3 citations)
- A Design Space Study for LISTA and Beyond (2 citations)
- HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark
and 1 more in the interactive map.
Quantization · 7 papers
- Training with Quantization Noise for Extreme Model Compression (114 citations)
- BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction (48 citations)
- A Block Minifloat Representation for Training Deep Neural Networks (12 citations)
- Neural gradients are near-lognormal: improved quantized and sparse training (5 citations)
- BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization (4 citations)
- Simple Augmentation Goes a Long Way: ADRL for DNN Quantization (1 citation)
- WrapNet: Neural Net Inference with Ultra-Low-Precision Arithmetic (1 citation)
