ICLR 2023 accepted papers on Architectures and efficiency

RNNs and normalization · 56 papers

• MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction (69 citations)
• Neural Networks and the Chomsky Hierarchy (44 citations)
• Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (32 citations)
• Re-parameterizing Your Optimizers rather than Architectures (29 citations)
• EVC: Towards Real-Time Neural Image Compression with Mask Decay (23 citations)
• Hebbian Deep Learning Without Feedback (17 citations)
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (16 citations)
• Backpropagation through Combinatorial Algorithms: Identity with Projection Works (11 citations)
• How to Train your HIPPO: State Space Models with Generalized Orthogonal Basis Projections (11 citations)
• REPAIR: REnormalizing Permuted Activations for Interpolation Repair (11 citations)
• Scaling Forward Gradient With Local Losses (11 citations)
• The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers (10 citations)

and 44 more in the interactive map.

Attention and transformers · 45 papers

• Conditional Positional Encodings for Vision Transformers (406 citations)
• Hungry Hungry Hippos: Towards Language Modeling with State Space Models (119 citations)
• Simplified State Space Layers for Sequence Modeling (81 citations)
• Mass-Editing Memory in a Transformer (54 citations)
• Accurate Image Restoration with Attention Retractable Transformer (47 citations)
• Long Range Language Modeling via Gated State Spaces (45 citations)
• Mega: Moving Average Equipped Gated Attention (37 citations)
• A Self-Attention Ansatz for Ab-initio Quantum Chemistry (27 citations)
• MOAT: Alternating Mobile Convolution and Attention Brings Strong Vision Models (21 citations)
• What Makes Convolutional Models Great on Long Sequence Modeling? (20 citations)
• An efficient encoder-decoder architecture with top-down attention for speech separation (16 citations)
• What Do Self-Supervised Vision Transformers Learn? (16 citations)

and 33 more in the interactive map.

Pruning and efficient tuning · 25 papers

• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing (393 citations)
• Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (37 citations)
• Editing models with task arithmetic (31 citations)
• Dataset Pruning: Reducing Training Data by Examining Generalization Influence (19 citations)
• Parameter-Efficient Fine-Tuning Design Spaces (16 citations)
• LilNetX: Lightweight Networks with EXtreme Model Compression and Structured Sparsification (7 citations)
• OTOv2: Automatic, Generic, User-Friendly (7 citations)
• Pruning Deep Neural Networks from a Sparsity Perspective (7 citations)
• Trainability Preserving Neural Pruning (6 citations)
• A Unified Framework for Soft Threshold Pruning (5 citations)
• Coverage-centric Coreset Selection for High Pruning Rates (3 citations)
• How to prepare your task head for finetuning (3 citations)

and 13 more in the interactive map.

CNN design · 19 papers

• 3D UX-Net: A Large Kernel Volumetric ConvNet Modernizing Hierarchical Transformer for Medical Image Segmentation (102 citations)
• NORM: Knowledge Distillation via N-to-One Representation Matching (19 citations)
• Dilated convolution with learnable spacings (18 citations)
• Can CNNs Be More Robust Than Transformers? (14 citations)
• Basic Binary Convolution Unit for Binarized Image Restoration Network (6 citations)
• The Lie Derivative for Measuring Learned Equivariance (6 citations)
• Modelling Long Range Dependencies in $N$D: From Task-Specific to a General Purpose CNN (5 citations)
• Empowering Networks With Scale and Rotation Equivariance Using A Similarity Convolution (2 citations)
• A General Framework For Proving The Equivariant Strong Lottery Ticket Hypothesis (1 citation)
• Image as Set of Points (1 citation)
• Integrating Symmetry into Differentiable Planning with Steerable Convolutions (1 citation)
• NeRN: Learning Neural Representations for Neural Networks (1 citation)

and 7 more in the interactive map.

Neural architecture search · 8 papers

• ZiCo: Zero-shot NAS via inverse Coefficient of Variation on Gradients (18 citations)
• Recon: Reducing Conflicting Gradients From the Root For Multi-Task Learning (11 citations)
• PASHA: Efficient HPO and NAS with Progressive Resource Allocation (4 citations)
• $\Lambda$-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells
• Equivariance-aware Architectural Optimization of Neural Networks
• EA-HAS-Bench: Energy-aware Hyperparameter and Architecture Search Benchmark
• Improving Differentiable Neural Architecture Search by Encouraging Transferability
• Transfer NAS with Meta-learned Bayesian Surrogates

Quantization · 7 papers

• Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats (6 citations)
• FIT: A Metric for Model Sensitivity (1 citation)
• PowerQuant: Automorphism Search for Non-Uniform Quantization (1 citation)
• Bit-Pruning: A Sparse Multiplication-Less Dot-Product
• Block and Subword-Scaling Floating-Point (BSFP) : An Efficient Non-Uniform Quantization For Low Precision Inference
• OPTQ: Accurate Quantization for Generative Pre-trained Transformers
• Winning Both the Accuracy of Floating Point Activation and the Simplicity of Integer Arithmetic

Spiking networks · 4 papers

• Spikformer: When Spiking Neural Network Meets Transformer (104 citations)
• Bridging the Gap between ANNs and SNNs by Calibrating Offset Spikes (11 citations)
• Spiking Convolutional Neural Networks for Text Classification (8 citations)
• Heterogeneous Neuronal and Synaptic Dynamics for Spike-Efficient Unsupervised Learning: Theory and Design Principles (1 citation)

Mixture of experts · 3 papers

• Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints (12 citations)
• Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers (6 citations)
• SCoMoE: Efficient Mixtures of Experts with Structured Communication