ICML 2023 accepted papers on Architectures and efficiency
143 of the 1,865 papers accepted at ICML 2023 (7.7%) are on Architectures and efficiency, compared with 9.3% at ICML 2022. They fall into 8 topics.
See ICML 2023 in the interactive map
Working on something in this area? Describe your idea in scime atlas to see the closest accepted papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 58 papers
- FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU (45 citations)
- Resurrecting Recurrent Neural Networks for Long Sequences (43 citations)
- Cramming: Training a Language Model on a single GPU in one day. (30 citations)
- BiBench: Benchmarking and Analyzing Network Binarization (17 citations)
- Minimal Width for Universal Property of Deep RNN (13 citations)
- Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory (12 citations)
- SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient (6 citations)
- Equivariant Architectures for Learning in Deep Weight Spaces (5 citations)
- Scaling Laws for Multilingual Neural Machine Translation (5 citations)
- Dataset Distillation with Convexified Implicit Gradients (3 citations)
- Maximal Initial Learning Rates in Deep ReLU Networks (3 citations)
- Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation (3 citations)
and 46 more in the interactive map.
Attention and transformers · 31 papers
- Scaling Vision Transformers to 22 Billion Parameters (115 citations)
- Hyena Hierarchy: Towards Larger Convolutional Language Models (74 citations)
- Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles (61 citations)
- GNOT: A General Neural Operator Transformer for Operator Learning (35 citations)
- Fast Inference from Transformers via Speculative Decoding (32 citations)
- Architecture-Agnostic Masked Image Modeling -- From ViT back to CNN (15 citations)
- A Closer Look at Self-Supervised Lightweight Vision Transformers (9 citations)
- Stabilizing Transformer Training by Preventing Attention Entropy Collapse (8 citations)
- LongCoder: A Long-Range Pre-trained Language Model for Code Completion (7 citations)
- Efficient Sequence Transduction by Jointly Predicting Tokens and Durations (6 citations)
- Simple Hardware-Efficient Long Convolutions for Sequence Modeling (6 citations)
- Adaptive Computation with Elastic Input Sequence (5 citations)
and 19 more in the interactive map.
Pruning and efficient tuning · 19 papers
- SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot (68 citations)
- Less is More: Task-aware Layer-wise Distillation for Language Model Compression (14 citations)
- LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation (7 citations)
- A Kernel-Based View of Language Model Fine-Tuning (6 citations)
- Why Random Pruning Is All We Need to Start Sparse (4 citations)
- Fast as CHITA: Neural Network Pruning with Combinatorial Optimization (3 citations)
- Mechanistic Mode Connectivity (2 citations)
- Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models (1 citation)
- STEP: Learning N:M Structured Sparsity Masks from Scratch with Precondition (1 citation)
- A Three-regime Model of Network Pruning
- Gradient-Free Structured Pruning with Unlabeled Data
- UPSCALE: Unconstrained Channel Pruning
and 7 more in the interactive map.
CNN design · 16 papers
- Defects of Convolutional Decoder Networks in Frequency Representation (8 citations)
- Scaling Spherical CNNs (6 citations)
- A Hybrid Quantum-Classical Approach based on the Hadamard Transform for the Convolutional Layer (3 citations)
- Existence, Stability and Scalability of Orthogonal Convolutional Neural Networks (3 citations)
- What Can Be Learnt With Wide Convolutional Neural Networks? (3 citations)
- Are Large Kernels Better Teachers than Transformers for ConvNets? (1 citation)
- Direct Parameterization of Lipschitz-Bounded Deep Networks (1 citation)
- Sequence Modeling with Multiresolution Convolutional Memory (1 citation)
- The Hessian perspective into the Nature of Convolutional Neural Networks (1 citation)
- What do CNNs Learn in the First Layer and Why? A Linear Systems Perspective (1 citation)
- Brauer's Group Equivariant Neural Networks
- Data Representations' Study of Latent Image Manifolds
and 4 more in the interactive map.
Quantization · 9 papers
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (97 citations)
- The case for 4-bit precision: k-bit Inference Scaling Laws (23 citations)
- Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks (9 citations)
- Oscillation-free Quantization for Low-bit Vision Transformers (6 citations)
- Understanding Int4 Quantization for Language Models: Latency Speedup, Composability, and Failure Cases (6 citations)
- Few-bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction (5 citations)
- FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantization (5 citations)
- DIVISION: Memory Efficient Training via Dual Activation Precision (1 citation)
- Unit Scaling: Out-of-the-Box Low-Precision Training
Neural architecture search · 6 papers
- PreNAS: Preferred One-Shot Learning Towards Efficient Neural Architecture Search (3 citations)
- Shortest Edit Path Crossover: A Theory-driven Solution to the Permutation Problem in Evolutionary Neural Architecture Search (1 citation)
- Rethink DARTS Search Space and Renovate a New Benchmark
- Symmetry-Aware Robot Design with Structured Subgroups
- Improving Bi-level Optimization Based Methods with Inspiration from Humans' Classroom Study Techniques
- QuantumDARTS: Differentiable Quantum Architecture Search for Variational Quantum Algorithms