ICML 2024 accepted papers on Architectures and efficiency

Attention and transformers · 80 papers

• Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling (54 citations)
• Learning Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking (34 citations)
• Can Mamba Learn How To Learn? A Comparative Study on In-Context Learning Tasks (10 citations)
• AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers (9 citations)
• SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization (7 citations)
• Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption (6 citations)
• DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training (6 citations)
• Gated Linear Attention Transformers with Hardware-Efficient Training (5 citations)
• Generalist Equivariant Transformer Towards 3D Molecular Interaction Learning (5 citations)
• How Transformers Learn Causal Structure with Gradient Descent (5 citations)
• Attention Meets Post-hoc Interpretability: A Mathematical Perspective (4 citations)
• Simple linear attention language models balance the recall-throughput tradeoff (4 citations)

and 68 more in the interactive map.

RNNs and normalization · 74 papers

• Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models (55 citations)
• Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations (5 citations)
• Think Before You Act: Decision Transformers with Working Memory (5 citations)
• A Dynamical Model of Neural Scaling Laws (4 citations)
• Mechanistic Design and Scaling of Hybrid Architectures (4 citations)
• Navigating Scaling Laws: Compute Optimality in Adaptive Model Training (4 citations)
• Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks (4 citations)
• Asymptotics of Learning with Deep Structured (Random) Features (3 citations)
• Codebook Features: Sparse and Discrete Interpretability for Neural Networks (2 citations)
• DeepPolar: Inventing Nonlinear Large-Kernel Polar Codes via Deep Learning (2 citations)
• Hidden Traveling Waves bind Working Memory Variables in Recurrent Neural Networks (2 citations)
• How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Model (2 citations)

and 62 more in the interactive map.

Pruning and efficient tuning · 38 papers

• DoRA: Weight-Decomposed Low-Rank Adaptation (38 citations)
• LoRA+: Efficient Low Rank Adaptation of Large Models (18 citations)
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (11 citations)
• TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge (8 citations)
• RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation (7 citations)
• Parameter-Efficient Fine-Tuning with Discrete Fourier Transform (5 citations)
• APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (2 citations)
• Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts (2 citations)
• No Free Prune: Information-Theoretic Barriers to Pruning at Initialization (2 citations)
• OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization (2 citations)
• Trainable Transformer in Transformer (2 citations)
• Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation (1 citation)

and 26 more in the interactive map.

Quantization · 21 papers

• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (25 citations)
• SqueezeLLM: Dense-and-Sparse Quantization (23 citations)
• Extreme Compression of Large Language Models via Additive Quantization (4 citations)
• Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization (1 citation)
• LQER: Low-Rank Quantization Error Reconstruction for LLMs (1 citation)
• A2Q+: Improving Accumulator-Aware Weight Quantization
• Collage: Light-Weight Low-Precision Strategy for LLM Training
• Ditto: Quantization-aware Secure Inference of Transformers upon MPC
• ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking
• FrameQuant: Flexible Low-Bit Quantization for Transformers
• IM-Unpack: Training and Inference with Arbitrarily Low Precision Integers
• Residual Quantization with Implicit Neural Codebooks

and 9 more in the interactive map.

CNN design · 9 papers

• KernelWarehouse: Rethinking the Design of Dynamic Convolution (3 citations)
• Revealing the Dark Secrets of Extremely Large Kernel ConvNets on Robustness (2 citations)
• A Probabilistic Approach to Learning the Degree of Equivariance in Steerable CNNs (1 citation)
• Clifford-Steerable Convolutional Neural Networks
• LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging
• Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learning
• Is Kernel Prediction More Powerful than Gating in Convolutional Neural Networks?
• Receptive Fields As Experts in Convolutional Neural Architectures
• Stacking Deep Set Networks and Pooling by Quantiles

Spiking networks · 9 papers

• Towards efficient deep spiking neural networks construction with spiking activity based pruning (8 citations)
• Balanced Resonate-and-Fire Neurons (3 citations)
• High-Performance Temporal Reversible Spiking Neural Networks with $\mathcal{O}(L)$ Training Memory and $\mathcal{O}(1)$ Inference Cost (2 citations)
• SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN (2 citations)
• Autaptic Synaptic Circuit Enhances Spatio-temporal Predictive Learning of Spiking Neural Networks (1 citation)
• CLIF: Complementary Leaky Integrate-and-Fire Neuron for Spiking Neural Networks (1 citation)
• SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms (1 citation)
• NDOT: Neuronal Dynamics-based Online Training for Spiking Neural Networks
• Towards Efficient Spiking Transformer: a Token Sparsification Framework for Training and Inference Acceleration

Mixture of experts · 7 papers

• MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Experts (3 citations)
• Scaling Laws for Fine-Grained Mixture of Experts (3 citations)
• Merging Multi-Task Models via Weight-Ensembling Mixture of Experts (2 citations)
• A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
• Exploring the Benefit of Activation Sparsity in Pre-training
• Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?
• Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training

Neural architecture search · 5 papers

• Differentiable Model Scaling using Differentiable Topk (1 citation)
• Encodings for Prediction-based Neural Architecture Search (1 citation)
• Integrated Hardware Architecture and Device Placement Search
• Seesaw: Compensating for Nonlinear Reduction with Linear Computations for Private Inference
• Towards Neural Architecture Search through Hierarchical Generative Modeling