ICLR 2024 accepted papers on Architectures and efficiency
221 of the 2,259 papers accepted at ICLR 2024 (9.8%) are on Architectures and efficiency, compared with 10.6% at ICLR 2023. They fall into 8 topics.
See ICLR 2024 in the interactive map
Working on something in this area? Describe your idea in scime atlas to see the closest accepted papers by meaning and how crowded the spot has become.
Topics and papers
RNNs and normalization · 68 papers
- EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations (84 citations)
- Sparse Autoencoders Find Highly Interpretable Features in Language Models (53 citations)
- Traveling Waves Encode The Recent Past and Enhance Sequence Learning (10 citations)
- A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks (8 citations)
- KW-Design: Pushing the Limit of Protein Design via Knowledge Refinement (8 citations)
- Teaching Arithmetic to Small Transformers (7 citations)
- ZipIt! Merging Models from Different Tasks without Training (6 citations)
- MgNO: Efficient Parameterization of Linear Operators via Multigrid (5 citations)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (4 citations)
- Epitopological learning and Cannistraci-Hebb network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning (4 citations)
- Small-scale proxies for large-scale Transformer training instabilities (4 citations)
- The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks. (3 citations)
and 56 more in the interactive map.
Attention and transformers · 66 papers
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (143 citations)
- Vision Transformers Need Registers (54 citations)
- Zipformer: A faster and better encoder for automatic speech recognition (28 citations)
- RingAttention with Blockwise Transformers for Near-Infinite Context (14 citations)
- Simplifying Transformer Blocks (10 citations)
- What Algorithms can Transformers Learn? A Study in Length Generalization (8 citations)
- Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps (7 citations)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (7 citations)
- Xformer: Hybrid X-Shaped Transformer for Image Denoising (7 citations)
- Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in Transformers (6 citations)
- Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words (6 citations)
- SparseFormer: Sparse Visual Recognition via Limited Latent Tokens (6 citations)
and 54 more in the interactive map.
Pruning and efficient tuning · 42 papers
- ReLoRA: High-Rank Training Through Low-Rank Updates (11 citations)
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns (8 citations)
- VeRA: Vector-based Random Matrix Adaptation (8 citations)
- InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning (7 citations)
- Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning (7 citations)
- How to Fine-Tune Vision Models with SGD (5 citations)
- The Expressive Power of Low-Rank Adaptation (5 citations)
- Dynamic Sparse Training with Structured Sparsity (4 citations)
- When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations (4 citations)
- Adaptive Sharpness-Aware Pruning for Robust Sparse Networks (3 citations)
- BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation (3 citations)
- FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis (3 citations)
and 30 more in the interactive map.
Spiking networks · 13 papers
- Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips (19 citations)
- Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings (18 citations)
- EventRPG: Event Data Augmentation with Relevance Propagation Guidance (2 citations)
- Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN (2 citations)
- A Progressive Training Framework for Spiking Neural Networks with Learnable Multi-hierarchical Model
- Adaptive deep spiking neural network with global-local learning via balanced excitatory and inhibitory mechanism
- Can we get the best of both Binary Neural Networks and Spiking Neural Networks for Efficient Computer Vision?
- Dynamic Neural Response Tuning
- Online Stabilization of Spiking Neural Networks
- Spatio-Temporal Approximation: A Training-Free SNN Conversion for Transformers
- SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition
- TAB: Temporal Accumulated Batch Normalization in Spiking Neural Networks
and 1 more in the interactive map.
Quantization · 13 papers
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (26 citations)
- LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models (19 citations)
- ZeRO++: Extremely Efficient Collective Communication for Large Model Training (15 citations)
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (14 citations)
- Finite Scalar Quantization: VQ-VAE Made Simple (13 citations)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (11 citations)
- EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models (4 citations)
- LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning (2 citations)
- AffineQuant: Affine Transformation Quantization for Large Language Models (1 citation)
- Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models (1 citation)
- Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappings
- Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators
and 1 more in the interactive map.
Neural architecture search · 8 papers
- Curriculum reinforcement learning for quantum architecture search under hardware errors (7 citations)
- SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS (3 citations)
- A Benchmark Study on Calibration (1 citation)
- Aux-NAS: Exploiting Auxiliary Labels with Negligibly Extra Inference Cost
- Neural Fine-Tuning Search for Few-Shot Learning
- Robustifying and Boosting Training-Free Neural Architecture Search
- ArchLock: Locking DNN Transferability at the Architecture Level with a Zero-Cost Binary Predictor
- Masked Distillation Advances Self-Supervised Transformer Architecture Search
CNN design · 7 papers
- Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Products (4 citations)
- Fully Hyperbolic Convolutional Neural Networks for Computer Vision (2 citations)
- A Characterization Theorem for Equivariant Networks with Point-wise Activations
- Convolutional Deep Kernel Machines
- Lie Group Decompositions for Equivariant Neural Networks
- Unveiling the Unseen: Identifiable Clusters in Trained Depthwise Convolutional Kernels
- Matrix Manifold Neural Networks++
Mixture of experts · 4 papers
- From Sparse to Soft Mixtures of Experts (25 citations)
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (11 citations)
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (3 citations)
- Sparse MoE with Language Guided Routing for Multilingual Machine Translation