ICML 2025 accepted papers on Architectures and efficiency

RNNs and normalization · 97 papers

• ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy (3 citations)
• Layer by Layer: Uncovering Hidden Representations in Language Models (2 citations)
• Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream (2 citations)
• Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations (1 citation)
• DriveGPT: Scaling Autoregressive Behavior Models for Driving (1 citation)
• Efficiently Serving Large Multimodal Models Using EPD Disaggregation (1 citation)
• How to set AdamW's weight decay as you scale model and dataset size (1 citation)
• Incorporating Arbitrary Matrix Group Equivariance into KANs (1 citation)
• Memory Layers at Scale (1 citation)
• Nonparametric Modern Hopfield Models (1 citation)
• Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment (1 citation)
• A Hitchhiker's Guide to Scaling Law Estimation

and 85 more in the interactive map.

Attention and transformers · 95 papers

• Learning to (Learn at Test Time): RNNs with Expressive Hidden States (15 citations)
• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data (9 citations)
• A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks (2 citations)
• LASER: Attention with Exponential Transformation (2 citations)
• Constrained Belief Updates Explain Geometric Structures in Transformer Representations (1 citation)
• Core Context Aware Transformers for Long Context Language Modeling (1 citation)
• DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers (1 citation)
• Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization (1 citation)
• HashAttention: Semantic Sparsity for Faster Inference (1 citation)
• M+: Extending MemoryLLM with Scalable Long-Term Memory (1 citation)
• Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (1 citation)
• On the Emergence of Position Bias in Transformers (1 citation)

and 83 more in the interactive map.

Pruning and efficient tuning · 59 papers

• Dynamic Sparse Training of Diagonally Sparse Networks (1 citation)
• Low-Rank Adapting Models for Sparse Autoencoders (1 citation)
• A Bayesian Model Selection Criterion for Selecting Pretraining Checkpoints
• A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizations
• Are Large Brainwave Foundation Models Capable Yet ? Insights from Fine-Tuning
• BackSlash: Rate Constrained Optimized Training of Large Language Models
• CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
• COGNATE: Acceleration of Sparse Tensor Programs on Emerging Hardware using Transfer Learning
• Compressing tree ensembles through Level-wise Optimization and Pruning
• Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
• Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective
• Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias

and 47 more in the interactive map.

Quantization · 34 papers

• FlatQuant: Flatness Matters for LLM Quantization (2 citations)
• Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries (1 citation)
• MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design (1 citation)
• Optimizing Large Language Model Training Using FP4 Quantization (1 citation)
• QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (1 citation)
• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (1 citation)
• BiMaCoSR: Binary One-Step Diffusion Model Leveraging Flexible Matrix Compression for Real Super-Resolution
• BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference
• BoA: Attention-aware Post-training Quantization without Backpropagation
• Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
• GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
• GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration

and 22 more in the interactive map.

Mixture of experts · 20 papers

• MoH: Multi-Head Attention as Mixture-of-Head Attention (3 citations)
• Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts (2 citations)
• A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
• Autonomy-of-Experts Models
• Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts
• FloE: On-the-Fly MoE Inference on Memory-constrained GPU
• Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
• Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer
• Mixture of Experts Made Intrinsically Interpretable
• Mixture of Lookup Experts
• MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition
• MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance

and 8 more in the interactive map.

Spiking networks · 13 papers

• Efficient ANN-SNN Conversion with Error Compensation Learning
• Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
• Efficient Parallel Training Methods for Spiking Neural Networks with Constant Time Complexity
• Faster and Stronger: When ANN-SNN Conversion Meets Parallel Spiking Calculation
• ReverB-SNN: Reversing Bit of the Weight and Activation for Spiking Neural Networks
• Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model
• Temporal Misalignment in ANN-SNN Conversion and its Mitigation via Probabilistic Spiking Neurons
• Differential Coding for Training-Free ANN-to-SNN Conversion
• SpikF: Spiking Fourier Network for Efficient Long-term Prediction
• Time to Spike? Understanding the Representational Power of Spiking Neural Networks in Discrete Time
• Training High Performance Spiking Neural Network by Temporal Model Calibration
• TS-SNN: Temporal Shift Module for Spiking Neural Networks

and 1 more in the interactive map.

CNN design · 5 papers

• An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures
• Equivariant Neural Tangent Kernels
• Flopping for FLOPs: Leveraging Equivariance for Computational Efficiency
• GLGENN: A Novel Parameter-Light Equivariant Neural Networks Architecture Based on Clifford Geometric Algebras
• One-dimensional Path Convolution

Neural architecture search · 2 papers

• Prior Knowledge Guided Neural Architecture Generation
• Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective