ICLR 2026 accepted papers on Architectures and efficiency

Attention and transformers · 163 papers

• Mamba-3: Improved Sequence Modeling using State Space Principles (1 citation)
• A Theoretical Analysis of Mamba’s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models
• An evolutionary perspective on modes of learning in Transformers
• ASMIL: Attention-Stabilized Multiple Instance Learning for Whole-Slide Imaging
• Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
• AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
• Avey-B
• Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
• BioTamperNet: Affinity-Guided State-Space Model Detecting Tampered Biomedical Images
• Block Recurrent Dynamics in Vision Transformers
• Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors
• ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

and 151 more in the interactive map.

RNNs and normalization · 97 papers

• Bidirectional Predictive Coding (5 citations)
• DistMLIP: A Distributed Inference Platform for Machine Learning Interatomic Potentials (4 citations)
• Sparse Autoencoders Trained on the Same Data Learn Different Features (1 citation)
• Tversky Neural Networks: Psychologically Plausible Deep Learning with Differentiable Tversky Similarity (1 citation)
• WAFT: Warping-Alone Field Transforms for Optical Flow (1 citation)
• A Biologically Plausible Dense Associative Memory with Exponential Capacity
• Accelerating Inference for Multilayer Neural Networks with Quantum Computers
• Adaptive Hopfield Network: Rethinking Similarities in Associative Memory
• Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers
• Characterizing the Discrete Geometry of ReLU Networks
• Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
• Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement

and 85 more in the interactive map.

Pruning and efficient tuning · 74 papers

• ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models
• AdaRank: Adaptive Rank Pruning for Enhanced Model Merging
• AIRE-Prune: Asymptotic Impulse-Response Energy for State Pruning in State Space Models
• Are EEG Foundation Models Worth It? Comparative Evaluation with Traditional Decoders in Diverse BCI Tasks
• ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
• Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency
• BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity
• Cautious Weight Decay
• Composable Sparse Subnetworks via Maximum-Entropy Principle
• CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure
• CSRv2: Unlocking Ultra-Sparse Embeddings
• Cut Less, Fold More: Model Compression through the Lens of Projection Geometry

and 62 more in the interactive map.

Quantization · 56 papers

• KV-Cache Transform Coding for Compact Storage in LLM Inference (1 citation)
• Robust Training of Neural Networks at Arbitrary Precision and Sparsity (1 citation)
• AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs
• BEP: A Binary Error Propagation Algorithm for Binary Neural Networks Training
• Beyond Outliers: A Study of Optimizers Under Quantization
• CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts
• Compute-Optimal Quantization-Aware Training
• Dataset Color Quantization: A Training-Oriented Framework for Dataset-Level Compression
• GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs
• Highly Efficient and Effective LLMs with Multi-Boolean Architectures
• Improving Block-Wise LLM Quantization by 4-bit Block-Wise Optimal Float (BOF4): Analysis and Variations
• Is Finer Better? The Limits of Microscaling Formats in Large Language Models

and 44 more in the interactive map.

Mixture of experts · 37 papers

• Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource (2 citations)
• Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
• Deft Scheduling of Dynamic Cloud Workflows with Varying Deadlines via Mixture-of-Experts
• DirMoE: Dirichlet-Routed Mixture of Experts
• Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
• Expert Divergence Learning for MoE-based Language Models
• Expert Merging in Sparse Mixture of Experts with Nash Bargaining
• LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts
• LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
• MoEEdit: Efficient and Routing-Stable Knowledge Editing for Mixture-of-Experts LLMs
• MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
• Multilingual Routing in Mixture-of-Experts

and 25 more in the interactive map.

Spiking networks · 19 papers

• Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks (1 citation)
• A Brain-Inspired Gating Mechanism Unlocks Robust Computation in Spiking Neural Networks
• Advancing Spatiotemporal Representations in Spiking Neural Networks via Parametric Invertible Transformation
• Breaking Gradient Temporal Collinearity for Robust Spiking Neural Networks
• Cannistraci-Hebb Training on Ultra-Sparse Spiking Neural Networks
• Fractional-Order Spiking Neural Network
• Neural Dynamics Self-Attention for Spiking Transformers
• Random Spiking Neural Networks are Stable and Spectrally Simple
• Robustify Spiking Neural Networks via Dominant Singular Deflation under Heterogeneous Training Vulnerability
• TP-Spikformer: Token Pruned Spiking Transformer
• Training Deep Normalization-Free Spiking Neural Networks with Lateral Inhibition
• Beyond Linear Processing: Dendritic Bilinear Integration in Spiking Neural Networks

and 7 more in the interactive map.

CNN design · 12 papers

• A Scalable Distributed Framework for Multimodal GigaVoxel Image Registration
• A Single Architecture for Representing Invariance Under Any Space Group
• Any-Subgroup Equivariant Networks via Symmetry Breaking
• Designing Affine-Invariant Neural Networks for Photometric Corruption Robustness and Generalization
• FS-KAN: Permutation Equivariant Kolmogorov-Arnold Networks via Function Sharing
• GmNet: Revisiting Gating Mechanisms From A Frequency View
• Lookup multivariate Kolmogorov-Arnold Networks
• Reducing Symmetry Increase in Equivariant Neural Networks
• Scaling Laws and Symmetry, Evidence from Neural Force Fields
• SONIC: Spectral Oriented Neural Invariant Convolutions
• Batch Pruning by Activation Stability
• Johnson-Lindenstrauss Lemma Guided Network for Efficient 3D Medical Segmentation

Neural architecture search · 6 papers

• Adaptive Width Neural Networks
• Composer: A Search Framework for Hybrid Neural Architecture Design
• Constraint-guided Hardware-aware NAS through Gradient Modification
• DIVERSE: Disagreement-Inducing Vector Evolution for Rashomon Set Exploration
• Expressive yet Efficient Feature Expansion with Adaptive Cross-Hadamard Products
• Relatron: Automating Relational Machine Learning over Relational Databases