ICLR 2025 accepted papers on Architectures and efficiency

Attention and transformers · 140 papers

• Vision-LSTM: xLSTM as Generic Vision Backbone (20 citations)
• Kolmogorov-Arnold Transformer (16 citations)
• Differential Transformer (13 citations)
• An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (10 citations)
• Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures (10 citations)
• Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences (8 citations)
• Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling (7 citations)
• Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (7 citations)
• PolaFormer: Polarity-aware Linear Attention for Vision Transformers (6 citations)
• TabM: Advancing tabular deep learning with parameter-efficient ensembling (5 citations)
• CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding (3 citations)
• Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (3 citations)

and 128 more in the interactive map.

Pruning and efficient tuning · 83 papers

• The Unreasonable Ineffectiveness of the Deeper Layers (5 citations)
• Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models (2 citations)
• A Quantum Circuit-Based Compression Perspective for Parameter-Efficient Learning (1 citation)
• Adam-mini: Use Fewer Learning Rates To Gain More (1 citation)
• DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation (1 citation)
• Efficient Learning with Sine-Activated Low-Rank Matrices (1 citation)
• GeoLoRA: Geometric integration for parameter efficient fine-tuning (1 citation)
• LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging (1 citation)
• Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning (1 citation)
• MCNC: Manifold-Constrained Reparameterization for Neural Compression (1 citation)
• RandLoRA: Full rank parameter-efficient fine-tuning of large models (1 citation)
• Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts (1 citation)

and 71 more in the interactive map.

RNNs and normalization · 80 papers

• KAN: Kolmogorov–Arnold Networks (535 citations)
• Scaling and evaluating sparse autoencoders (11 citations)
• Towards Fast, Specialized Machine Learning Force Fields: Distilling Foundation Models via Energy Hessians (9 citations)
• On the expressiveness and spectral bias of KANs (4 citations)
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (4 citations)
• Strong Model Collapse (4 citations)
• Expressivity of Neural Networks with Random Weights and Learned Biases (3 citations)
• Language models scale reliably with over-training and on downstream tasks (3 citations)
• Not All Language Model Features Are One-Dimensionally Linear (3 citations)
• Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control (3 citations)
• Cut Your Losses in Large-Vocabulary Language Models (2 citations)
• Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition (2 citations)

and 68 more in the interactive map.

Quantization · 30 papers

• SpinQuant: LLM Quantization with Learned Rotations (6 citations)
• Training-Free Activation Sparsity in Large Language Models (3 citations)
• Effective Interplay between Sparsity and Quantization: From Theory to Practice (2 citations)
• Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression (1 citation)
• CBQ: Cross-Block Quantization for Large Language Models (1 citation)
• Progressive Mixed-Precision Decoding for Efficient LLM Inference (1 citation)
• Scaling FP8 training to trillion-token LLMs (1 citation)
• STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs (1 citation)
• SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression (1 citation)
• Approaching Rate-Distortion Limits in Neural Compression with Lattice Transform Coding
• ARB-LLM: Alternating Refined Binarizations for Large Language Models
• BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments

and 18 more in the interactive map.

Mixture of experts · 24 papers

• OLMoE: Open Mixture-of-Experts Language Models (7 citations)
• Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models (4 citations)
• Layerwise Recurrent Router for Mixture-of-Experts (2 citations)
• ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing (2 citations)
• Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free (2 citations)
• Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts (1 citation)
• Mixture of In-Context Prompters for Tabular PFNs (1 citation)
• Mixture of Parrots: Experts improve memorization more than reasoning (1 citation)
• Ada-K Routing: Boosting the Efficiency of MoE-based LLMs
• Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
• Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
• Mixture Compressor for Mixture-of-Experts LLMs Gains More

and 12 more in the interactive map.

CNN design · 10 papers

• Lie Algebra Canonicalization: Equivariant Neural Operators under arbitrary Lie Groups (1 citation)
• Group Downsampling with Equivariant Anti-aliasing
• Learning Color Equivariant Representations
• Revisiting Convolution Architecture in the Realm of DNA Foundation Models
• REVISITING MULTI-PERMUTATION EQUIVARIANCE THROUGH THE LENS OF IRREDUCIBLE REPRESENTATIONS
• Towards a Unified and Verified Understanding of Group-Operation Networks
• Affine Steerable Equivariant Layer for Canonicalization of Neural Networks
• Brain-inspired $L_p$-Convolution benefits large kernels and aligns better with visual cortex
• DeeperForward: Enhanced Forward-Forward Training for Deeper and Better Performance
• On the Fourier analysis in the SO(3) space : the EquiLoPO Network

Spiking networks · 10 papers

• P-SPIKESSM: HARNESSING PROBABILISTIC SPIKING STATE SPACE MODELS FOR LONG-RANGE DEPENDENCY TASKS (5 citations)
• QP-SNN: Quantized and Pruned Spiking Neural Networks (2 citations)
• Quantized Spike-driven Transformer (1 citation)
• Rethinking Spiking Neural Networks from an Ensemble Learning Perspective (1 citation)
• Spiking Vision Transformer with Saccadic Attention (1 citation)
• Improving the Sparse Structure Learning of Spiking Neural Networks from the View of Compression Efficiency
• DeepTAGE: Deep Temporal-Aligned Gradient Enhancement for Optimizing Spiking Neural Networks
• Improving Neural Network Accuracy by Concurrently Training with a Twin Network
• NExUME: Adaptive Training and Inference for DNNs under Intermittent Power Environments
• Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness

Neural architecture search · 4 papers

• Multi-objective Differentiable Neural Architecture Search (2 citations)
• NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance
• STAR: Synthesis of Tailored Architectures
• W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Models