NeurIPS 2025 accepted papers on Architectures and efficiency

Attention and transformers · 183 papers

• Titans: Learning to Memorize at Test Time (18 citations)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (15 citations)
• Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention (9 citations)
• MoBA: Mixture of Block Attention for Long-Context LLMs (8 citations)
• CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding (6 citations)
• Faster Video Diffusion with Trainable Sparse Attention (6 citations)
• A Generalist Intracortical Motor Decoder (5 citations)
• Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation (5 citations)
• Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (3 citations)
• Attention (as Discrete-Time Markov) Chains (3 citations)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (3 citations)
• RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (3 citations)

and 171 more in the interactive map.

RNNs and normalization · 158 papers

• UMA: A Family of Universal Models for Atoms (72 citations)
• Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations (10 citations)
• Nested Learning: The Illusion of Deep Learning Architectures (9 citations)
• A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities (6 citations)
• BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Models (4 citations)
• Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation (3 citations)
• Unfolding the Black Box of Recurrent Neural Networks for Path Integration (3 citations)
• DrivAerStar: An Industrial-Grade CFD Dataset for Vehicle Aerodynamic Optimization (2 citations)
• DynaPipe: Dynamic Layer Redistribution for Efficient Serving of LLMs with Pipeline Parallelism (2 citations)
• FALCON: An ML Framework for Fully Automated Layout-Constrained Analog Circuit Design (2 citations)
• Generalizable, real-time neural decoding with hybrid state-space models (2 citations)
• Towards Fully FP8 GEMM LLM Training at Scale (2 citations)

and 146 more in the interactive map.

Pruning and efficient tuning · 66 papers

• LoRA vs Full Fine-tuning: An Illusion of Equivalence (10 citations)
• Revisiting End-to-End Learning with Slide-level Supervision in Computational Pathology (5 citations)
• RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness (3 citations)
• Accurate and Efficient Low-Rank Model Merging in Core Space (2 citations)
• Parameter Efficient Fine-tuning via Explained Variance Adaptation (2 citations)
• Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (2 citations)
• Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples (1 citation)
• CrossSpectra: Exploiting Cross-Layer Smoothness for Parameter-Efficient Fine-Tuning (1 citation)
• Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (1 citation)
• Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (1 citation)
• LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing (1 citation)
• Machine Unlearning via Task Simplex Arithmetic (1 citation)

and 54 more in the interactive map.

Mixture of experts · 33 papers

• FlashMoE: Fast Distributed MoE in a Single Kernel (2 citations)
• Mamba Goes HoME: Hierarchical Soft Mixture-of-Experts for 3D Medical Image Segmentation (2 citations)
• Discovering Important Experts for Mixture-of-Experts Models Pruning Through a Theoretical Perspective (1 citation)
• Enhanced Expert Merging for Mixture-of-Experts in Graph Foundation Models (1 citation)
• FlexOLMo: Open Language Models for Flexible Data Use (1 citation)
• FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts (1 citation)
• UMoE: Unifying Attention and FFN with Shared Experts (1 citation)
• Who Speaks for the Trigger? Dynamic Expert Routing in Backdoored Mixture-of-Experts Transformers (1 citation)
• Advancing Expert Specialization for Better MoE
• Brain-Like Processing Pathways Form in Models With Heterogeneous Experts
• CryptoMoE: Privacy-Preserving and Scalable Mixture of Experts Inference via Balanced Expert Routing
• Dense Backpropagation Improves Training for Sparse Mixture-of-Experts

and 21 more in the interactive map.

Quantization · 32 papers

• SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training (3 citations)
• Analog Foundation Models (2 citations)
• GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (2 citations)
• PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models (2 citations)
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (1 citation)
• Efficient and Generalizable Mixed-Precision Quantization via Topological Entropy (1 citation)
• Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling (1 citation)
• FP4 All the Way: Fully Quantized Training of Large Language Models (1 citation)
• $\text{S}^2$Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation
• Binary Quadratic Quantization: Beyond First-Order Quantization for Real-Valued Matrix Compression
• CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
• DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization

and 20 more in the interactive map.

Spiking networks · 20 papers

• Spiking Meets Attention: Efficient Remote Sensing Image Super-Resolution with Attention Spiking Neural Networks (5 citations)
• Multiplication-Free Parallelizable Spiking Neurons with Efficient Spatio-Temporal Dynamics (2 citations)
• SPACE: SPike-Aware Consistency Enhancement for Test-Time Adaptation in Spiking Neural Networks (2 citations)
• Spiking Neural Networks Need High-Frequency Information (2 citations)
• Toward Relative Positional Encoding in Spiking Transformers (2 citations)
• HetSyn: Versatile Timescale Integration in Spiking Neural Networks via Heterogeneous Synapses (1 citation)
• S$^2$M-Former: Spiking Symmetric Mixing Branchformer for Brain Auditory Attention Detection (1 citation)
• Activity Pruning for Efficient Spiking Neural Networks
• Adaptive Fission: Post-training Encoding for Low-latency Spike Neural Networks
• BioOSS: A Bio-Inspired Oscillatory State System with Spatio-Temporal Dynamics
• Bipolar Self-attention for Spiking Transformers
• Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training

and 8 more in the interactive map.

CNN design · 13 papers

• ImageNet-trained CNNs are not biased towards texture: Revisiting feature reliance through controlled suppression (3 citations)
• Lorentz Local Canonicalization: How to make any Network Lorentz-Equivariant (3 citations)
• TV-Rec: Time-Variant Convolutional Filter for Sequential Recommendation (2 citations)
• Explicitly Modeling Subcortical Vision with a Neuro-Inspired Front-End Improves CNN Robustness (1 citation)
• PLEIADES: Building Temporal Kernels with Orthogonal Polynomials (1 citation)
• Convolution Goes Higher-Order: A Biologically Inspired Mechanism Empowers Image Classification
• Learning Chern Numbers of Multiband Topological Insulators with Gauge Equivariant Neural Networks
• Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning
• msf-CNN: Patch-based Multi-Stage Fusion with Convolutional Neural Networks for TinyML
• Normalize Filters! Classical Wisdom for Deep Vision
• On the VC dimension of deep group convolutional neural networks
• Structured Initialization for Vision Transformers

and 1 more in the interactive map.

Neural architecture search · 7 papers

• HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models (1 citation)
• LoMix: Learnable Weighted Multi-Scale Logits Mixing for Medical Image Segmentation (1 citation)
• Per-Architecture Training-Free Metric Optimization for Neural Architecture Search (1 citation)
• High-Performance Arithmetic Circuit Optimization via Differentiable Architecture Search
• Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search
• TensorRL-QAS: Reinforcement learning with tensor networks for improved quantum architecture search
• TF-MAS: Training-free Mamba2 Architecture Search