NeurIPS 2026 accepted papers on Architectures and efficiency
551 of the 6,080 papers accepted at NeurIPS 2026 (9.1%) are on Architectures and efficiency, compared with 8.8% at NeurIPS 2025. They fall into 8 topics.
See NeurIPS 2026 in the interactive map
Working on something in this area? Describe your idea in scime atlas to see the closest accepted papers by meaning and how crowded the spot has become.
Topics and papers
Attention and transformers · 190 papers
- Ordinary Least Squares as an Attention Mechanism (2 citations)
- A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
- AgentKVShift: Efficient KV Cache Reuse for Agentic Memory Systems
- Assign and Add: A Mechanistic Study of Compositional Arithmetic
- AtomWorld-Mem: Memory-Restored World States for Long-Horizon Atomistic Evolution
- Attention Transfer Is Not Universally Effective for Vision Transformers
- Attention-based PCA
- Benchmarking Attention for Tabular Foundation Models
- Block Sparse Flash Attention
- CanViT: Toward Active-Vision Foundation Models
- Capturing In-Context Learning Dynamics with Task Operators
- Certification from Examples is Hard for Circuits and Transformers under Minimal Overparametrization
and 178 more in the interactive map.
RNNs and normalization · 161 papers
- A Scalable Multi-Task Model for Virtual Sensors
- ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism
- Augmented Lagrangian Predictive Coding
- Bridging Compute- and Data-Optimal Pretraining
- Communication-Efficient LLM Adaptation over Decentralized GPU Meshes
- Concept-Based Mechanistic Interpretability Needs a Concrete Evaluation Paradigm
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
- Deep Minds and Shallow Probes
- Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation
- Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures
- Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology
and 149 more in the interactive map.
Pruning and efficient tuning · 70 papers
- AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning
- AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation
- AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
- Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning
- CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation
- Crowded in B-Space: Calibrating Shared Directions for LoRA Merging
- Dynamic Model Merging Made Slim
- LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection
- Machine Unlearning in Low-Dimensional Feature Subspace
- MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search
- Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models
and 58 more in the interactive map.
Quantization · 58 papers
- ActQuant: Sub-4-bit Action-Guided Quantization for Vision-Language-Action Models
- Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation
- Amortized-Precision Quantization for Early-Exit Vision Transformers
- Block Sphere Vector Quantization
- ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs
- DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers
- GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
- HESTIA: A Hessian-Guided Differentiable Quantization-Aware Training Framework for Extremely Low-Bit LLMs
- HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models
- HyperVQ: Enabling Hyperprior Entropy Modeling for VQ-Based Generative Image Compression
- KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers
- KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation
and 46 more in the interactive map.
Mixture of experts · 32 papers
- Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE
- dMoE: dLLMs with Learnable Block Experts
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
- FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
- How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
- POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models
- Pruning and Distilling Mixture-of-Experts into Dense Language Models
- RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism
- Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
- Sparse Layers are Critical to Scaling Looped Language Models
and 20 more in the interactive map.
Spiking networks · 19 papers
- Exploring the Limitations of Layer Synchronization in Spiking Neural Networks
- FiTS: Interpretable Spiking Neurons via Frequency Selectivity and Temporal Shaping
- From Cortical Synchronous Rhythm to Brain Inspired Learning Mechanism: An Oscillatory Spiking Neural Network with Time-Delayed Coordination
- LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning
- Neuronal Self-Adaptation Enhances Capacity and Robustness of Representation in Spiking Neural Networks
- TIDE: Asymmetric Neural Circuits for Stabilized Temporal Inhibitory-Excitatory Dynamics
- All-Addition Spiking Diffusion Models with Attention Enhancement
- Beyond Low-Pass Dynamics: Frequency-Selective Spiking Reservoirs with Resonant Neurons
- Distributed-Order Fractional Spiking Neural Network
- Do Not Let Spikes Flip: Margin-Resculpted Learning for Robust Spiking Neural Networks
- Efficient Brain-to-Speech Decoding with Fixed-Delay Spiking Neural Networks
- Efficient Training of Deep Spiking Neural Networks with Input-Driven Derivative-Free Updates
and 7 more in the interactive map.
CNN design · 12 papers
- DD-CAM: Minimal Sufficient Explanations for Vision Models Using Delta Debugging
- Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision
- Selective Disk Bispectrum: A Complete and Rotation Invariant Image Descriptor
- $\texttt{bispectrum}$: Selective $G$-Bispectra Made Practical
- AuGhostmentation: The Eyes Never Stand Still—$\textit{Why Should CNNs?}$
- Foundations of Categorical Equivariant Deep Learning
- Foveated BagNet: Inherent Interpretability Does Not Exclude Global Context
- KernelDNA: Cross-Layer Kernel Sharing via Decoupled Neural Adapters
- kVNN: Learnable Volterra Network Kernels
- Layer Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant
- On Making $SE(2)$-Invariant Networks Optimal
- Rotation-Invariant Vector Normalization for Molecular Force Learning
Neural architecture search · 9 papers
- Network of Theseus (Like the ship)
- Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback
- Simple Baselines are Competitive with Code Evolution
- Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search
- AIRA-Compose: Agentic Discovery of Neural Architectures
- Cool Graphs: Active Property Search Towards Quantum Nano-Refrigerators
- EENAS: Zero-Shot Energy-Efficiency-Aware Neural Architecture Search
- Revisiting Autoregressive GCNs for Vehicle Routing Problems
- Sparsity for Free: A Budget-Induced Equilibrium in Joint Topology–Parameter Search