NeurIPS 2024 accepted papers on Architectures and efficiency

Attention and transformers · 130 papers

• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (71 citations)
• MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection (69 citations)
• Demystify Mamba in Vision: A Linear Attention Perspective (58 citations)
• Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling (43 citations)
• ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention (37 citations)
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (29 citations)
• Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (28 citations)
• Bridging the Divide: Reconsidering Softmax and Linear Attention (14 citations)
• FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model (13 citations)
• DiTFastAttn: Attention Compression for Diffusion Transformer Models (11 citations)
• Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers (11 citations)
• Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (11 citations)

and 118 more in the interactive map.

RNNs and normalization · 127 papers

• The GAN is dead; long live the GAN! A Modern GAN Baseline (31 citations)
• Learning to Handle Complex Constraints for Vehicle Routing Problems (20 citations)
• Training Compute-Optimal Protein Language Models (19 citations)
• Recurrent neural networks: vanishing and exploding gradients are not the end of the story (18 citations)
• OPUS: Occupancy Prediction Using a Sparse Set (10 citations)
• Amortized Planning with Large-Scale Transformers: A Case Study on Chess (8 citations)
• Multiple Physics Pretraining for Spatiotemporal Surrogate Models (8 citations)
• Transcoders find interpretable LLM feature circuits (8 citations)
• Swift Sampler: Efficient Learning of Sampler by 10 Parameters (7 citations)
• The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domains (7 citations)
• Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning (6 citations)
• Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment (6 citations)

and 115 more in the interactive map.

Pruning and efficient tuning · 50 papers

• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (72 citations)
• Better by default: Strong pre-tuned MLPs and boosted trees on tabular data (41 citations)
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation (20 citations)
• SlimGPT: Layer-wise Structured Pruning for Large Language Models (12 citations)
• Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (11 citations)
• SlimSAM: 0.1% Data Makes Segment Anything Slim (11 citations)
• Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation (6 citations)
• DEPrune: Depth-wise Separable Convolution Pruning for Maximizing GPU Parallelism (4 citations)
• FM-Delta: Lossless Compression for Storing Massive Fine-tuned Foundation Models (4 citations)
• SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors (4 citations)
• TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks (4 citations)
• Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (3 citations)

and 38 more in the interactive map.

Quantization · 22 papers

• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (51 citations)
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (47 citations)
• DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs (16 citations)
• MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (14 citations)
• ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification (10 citations)
• OneBit: Towards Extremely Low-bit Large Language Models (8 citations)
• KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (7 citations)
• PTQ4DiT: Post-training Quantization for Diffusion Transformers (7 citations)
• ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (6 citations)
• SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training (5 citations)
• QTIP: Quantization with Trellises and Incoherence Processing (4 citations)
• Compressing Large Language Models using Low Rank and Low Precision Decomposition (3 citations)

and 10 more in the interactive map.

CNN design · 19 papers

• MambaLLIE: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Space (27 citations)
• Convolutional Differentiable Logic Gate Networks (8 citations)
• E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation (8 citations)
• Interpretable Image Classification with Adaptive Prototype-based Vision Transformers (6 citations)
• Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers (3 citations)
• TinyLUT: Tiny Look-Up Table for Efficient Image Restoration at the Edge (3 citations)
• B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable (2 citations)
• DEX: Data Channel Extension for Efficient CNN Inference on Tiny AI Accelerators (2 citations)
• MonkeySee: Space-time-resolved reconstructions of natural images from macaque multi-unit activity (2 citations)
• AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation (1 citation)
• Dendritic Integration Inspired Artificial Neural Networks Capture Data Correlation (1 citation)
• Model LEGO: Creating Models Like Disassembling and Assembling Building Blocks (1 citation)

and 7 more in the interactive map.

Mixture of experts · 17 papers

• Toward Efficient Inference for Mixture of Experts (15 citations)
• SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (8 citations)
• Multi-Head Mixture-of-Experts (6 citations)
• Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (6 citations)
• MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Tasks (3 citations)
• How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider and MoE Transformers (2 citations)
• LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing (2 citations)
• Mixture of Nested Experts: Adaptive Processing of Visual Tokens (2 citations)
• Mixture of Tokens: Continuous MoE through Cross-Example Aggregation (2 citations)
• MoEUT: Mixture-of-Experts Universal Transformers (2 citations)
• Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learner (2 citations)
• Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (2 citations)

and 5 more in the interactive map.

Spiking networks · 14 papers

• QKFormer: Hierarchical Spiking Transformer using Q-K Attention (29 citations)
• SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation (9 citations)
• FEEL-SNN: Robust Spiking Neural Networks with Frequency Encoding and Evolutionary Leak Factor (6 citations)
• Autonomous Driving with Spiking Neural Networks (5 citations)
• Rethinking the Membrane Dynamics and Optimization Objectives of Spiking Neural Networks (5 citations)
• Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagation (3 citations)
• Spiking Neural Network as Adaptive Event Stream Slicer (3 citations)
• LM-HT SNN: Enhancing the Performance of SNN to ANN Counterpart through Learnable Multi-hierarchical Threshold Model (2 citations)
• Spike-based Neuromorphic Model for Sound Source Localization (2 citations)
• Spiking Token Mixer: An event-driven friendly Former structure for spiking neural networks (2 citations)
• Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks (2 citations)
• Long-Range Feedback Spiking Network Captures Dynamic and Static Representations of the Visual Cortex under Movie Stimuli (1 citation)

and 2 more in the interactive map.

Neural architecture search · 9 papers

• Customized Subgraph Selection and Encoding for Drug-drug Interaction Prediction (9 citations)
• Search for Efficient Large Language Models (5 citations)
• Automated Multi-Task Learning for Joint Disease Prediction on Electronic Health Records (3 citations)
• CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework (2 citations)
• HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models (1 citation)
• Towards Next-Generation Logic Synthesis: A Scalable Neural Circuit Generation Framework (1 citation)
• Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights (1 citation)
• einspace: Searching for Neural Architectures from Fundamental Operations
• MOTE-NAS: Multi-Objective Training-based Estimate for Efficient Neural Architecture Search