NeurIPS 2023 accepted papers on Architectures and efficiency

RNNs and normalization · 125 papers

• Deep Patch Visual Odometry (39 citations)
• Scaling Data-Constrained Language Models (32 citations)
• xTrimoGene: An Efficient and Scalable Representation Learner for Single-Cell RNA-Seq Data (30 citations)
• Separable Physics-Informed Neural Networks (27 citations)
• Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models (24 citations)
• Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time (23 citations)
• A Unified, Scalable Framework for Neural Population Decoding (22 citations)
• TensorNet: Cartesian Tensor Representations for Efficient Learning of Molecular Potentials (18 citations)
• ZoomTrack: Target-aware Non-uniform Resizing for Efficient Visual Tracking (17 citations)
• Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors (15 citations)
• Balancing memorization and generalization in RNNs for high performance brain-machine Interfaces (11 citations)
• Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective (11 citations)

and 113 more in the interactive map.

Attention and transformers · 65 papers

• HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution (157 citations)
• VanillaNet: the Power of Minimalism in Deep Learning (85 citations)
• BIOT: Biosignal Transformer for Cross-data Learning in the Wild (45 citations)
• MixFormerV2: Efficient Fully Transformer Tracking (32 citations)
• The Impact of Positional Encoding on Length Generalization in Transformers (32 citations)
• Hierarchically Gated Recurrent Neural Network for Sequence Modeling (25 citations)
• Unlimiformer: Long-Range Transformers with Unlimited Length Input (23 citations)
• White-Box Transformers via Sparse Rate Reduction (22 citations)
• Scalable Transformer for PDE Surrogate Modeling (20 citations)
• ATMAN: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation (16 citations)
• Learning to Compress Prompts with Gist Tokens (16 citations)
• The geometry of hidden representations of large transformer models (16 citations)

and 53 more in the interactive map.

Pruning and efficient tuning · 34 papers

• Symbolic Discovery of Optimization Algorithms (167 citations)
• Fine-Tuning Language Models with Just Forward Passes (36 citations)
• Model Sparsity Can Simplify Machine Unlearning (32 citations)
• Pruning vs Quantization: Which is Better? (23 citations)
• Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior (14 citations)
• Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models (11 citations)
• Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers (10 citations)
• Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing (8 citations)
• ZipLM: Inference-Aware Structured Pruning of Language Models (7 citations)
• The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter (6 citations)
• Universality and Limitations of Prompt Tuning (6 citations)
• Composing Parameter-Efficient Modules with Arithmetic Operation (5 citations)

and 22 more in the interactive map.

CNN design · 16 papers

• Convolutional Neural Operators for robust and accurate learning of PDEs (37 citations)
• Theoretical Analysis of the Inductive Biases in Deep Convolutional Networks (9 citations)
• Explaining V1 Properties with a Biologically Constrained Deep Learning Architecture (6 citations)
• Emergence of Shape Bias in Convolutional Neural Networks through Activation Sparsity (4 citations)
• Robust Learning with Progressive Data Expansion Against Spurious Correlation (4 citations)
• A General Framework for Equivariant Neural Networks on Reductive Lie Groups (3 citations)
• Normalization-Equivariant Neural Networks with Application to Image Denoising (3 citations)
• Inner Product-based Neural Network Similarity (2 citations)
• Investigating how ReLU-networks encode symmetries (2 citations)
• A General Framework for Robust G-Invariance in G-Equivariant Networks (1 citation)
• Causes and Effects of Unanticipated Numerical Deviations in Neural Network Inference Frameworks (1 citation)
• Implicit Convolutional Kernels for Steerable CNNs (1 citation)

and 4 more in the interactive map.

Quantization · 15 papers

• QLoRA: Efficient Finetuning of Quantized LLMs (824 citations)
• Training Transformers with 4-bit Integers (8 citations)
• Matrix Compression via Randomized Low Rank and Low Precision Factorization (5 citations)
• PackQViT: Faster Sub-8-bit Vision Transformers via Full and Packed Quantization on the Mobile (5 citations)
• Unleashing the Full Potential of Product Quantization for Large-Scale Image Retrieval (5 citations)
• Temporal Dynamic Quantization for Diffusion Models (4 citations)
• TexQ: Zero-shot Network Quantization with Texture Feature Distribution Calibration (4 citations)
• Memory Efficient Optimizers with 4-bit States (3 citations)
• QuantSR: Accurate Low-bit Quantization for Efficient Image Super-Resolution (3 citations)
• REx: Data-Free Residual Quantization Error Expansion (3 citations)
• Towards Efficient and Accurate Winograd Convolution via Full Quantization (2 citations)
• Binarized Neural Machine Translation (1 citation)

and 3 more in the interactive map.

Spiking networks · 10 papers

• Spike-driven Transformer (48 citations)
• Neural Data Transformer 2: Multi-context Pretraining for Neural Spiking Activity (24 citations)
• Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies (14 citations)
• SEENN: Towards Temporal Spiking Early Exit Neural Networks (13 citations)
• Direct Training of SNN using Local Zeroth Order Method (3 citations)
• Evolving Connectivity for Recurrent Spiking Neural Networks (3 citations)
• SparseProp: Efficient Event-Based Simulation and Training of Sparse Recurrent Spiking Neural Networks (3 citations)
• EICIL: Joint Excitatory Inhibitory Cycle Iteration Learning for Deep Spiking Neural Networks (1 citation)
• Exploring Loss Functions for Time-based Training Strategy in Spiking Neural Networks
• Addressing the speed-accuracy simulation trade-off for adaptive spiking neurons

Neural architecture search · 8 papers

• EvoPrompting: Language Models for Code-Level Neural Architecture Search (18 citations)
• Evolutionary Neural Architecture Search for Transformer in Knowledge Tracing (12 citations)
• Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition (9 citations)
• MeCo: Zero-Shot NAS with One Data and Single Forward Pass via Minimum Eigenvalue of Correlation (5 citations)
• MCUFormer: Deploying Vision Tranformers on Microcontrollers with Limited Memory (3 citations)
• MathNAS: If Blocks Have a Role in Mathematical Architecture Design
• Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars
• Operation-Level Early Stopping for Robustifying Differentiable NAS

Mixture of experts · 2 papers

• ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer (5 citations)
• Alternating Updates for Efficient Transformers