Research map / Architectures and efficiency / RNNs and normalization
Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a latency-first recipe for small language models (depth-width ratio scaling law, evolutionary search over hybrid DeltaNet/Mamba2/attention operators, and weight normalization plus meta tokens), yielding the Nemotron-Flash 1B/3B family.
Problem
Small language models are usually designed to minimize parameter count, which does not translate into proportional real-device latency or throughput; SLM design relies on manual trial-and-error for hybrid operator combinations.
Approach
Select depth-width ratios by latency using an augmented scaling law (loss as a function of depth, width, and data), search hybrid operator combinations with aging evolution using short-training PPL as a proxy and LUT latency, and add weight normalization (unit-norm projection) and learnable meta tokens to improve training.
Key results
- Nemotron-Flash-3B achieves 60.98% average accuracy across 16 tasks, +2.0% over Qwen2.5-3B, with 1.7x lower latency and 6.4x higher throughput.
- Nemotron-Flash-1B achieves 49.63% average accuracy, +5.5% over Qwen3-0.6B, with 1.9x lower decoding latency and about 46x higher throughput.
- Weight normalization improves CR accuracy by +1.20% and reduces Wikitext PPL by 0.66 on average across Llama, DeltaNet, and Mamba2 1B models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the searched hybrid DeltaNet–Mamba2–attention architecture rather than pure operator baselines | 500M language models at matched decoding latency · Wikitext | PPL: 20.90 -> 20.70 | Helped |
| Pair DeltaNet with Mamba2 instead of using DeltaNet alone (ablation) | 500M hybrid operator exploration · Wikitext | PPL: 23.87 -> 23.37 | Helped |
| Pair Gated DeltaNet with Mamba2 instead of using Gated DeltaNet alone (ablation) | 500M hybrid operator exploration · Wikitext | PPL: 23.80 -> 23.37 | Helped |
| Pair DeltaNet with attention instead of using DeltaNet alone (ablation) | 500M hybrid operator exploration · Wikitext | PPL: 23.87 -> 24.32 | Hurt |
| Add weight normalization (ablation) | 1B Llama, DeltaNet, and Mamba2 models · 8-task commonsense reasoning | accuracy (%): 53.81 -> 54.85; 53.46 -> 54.39; 53.30 -> 54.71 | Helped |
| Add weight normalization (ablation) | 1B Llama, DeltaNet, and Mamba2 models · Wikitext | PPL: 18.67 -> 18.03; 18.86 -> 18.19; 18.44 -> 17.88 | Helped |
| Add 256 learnable meta tokens (ablation) | Mamba2-48L-1B, DeltaNet-48L-1B, and Searched-24L-830M · Wikitext | PPL: 19.04 -> 18.98; 19.60 -> 19.47; 20.61 -> 20.49 | Helped |
| Add 256 learnable meta tokens (ablation) | Mamba2-48L-1B, DeltaNet-48L-1B, and Searched-24L-830M · 8-task commonsense reasoning | accuracy (%): 51.71 -> 52.33; 52.12 -> 52.46; 50.74 -> 51.13 | Helped |
| Replace two full-attention layers with sliding-window attention (ablation) | Nemotron-Flash-3B · 32k input, maximum batch size | throughput (tok/sec): 2939 -> 4657 | Helped |
| Replace two full-attention layers with sliding-window attention (ablation) | Nemotron-Flash-3B · general benchmarks | average accuracy (%): 60.98 -> 60.84 | No clear effect |
| Reduce the number of full-attention layers to one (ablation) | Nemotron-Flash-3B · Ruler NIAH at longer context lengths | NIAH performance | Hurt |
| Choose depth and width to fit the latency budget rather than defaulting to deep-thin models | Llama model configurations · 3-second latency budget | 8-task commonsense reasoning accuracy | Helped |
Surprising findings
- Deep-thin models, although more parameter-efficient, give suboptimal accuracy-latency trade-offs; the optimal depth-width ratio grows with the latency budget.
- Nemotron-Flash-1B and 3B keep only 2 and 3 full-attention layers yet achieve competitive recall accuracy.
Limitations
- Architecture search uses language modeling perplexity as proxy, which may not capture long-context or other capabilities.
- Search focuses on macro-architecture, leaving fine-grained factors unexplored.
- Training data for final models and instruction tuning includes proprietary datasets, limiting reproducibility.
Experimental setup
Datasets: Zyda2 Smollm-corpus Climb-Mix MegaMath Wikitext Ruler AG News
Benchmarks: MMLU PIQA ARC-C ARC-E Hellaswag Winogrande OBQA GSM8K MathQA HumanEval HumanEval-Plus MBPP
Metrics: accuracy perplexity decoding latency throughput
Baselines: Qwen2.5 Qwen3 Llama-3.2 SmolLM2 H2O-Danube3 AMD-OLMo SWA GLA DeltaNet Gated DeltaNet Mamba2 nGPT
Builds on: Mamba2 Gated DeltaNet nGPT aging evolution TensorRT-LLM AutoDeploy
Closest papers by meaning
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.88)
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (NeurIPS 2024 · similarity 0.87)
- Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (ICML 2025 · similarity 0.87)
- Zebra-Llama: Towards Extremely Efficient Hybrid Models (NeurIPS 2025 · similarity 0.87)
- Rethinking Optimization and Architecture for Tiny Language Models (ICML 2024 · similarity 0.87)
- Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.