Research map / Architectures and efficiency / RNNs and normalization
Switchable Decision: Dynamic Neural Generation Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Switchable Decision (SD), an RL-based policy that dynamically skips attention, FFN, and input-token candidates per instance in BART-style encoder-decoder generation models, trained with lexicographic constrained optimization to trade quality against FLOPs.
Problem
Auto-regressive encoder-decoder generation models are slow at inference, and static compression methods yield one fixed-size model; the paper targets per-instance computation allocation that keeps accuracy while reducing FLOPs.
Approach
A lightweight RL policy network (one-layer MLP on first-layer hidden states) samples Bernoulli skip decisions for attention and FFN layers and a categorical choice of token-skipping strategy per input; a lexicographic constrained-optimization update keeps quality at a threshold while minimizing FLOPs, instead of a hand-tuned linear trade-off coefficient.
Key results
- On CNN/DailyMail, SD reaches ROUGE-1 44.31 vs BART-large 44.16 while using 61.1% of FLOPs.
- On GLUE SST, SD matches BART at 96.6% accuracy with 80.7% FLOPs; on RTE it gets 87.2 vs 87.0 with 83.6% FLOPs.
- On SQuAD 1.1, SD gets 88.7/94.5 EM/F1 vs BART 88.8/94.6 at 80.5% FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Switchable Decision to BART-large | BART-large · CNN/DailyMail | ROUGE-1: 44.16 -> 44.31 | Helped |
| apply Switchable Decision to BART-large | BART-large · CNN/DailyMail | FLOPs (%): 100 -> 61.1 | Helped |
| apply Switchable Decision to BART-large | BART-large · XSum | ROUGE-1: 45.14 -> 45.20 | Helped |
| apply Switchable Decision to BART-large | BART-large · XSum | FLOPs (%): 100 -> 81.9 | Helped |
| apply Switchable Decision to BART-large | BART-large · MNLI | FLOPs (%): 100 -> 82.4 | Helped |
| apply Switchable Decision to BART-large | BART-large · RTE | accuracy: 87.0 -> 87.2 | Helped |
| apply Switchable Decision to BART-large | BART-large · SST | accuracy: 96.6 -> 96.6 | No clear effect |
| apply Switchable Decision to BART-large | BART-large · SQuAD 1.1 | EM/F1: 88.8/94.6 -> 88.7/94.5 | No clear effect |
| apply Switchable Decision to BART-large | BART-large · SQuAD 2.0 | EM/F1: 86.1/89.2 -> 86.0/89.3 | No clear effect |
| add attention and FFN skipping in the encoder only (ablation) | BART-large · CNN/DailyMail | ROUGE: 44.16/21.28/40.90 -> 44.21/21.32/40.95 | No clear effect |
| add attention and FFN skipping in the decoder only (ablation) | BART-large · CNN/DailyMail | ROUGE: 44.16/21.28/40.90 -> 44.13/21.08/40.86 | Hurt |
| add token skipping only (ablation) | BART-large · CNN/DailyMail | FLOPs (%): 100 -> 71.5 | Helped |
Surprising findings
- A random skipping policy with similar FLOPs (75.3%) drops CNN/DailyMail ROUGE-1 to 41.77, showing learned decisions matter.
- Layer-wise decisions made from previous-layer outputs give nearly identical quality to first-layer decisions but cost slightly more FLOPs (61.8% vs 61.1%).
Limitations
- Gains are reported mainly on BART-large with a few tasks; GPT-2, T5 and LLaMA results are limited to CNN/DailyMail.
- Some reported gains are small (e.g., SQuAD and GLUE within about 0.2 points of BART), and FLOPs are estimates rather than measured latency for most settings.
- Combining with quantization or distillation is left as future work.
Experimental setup
Datasets: CNN/DailyMail XSum SQuAD 1.1 SQuAD 2.0 MNLI RTE SST-2
Benchmarks: CNN/DailyMail XSum SQuAD 1.1 SQuAD 2.0 MNLI RTE SST-2
Metrics: ROUGE-1 ROUGE-2 ROUGE-L exact match F1 accuracy FLOPs
Baselines: BART UniLM BERTSUM CALM FREE Pegasus DQ-BART RoBERTa
Builds on: BART SkipNet BlockDrop self-critical policy gradient
Closest papers by meaning
- Inference Time Optimization with Confidence Dynamics (ICML 2026 · similarity 0.85)
- Learning How Hard to Think: Input-Adaptive Allocation of LM Computation (ICLR 2025 · similarity 0.85)
- A Simple Model of Inference Scaling Laws (ICML 2025 · similarity 0.84)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.84)
- Learning Dynamics of LLM Finetuning (ICLR 2025 · similarity 0.84)
- DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.