Research map / Architectures and efficiency / RNNs and normalization
Bring Future Vision: Dynamic Computation Allocation Guided by Lightweight Feature Forecaster
Chao Han, Yijuan Liang, Zihao Xuan, Daokuan Wu, Wei Zhang, Xiaoyu Shen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Lightweight Feature Forecaster (informed routing) instead of skipping (ablation) | SkipGPT router, Llama3.1-8B · WikiText-2, 25% sparsity | perplexity: 20.76 -> 9.43 | Improved |
| add Lightweight Feature Forecaster instead of skipping (ablation) | SkipGPT router, Llama3.1-8B · WikiText-2, 40% sparsity | perplexity: 53.94 -> 13.87 | Improved |
| add LFF (ablation) | SkipGPT router, Llama3.1-8B · Reasoning average, 25% sparsity | average accuracy (%): 51.23 -> 64.62 | Improved |
| add LFF (ablation) | SkipGPT router, Llama3.1-8B · Reasoning average, 40% sparsity | average accuracy (%): 46.75 -> 56.02 | Improved |
| add LFF, after LoRA fine-tuning (ablation) | SkipGPT-LoRA · Reasoning average, 25% sparsity | average accuracy (%): 60.43 -> 66.16 | Improved |
| add LFF, after LoRA fine-tuning (ablation) | SkipGPT-LoRA · Reasoning average, 40% sparsity | average accuracy (%): 58.85 -> 60.24 | Improved |
| add LFF with balanced attention/FFN sparsity (ablation) | SkipGPT-balance · Balanced reasoning | average accuracy (%): 58.42 -> 63.49 | Improved |
| add LFF with balanced attention/FFN sparsity (ablation) | SkipGPT-balance · Balanced WikiText-2 | perplexity: 11.40 -> 9.49 | Improved |
| add LFF under KV cache reduction (ablation) | SkipGPT · WikiText-2 PPL, 25% sparsity, KV reduction | perplexity: 11.68 -> 10.21 | Improved |
| enforce KV cache reduction (mask skipped tokens from attention) | LFF · WikiText-2 PPL, 25% sparsity | perplexity: 8.93 -> 10.21 | Hurt |
| add a linear LFF to FFN units (ablation) | FFN units · WikiText-2 PPL without fine-tuning | perplexity: 36.50 -> 12.19 | Improved |
| add ReLU non-linearity in the LFF middle layer (ablation) | linear FFN LFF · WikiText-2 PPL without fine-tuning | perplexity: 12.19 -> 11.99 | No clear effect |
Closest papers by meaning
- SkipGPT: Each Token is One of a Kind (ICML 2025 · similarity 0.87)
- $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.87)
- Dr.LLM: Dynamic Layer Routing in LLMs (ICLR 2026 · similarity 0.86)
- Ada-K Routing: Boosting the Efficiency of MoE-based LLMs (ICLR 2025 · similarity 0.86)
- TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.86)
- Reinforced Fast Weights via Next-Sequence Prediction (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
