Research map / Architectures and efficiency / Quantization
Efficient LLM Adaptation with Forward-Only Passes
Baichuan Huang, Ananth Balashankar, Amir Aminifar
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a forward-only approach that adapts LLMs to tasks using only inference-time forward passes, exploiting angle concentration of activations, instead of backpropagation.
Problem
Efficient task-specific adaptation of LLMs is costly because standard fine-tuning requires backpropagation; the paper asks whether the forward passes used in inference serving can enable adaptation.
Approach
Uses forward-only passes already executed during LLM inference serving to perform task-specific adaptation, leveraging the angle concentration of activations induced by each (input) to avoid backward computation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use forward-only search instead of gradient-based LASER-Grad to pick matrix | rank reduction · GPT-J, 8 datasets | peak memory (GB): 22.0 -> 11.5 | Improved |
| Use hierarchical forward-only search instead of exhaustive LASER (std) | rank reduction · GPT-J, 8 datasets | speedup: 1 -> 36.7x | Improved |
| Use forward-only search instead of LASER (std) | rank reduction · GPT-J, 8 datasets | average accuracy gain: 8.24 -> 11.3 | Improved |
| Use forward-only search instead of LASER (std) | rank reduction · GPT-J Epistemic Reasoning | accuracy (%): 38.3 -> 62.8 | Improved |
| Use forward-only search instead of LASER (std) | rank reduction · GPT-J Bios Profession | accuracy (%): 82.1 -> 79.1 | Hurt |
| Use hierarchical search instead of std evaluation over full validation (ablation) | forward-only search · GPT-J, 8 datasets | speedup: 5.6x -> 36.7x | Improved |
| Use 100-sample subset instead of std validation set (ablation) | forward-only search · GPT-J, 8 datasets | average accuracy gain: 11.2 -> 9.7 | Hurt |
| Use top-j=5 instead of j=10 (ablation) | hierarchical search · GPT-J | accuracy | Hurt |
| Use last m=20 instead of m=50 SVD components (ablation) | hierarchical search · GPT-J | accuracy | Mixed |
| Increase last-m SVD components (20, 50, 100) (ablation) | forward-only search · GPT-J | peak memory | Hurt |
| Reduce D_subset size from 100 to 10, 5, 1 (ablation) | hierarchical search · RoBERTa-base | final candidate | No clear effect |
| Add attention matrices (key, query, value, out) to candidates (ablation) | forward-only search · RoBERTa-base | top-j matrices | No clear effect |
Closest papers by meaning
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.91)
- Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (NeurIPS 2025 · similarity 0.89)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.89)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.89)
- Optimization Inspired Few-Shot Adaptation for Large Language Models (NeurIPS 2025 · similarity 0.89)
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
