Research map / Large language models / Instruction and fine-tuning
Optimization Inspired Few-Shot Adaptation for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OFA, which reinterprets LLM forward passes as preconditioned gradient descent and learns LayerNorm-based preconditioners with step-ratio and Hutchinson-estimated sharpness penalties for few-shot adaptation.
Problem
Few-shot adaptation of LLMs where fine-tuning and PEFT overfit on few demonstrations and in-context learning adds inference cost and has limited gains.
Approach
Treats the forward pass as preconditioned gradient descent, learns layer-wise preconditioners via LayerNorm parameters (no extra parameters), and adds a step-ratio convergence penalty and a Hutchinson-estimated preconditioned-Hessian-trace flatness penalty to the cross-entropy objective.
Key results
- On Llama2-7B, OFA reaches 95.84 on SST-2 and 97.72 on DBPedia, outperforming the best baseline (I2CL or IA3) on all nine datasets.
- On Llama3-8B-Instruct, OFA reaches 58.32 on SST-5 and 92.64 on Subj, versus 56.42 (Task-vector) and 82.04 (IA3) for the best baselines.
- OFA uses 0.27M trainable parameters versus 0.53M for rank-1 LoRA and 67.10M for rank-128 LoRA on Llama2-7B, with zero added inference cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OFA instead of the strongest reported baseline | Llama2-7B · SST-2 | accuracy: 94.44 ±1.44 -> 95.84 ±0.41 | Helped |
| Use OFA instead of the strongest reported baseline | Llama2-7B · SST-5 | accuracy: 46.08 ±2.11 -> 50.36 ±3.28 | Helped |
| Use OFA instead of the strongest reported baseline | Llama2-7B · TREC | accuracy: 84.40 ±5.99 -> 85.92 ±1.90 | Helped |
| Use OFA instead of the strongest reported baseline | Llama2-7B · Subj | accuracy: 73.84 ±3.84 -> 88.40 ±4.76 | Helped |
| Use OFA instead of the strongest reported baseline | Llama2-7B · HateSp18 | accuracy: 72.44 ±2.59 -> 83.04 ±3.72 | Helped |
| Use OFA instead of the strongest reported baseline | Llama2-7B · DBPedia | accuracy: 96.64 ±0.48 -> 97.72 ±0.52 | Helped |
| Use OFA instead of the strongest reported baseline | Llama2-7B · EmoC | accuracy: 75.48 ±1.63 -> 76.60 ±2.39 | Helped |
| Use OFA instead of the strongest reported baseline | Llama3-8B-Instruct · SST-5 | accuracy: 56.42 ±1.15 -> 58.32 ±2.74 | Helped |
| Use OFA instead of the strongest reported baseline | Llama3-8B-Instruct · TREC | accuracy: 87.60 ±3.46 -> 89.06 ±1.49 | Helped |
| Use OFA instead of the strongest reported baseline | Llama3-8B-Instruct · HateSp18 | accuracy: 77.20 ±4.37 -> 89.47 ±0.47 | Helped |
| Use OFA instead of the strongest reported baseline | Llama3-8B-Instruct · EmoC | accuracy: 68.04 ±2.24 -> 79.24 ±4.87 | Helped |
| Use OFA instead of the strongest reported baseline | Llama3-8B-Instruct · DBPedia | accuracy: 97.00 ±0.28 -> 97.92 ±1.06 | Helped |
Surprising findings
- LoRA at rank 1 (~0.53M parameters) performs worse than ICL on several datasets and shows very high variance, suggesting overfitting on few demonstrations.
- Adding the OFA objective to rank-1 LoRA does not close the gap to OFA, which the authors attribute to LoRA modifying the gradient rather than only the preconditioners.
Limitations
- Two penalty terms with multiple hyperparameters (lambda1, lambda2) increase tuning burden and compute overhead.
- Theoretical results rely on local quadratic approximations and loose assumptions, with proof sketches that are informal.
- Evaluated only on classification tasks with a fixed set of text datasets.
Experimental setup
Datasets: SST-2 SST-5 TREC AGNews Subj HateSpeech18 DBPedia EmoC MR
Benchmarks: SST-2 SST-5 TREC AGNews Subj HateSpeech18 DBPedia EmoC MR
Metrics: classification accuracy standard deviation over 5 seeds inference time (s) trainable parameter count
Baselines: Zero-shot Few-shot (ICL) Soft-prompt Label-anchor Task-vector I2CL IA3 LoRA
Builds on: in-context learning as gradient descent LoRA I2CL Hutchinson trace estimator
Closest papers by meaning
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.90)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.90)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
- Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (NeurIPS 2025 · similarity 0.89)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.89)
- Many-Shot In-Context Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.