Research map / Large language models / Instruction and fine-tuning
Distribution-Aligned Decoding for Efficient LLM Task Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SVDecode, which warm-starts a PEFT model, builds a logit-space steering vector from the negative KL gradient between warm-started and pre-trained output distributions, and adds it at decoding time with a calibrated global strength.
Problem
Task adaptation of LLMs is costly because PEFT still requires backward passes and weight updates; the paper asks whether output distribution can be steered toward the task directly at decoding time.
Approach
Treat adaptation as output-distribution alignment: take the negative gradient of KL(P_warm || P_pretrained), project it to logit space via the softmax Jacobian, mask low-confidence tokens, and add it to logits during decoding with a Newton-style strength estimated on a calibration set.
Key results
- On TruthfulQA multiple-choice with Qwen2.5-7B, SVDecode with Prompt Tuning raises average from 45.49% to 50.29%.
- On commonsense reasoning with Qwen2.5-7B, SVDecode with IA3 improves average accuracy from 80.08 to 81.90.
- Removing logit-space projection drops performance across methods, some by about 10 points; removing confidence masking yields degenerate outputs (e.g., repeated '!!!!' tokens).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SVDecode to LoRA | Qwen2.5-1.5B · TruthfulQA multiple-choice | Avg.: 46.99% to 49.77% | Helped |
| Add SVDecode to Prompt Tuning | Qwen2.5-7B · TruthfulQA multiple-choice | Avg.: 45.49 to 50.29 | Helped |
| Add SVDecode to LoRA | LLaMA3.1-8B · TruthfulQA multiple-choice | Avg.: 42.89% to 47.80% | Helped |
| Add SVDecode to LoRA | Qwen2.5-1.5B · TruthfulQA open-ended generation | Avg.: 44.71% to 45.73% | Helped |
| Add SVDecode to P-Tuning v2 | Qwen2.5-7B · TruthfulQA open-ended generation | Avg.: 62.80% to 64.43% | Helped |
| Add SVDecode to LoRA | LLaMA3.1-8B · TruthfulQA open-ended generation | Avg.: 45.78% to 46.54% | Helped |
| Add SVDecode to PEFT methods | Qwen2.5-7B and LLaMA3.1-8B · Commonsense reasoning tasks | accuracy | Helped |
| Remove confidence-aware steering vector constraint (ablation) | SVDecode with LoRA on Qwen2.5-7B · TruthfulQA open-ended generation | %Truth: 55.48 -> 0.02 | Hurt |
| Remove confidence-aware steering vector constraint (ablation) | SVDecode with LoRA on Qwen2.5-7B · TruthfulQA open-ended generation | %Info: 50.61 -> 0.01 | Hurt |
| Remove confidence-aware steering vector constraint (ablation) | SVDecode with LoRA on Qwen2.5-7B · TruthfulQA open-ended generation | %T*I: 46.95 -> 0.00 | Hurt |
| Remove logit-space projection (ablation) | SVDecode across PEFT methods · TruthfulQA multiple-choice | accuracy | Hurt |
| Add SVDecode to Greedy Search | Qwen2.5-7B · Commonsense reasoning tasks | Avg.: 75.45 -> 77.06 | Helped |
Surprising findings
- Without the confidence-aware mask, the %Truth, %Info and %T*I metrics collapse to near zero for LoRA on Qwen2.5-7B, showing decoding becomes degenerate.
Limitations
- Decoding-time steering adds per-token computation over the warm-start and pre-trained models, which is not quantified.
- Theoretical equivalence to fine-tuning is only first-order and local; global equivalence requires strong conditions the authors acknowledge.
- The global strength mu is estimated from labeled calibration data, so it is not fully free of task supervision.
Experimental setup
Datasets: TruthfulQA BoolQ PIQA SIQA HellaSwag WinoGrande ARC-easy ARC-challenge OBQA
Benchmarks: TruthfulQA BoolQ PIQA SIQA HellaSwag WinoGrande ARC-easy ARC-challenge OBQA
Metrics: MC1 MC2 MC3 %Truth %Info %Truth*Info accuracy
Baselines: Prompt Tuning IA3 P-Tuning v2 LoRA
Builds on: LoRA P-Tuning v2 Prompt Tuning IA3 Contrastive Decoding
Closest papers by meaning
- Model Whisper: Steering Vectors Unlock Large Language Models’ Potential in Test-Time (AAAI 2026 · similarity 0.88)
- Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (NeurIPS 2025 · similarity 0.88)
- Focus On This, Not That! Steering LLMs with Adaptive Feature Specification (ICML 2025 · similarity 0.88)
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.87)
- Decoding-Time Language Model Alignment with Multiple Objectives (NeurIPS 2024 · similarity 0.87)
- DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.