Research map / Large language models / Instruction and fine-tuning

In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering

ICML 2024 · Large language models › Instruction and fine-tuning · approach: Gradient-free adaptation and merging · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes In-Context Vectors (ICV), an inference-only method that extracts a single steering vector from demonstration latent states (first principal direction of paired differences or a contrastive gradient) and adds it to all layers' latent states at query time, outperforming ICL and LoRA fine-tuning on detoxification, style transfer, and role-play.

Problem

In-context learning is uneven and sensitive to demonstrations and templates, consumes context window space, and is hard to control quantitatively; fine-tuning is costly.

Approach

Summarize demonstrations into a single in-context vector from last-token latent-state differences (paired: first principal direction; unpaired: contrastive gradient), then add it, scaled by lambda and norm-preserved, to every layer and token at inference, avoiding prompt demonstrations and weight updates.

Key results

• On ParaDetox with Falcon-7b, ICV (lambda=0.1) reduces toxicity to 34.77% versus 73.09% for ICL and 52.78% for LoRA FT.
• On Llama-7b informal-to-formal transfer, ICV reaches 48.30% formality versus 32.96% for ICL and 21.99% for LoRA FT.
• On jailbreaking Vicuna-7b, ICV (lambda=0.20) reaches 99% attack success rate versus 44% for in-context learning.

Measured effects reported in the paper

Surprising findings

• Applying the ICV to only the first, middle, or last layer performs about as poorly as no ICV, so the effect requires adding it across all layers.
• Latent-state kNN on AGNews substantially outperforms vanilla ICL for GPT2 models, suggesting latent states hold information attention does not fully exploit.

Limitations

• ICV requires white-box access to model activations, so it cannot be applied to closed API models.
• Larger steering strength lowers semantic similarity and fluency (ROUGE-1 drops).
• Paired ICV is slightly better than the unpaired contrastive variant, and performance depends on hyperparameter lambda.

Experimental setup

Datasets: ParaDetox GYAFC Yelp review Romeo and Juliet AGNews Wikipedia Toxic Comments Build-it Break-it Fix-it Bot-Adversarial Dialogue

Benchmarks: ParaDetox GYAFC Yelp review Romeo and Juliet AGNews

Metrics: toxicity rate ROUGE-1 BERT score BLEU formality accuracy positivity accuracy win rate attack success rate

Baselines: In-context learning LoRA fine-tuning w/o context GBDA PEZ task vectors

Builds on: in-context learning LoRA task arithmetic inference-time intervention

Closest papers by meaning

• Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning (NeurIPS 2026 · similarity 0.90)
• Implicit In-context Learning (ICLR 2025 · similarity 0.90)
• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.89)
• Disentangling Latent Shifts of In-Context Learning with Weak Supervision (NeurIPS 2025 · similarity 0.89)
• Vector-ICL: In-context Learning with Continuous Vector Representations (ICLR 2025 · similarity 0.89)
• Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.89)