Research map / Architectures and efficiency / RNNs and normalization
Iterative Vectors: In-Context Gradient Steering without Backpropagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Iterative Vectors (IVs), activation-space steering vectors extracted from in-context demonstrations by iteratively refining batch-wise attention-activation differences, applied at inference without backpropagation.
Problem
In-context learning is volatile and limited by context length and inference cost when many demonstrations are used; existing activation vector methods were mostly tested on synthetic tasks rather than real ICL classification.
Approach
Extract meta-gradients as differences between attention activations at input-output separators with and without demonstrations, then iteratively re-inject the running average of extracted vectors during extraction (emulating batched gradient updates) and apply the final vectors at inference without backpropagation.
Key results
- Across 13 tasks and four models, IV achieves an average macro-F1 gain of about 3.2% over one-shot clean ICL, outperforming Function Vectors and Task Vectors in most settings.
- On Rotten Tomatoes with Llama-2-7b, IV raises 2-shot macro-F1 from 70.28 to 87.50 (+17.22).
- On the 20-class emoji task, IV reaches 12.90 macro-F1 in 1,452s versus 2,434s for 2-shot ICL, and cuts extraction time 95% relative to Function Vectors (23.75 vs 438.3 min).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Iterative Vectors instead of clean ICL | GPT-J-6B · 13-task average | macro-F1: 39.90 -> 41.76 | Helped |
| apply Iterative Vectors instead of clean ICL | Llama 2-7B · 13-task average | macro-F1: 43.07 -> 46.59 | Helped |
| apply Iterative Vectors instead of clean ICL | Llama 3.1-8B · 13-task average | macro-F1: 46.02 -> 48.79 | Helped |
| apply Iterative Vectors instead of clean ICL | Llama 2-13B · 13-task average | macro-F1: 48.20 -> 52.89 | Helped |
| replace clean ICL with IV | Llama 2-7B · AG News, 5-shot | macro-F1: 82.47 -> 85.84 | Helped |
| replace clean ICL with IV | Llama 2-7B · Rotten Tomatoes, 2-shot | macro-F1: 70.28 -> 87.50 | Helped |
| increase extraction episodes from 1 to 200 | IV on Llama 2-7B · AG News | macro-F1: 40.64 -> 69.62 | Helped |
| disable iterative refinement (b = 0) (ablation) | IV on Llama 2-7B · AG News validation | macro-F1 | Hurt |
| use single-episode iterative refinement (b = 1) rather than disabling iteration (ablation) | IV on Llama 2-7B · AG News validation | macro-F1: 62.7 -> 69.4 | Helped |
| vary extraction batch size across the reported settings (ablation) | IV on Llama 2-7B · AG News validation | macro-F1: 72.8; 67.8; 67.0; 64.5; 63.3 | Mixed |
Surprising findings
- Function Vectors and Task Vectors degrade below standard ICL in 3 of 8 averaged settings.
- Setting the extraction batch size to 0 (no iterative refinement) performs significantly worse than any batch size of 1 or more.
Limitations
- Evaluated only on classification tasks with single-token label words.
- Hyperparameters (k, b, alpha1, alpha2) require grid search on validation episodes.
- Function Vectors are slow to extract, so comparisons at higher shot counts are limited.
Experimental setup
Datasets: AG News Rotten Tomatoes emoji SST-5 TREC hate speech sentiment offensive irony emotion climate abortion
Benchmarks: AG News Rotten Tomatoes emoji SST-5 TREC hate speech sentiment offensive irony emotion climate abortion
Metrics: macro-F1 micro-F1 weighted-F1 inference time extraction time
Baselines: Function Vectors Task Vectors standard one-shot ICL
Builds on: Function Vectors Task Vectors dual form of neural networks meta-optimization view of ICL
Closest papers by meaning
- Improving Instruction-Following in Language Models through Activation Steering (ICLR 2025 · similarity 0.89)
- In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.89)
- Implicit In-context Learning (ICLR 2025 · similarity 0.88)
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.87)
- Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior (NeurIPS 2026 · similarity 0.87)
- Steering When Necessary: Flexible Steering Large Language Models with Backtracking (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.