Research map / Large language models / Language, safety and interpretability
Causality ≠ Invariance: Function and Concept Vectors in LLMs
Gustaw Opielka, Hannes Rosenbusch, Claire E Stevenson
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select heads by RSA (concept consistency) instead of activation patching to form vectors | Function Vectors · Format invariance, 4 LLMs | concept vs format RSA | Improved |
| extract vector from different input format (OE vs MC) | Function Vectors · Llama 3.1 70B similarity matrices | cosine similarity: near orthogonal across formats; within-format 0.90 | Hurt |
| use RSA-selected CV heads vs AIE-selected FV heads | Concept Vectors · Top-K head overlap | number of overlapping heads: 0 overlap for K<=10 in all models | No clear effect |
| use CV instead of FV for OOD steering (MC, French) | Concept Vectors · AmbiguousICL | delta P / KL | Improved |
| use FV instead of CV for in-distribution steering | Function Vectors · AmbiguousICL OE-ENG | delta P: salty->sweet: 49 -> 56 (+%) | Improved |
| cross-format activation patching (ablation) | Activation patching · Llama 3.1 70B | identified heads: 0 overlap with CV top-5 | No clear effect |
| apply CV to zero-shot steering | Concept Vectors · Zero-shot steering | delta P | No clear effect |
| extract FV from French/MC prompts and steer | Function Vectors · AmbiguousICL | format tokens: MC: '(' +53%; OE-FR: 'su' +31% | Hurt |
| exclude Qwen 2.5 72B outlier datasets from AIE (ablation) | Activation patching · Qwen 2.5 72B | top-5 head ranking: 100% overlap | No clear effect |
Closest papers by meaning
- Function Vectors in Large Language Models (ICLR 2024 · similarity 0.92)
- Which Attention Heads Matter for In-Context Learning? (ICML 2025 · similarity 0.88)
- Provable In-Context Vector Arithmetic via Retrieving Task Concepts (ICML 2025 · similarity 0.88)
- Do different prompting methods yield a common task representation in language models? (NeurIPS 2025 · similarity 0.88)
- Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning (NeurIPS 2026 · similarity 0.88)
- Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
