Research map / Architectures and efficiency / Attention and transformers
DISCO: Disentangled Communication Steering for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DISCO Steering, which injects mean-difference steering vectors into attention query and value spaces of LLaMA 3.1 8B and Gemma 2 9B, and shows it disentangles and outperforms steering attention inputs (Communication Steering).
Problem
Choosing the representation space for steering vectors in LLMs; prior work steers layer outputs or attention head outputs, while the query and value spaces inside attention heads are unexplored despite their linear discriminability.
Approach
Directly add mean-difference steering vectors to per-head query and value representations; analytically show query steering re-weights attention via key inner products and that joint query/value steering disentangles Communication Steering (steering attention inputs), which rigidly couples query and value magnitudes, enabling separate control.
Key results
- DISCO achieves best performance in 13/16 experiments across datasets, models and metrics on LLaMA 3.1 8B and Gemma 2 9B.
- Steering efficacy of DISCO scores up to 19.1% higher than the runner-up (Gemma 2, Corr promotion).
- Query and value spaces show greater portion of attention heads with high linear discriminability than head output spaces on Corr, Power, Wealth and TQA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Steer query and value spaces jointly instead of the attention input | LLaMA 3.1 8B · Corr promotion | LLM Judge score: 2.99 -> 3.49 | Helped |
| Steer query and value spaces jointly instead of the attention input | Gemma 2 9B · Corr promotion | LLM Judge score: 3.03 -> 3.61 | Helped |
| Steer query and value spaces jointly instead of the attention input | LLaMA 3.1 8B · TruthfulQA | T*I: 82.3 -> 80.7 | Hurt |
| Steer query and value spaces jointly instead of the attention input | Gemma 2 9B · TruthfulQA | T*I: 90.5 -> 86.4 | Hurt |
| Add value steering to query-only steering (ablation) | LLaMA 3.1 8B · Corr promotion | LLM Judge score: 3.29 -> 3.49 | Helped |
| Add query steering to value-only steering (ablation) | LLaMA 3.1 8B · Corr promotion | LLM Judge score: 2.93 -> 3.49 | Helped |
| Add value steering to query-only steering (ablation) | Gemma 2 9B · Corr promotion | LLM Judge score: 2.66 -> 3.61 | Helped |
| Add query steering to value-only steering (ablation) | Gemma 2 9B · Corr promotion | LLM Judge score: 3.17 -> 3.61 | Helped |
| Add value steering to query-only steering (ablation) | LLaMA 3.1 8B · TruthfulQA | MC accuracy: 84.4 -> 84.4 | No clear effect |
| Add query steering to value-only steering (ablation) | LLaMA 3.1 8B · TruthfulQA | T*I: 83.1 -> 80.7 | Hurt |
| Add value steering to query-only steering (ablation) | Gemma 2 9B · TruthfulQA | MC accuracy: 86.8 -> 87.7 | Helped |
| Add query steering to value-only steering (ablation) | Gemma 2 9B · TruthfulQA | T*I: 82.7 -> 86.4 | Helped |
Surprising findings
- Steering the query, which only re-weights attention, is useful for promoting and suppressing behaviors; DISCO-QV is best in 7 of 12 behavior cases.
- Communication Steering (steering attention inputs) is a strong baseline, best non-DISCO method in 6 cases, though rarely reported in prior work.
Limitations
- Analysis centered on mean-difference steering vectors; generalization to optimization-based or affine steering left to future work.
- Results limited to two models (LLaMA 3.1 8B and Gemma 2 9B) and four behaviors.
- Hyperparameter search (top-k heads, magnitudes) done per method on validation sets, with DISCO-QV using pairs derived from DISCO-V and DISCO-Q settings.
Experimental setup
Datasets: TruthfulQA Model-Written Evaluations (Power-seeking) Model-Written Evaluations (Wealth-seeking) Model-Written Evaluations (Corrigibility less-hhh)
Benchmarks: TruthfulQA Power-seeking Wealth-seeking Corrigibility
Metrics: LLM judge score (1-4) promote/suppress degradation rate multiple-choice accuracy True*Info
Baselines: CAA ITI Post Attn. MLP Input MLP Output Communication Steering Attn Output
Builds on: Inference Time Intervention Contrastive Activation Addition linear representation hypothesis Communication Steering
Closest papers by meaning
- Steer2Edit: From Activation Steering to Component-Level Editing (NeurIPS 2026 · similarity 0.88)
- Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization (NeurIPS 2024 · similarity 0.87)
- Improved Representation Steering for Language Models (NeurIPS 2025 · similarity 0.87)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.87)
- On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs (NeurIPS 2026 · similarity 0.87)
- When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.