Research map / Large language models / Language, safety and interpretability
Attend to the Active: Structure-Aware Dynamic Attention in LLMs for Compositional Instruction Following
Fangrui Lv, Yulei Qin, Ruixin Hong, Jian Liang, Jinyang Wu, Ke Li, Xing Sun, Changshui Zhang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add structure information for restricting steering spans (ablation) | ATA (Llama-3-8B) · Chain/Branch/Parallel | accuracy: Parallel 60.35 -> 69.91; Chain 60.45 -> 62.98; Branch 55.28 -> 58.74 | Improved |
| add mutually exclusive masking (ablation) | ATA (Llama-3-8B) · Chain/Branch/Parallel | accuracy: Parallel 67.42 -> 69.91; Chain 61.82 -> 62.98; Branch 57.03 -> 58.74 | Improved |
| add dynamic attention masking (ablation) | ATA (Llama-3-8B) · Chain/Branch/Parallel | accuracy: Parallel 64.31 -> 69.91; Chain 60.74 -> 62.98; Branch 56.65 -> 58.74 | Improved |
| add active control for sub-task identification (ablation) | ATA (Llama-3-8B) · Chain/Branch/Parallel | accuracy: Parallel 66.82 -> 69.91; Chain 61.14 -> 62.98; Branch 57.23 -> 58.74 | Improved |
| use ATA steering instead of SampleAttention | Llama-3-8B · Branch | accuracy: 51.92 -> 58.74 | Improved |
| use ATA steering instead of PASTA | Llama-3-8B · Parallel | accuracy: 60.43 -> 69.91 | Improved |
| steer toward misguided (inactive) sub-tasks (ablation) | ATA (Llama-3-8B) · Chain/Branch/Parallel | accuracy: Chain 62.98 -> 56.43; Branch 58.74 -> 47.86; Parallel 69.91 -> 54.12 | Hurt |
| steer toward random sub-tasks (ablation) | ATA (Llama-3-8B) · Chain/Branch/Parallel | accuracy: Chain 62.98 -> 57.38; Branch 58.74 -> 51.04; Parallel 69.91 -> 56.81 | Hurt |
| use entropy confidence instead of max probability for active sub-task identification (ablation) | ATA (Llama-3-8B) · Chain/Branch/Parallel | accuracy: Chain 61.74 -> 62.98; Branch 57.91 -> 58.74; Parallel 68.62 -> 69.91 | Improved |
| use entropy confidence instead of margin confidence (ablation) | ATA (Llama-3-8B) · Chain/Branch/Parallel | accuracy: Chain 61.45 -> 62.98; Branch 58.16 -> 58.74; Parallel 69.27 -> 69.91 | Improved |
| apply ATA on nested chain & branch instructions | Llama3-8B · Nested Chain & Branch | accuracy: 48.42 -> 51.95 | Improved |
| apply ATA on nested chain & parallel instructions | Llama3-13B · Nested Chain & Parallel | accuracy: 56.48 -> 59.30 | Improved |
Closest papers by meaning
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.89)
- Chain-of-Instructions: Compositional Instruction Tuning on Large Language Models (AAAI 2025 · similarity 0.88)
- Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models (ICLR 2025 · similarity 0.88)
- Attention Illuminates LLM Reasoning: The Uncovered Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization (ICML 2026 · similarity 0.88)
- Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMs (NeurIPS 2026 · similarity 0.87)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
