Research map / Architectures and efficiency / Attention and transformers
Order-Level Attention Similarity Across Language Models: A Latent Commonality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that order-level decompositions of attention rollout (OLA) are similar across LMs and encode syntax, and uses them to build a training-free cross-LM adapter transfer method (TOA).
Problem
Whether context aggregation (attention) patterns share commonalities across different language models, and whether adapters trained on one LM can be transferred to another without training, given that feature spaces differ across models.
Approach
Decompose attention rollout by aggregation order (Order-Level Attention, OLA) so that same-order maps from different LMs become comparable; OLA is highly similar across LMs and encodes syntax, so an adapter trained on OLA from a source LM can be applied directly to a target LM's OLA.
Key results
- First-order OLA classification accuracy exceeds 90% for several source-target pairs (e.g., 94.60% on Llama3.2-3B target from Qwen/Gemma sources), versus at most 85.6% for the best baseline (ALTI).
- TOA transferred from Llama3.2-3B to Qwen2-1.5B raises RE accuracy from 7.69% zero-shot to 34.90%.
- For MLMs, TOA exceeds zero-shot on all source-target pairs across four tasks; e.g., BERT-base to BERT-large RE transfer reaches 29.90% versus 32.29% self-transfer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use first-order OLA instead of Attention Rollout as the representation for dependency parsing | OLA-based syntactic dependency parser on Qwen2-1.5B · Dependency parsing from OLA, CLM | UAS: 50.53 -> 63.58 | Helped |
| Transfer TOA trained on Llama3.2-3B to Qwen2-1.5B | TOA · Relation extraction (SemEval-2010 Task 8), target Qwen2-1.5B | relation prediction accuracy: 7.69 -> 30.49 | Helped |
| Apply TOA trained on Qwen2-1.5B | Qwen2-1.5B · Named entity recognition (CoNLL2012) | F1: 5.35 -> 53.81 | Helped |
| Transfer TOA trained on Bert-base to Bert-large | TOA · Relation extraction (SemEval-2010 Task 8), target Bert-large | relation prediction accuracy: 0.48 -> 29.90 | Helped |
| Use second-order OLA instead of first-order OLA (ablation) | OLA-based syntactic dependency parser on Bert-base · Dependency parsing from OLA, MLM | UAS: 81.29 -> 72.86 | Hurt |
| Use third-order OLA instead of first-order OLA (ablation) | OLA-based syntactic dependency parser on Bert-base · Dependency parsing from OLA, MLM | UAS: 81.29 -> 66.44 | Hurt |
| Remove data augmentation (ablation) | OLAS visual classification evaluation · OLAS visual classification, CLM; target Qwen2-1.5B | accuracy: 52.60 -> 41.00 | Hurt |
| Use first-order OLA rather than ALTI | Cross-model OLA similarity evaluation · OLAS visual classification, CLM; target Qwen2-1.5B | accuracy: 22.60 -> 52.60 | Helped |
Surprising findings
- Higher-order OLA and full Attention Rollout carry less distinguishable and less syntactic information than first-order OLA.
- TOA underperforms zero-shot in a small fraction (<6%) of CLM NER and POS cases, mainly for large target models such as Gemma2-9B.
Limitations
- Cross-model transfer still underperforms self-transfer in many settings, and TOA can hurt large target models on some tasks.
- Similarity of OLA varies substantially across source-target pairs, so transfer quality is not uniform.
Experimental setup
Datasets: CoNLL-2012 UD-English-EWT v2.15 SemEval-2010 Task 8 CoNLL2000 IMDB
Benchmarks: SemEval-2010 Task 8 CoNLL-2012 UD-English-EWT v2.15 CoNLL2000 IMDB
Metrics: classification accuracy Hits@1 Hits@5 UAS LAS F1 POS accuracy
Baselines: Attention Rollout IRNL ALTI zero-shot prompting
Builds on: Attention Rollout ResNet-18 axial transformer biaffine dependency parser
Closest papers by meaning
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.88)
- Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models (NeurIPS 2024 · similarity 0.87)
- Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.87)
- Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism (ICML 2025 · similarity 0.86)
- Taming Knowledge Conflicts in Language Models (ICML 2025 · similarity 0.86)
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.