Research map / Architectures and efficiency / Attention and transformers
Multi-head Temporal Latent Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-head Temporal Latent Attention (MTLA), which compresses MLA's latent KV cache along the time axis with a hyper-network-generated merging weight and a stride-aware causal mask for parallel training, evaluated on speech translation, ASR, SLU and summarisation.
Problem
The KV cache in Transformer decoder inference grows linearly with sequence length, and existing methods (MQA, GQA, MLA) compress only head or latent dimensions, not the temporal dimension.
Approach
Dynamically merge temporally adjacent latent KV vectors using hyper-network-generated sigmoid weights built on MLA, and train in parallel with a stride-aware causal mask that matches incremental inference behaviour.
Key results
- On MuST-C En-De, MTLA (s=2) reaches 23.28 BLEU vs 23.18 for MHA, with 4.29x inference speedup and 6.58x lower average GPU memory.
- With FlashAttention-2 extended kernels, MTLA achieves 7.71x speedup and 14.81x memory reduction versus MHA at BLEU 23.29.
- On XSum, AMI and SLURP, MTLA reaches up to 3.75x speedup and over 7x memory reduction vs MHA while keeping quality roughly unchanged.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MTLA instead of MHA | decoder-only speech translation model · MuST-C En-De tst-COMMON | BLEU: 23.18 -> 23.28 | Helped |
| Use MTLA instead of MHA | decoder-only speech translation model · MuST-C En-De tst-COMMON | inference speedup: 4.29$\times$ speedup | Helped |
| Use MTLA instead of MHA | decoder-only speech translation model · MuST-C En-De tst-COMMON | average GPU memory reduction factor: 6.58 | Helped |
| Use MTLA instead of MLA | decoder-only text summarisation model · XSum test | ROUGE-1: 29.39 -> 29.14 | Hurt |
| Use MTLA instead of MLA | decoder-only text summarisation model · XSum test | ROUGE-2: 9.87 -> 9.79 | Hurt |
| Use MTLA instead of MLA | decoder-only text summarisation model · XSum test | ROUGE-L: 23.78 -> 23.60 | Hurt |
| Use MTLA instead of MHA | decoder-only AMI speech recognition model · AMI IHM test | WER: 12.98 -> 12.66 | Helped |
| Use MTLA instead of MLA | decoder-only SLURP intent classification model · SLURP test | accuracy: 86.93 -> 86.80 | Hurt |
| Increase temporal compression rate from s=2 to s=3 (ablation) | MTLA · MuST-C En-De tst-COMMON | BLEU: 23.28 -> 23.25 | No clear effect |
| Increase temporal compression rate from s=2 to s=4 (ablation) | MTLA · MuST-C En-De tst-COMMON | BLEU: 23.28 -> 23.05 | Hurt |
| Increase temporal compression rate from s=2 to s=3 (ablation) | MTLA · MuST-C En-De tst-COMMON | inference speedup: 4.29$\times$ -> 5.34$\times$ | Helped |
| Increase temporal compression rate from s=2 to s=4 (ablation) | MTLA · MuST-C En-De tst-COMMON | inference speedup: 4.29$\times$ -> 5.78$\times$ | Helped |
Surprising findings
- MTLA with s=2 slightly outperforms MHA in BLEU on MuST-C En-De, suggesting compressing redundant historical KV information may benefit quality.
- MLA with SnapKV loses translation quality (21.76 BLEU), while MTLA retains it.
Limitations
- No LLM-scale pre-training; models are small decoder-only models trained from scratch due to compute limits.
- Results for speech tasks only, with text summarisation and MT in appendix; generalisation to LLMs untested.
- Choosing too large a compression ratio s causes marked performance degradation.
Experimental setup
Datasets: MuST-C v1.0 En-De XSum AMI SLURP WMT14 En-De Long Range Arena
Benchmarks: MuST-C En-De tst-COMMON XSum test AMI IHM test SLURP test Long Range Arena WMT14 En-De
Metrics: BLEU ROUGE-1 ROUGE-2 ROUGE-L WER intent accuracy inference time GPU memory speedup
Baselines: MHA MLA MQA GQA MLA w/ SnapKV Mamba-2 ESPnet-ST TransformerABS ESPnet ESPnet-SLU
Builds on: multi-head latent attention multi-query attention grouped-query attention decoupled rotary position embedding FlashAttention-2
Closest papers by meaning
- Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (NeurIPS 2024 · similarity 0.87)
- Tensor Product Attention Is All You Need (NeurIPS 2025 · similarity 0.86)
- MHA2MLA-VLM: Enabling DeepSeek’s Economical Multi-Head Latent Attention Across Vision-Language Models (AAAI 2026 · similarity 0.86)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.86)
- CHAI: Clustered Head Attention for Efficient LLM Inference (ICML 2024 · similarity 0.86)
- CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.