Research map / Architectures and efficiency / Attention and transformers
LatentLLM: Activation-Aware Transform to Multi-Head Latent Attention
Toshiaki Koike-Akino, Xiangyu Chen, Jing Liu, Ye Wang, Pu (Perry) Wang, Matthew Brand
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use root-covariance pre-conditioner instead of identity/Hessian/L1/L2/covariance in activation-aware SVD (ablation) | ASVD · WikiText2, 20% compression, OPT-1.3B | perplexity: 28.7 -> 21.9 | Improved |
| replace local per-layer SVD with joint attention-aware SVD (LatentLLM) (ablation) | ASVD (RootCov) · WikiText2, 20% compression, OPT-1.3B | perplexity: 21.9 -> 15.8 | Improved |
| joint SVD vs RootCov (ablation) | ASVD (RootCov) · WikiText2, 20% compression, Qwen3-8B | perplexity: 26.0 -> 14.2 | Improved |
| joint SVD vs RootCov at 40% compression (ablation) | ASVD (RootCov) · WikiText2, 40% compression, Qwen3-8B | perplexity: 119.2 -> 53.9 | Improved |
| joint SVD vs RootCov (ablation) | ASVD (RootCov) · ScienceQA avg, 10% compression, LLaVA-7B | accuracy: 60.67 -> 65.76 | Improved |
| joint SVD vs RootCov (ablation) | ASVD (RootCov) · ScienceQA avg, 50% compression, LLaVA-7B | accuracy: 49.30 -> 52.25 | Improved |
| joint SVD vs RootCov (ablation) | ASVD (RootCov) · TextVQA, 30% compression, Qwen2.5-VL-7B | accuracy: 66.31 -> 73.90 | Improved |
| use covariance (not root) pre-conditioner (ablation) | ASVD · ScienceQA avg, 10% compression, LLaVA-7B | accuracy: 60.67 -> 37.42 | Hurt |
| use block-identity junction matrix | SVD low-rank layer · parameter count | parameters: r(d'+d) -> r(d'+d) - r^2 | Improved |
| compress model with LatentLLM | OPT-6.7B · speed and KV cache | KV cache bytes: 5.37G -> 0.57G (0% -> 80% compression) | Improved |
Closest papers by meaning
- Tensor Product Attention Is All You Need (NeurIPS 2025 · similarity 0.87)
- From Attention to Activation: Unraveling the Enigmas of Large Language Models (ICLR 2025 · similarity 0.87)
- Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs (CVPR 2024 · similarity 0.87)
- Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior (NeurIPS 2026 · similarity 0.86)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.86)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
