Research map / Architectures and efficiency / Attention and transformers

Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs

CVPR 2024 · Architectures and efficiency › Attention and transformers · approach: Long-context KV and memory attention · 15 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LoRA-Sparse, which adds low-rank query/key projections trained with an order-mimic loss to predict top-k attention entries, enabling about 50% sparse self-attention in LLaMA and LLaVA-1.5 that matches or slightly exceeds full-attention baselines.

Problem

The quadratic cost of self-attention in LLMs and multimodal LLMs, where computing the full attention map is needed to identify important query-key pairs for sparse attention, and where sparse attention is hard to adapt to pre-trained weights.

Approach

Approximate the attention map in a low-rank space with two extra linear projections, select top-s% keys per query from that approximation, and train the projections with an order-mimic loss (soft-margin ordering loss plus a sigmoid magnitude loss) so that the predicted ranking matches the full attention ranking, leaving the pre-trained weights unchanged.

Key results

• On LLaVA-1.5 7B, the low-rank sparse attention at 50% selection ratio reaches 68.4 ScienceQA-Image accuracy versus 67.6 for the full-attention baseline.
• On LLaMA 7B with Alpaca SFT, sparse attention at 0.5 ratio improves arc-challenge from 45.6 to 47.3 while saving 50% self-attention computation, with 13 benchmarks roughly matched.
• Long-context PG19 perplexity stays around 10.8-11.0 for LLaMA 7B from 4096 to 16384 context, while the baseline LLaMA is limited to 2048 context.

Measured effects reported in the paper

Surprising findings

• Training-free top-s% sparsification of full attention already improved ScienceQA accuracy by about 0.9% over the baseline without fine-tuning, suggesting redundant or harmful attention in LLMs.
• Replacing sparse attention with deformable interpolation-based attention drops LLaMA benchmark accuracy drastically (e.g., arc-easy 73.4 to 24.4).

Limitations

• Sparse attention offers limited GPU acceleration at lower sparsity levels.
• Gains over baseline are small (e.g., 0.8 points on ScienceQA, mostly within noise), and the sparse benefit depends on choosing the selection ratio.

Experimental setup

Datasets: RedPajama-V2 Alpaca LLaVA-1.5 665K instruction tuning mix PG19

Benchmarks: arc-easy arc-challenge openbookqa piqa winogrande record mnli mrpc rte wsc wic sst

Metrics: accuracy perplexity

Baselines: LLaMA 7B full attention LLaVA-1.5 7B BLIP-2 InstructBLIP Qwen-VL Deformable attention

Builds on: LLaVA-1.5 LLaMA Alpaca hard-hinge loss position interpolation

Closest papers by meaning

• LoLCATs: On Low-Rank Linearizing of Large Language Models (ICLR 2025 · similarity 0.90)
• Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.90)
• Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.89)
• Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.89)
• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.89)
• Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.88)