Research map / Learning theory and algorithms / Matrix and tensor methods
COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COALA, an inversion-free, Gram-matrix-free, regularized context-aware low-rank approximation via TSQR and augmented activations, applied to LLM compression and LoRA initialization.
Problem
Context-aware (activation-weighted) low-rank approximation of neural network weights relies on Gram matrix formation and inversion, which is numerically unstable, memory-heavy for large calibration sets, and ill-posed with limited data.
Approach
Solve the weighted low-rank problem via a QR (TSQR) factorization of X^T and an SVD of W R^T, giving W' = U_r U_r^T W with no inversion or Gram matrix; add ridge-style regularization by augmenting X with sqrt(mu) I, and show convergence of the regularized solution as mu goes to 0.
Key results
- On LLaMA-3.2-1B-Instruct at 90% compression (fp16), COALA with regularization reaches higher accuracy than SVD-LLM and ASVD on most commonsense tasks (e.g., HSwag 36.6 vs 34.6 for SVD-LLM).
- On Mistral-7B at 70% compression, COALA achieves ARC-E 63.63 and ARC-C 35.49, versus 58.25 and 30.72 for SVD-LLM.
- Runtime on LLaMA3-8B with 128 samples drops from 3624.88 s (SVD-LLM) and 4210.5 s (SVD-LLM V2) to 1811.0 s with COALA on a single A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add regularization (μ) to COALA (ablation) | COALA compression of LLaMA-3.2-1B-Instruct · Commonsense reasoning, 90% compression | accuracy: 57.6_{±0.7} -> 59.0_{±0.7} (boolQ); 60.9_{±1.0} -> 62.8_{±1.0} (PIQA); 53.2_{±1.3} -> 54.0_{±1.3} (WiNoG); 34.6_{±0.5} - | Helped |
| Use COALA instead of SVD-LLM or SVD-LLM V2 for the QR-based compression computation | LLaMA3-1B, 64 samples · Compression computation | time (s): 273.93 ± 22.12 -> 196.34 ± 6.48 (SVD-LLM); 404.88 ± 5.49 -> 196.34 ± 6.48 (SVD-LLM V2) | Helped |
| Use COALA instead of SVD-LLM or SVD-LLM V2 for the QR-based compression computation | LLaMA3-8B, 128 samples · Compression computation | time (s): 3624.88 ± 512.4 -> 1811.0 ± 15.6 (SVD-LLM); 4210.5 ± 63.3 -> 1811.0 ± 15.6 (SVD-LLM V2) | Helped |
| Use COALA α=1 initialization instead of LoRA | LLaMA3-1B-Instruct fine-tuning, rank 8 · Commonsense reasoning | average accuracy: 75.0 -> 75.5 | Helped |
| Use COALA α=1 initialization instead of PiSSA | LLaMA3-1B-Instruct fine-tuning, rank 8 · Commonsense reasoning | average accuracy: 75.4 -> 75.5 | No clear effect |
| Use COALA α=1 initialization instead of CorDA | LLaMA3-1B-Instruct fine-tuning, rank 8 · Commonsense reasoning | average accuracy: 60.9 -> 75.5 | Helped |
| Replace COALA α=2 initialization with robustified COALA α=2 rather than CorDA | LLaMA3-1B-Instruct fine-tuning, rank 8 · Commonsense reasoning | average accuracy: 60.9 -> 75.4 | Helped |
Surprising findings
- Gram-matrix-based methods (SVD-LLM, SVD-LLM V2) show large errors already during construction of the approximation, independent of rank, due to loss of precision when squaring singular values.
- Using PyTorch's default gesvdj GPU SVD produces noticeably larger error, so the gesvd method was used instead.
Limitations
- Limitations section is very brief and only states that efficiency is limited to tasks where weighted approximation works well.
- Regularization parameter mu requires selection via lambda; optimal lambda reported as stable only in a range from 1 to 10.
- Convergence bounds depend on the singular value gap, which can be small, yielding slow convergence.
Experimental setup
Datasets: WikiText2 commonsense reasoning dataset boolQ OpenbookQA WinoGrande HellaSwag ARC-e ARC-c PIQA MMLU SIQA
Benchmarks: boolQ PIQA WinoGrande HellaSwag ARC-E ARC-C OpenBookQA MMLU SIQA
Metrics: accuracy spectral norm error Frobenius norm error runtime (s)
Baselines: ASVD SVD-LLM SVD-LLM V2 SoLA SliceGPT FLAP LoRA PiSSA CorDA
Builds on: SVD-LLM CorDA PiSSA ASVD
Closest papers by meaning
- From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications (ICML 2025 · similarity 0.88)
- Approximation to Smooth Functions by Low-Rank Swish Networks (ICML 2025 · similarity 0.87)
- Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.86)
- Geometry-aware training of factorized layers in tensor Tucker format (NeurIPS 2024 · similarity 0.86)
- Generalization Bounds for Rank-sparse Neural Networks (NeurIPS 2025 · similarity 0.86)
- Flora: Low-Rank Adapters Are Secretly Gradient Compressors (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.