Research map / Architectures and efficiency / Pruning and efficient tuning
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COAP, a low-rank gradient projection optimizer wrapper that updates projection matrices via a correlation-aware SGD objective plus occasional low-cost SVD, cutting optimizer memory and projection overhead for pre-training and fine-tuning.
Problem
Optimizer state memory dominates training memory for large vision, language, and multimodal models; existing low-rank gradient projection methods (GaLore, Flora) lack inter-projection correlation and incur high SVD or random-projection costs.
Approach
Update the projection matrix by gradient descent on an objective combining gradient reconstruction error (MSE) with directional consistency of the back-projected first moment (cosine similarity), so each new subspace stays correlated with the previous one; occasionally recalibrate with a cheap SVD computed in the reduced QR-projected space (O(mr^2) instead of O(mn^2)).
Key results
- On LLaMA-1B pre-training on C4, COAP cuts optimizer memory by 61% with +2% training time and matches AdamW perplexity (15.56).
- On LLaVA-v1.5-7B fine-tuning on ScienceQA, 8-bit COAP reduces optimizer memory by 81% and trains 4x faster than 8-bit GaLore with higher accuracy (92.0 vs 90.7).
- On ControlNet-SDXL, COAP with rank ratio 2 reaches mAP 73.4 at 80K steps versus Adafactor's 72.7, using 29% less optimizer memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COAP instead of GaLore with AdamW | LDM · ImageNet-1K validation | FID: 17.8 -> 16.2 | Helped |
| Use COAP instead of GaLore with Adafactor | LDM · ImageNet-1K validation | FID: 23.3 -> 18.3 | Helped |
| Use COAP with AdamW rather than full-rank AdamW | SiT-XL/2 · ImageNet-1K | FID: 1.9 -> 2.2 | Hurt |
| Use COAP instead of GaLore with Adafactor | SiT-XL/2 · ImageNet-1K | FID: 3.0 -> 2.1 | Helped |
| Use COAP instead of GaLore | ControlNet-XL · Human-pose conditioning, 80K steps | mAP: 72.7 -> 73.4 | Helped |
| Use 8-bit COAP instead of 8-bit GaLore | ControlNet-XL · Human-pose conditioning, 80K steps | mAP: 19.7 -> 71.5 | Hurt |
| Use 8-bit COAP instead of 8-bit Adam | LLaMA-7B · C4, 80K steps | PPL: 15.39 -> 15.28 | Helped |
| Use COAP instead of GaLore | LLaVA-v1.5-7B · ScienceQA | IMG-Acc (%): 91.1 -> 92.3 | Helped |
| Use 8-bit COAP instead of 8-bit GaLore | LLaVA-v1.5-7B · ScienceQA | IMG-Acc (%): 90.7 -> 92.0 | Helped |
| Add Occasional Low-cost SVD (Eqn. 7) (ablation) | COAP rank 192 on DeiT-Base · CIFAR-100 pre-training | Top-1 accuracy: 63.28 -> 70.39 | Helped |
| Add the direction term to Eqn. 6 (ablation) | COAP rank 192 on DeiT-Base · CIFAR-100 pre-training | Top-1 accuracy: 63.31 -> 70.39 | Helped |
| Add the reconstruction term to Eqn. 6 (ablation) | COAP rank 192 on DeiT-Base · CIFAR-100 pre-training | Top-1 accuracy: 63.27 -> 70.39 | Helped |
Surprising findings
- Flora fails to converge on LLaVA fine-tuning (66.6% accuracy vs 82.4% DeepSpeed baseline), and GaLore fails to converge on ControlNet at the same compression rate, showing that naive low-rank projection can break training.
- On the DeiT-Base ablation, COAP sometimes produces a cumulative effective update exceeding that of Adam.
Limitations
- Method requires tuning of rank, update interval Tu and recalibration interval lambda; sensitive at high compression (rank 64)
- Gains from the occasional SVD are minimal in fine-tuning settings
- Evaluation on a limited set of models, with some experiments on a single A100 GPU and limited hyperparameter exploration
Experimental setup
Datasets: ImageNet-1K C4 ScienceQA CIFAR-100 in-house human pose images
Benchmarks: ImageNet-1K FID C4 perplexity ScienceQA accuracy CIFAR-100 top-1 accuracy COCO-style pose mAP for ControlNet
Metrics: FID perplexity accuracy mAP training time overhead optimizer memory
Baselines: AdamW Adafactor GaLore Flora LoRA ReLoRA 8-bit Adam 8-bit GaLore DeepSpeed CPU-offload
Builds on: GaLore Flora LoRA Adam Adafactor
Closest papers by meaning
- CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models (ICML 2025 · similarity 0.84)
- Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.84)
- DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity (NeurIPS 2024 · similarity 0.84)
- Frozen Language Models Are Gradient Coherence Rectifiers in Vision Transformers (AAAI 2025 · similarity 0.83)
- Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappings (ICLR 2024 · similarity 0.83)
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.