Research map / Architectures and efficiency / Pruning and efficient tuning
LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoCA, an iDCT-based PEFT method that learns both coefficients and locations of DCT frequency components, using finite-difference gradients for discrete locations, and compares it theoretically to low-rank and Fourier-domain adaptation.
Problem
Low-rank adaptation constrains the hypothesis space of weight updates, and FourierFT's random frequency-location selection lacks theoretical grounding relative to low-rank methods; PEFT needs more expressive yet parameter-efficient weight reparameterizations.
Approach
Reparameterize the weight update as a sparse set of learnable DCT coefficients placed by iDCT, and learn their discrete locations via central finite-difference gradient estimates, alternating coefficient and location optimization before freezing locations.
Key results
- On GLUE with RoBERTa-base, LoCA reaches 86.0 average with 0.024M parameters, versus FourierFT 85.4 at the same budget and LoRA 86.1 at 0.3M.
- On ViT-base image classification, LoCA with 72K parameters averages 85.71 versus FourierFT 84.23 at the same budget; at 239K it reaches 86.66 versus FourierFT's 86.02 at 239K.
- On GPT-2 large E2E NLG, LoCA with 0.072M parameters achieves the best BLEU (70.4), METEOR (47.2), and ROUGE-L (72.1) among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace random frequency-location selection with learned location selection | FourierFT-style frequency adaptation · GLUE (RoBERTa-base), All Avg. | average: 85.4 -> 86.0 | Helped |
| replace random frequency-location selection with learned location selection | FourierFT-style frequency adaptation · E2E (GPT-2 Large) | BLEU: 70.2±.2 -> 70.4±.2 | Helped |
| replace random frequency-location selection with learned location selection | FourierFT-style frequency adaptation · Average (ViT-base, 239K parameters) | accuracy: 86.02 -> 86.66 | Helped |
| optimize coefficients only with randomly initialized locations (ablation) | LoCA · QQP (RoBERTa-base) | accuracy: 88.7 -> 87.7 | Hurt |
| alternate coefficient and location optimization throughout training instead of fixing locations after the alternating phase (ablation) | LoCA · QQP (RoBERTa-base) | accuracy: 88.7 -> 86.5 | Hurt |
| jointly optimize locations and coefficients in each step (ablation) | LoCA · QQP (RoBERTa-base) | accuracy: 88.7 -> 88.0 | Hurt |
| use forward-difference gradient approximation instead of central difference (ablation) | LoCA · QQP (RoBERTa-base) | accuracy: 88.7 -> 88.6 | Hurt |
| use backward-difference gradient approximation instead of central difference (ablation) | LoCA · QQP (RoBERTa-base) | accuracy: 88.7 -> 88.4 | Hurt |
| increase the frequency-component budget from 72K to 239K parameters | LoCA on ViT-base · Average | accuracy: 85.71 -> 86.66 | Helped |
Surprising findings
- Randomly selected Fourier-domain locations (FourierFT-style) have lower expected expressivity than low-rank decomposition in the theoretical analysis, while carefully selected components can exceed it.
Limitations
- Location selection relies on a finite-difference gradient surrogate, and alternating optimization can be unstable if run throughout training.
- Gains over LoRA and FourierFT are often small (e.g., under 1 point on GLUE), and FourierFT needed extensive tuning, complicating fair comparison.
- Theoretical comparison assumes i.i.d. Gaussian weight updates, which is justified empirically and via assumptions rather than proven.
Experimental setup
Datasets: GLUE E2E NLG Challenge Alpaca-52K OxfordPets StanfordCars CIFAR10 DTD EuroSAT FGVC RESISC45 CIFAR100
Benchmarks: GLUE E2E NLG MT-Bench Vicuna OxfordPets StanfordCars CIFAR10 DTD EuroSAT FGVC RESISC45 CIFAR100
Metrics: accuracy Matthews correlation coefficient Pearson correlation BLEU NIST METEOR ROUGE-L CIDEr GPT-4 judge score (1-10)
Baselines: Full fine-tuning BitFit Adapter LoRA AdaLoRA VeRA DoRA FourierFT Linear Probing
Builds on: LoRA FourierFT straight-through estimator VeRA DoRA
Closest papers by meaning
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform (ICML 2024 · similarity 0.90)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.89)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.88)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.88)
- C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models (NeurIPS 2025 · similarity 0.88)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.