Research map / Architectures and efficiency / Pruning and efficient tuning
Parameter-Efficient Fine-Tuning with Discrete Fourier Transform
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FourierFT, a PEFT method that learns a small set of spectral coefficients of the weight change ΔW at randomly chosen shared frequency entries and recovers ΔW via inverse DFT, matching or beating LoRA with 8–500x fewer trainable parameters.
Problem
Storage and memory cost of LoRA adapters grows with model width and customization count; the goal is to compress trainable parameters for fine-tuning foundation models even further.
Approach
Treat the weight change ΔW as a spatial-domain matrix, learn only n spectral coefficients at n randomly selected frequency entries (entry locations shared across all layers), and reconstruct ΔW with an inverse 2D DFT, instead of using low-rank factors as in LoRA.
Key results
- On GLUE with RoBERTa Base, FourierFT (0.024M params) averages 85.0 vs LoRA 85.2 (0.3M); with RoBERTa Large, FourierFT (0.048M) averages 88.0 vs LoRA 87.8 (0.8M).
- On instruction tuning with LLaMA2-7B, FourierFT with 0.064M trainable parameters scores 7.49 on Vicuna vs LoRA's 7.35 with 33.5M parameters.
- On ViT-Base image classification, FourierFT with 239K parameters averages 80.29 vs LoRA 77.58 with 581K parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FourierFT instead of LoRA | RoBERTa Base · GLUE CoLA | MCC: 63.4 ± 1.2 -> 63.8 ± 1.6 | Helped |
| Use FourierFT instead of LoRA | RoBERTa Large · GLUE RTE | accuracy: 85.2 ± 1.1 -> 87.4 ± 1.6 | Helped |
| Use FourierFT instead of LoRA | GPT-2 Medium · E2E | BLEU: 68.9 ± .3 -> 69.1 ± .1 | Helped |
| Use FourierFT instead of LoRA | GPT-2 Large · E2E | BLEU: 70.1 ± .3 -> 70.2 ± .2 | No clear effect |
| Use FourierFT instead of LoRA | LLaMA2-7B · MT-Bench | GPT-4 score: 5.20 ± .3 -> 5.18 ± .3 | No clear effect |
| Use FourierFT instead of LoRA | LLaMA2-7B · Vicuna Eval | GPT-4 score: 7.35 ± .6 -> 7.49 ± .4 | Helped |
| Use FourierFT instead of LoRA | ViT Large · 8 image classification datasets (average) | accuracy: 84.94 -> 86.68 | Helped |
| Use FourierFT instead of LoRA | Stable Diffusion 1.5 · Dreambooth subject-driven generation | FID: 245.2 -> 244.9 | No clear effect |
| replace the Fourier basis with a random basis (ablation) | FourierFT · GLUE RTE, RoBERTa Base | accuracy: 79.1 -> 72.7 | Hurt |
| replace the Fourier basis with an orthogonal basis (ablation) | FourierFT · GLUE RTE, RoBERTa Base | accuracy: 79.1 -> 75.6 | Hurt |
| replace the Fourier basis with a random basis (ablation) | FourierFT · GLUE CoLA, RoBERTa Base | MCC: 63.8 -> 58.7 | Hurt |
| replace the Fourier basis with an orthogonal basis (ablation) | FourierFT · GLUE CoLA, RoBERTa Base | MCC: 63.8 -> 60.0 | Hurt |
Surprising findings
- Fourier basis outperforms random and orthogonal bases of the same parameter budget, with random basis losing up to 8.1% on RTE.
- On a synthetic 2D classification task, LoRA (r=1) never reaches 100% accuracy while FourierFT (n=128) does, under equal parameter count.
Limitations
- Performance gains over LoRA are mostly comparable rather than clearly superior; results depend on per-task hyperparameter tuning.
- Frequency-bias entry sampling requires traversing central frequency values, which is inefficient and not made trainable.
- Parameter count of FourierFT grows more slowly with width, but the paper's instruction-tuning comparison uses a single epoch and GPT-4 judging.
Experimental setup
Datasets: GLUE E2E Alpaca MT-Bench Vicuna Eval OxfordPets StanfordCars CIFAR10 DTD EuroSAT FGVC RESISC45
Benchmarks: GLUE E2E MT-Bench Vicuna Eval image classification (8 datasets) FID on DreamBooth
Metrics: accuracy Matthews correlation Pearson correlation BLEU NIST METEOR ROUGE-L CIDEr MT-Bench score Vicuna score
Baselines: LoRA AdaLoRA DyLoRA BitFit Adapter Full fine-tuning Linear probing DreamBooth
Builds on: LoRA AdaLoRA DyLoRA Bitfit
Closest papers by meaning
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.90)
- LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning (ICLR 2025 · similarity 0.90)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.90)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.90)
- F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning (NeurIPS 2025 · similarity 0.89)
- AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.