Research map / Vision-language and multimodal / CLIP and zero-shot
Visual Fourier Prompt Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual Fourier Prompt Tuning (VFPT), which applies a 2D FFT to a fraction of visual prompt embeddings in a frozen ViT/Swin to bridge spatial and frequency information for parameter-efficient adaptation across datasets with varying disparity.
Problem
Visual prompt tuning degrades significantly when the pretraining and finetuning datasets differ substantially (large data disparity), limiting its generality for parameter-efficient adaptation of vision Transformers.
Approach
Replace a fraction of learnable visual prompt embeddings with their 2D FFT (over sequence and hidden dimensions, keeping the real part) before concatenating them with the remaining spatial prompts, so prompts carry both spatial and frequency-domain information at negligible extra cost.
Key results
- VFPT reaches 73.20% mean accuracy on VTAB-1k with 0.66% tuned parameters on ViT-Base/16, versus 72.32% for AdaptFormer and 69.43% for VPT-Deep.
- On VTAB-1k Structured, VFPT achieves 60.19% vs 54.98% for VPT-Deep and 47.64% for full fine-tuning.
- VFPT outperforms full fine-tuning in 22 of 24 instances on ViT-Base/16 with fewer trainable parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Fourier prompts to visual prompt tuning | VPT-D on ViT-Base/16 with supervised ImageNet-21k · VTAB-1k Structured | accuracy: 54.98% -> 60.19% | Helped |
| add Fourier prompts to visual prompt tuning | VPT-D on ViT-Base/16 with supervised ImageNet-21k · VTAB-1k Natural | accuracy: 78.48% -> 81.35% | Helped |
| add Fourier prompts to visual prompt tuning | VPT-D on ViT-Base/16 with supervised ImageNet-21k · VTAB-1k Specialized | accuracy: 82.43% -> 84.93% | Mixed |
| replace FFT with a fixed linear transform (ablation) | VFPT on ViT-Base/16 · VTAB-1k Natural and Specialized | accuracy: 81.35% -> 80.98%; 84.93% -> 84.02% | Hurt |
| replace FFT with a learnable linear transform (ablation) | VFPT on ViT-Base/16 · VTAB-1k Natural and Specialized | accuracy: 81.35% -> 80.54%; 84.93% -> 82.64% | Hurt |
| add Fourier domain adaptation to FFT prompts (ablation) | VFPT on ViT-Base/16 · VTAB-1k Natural and Specialized | accuracy: 81.35% -> 80.90%; 84.93% -> 84.03% | No clear effect |
| apply FFT on both sequence and hidden dimensions rather than one dimension alone (ablation) | VFPT on ViT-Base/16 · VTAB-1k Natural and Specialized | accuracy: 80.88% -> 81.35%; 80.74% -> 81.35% | Helped |
| append Fourier prompts instead of prepending (ablation) | VFPT on ViT-Base/16 · VTAB-1k Natural and Specialized | accuracy: 81.35% -> 81.02%; 84.93% -> 83.80% | No clear effect |
| randomly place Fourier prompts in each layer rather than prepend them (ablation) | VFPT on ViT-Base/16 · VTAB-1k Natural and Specialized | accuracy: 81.35% -> 78.62%; 84.93% -> 82.47% | Hurt |
| use Fourier prompts in all layers rather than only selected layers (ablation) | VFPT on ViT-Base/16 · VTAB-1k Natural and Specialized | accuracy: 80.48%, 80.79%, 80.83% -> 81.35%; 83.73%, 84.34%, 83.93% -> 84.93% | Helped |
| increase the percentage of Fourier prompts (ablation) | VFPT on VTAB-1k · VTAB-1k Specialized and Structured | accuracy | Helped |
| use half of the visual prompts as Fourier prompts (ablation) | VFPT on VTAB-1k Natural · VTAB-1k Natural | accuracy | Helped |
Surprising findings
- Using a fixed FFT (no learnable parameters) outperforms learnable linear-layer transforms (LLL) in the transform ablation.
- Applying a 2D FFT across both sequence and hidden dimensions gives a synergistic gain over either dimension alone.
Limitations
- Fourier percentage must be tuned per task, and the best choice varies across datasets.
- Gains on the Natural group are smaller and the method is evaluated only on image classification benchmarks.
Experimental setup
Datasets: VTAB-1k FGVC ImageNet-21k
Benchmarks: VTAB-1k FGVC
Metrics: mean test accuracy number of wins vs full fine-tuning tuned/total parameter percentage
Baselines: Full fine-tuning Linear probing Partial-1 MLP-3 Sidetune Bias Adapter LoRA AdaptFormer ARC VPT-Shallow VPT-Deep
Builds on: Visual Prompt Tuning (VPT) Adapter LoRA AdaptFormer E2VPT
Closest papers by meaning
- DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers (CVPR 2025 · similarity 0.89)
- WST: Wavelet-Based Multi-scale Tuning for Visual Transfer Learning (AAAI 2025 · similarity 0.88)
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.88)
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.88)
- LION: Implicit Vision Prompt Tuning (AAAI 2024 · similarity 0.87)
- Selective Visual Prompting in Vision Mamba (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.