Research map / Continual learning / Continual fine-tuning of LLMs
Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proxy-FDA regularizes fine-tuning of vision foundation models by aligning kNN-graph structure between pre-trained and fine-tuned feature spaces, augmented with learned instance-wise proxies, to reduce concept forgetting.
Problem
Fine-tuning vision foundation models on a downstream task causes concept forgetting on other tasks; existing point-wise weight or feature regularizers are too strict and ignore feature neighborhood structure.
Approach
Align pre-trained and fine-tuned feature distributions via nearest-neighbor graphs (neighbor indices and similarities transferred) using a sigmoid-style loss, and augment the batch with online-learned proxy features generated by a lightweight attention+conv pooling generator to increase diversity.
Key results
- End-to-end CLIP ViT-B/32 across 10 datasets: Proxy-FDA achieves mean forgetting delta Delta_LP of 1.54 vs 1.39 for FDA, 0.86 for LDIFS, 0.29 for L2SP, and -4.37 for naive fine-tuning, with mean accuracy 91.82 close to naive 91.90.
- Few-shot base-to-new prompt tuning (11 datasets, 16 shots): adding Proxy-FDA to CoOp raises average new-class accuracy from 63.22 to 73.67 (Delta_New from -10.99 to -0.55), and to PromptSRC raises harmonic mean from 79.97 to 80.81.
- Continual fine-tuning on SVHN->CIFAR10->RESISC45: Proxy-FDA gives the highest Delta_LP on most tasks with all positive forgetting deltas (e.g., RESISC45 4.22).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace point-wise regularization with feature-neighborhood alignment | CLIP ViT-B/32 end-to-end fine-tuning · 10 classification datasets, mean across other datasets | Δ_LP: 0.86 -> 1.39 | Helped |
| add learned proxies to FDA | CLIP ViT-B/32 end-to-end fine-tuning · 10 classification datasets, mean across other datasets | Δ_LP: 1.39 -> 1.54 | Helped |
| add Proxy-FDA regularization | CoOp prompt tuning · 11 datasets, base-to-new class generalization | Δ_New: -10.99 -> -0.55 | Helped |
| add Proxy-FDA regularization | CoCoOp prompt tuning · 11 datasets, base-to-new class generalization | Δ_New: -2.53 -> 2.22 | Helped |
| add Proxy-FDA regularization | VPT prompt tuning · 11 datasets, base-to-new class generalization | Δ_New: -4.61 -> -0.33 | Helped |
| add Proxy-FDA regularization | MaPLe prompt tuning · 11 datasets, base-to-new class generalization | Δ_New: 0.92 -> 2.91 | Helped |
| add Proxy-FDA regularization | CLIPood prompt tuning · 11 datasets, base-to-new class generalization | Δ_New: 0.28 -> 2.33 | Helped |
| add Proxy-FDA regularization | PromptSRC prompt tuning · 11 datasets, base-to-new class generalization | Δ_New: 1.88 -> 3.23 | Helped |
| replace hard class mining with random class sampling (ablation) | Proxy-FDA batch construction · Few-shot prompt tuning | performance | Hurt |
| replace hard class mining with entropy-based batch sampling (ablation) | Proxy-FDA batch construction · Few-shot prompt tuning | performance | Hurt |
| remove neighbor similarity weights and align neighbor indices only (ablation) | FDA · Few-shot prompt tuning | performance | Hurt |
| replace attention in the proxy generator with convolutional and pooling layers (ablation) | Proxy-FDA proxy generator · Few-shot prompt tuning | performance | Hurt |
Surprising findings
- Naive fine-tuning and LP-FT yield negative average forgetting deltas, while feature regularization methods like FDA achieve positive forward transfer on other datasets.
- OTDD correlates with forgetting better than point-wise L2 feature distance, even where L2 distance is larger than LDIFS.
Limitations
- Relies on batch-level kNN graphs requiring a large batch size and hard class mining, which adds sampling complexity.
- Per-dataset neighborhood size K and regularization weight lambda are tuned per dataset on validation sets.
- Proxies are generated from observed fine-tuning data only; using external data for FDA is left to future work.
Experimental setup
Datasets: Stanford Cars CIFAR-10 CIFAR-100 DTD EuroSAT GTSRB MNIST RESISC45 SVHN ImageNet Caltech101 OxfordPets
Benchmarks: ImageNet CIFAR-10 CIFAR-100 DTD EuroSAT GTSRB MNIST RESISC45 SVHN Stanford Cars Caltech101 OxfordPets
Metrics: linear probe accuracy forgetting delta (Delta_LP) base/new harmonic mean accuracy CIDEr-D CLIPScore Ref-CLIPScore VQA accuracy OTDD L2 feature distance top-1 accuracy
Baselines: Naive End-to-End LP-FT L2SP LDIFS FDA CLIPood PromptSRC CoOp CoCoOp VPT MaPLe WiSE-FT
Builds on: LDIFS L2SP LP-FT OTDD PromptSRC
Closest papers by meaning
- Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers (NeurIPS 2025 · similarity 0.85)
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.85)
- Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.85)
- Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.85)
- Homology Consistency Constrained Efficient Tuning for Vision-Language Models (NeurIPS 2024 · similarity 0.85)
- DiVE-k: DIFFERENTIAL VISUAL REASONING FOR FINE-GRAINED IMAGE RECOGNITION (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.