Research map / Architectures and efficiency / Pruning and efficient tuning
Directional Gradient Projection for Robust Fine-Tuning of Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiGraP, a layer-wise trainable directional gradient projection method that reframes robust fine-tuning as multi-objective optimization, and extends robust fine-tuning evaluation to VQA with near/far OOD splits.
Problem
Robust fine-tuning of foundation models must balance in-distribution accuracy against preserving robustness to distribution shift; existing magnitude-based regularization and bi-level projection methods are computationally heavy, need extensive tuning, and can underfit. Robust fine-tuning has also m
Approach
Treat the L2 distance to pre-trained weights as a second objective and apply PCGrad-style projection of the task gradient onto the orthogonal plane of the regularization gradient when they conflict, with a per-layer projection strength omega in [0,1] that is itself learned via Adam from a normalized gradient signal, so the effective regularization varies across layers and iterations.
Key results
- On DomainNet with MOCO-V3 ResNet50, DiGraP (0.1) reaches 38.01% OOD average vs 37.62% for the next-best FTP, with 82.20% ID on Real.
- On VQAv2 with PaliGemma-3B under LoRA, DiGraP (0.5) gets 87.40% ID and 75.87% near-OOD average, the best among compared methods, while FTP leads on far OOD (41.95 vs 38.77 for DiGraP).
- On DomainNet-oVQA with PaliGemma-3B, DiGraP (0.5) reaches 61.17% OOD average vs 59.76% for LP-FT and 58.59% for FTP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply directional gradient projection during fine-tuning | MOCO-V3 ResNet50 · DomainNet (Real → OOD Avg.) | accuracy: 37.62 -> 38.01 | Helped |
| apply directional gradient projection during fine-tuning | PaliGemma-3B · DomainNet-oVQA (Real → OOD Avg.) | ClipMatch: 59.76 -> 61.17 | Helped |
| apply directional gradient projection during fine-tuning | PaliGemma-3B · VQAv2 (val) | accuracy: 86.29 -> 87.40 | Helped |
| apply directional gradient projection during fine-tuning | PaliGemma-3B · Near OOD VQA (Avg.) | accuracy: 75.08 -> 75.87 | Helped |
| apply directional gradient projection during fine-tuning | PaliGemma-3B · Far OOD VQA (Avg.) | accuracy: 41.95 -> 38.77 | Hurt |
| use trainable rather than fixed projection strength ω (ablation) | DiGraP on DomainNet-oVQA · Real → OOD Avg. | ClipMatch: 60.39 -> 61.17; 60.11 -> 61.17; 60.82 -> 61.17 | Helped |
| vary μ across 0.01, 0.1, 0.5, 1, and 100 (ablation) | DiGraP on DomainNet-oVQA · DomainNet-oVQA | ID and OOD Avg. performance: OOD Avg.: 60.95, 59.96, 61.17, 60.31, 60.40; ID: 92.52, 92.68, 92.72, 92.65, 92.45 | No clear effect |
| vary μ across 0.01, 0.1, 0.5, 1, and 100 (ablation) | DiGraP on VQA · VQA | ID and OOD Avg. performance: OOD Avg.: 63.15, 62.78, 63.50, 63.52, 63.44; ID: 86.91, 86.85, 87.40, 87.08, 87.18 | No clear effect |
| interpolate pre-trained and fine-tuned weights with WiSE-FT | PaliGemma-3B on DomainNet-oVQA · Real → OOD Avg. | ClipMatch: 59.60 -> 52.89 | Hurt |
Surprising findings
- WiSE-FT degrades both ID and OOD performance on the VQA tasks, because zero-shot PaliGemma is far from the fine-tuned model.
- Vanilla fine-tuning of PaliGemma on DomainNet-oVQA improves OOD performance over zero-shot, unlike the image classification setting where fine-tuning degrades OOD.
Limitations
- DiGraP is weaker on far OOD datasets (TextVQA, VizWiz, OK-VQA) than FTP, due to weaker regularization.
- The method is suited to well-pretrained initializations; training from scratch or non-robust initialization is not explored.
- Mahalanobis-based shift categorization relies on features from a fine-tuned model and a single feature choice.
Experimental setup
Datasets: DomainNet DomainNet-oVQA VQAv2 BOSS
Benchmarks: DomainNet DomainNet-oVQA VQAv2 IV-VQA CV-VQA VQA-Rephrasings VQA-CP v2 VQA-CE AdVQA TextVQA VizWiz OK-VQA
Metrics: accuracy OOD average accuracy ID delta (%) OOD delta (%) Mahalanobis distance shift score
Baselines: Vanilla FT L2-SP MARS-SP LP-FT WiSE-FT TPGM FTP Linear Probing
Builds on: PCGrad L2-SP TPGM FTP LoRA
Closest papers by meaning
- Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.88)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.86)
- Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.86)
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.86)
- Foundation Model-oriented Robustness: Robust Image Model Evaluation with Pretrained Models (ICLR 2024 · similarity 0.85)
- Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.