Research map / Vision-language and multimodal / CLIP and zero-shot
PROFIT: A Specialized Optimizer for Deep Fine Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PROFIT, a modular optimizer wrapper that fine-tunes converged models on new but proximal data by orthogonalizing the fine-tuning gradient against the displacement from a reference state, reducing catastrophic forgetting.
Problem
Fine-tuning converged models on new tasks or datasets causes catastrophic forgetting of old performance, and standard optimizers (SGD, Adam, AdamW) ignore that the starting point is a good converged state.
Approach
Treat fine-tuning as temporal multitask learning: compute the displacement Δ from a saved reference state after reference SGD steps, then project the new-task gradient orthogonal to Δ (PCGrad-style, applied only to the new gradient) and take the main optimizer step from the restored state.
Key results
- On CIFAR100 fine-tuning, PROFIT with ViT-Tiny reaches 58.53% CIFAR10 / 62.20% CIFAR100 versus Adam's 55.64% / 61.35%.
- On DriveLM with LLaMA-Adapter-v2, PROFIT raises accuracy from 62.21% to 67.88% and final score from 56.98 to 59.16 over AdamW.
- On VTAB-1K, naive PROFIT performs poorly (e.g., 12.6% on Clevr-Count), but PROFIT with a 10-epoch AdamW warmup beats full AdamW fine-tuning on most tasks; the warmup variant is not the best on all 19 tasks, and it trails VPT, LoRA and NOAH on several.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PROFIT instead of full-model fine-tuning | 2D toy MLP fine-tuning · 2D toy, original data | error: 0.705 -> 0.046 | Helped |
| Use PROFIT instead of Adam fine-tuning | ViT-Tiny · CIFAR100 after CIFAR10 pretraining; CIFAR10 retention | accuracy (%): 56.00 -> 58.53 | Helped |
| Use PROFIT instead of Adam fine-tuning | ViT-Tiny · CIFAR100 after CIFAR10 pretraining | accuracy (%): 58.99 -> 62.20 | Helped |
| Use PROFIT instead of AdamW fine-tuning | LLaMA-Adapter-v2 · DriveLM validation | Accuracy (%): 62.21 -> 67.88 | Helped |
| Use PROFIT instead of AdamW fine-tuning | LLaMA-Adapter-v2 · DriveLM validation | Final score: 56.98 -> 59.16 | Helped |
| Use PROFIT instead of full-model fine-tuning | Wayformer car-to-pedestrian transfer · Waymo WOMD, pedestrian target | ADE@3s (m): 0.214 -> 0.203 | Helped |
| Add AdamW warm-up before PROFIT (ablation) | ViT-Base VTAB-1K adaptation · VTAB-1K, Clevr-Count | accuracy: 12.6 -> 80.5 | Helped |
| Increase n_ref from 1 to 5 (ablation) | PROFIT with ViT-Small · CIFAR100 | accuracy (%): 65.44% -> 63.75% | Hurt |
| Increase n_ref from 1 to 5 (ablation) | PROFIT with ViT-Small · CIFAR10 after CIFAR100 fine-tuning | accuracy (%): 59.02% -> 56.54% | Hurt |
| Set lambda_main/lambda_ref to 10 instead of 100 (ablation) | PROFIT with ViT-Small · CIFAR100 | accuracy (%): 65.44% -> 66.29% | Helped |
| Set lambda_main/lambda_ref to 10000 instead of 10 (ablation) | PROFIT on DriveLM · DriveLM validation | Final score: 58.54 -> 59.16 | Helped |
| Train PROFIT from scratch rather than from converged weights (ablation) | ResNet-18 · CIFAR100 | accuracy: null | Hurt |
Surprising findings
- Naive PROFIT trained from scratch on CIFAR100 reaches only 1.05% accuracy, and on VTAB-1K without warmup it performs very poorly, showing it fails outside proximal settings.
- The reference step size ratio (lambda_main/lambda_ref) has inconsistent effects across backbones, with ViT-Small peaking at 100 and ViT-Tiny at 1000.
Limitations
- Requires proximal distributions between pre-training and fine-tuning data; fails on distribution shifts such as VTAB-1K without warmup.
- Extra compute (n_ref+1 batches per step) and memory overhead from storing reference weights and a second optimizer.
- Requires a converged starting model; training from scratch performs at chance.
Experimental setup
Datasets: CIFAR10 CIFAR100 VTAB-1K DriveLM-nuScenes Waymo Open Motion Dataset ImageNet
Benchmarks: CIFAR100 CIFAR10 VTAB-1K DriveLM Waymo Open Motion Dataset
Metrics: accuracy BLEU-4 ROUGE-L CIDEr GPT score match score final score ADE FDE
Baselines: SGD Adam Lookahead Adan AdamW full fine-tuning Linear probe VPT LoRA NOAH ConvAdapter
Builds on: PCGrad Learning Without Forgetting AdamW VPT
Closest papers by meaning
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.87)
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.87)
- Pay Attention to Small Weights (NeurIPS 2025 · similarity 0.87)
- Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models (ICML 2025 · similarity 0.86)
- LEAD: Exploring Logit Space Evolution for Model Selection (CVPR 2024 · similarity 0.86)
- ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.