Research map / Detection and segmentation / CNN features
One Step Learning, One Step Review
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OLOR, an optimizer-integrated fine-tuning method that adds a weight-rollback term toward pre-trained weights at each step, with a layer-wise penalty decay, to reduce knowledge forgetting in visual fine-tuning.
Problem
Full fine-tuning of pre-trained vision models causes knowledge forgetting, while existing remedies (replay, L2-SP, parameter isolation) are inefficient, incompatible with adaptive optimizers, or add parameters.
Approach
Incorporate a weight rollback term (pulling weights back toward pre-trained weights via a discrepancy term) directly into the SGD/Adam update each step, with per-layer penalty factors that decay with depth and a power exponent controlling decay rate.
Key results
- On PACS with ViT-B, OLOR-Adam reaches 94.38% vs 87.79% for full fine-tuning and 87.74% for L2-SP.
- On COCO2017 with Mask R-CNN (ConvNeXt-B), OLOR achieves 41.10 box AP and 36.90 mask AP vs 40.20 and 36.00 for full fine-tuning.
- On ADE20K with UperNet (ViT-B), OLOR reaches 44.62 mIoU vs 43.65 for full fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OLOR-Adam instead of the strongest listed baseline | ViT-B fine-tuning · CIFAR-100, SVHN, CUB-200, Stanford Cars, Places-LT, IP102, OfficeHome, PACS | accuracy: +6.47% in accuracy; +2.57% and +7.38% on the two OOD datasets | Helped |
| Use OLOR-SGD instead of the strongest listed baseline | ConvNeXt-B fine-tuning · CIFAR-100, SVHN, CUB-200, Stanford Cars, Places-LT, IP102, OfficeHome, PACS | accuracy | Helped |
| Use OLOR instead of Full fine-tuning | Mask R-CNN with ConvNeXt-B · COCO2017 | Bbox_m: 40.20 -> 41.10 | Helped |
| Use OLOR instead of Full fine-tuning | Mask R-CNN with ConvNeXt-B · COCO2017 | Segm_m: 36.00 -> 36.90 | Helped |
| Use OLOR instead of Full fine-tuning | UperNet with ViT-B · ADE20K | IOU_m: 43.65 -> 44.62 | Helped |
| Use OLOR instead of the strongest competing method | ViT-B with different pre-trained weights · PACS | accuracy: 5.08%, 0.64%, and 3.47% | Helped |
| Apply weight rollback during fine-tuning | ViT-B with Adam on CIFAR-100 · CIFAR-100 | top1 accuracy | Helped |
| Apply weight rollback during fine-tuning | ViT-B with Adam on PACS · PACS | weight discrepancy | Helped |
| Use weight rollback for zero-shot reviewing | fine-tuned PACS model · PACS | reviewing performance | Helped |
Surprising findings
- Existing weight decay can push weights toward the negative current gradient rather than zero, and can increase the weight magnitude under some conditions, motivating the rollback design.
Limitations
- Slower loss convergence at first due to weight restriction, per the paper's analysis.
- Hyper-parameters (max rollback level, power) must be searched per task and per backbone.
Experimental setup
Datasets: CIFAR-100 SVHN CUB-200 Stanford Cars Places-LT IP102 OfficeHome PACS COCO2017 ADE20K ImageNet-1K ImageNet-21K
Benchmarks: CIFAR-100 SVHN CUB-200 Stanford Cars Places-LT IP102 OfficeHome PACS COCO2017 ADE20K
Metrics: top-1 accuracy box AP mask AP mIoU
Baselines: Full fine-tuning Linear probing L2-SP VPT
Builds on: L2-SP weight decay Adam VPT
Closest papers by meaning
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.87)
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.85)
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.85)
- 5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks (CVPR 2025 · similarity 0.84)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.84)
- PROFIT: A Specialized Optimizer for Deep Fine Tuning (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.