Research map / Detection and segmentation / Open-set and few-shot detection
5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mona, an adapter-based delta-tuning method with scaled LayerNorm input optimization and multi-scale depth-wise convolution filters, which surpasses full fine-tuning on COCO, ADE20K, VOC, DOTA/STAR and classification while training under 5% of backbone parameters.
Problem
Existing delta-tuning (parameter-efficient) methods for visual tasks fail to exceed full fine-tuning on challenging recognition tasks such as instance and semantic segmentation.
Approach
Replace linear adapter filters (designed for language) with vision-friendly multi-scale depth-wise convolution filters (3x3, 5x5, 7x7) plus a point-wise aggregation, and add a scaled LayerNorm at the adapter input to adjust the input distribution from frozen layers.
Key results
- Mona with 4.67% trainable backbone parameters reaches 53.4 APbox on COCO (Cascade Mask R-CNN, Swin-B), +1.0 over full fine-tuning (52.4).
- On ADE20K (Swin-L, UperNet), Mona achieves 51.36 mIoU vs 51.18 for full fine-tuning, the only adapter method above full tuning on that task.
- Mona is the only delta-tuning method that outperforms full fine-tuning on all reported tasks, including Pascal VOC (+3.6 APbox) and DOTA (78.44 vs 78.31).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace full fine-tuning with Mona | Swin-B · COCO | APBox: 52.40 % -> 53.40 % | Helped |
| Replace full fine-tuning with Mona | Swin-B · COCO | APMask: 45.10 % -> 46.00 % | Helped |
| Replace full fine-tuning with Mona | Swin-L · Pascal VOC | APBox: 83.70 % -> 87.30 % | Helped |
| Replace full fine-tuning with Mona | Swin-L · ADE20K | mIoU: 51.18 % -> 51.36 % | Helped |
| Replace full fine-tuning with Mona | Swin-B · DOTA-v1.0 (Oriented R-CNN) | APBox: 78.31 % -> 78.44 % | Helped |
| Replace full fine-tuning with Mona | Swin-B · STAR (Oriented R-CNN) | APBox: 38.63 % -> 39.45 % | Helped |
| Replace full fine-tuning with Mona | Swin-L · Average (Flowers102, OxfordPets, VOC2007) | top-1 accuracy: 92.7876 -> 94.0413 | Helped |
| Use intermediate dimension 64 instead of 32 or 128 (ablation) | Mona on Swin-L · Pascal VOC | APBox: 86.8 %; 87.3 %; 87.1 % | Mixed |
| Add scaled normalization (ablation) | Mona on Swin-L · Pascal VOC | APBox: 86.9 % -> 87.3 % | Helped |
| Vary the multi-cognitive convolution kernel setting (ablation) | Mona on Swin-L · Pascal VOC | APBox: 86.9 %; 87.0 %; 86.7 %; 86.8 %; 86.9 %; 87.1 %; 87.3 % | Mixed |
| Scale the backbone from Swin-T to Swin-L (ablation) | Mona · Pascal VOC | APBox: 83.5 % -> 86.5 % -> 87.3 % | Helped |
Surprising findings
- Most delta-tuning baselines outperform full fine-tuning on Pascal VOC, which the authors attribute to overfitting of full tuning on a small dataset.
- LoRA performs poorly on visual dense prediction tasks despite strong NLP results.
Limitations
- Adapter modules add a small inference cost, unlike reparameterization methods such as LoRA.
- Performance gains on ADE20K are small (+0.18 mIoU over full fine-tuning).
Experimental setup
Datasets: MS COCO ADE20K Pascal VOC 0712 DOTA-v1.0 STAR Oxford 102 Flowers Oxford-IIIT Pet VOC 2007 classification ImageNet-22k
Benchmarks: COCO ADE20K Pascal VOC DOTA-v1.0 STAR Oxford 102 Flowers Oxford-IIIT Pet VOC 2007 classification
Metrics: APbox APmask mIoU top-1 accuracy top-5 accuracy
Baselines: FULL FIXED BitFit Norm Tuning Partial-1 Adapter LoRA AdaptFormer LoRand
Builds on: AdaptFormer LoRA LoRand Adapter (Houlsby) Swin Transformer
Closest papers by meaning
- Minimal Interaction Seperated Tuning: A New Paradigm for Visual Adaptation (CVPR 2025 · similarity 0.88)
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.87)
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.86)
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.86)
- Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering (ICML 2025 · similarity 0.85)
- Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.