Research map / Detection and segmentation / CNN features

VFM-Adapter: Adapting Visual Foundation Models for Dense Prediction with Dynamic Hybrid Operation Mapping

AAAI 2025 · Detection and segmentation › CNN features · approach: Convolution-transformer backbones · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VFM-Adapter, a parameter-efficient adapter for dense prediction that combines hybrid linear/convolutional operation mapping with input-conditioned hypernetwork-generated convolution kernels, matching or beating full fine-tuning on COCO and ADE20K with about 3% trainable parameters on SAM-Base.

Problem

Parameter-efficient fine-tuning of large vision foundation models for dense prediction tasks (object detection, instance segmentation, semantic segmentation), where existing low-rank PEFT methods are argued to be unsuitable.

Approach

Replace low-rank linear adapters with hybrid operation mappings that add a 3x3 convolution to capture local structure alongside attention/FFN-parallel adapters, and generate the convolution kernels dynamically per input via a shared hypernetwork driven by similarity-weighted prompt-token embeddings of the feature map.

Key results

• On COCO object detection with SAM-Base backbone, VFM-Adapter reaches 43.5 bbox mAP versus 43.4 for full fine-tuning using about 10.42M vs 97.3M parameters.
• On COCO with DINOv2-Base, VFM-Adapter achieves 51.6 mAP (ATSS) versus 53.5 for full tuning at 11.93M vs 98.0M parameters, outperforming other PEFT baselines such as LoRA (47.3).
• With ViT-Huge, VFM-Adapter reduces parameters by about 94% and GPU memory by 51% versus full fine-tuning, reaching 52.6 mAP versus 51.5 for full tuning.

Measured effects reported in the paper

Surprising findings

• Frozen VFM backbones perform markedly worse on dense tasks (over 10 mAP drop for detection), undermining zero-shot claims of VFMs.
• Dense-prediction fine-tuned weight matrices do not show the low-rank singular-value concentration seen in classification, which the authors argue makes low-rank PEFT unsuitable.

Limitations

• Code is only promised ('will be available'), not released at publication.
• Gains over full fine-tuning are not uniform; full tuning remains better on DINOv2 detection (53.5 vs 51.6 mAP) and semantic segmentation for DINOv2 (52.8 vs 52.1 mIoU).

Experimental setup

Datasets: COCO ADE20K

Benchmarks: COCO ADE20K

Metrics: bbox mAP segm mAP mIoU mAcc aAcc GPU memory inference FPS training time

Baselines: Full tuning Fix BitFit LoRA SSF Lorand ARC

Builds on: Adapter LoRA ARC CondConv ViTDet

Closest papers by meaning

• Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.87)
• VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.86)
• Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction (ICLR 2025 · similarity 0.86)
• Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · similarity 0.86)
• Revisiting Semi-Supervised Learning in the Era of Foundation Models (NeurIPS 2025 · similarity 0.86)
• Unleashing the Power of Visual Foundation Models for Generalizable Semantic Segmentation (AAAI 2025 · similarity 0.86)