Research map / Label-efficient and robust learning / Domain adaptation and generalization

Structured Model Probing: Empowering Efficient Transfer Learning by Structured Regularization

CVPR 2024 · Label-efficient and robust learning › Domain adaptation and generalization · approach: Statistical transfer learning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes structured model probing (SMP), which uses structured group-sparsity and non-linearity regularization over multi-layer ViT features to make probing adapt well to both easy and difficult transfer tasks on VTAB-1k.

Problem

Model probing (frozen pre-trained features with a light head) lags behind parameter-efficient tuning on VTAB-1k: linear probing works for domains similar to the source but fails for dissimilar ones, while more complex probes hurt easy domains.

Approach

Extract structured features from all ViT sub-structures (token- and channel-pooled), apply a group-sparsity regularizer whose groups follow the model structure, and couple a linear model with an MLP whose complexity is bounded by the selected-feature norms, so the probe automatically becomes linear for easy tasks and non-linear for difficult ones.

Key results

• SMP reaches 70.9% mean accuracy on VTAB-1k, versus 69.4% for VPT and 67.5% for Head2Toe.
• On the structured (difficult) VTAB group SMP attains 55.4% mean accuracy, against 55.0% for VPT and 45.3% for LinearStruc.
• SMP needs about 15% of fine-tuning adaptation time on average, while VPT needs 97%, and adds only 0.01G inference FLOPs versus VPT's 11.01G.

Measured effects reported in the paper

Surprising findings

• Linear probing on the last layer is comparable to tuning methods on easy VTAB tasks, but a non-linear MLP probe degrades performance on easy tasks through overfitting.
• Adding non-linearity or more layer features helps difficult tasks but hurts easy ones, so no single fixed probe complexity works for both.

Limitations

• SMP requires 6.91M extra parameters on average, more than VPT's 0.64M.
• Gains over VPT on the mean are modest (70.9% vs 69.4%).
• Evaluation focuses on ViT-B/16 with ImageNet-21k pre-training and VTAB-1k, with limited other settings.

Experimental setup

Datasets: VTAB-1k ImageNet-21k Food101 CUB-200-2011 Stanford Cars Stanford Dogs Oxford Pets

Benchmarks: VTAB-1k Few-shot fine-grained recognition (Food101, CUB-200-2011, Stanford Cars, Stanford Dogs, Oxford Pets) 9 full-size downstream datasets

Metrics: median test accuracy over 3 seeds mean accuracy relative adaptation time extra inference FLOPs extra parameters

Baselines: Fine-tuning Bias VPT Linear Probing Side-Tuning Head2Toe LinearStruc.

Builds on: group lasso Head2Toe VPT Side-Tuning hierarchical sparse modeling

Closest papers by meaning

• Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models (NeurIPS 2025 · similarity 0.84)
• RAPTOR: Ridge-Adaptive Logistic Probes (NeurIPS 2026 · similarity 0.83)
• Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing (ICLR 2025 · similarity 0.83)
• Transferring Knowledge From Large Foundation Models to Small Downstream Models (ICML 2024 · similarity 0.83)
• Transferring Linear Features Across Language Models With Model Stitching (NeurIPS 2025 · similarity 0.82)
• Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptation (NeurIPS 2024 · similarity 0.82)