Research map / Architectures and efficiency / Pruning and efficient tuning
Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COMBO, a probing-based adapter that compresses multi-layer, multi-model ViT features into token-wise representations and classifies with a small transformer, plus a regularisation-based backbone relevance score, evaluated on VTAB-1k.
Problem
Combining complementary representations from multiple frozen foundation models for downstream adaptation without backpropagating through the backbones, while avoiding the scaling limits and dataset-specific tuning of existing probing methods (Head2Toe, SMP).
Approach
A learned affine projection compresses stacked activations from all layers and models at each spatial token into a compact descriptor, which a small transformer with a cls token processes to predict the label; a group-sparsity penalty on projection weights per backbone yields task-relevance scores for selecting backbones.
Key results
- COMBO probing all four ViT-B backbones reaches 78.2% VTAB-1k global average and top-two selection 78.6%, versus 77.5% for distilled RADIOv2.5 with Adapter+.
- COMBO on single ViT-B/16 ImageNet-21K gets 74.6% global average, above full fine-tuning (74.2%) and SMP (73.6%) but below Adapter+ (77.6%).
- Probing individually tuned backbones, COMBO reaches 79.3% global average vs 78.7% for Multi-Adapter+ with much lower training memory (~3 GB vs 13 GB).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| probe all four backbones together rather than the strongest single backbone | COMBO · VTAB-1k | global average accuracy: 77.9 -> 78.2 | Helped |
| select the top two backbones by task relevance instead of probing all four | COMBO · VTAB-1k | global average accuracy: 78.2 -> 78.6 | Helped |
| use COMBO on multiple foundation models rather than distill them and tune RADIOv2.5 | VTAB-1k · VTAB-1k | global average accuracy: 77.5 -> 78.2 | Helped |
| use COMBO on tuned backbones rather than joint Multi-Adapter+ tuning | VTAB-1k · VTAB-1k | global average accuracy: 78.7 -> 79.3 | Helped |
| use COMBO instead of prior probing methods | ImageNet-21K ViT-B/16 · VTAB-1k | global average accuracy: 73.6 -> 74.6 | Helped |
| probe only the last layer instead of all layers (ablation) | COMBO on the original model · VTAB-1k validation | average accuracy: 76.5 -> 63.6 | Hurt |
| probe only the last layer instead of all layers (ablation) | COMBO on an Adapter+-tuned model · VTAB-1k validation | average accuracy: 76.5 -> 76.9 | No clear effect |
| remove feature map normalisation (ablation) | COMBO · VTAB-1k validation | global average accuracy: 73.8 -> 71.5 | Hurt |
| replace the transformer with a linear head (ablation) | COMBO · VTAB-1k validation | global average accuracy: 73.8 -> 65.1 | Hurt |
| replace the transformer with the SMP MLP (ablation) | COMBO · VTAB-1k validation | global average accuracy: 73.8 -> 67.5 | Hurt |
| probe only the first six blocks instead of all blocks (ablation) | COMBO · VTAB-1k validation | global average accuracy: 73.8 -> 65.0 | Hurt |
| probe only the last six blocks instead of all blocks (ablation) | COMBO · VTAB-1k validation | global average accuracy: 73.8 -> 69.1 | Hurt |
Surprising findings
- Probing all layers matters for untuned models, but for Adapter+-tuned models the last layer alone gives nearly the same performance as all layers.
- SAM features perform far worse than other backbones on VTAB-1k, and removing SAM from the probed set slightly improves the global average (76.9% vs 76.2%).
Limitations
- Assumes consistent feature map shapes across models; interpolation may degrade representation fidelity.
- Uses a single affine projection for all layers, which may limit scalability to more layers.
- Evaluated only on ViT-B backbones and VTAB-1k low-data classification, not dense prediction or larger models.
Experimental setup
Datasets: VTAB-1k ImageNet-21K FGVCx Fungi
Benchmarks: VTAB-1k
Metrics: accuracy global average accuracy FLOPs peak GPU memory
Baselines: Linear probing Head2Toe SMP Full fine-tuning LoRA VPT-Deep Adapter+ RADIOv2.5 Multi-Adapter+
Builds on: Head2Toe Structured Model Probing Adapter+ Universal Representation Transformer FiT
Closest papers by meaning
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.86)
- Structured Model Probing: Empowering Efficient Transfer Learning by Structured Regularization (CVPR 2024 · similarity 0.84)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.84)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.84)
- RAPTOR: Ridge-Adaptive Logistic Probes (NeurIPS 2026 · similarity 0.83)
- Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domains (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.