Research map / Vision-language and multimodal / CLIP and zero-shot
Bayesian-guided Label Mapping for Visual Reprogramming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bayesian-guided label mapping (BLM and BLM+), a probabilistic many-to-many output label mapping for visual reprogramming that replaces one-to-one gradient-free mappings and is estimated from joint pretrained/downstream label statistics.
Problem
Gradient-free output label mapping in visual reprogramming uses one-to-one mappings between pretrained and downstream labels, which ignores many-to-many relationships and leads to suboptimal label assignments.
Approach
Replace the binary one-to-one label mapping matrix with a real-valued probabilistic matrix whose entries are Bayesian conditional probabilities estimated from the joint distribution of predicted pretrained labels and downstream ground-truth labels, iteratively updated during VR; BLM+ aggregates top-K softmax probabilities instead of hard counts.
Key results
- With padding VR on ResNet-18, BLM and BLM+ improve average accuracy over ILM by 4.7% and 6.1% across 12 datasets (BLM+ reaches 46.7% vs ILM 40.6%).
- With watermarking VR on ResNet-18, BLM and BLM+ average 49.3% and 50.7% versus ILM's 43.2%.
- On CLIP, BLM and BLM+ achieve average accuracies of 79.1% and 79.3% across 12 datasets and outperform baselines on 11 datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ILM's one-to-one mapping with Bayesian probabilistic label mapping | ResNet-18 padding VR · 12 downstream datasets average | accuracy: 40.6 -> 45.3 | Helped |
| Use top-K probability aggregation instead of a single predicted label | ResNet-18 padding VR with BLM · 12 downstream datasets average | accuracy: 45.3 -> 46.7 | Helped |
| Replace ILM's one-to-one mapping with Bayesian probabilistic label mapping | ResNeXt-101 padding VR · 12 downstream datasets average | accuracy: 50.0 -> 53.2 | Helped |
| Use BLM+ instead of ILM | ResNet-18 watermarking VR · 12 downstream datasets average | accuracy: 43.2 -> 50.7 | Helped |
| Remove Bayesian computation of the mapping matrix (-Bayes) (ablation) | BLM with ResNet-18 padding VR · most downstream datasets | accuracy | Hurt |
| Stop updating the mapping after the first epoch (-Iter) (ablation) | BLM with ResNet-18 padding VR · most downstream datasets | accuracy | Hurt |
| Aggregate all probabilities rather than only top-K probabilities (-Top-K) (ablation) | BLM+ with ResNet-18 padding VR · most downstream datasets | accuracy | Hurt |
| Stop updating the mapping after the first epoch (-Iter) (ablation) | BLM+ with ResNet-18 padding VR · most downstream datasets | accuracy | Hurt |
| Replace BLM with the one-to-one -Bayes variant (ablation) | BLM+ with ResNet-18 padding VR · UCF101 | accuracy: 32.0 -> 23.9 | Hurt |
| Replace BLM with the one-to-one -Bayes variant (ablation) | BLM+ with ResNet-18 padding VR · SVHN | accuracy: 74.2 -> 75.2 | Helped |
| Use top-K probabilities rather than all probabilities (ablation) | BLM+ with ResNet-18 padding VR · OxfordPets | accuracy: 59.9 -> 70.6 | Helped |
| Use top-K probabilities rather than all probabilities (ablation) | BLM+ with ResNet-18 padding VR · CIFAR100 | accuracy: 30.8 -> 30.6 | Hurt |
Surprising findings
- On SVHN, the one-to-one ILM baseline slightly outperforms BLM, which the authors attribute to minimal inter-class variation and only 10 classes.
Limitations
- Advantage shrinks on tasks with few classes or minimal inter-class variation (e.g., SVHN).
- Still trails deep learning-based linear label mapping on most datasets (e.g., ResNeXt average 53.8% vs 71.9%).
- Theoretical analysis is restricted to a binary label setting.
Experimental setup
Datasets: Flowers102 DTD UCF101 Food101 GTSRB EuroSAT OxfordPets StanfordCars SUN397 CIFAR10 CIFAR100 SVHN
Benchmarks: Flowers102 DTD UCF101 Food101 GTSRB EuroSAT OxfordPets StanfordCars SUN397 CIFAR10 CIFAR100 SVHN
Metrics: accuracy
Baselines: RLM FLM ILM deep learning-based label mapping
Builds on: ILM FLM RLM visual reprogramming
Closest papers by meaning
- Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts (ICML 2025 · similarity 0.89)
- Attribute-based Visual Reprogramming for Vision-Language Models (ICLR 2025 · similarity 0.88)
- Sample-specific Masks for Visual Reprogramming-based Prompting (ICML 2024 · similarity 0.88)
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.88)
- Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.86)
- Pathways of Visual Information Flow in Vision-Language Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.