Research map / Label-efficient and robust learning / Recognition and long-tail
Sample-specific Masks for Visual Reprogramming-based Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMM, a visual reprogramming framework that replaces a shared binary mask with a lightweight CNN generating sample-specific three-channel masks (plus patch-wise upsampling), with theory showing lower approximation error and gains on ResNet and ViT across 11 datasets.
Problem
Visual reprogramming (VR) uses a pattern added to inputs at a location fixed by a mask shared across all samples, which limits generalization and increases approximation error when re-purposing pre-trained classifiers to target tasks.
Approach
Replace the fixed shared binary mask with a sample-specific three-channel mask produced by a lightweight CNN generator, upscaled by a parameter-free patch-wise interpolation, multiplied with a shared learnable pattern; theoretically shown to have lower approximation error than shared-mask and sample-specific-pattern hypothesis spaces.
Key results
- SMM with ResNet-18 averages 52.53% across 11 datasets versus 46.85% for the best shared-mask baseline (full watermark).
- With ViT-B32, SMM averages 72.4% versus 65.2% for the best baseline (medium watermark), e.g., +21.8% on Flowers102.
- On SVHN, SMM improves ResNet-18 by 6.1% and ResNet-50 by 3.1% over the best baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the full shared watermark mask with sample-specific three-channel masks | Visual reprogramming with ResNet-18 · 11-dataset average | accuracy: 46.85 -> 52.53 | Helped |
| Replace the full shared watermark mask with sample-specific three-channel masks | Visual reprogramming with ResNet-50 · 11-dataset average | accuracy: 52.10 -> 56.35 | Helped |
| Replace the medium shared watermark mask with sample-specific three-channel masks | Visual reprogramming with ViT-B32 · 11-dataset average | accuracy: 65.2 -> 72.4 | Helped |
| Add the sample-specific mask generator to the shared pattern (ablation) | SMM with ResNet-18 · 11-dataset average | accuracy: 46.85 -> 52.53 | Helped |
| Add the shared pattern δ to the sample-specific mask generator (ablation) | SMM with ResNet-18 · 11-dataset average | accuracy: 42.59 -> 52.53 | Helped |
| Use a three-channel mask instead of a single-channel mask (ablation) | SMM with ResNet-18 · 11-dataset average | accuracy: 49.70 -> 52.53 | Helped |
| Use a single-channel mask instead of a three-channel mask (ablation) | SMM with ResNet-18 · GTSRB | accuracy: 80.4 -> 70.7 | Hurt |
| Use a single-channel mask instead of a three-channel mask (ablation) | SMM with ResNet-18 · Flowers102 | accuracy: 38.7 -> 30.2 | Hurt |
| Use a single-channel mask instead of a three-channel mask (ablation) | SMM with ResNet-18 · CIFAR10 | accuracy: 72.6 ±2.6 -> 72.8 ±0.7 | No clear effect |
| Use specific rather than unified learning parameters (ablation) | SMM with ViT-B32 · UCF101 | accuracy: 42.6 -> 49.9 | Helped |
| Replace bilinear interpolation with patch-wise interpolation (ablation) | Mask resizing with ResNet-18/50 · Batch processing | time per batch (s): 0.062±0.001 -> 0.026±0.004 | Helped |
| Replace bicubic interpolation with patch-wise interpolation (ablation) | Mask resizing with ViT-B32 · Batch processing | time per batch (s): 0.486±0.026 -> 0.069±0.004 | Helped |
Surprising findings
- Using only the shared pattern delta or only the sample-specific mask generator each underperforms the combined SMM on most datasets, indicating both components are needed.
- On ViT-B32 the training accuracy without SMM can match or exceed SMM, with the gain attributed to reduced overfitting rather than better fitting.
Limitations
- Lower approximation error does not guarantee lower excess risk; the estimation error analysis is empirical (overfitting comparison) rather than formal.
- Shared hyperparameters (learning rate, milestones) were not optimal for all datasets, e.g., UCF101 on ViT-B32 (42.6% vs 49.9% with tuned parameters).
- Performance on ViT is worse than SMM would be with tuned hyperparameters on some datasets; some datasets (e.g., EuroSAT) show underperformance versus padding.
Experimental setup
Datasets: CIFAR10 CIFAR100 SVHN GTSRB Flowers102 DTD UCF101 Food101 SUN397 EuroSAT OxfordPets ImageNet-1K
Benchmarks: CIFAR10 CIFAR100 SVHN GTSRB Flowers102 DTD UCF101 Food101 SUN397 EuroSAT OxfordPets
Metrics: test accuracy (mean ± std over three seeds) training loss/accuracy curves t-SNE feature visualization
Baselines: Pad Narrow watermark Medium watermark Full watermark AutoVP
Builds on: watermarking reprogramming AutoVP visual prompt tuning
Closest papers by meaning
- Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts (ICML 2025 · similarity 0.90)
- Bayesian-guided Label Mapping for Visual Reprogramming (NeurIPS 2024 · similarity 0.88)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.87)
- Selective Visual Prompting in Vision Mamba (AAAI 2025 · similarity 0.86)
- Attribute-based Visual Reprogramming for Vision-Language Models (ICLR 2025 · similarity 0.86)
- LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.