Research map / Adversarial robustness and security / Attacks on text and graphs

R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning

CVPR 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Vision-language model robustness · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes R-TPT, a test-time prompt tuning defense for CLIP that drops the KL term from marginal entropy (keeping pointwise entropy) and adds reliability-weighted ensembling of augmented views to resist adversarial attacks without labeled data.

Problem

CLIP-based classifiers are highly vulnerable to adversarial attacks, and existing defenses rely on adversarial fine-tuning that needs labeled data and lacks flexibility across downstream tasks.

Approach

Decompose marginal entropy into pointwise entropy plus a KL divergence term, show the KL term pulls augmented views toward a misleading mean under adversarial inputs, and discard it; then weight augmented-view predictions by a neighbor-similarity reliability score so adversarial or noisy views get near-zero weight.

Key results

• On eight fine-grained datasets with CLIP-ResNet50 (eps=1.0), R-TPT reaches 45.8% average adversarial accuracy versus 3.6% for TPT and 30.6% for MTA, while clean accuracy rises slightly from 53.2% to 53.7%.
• On ImageNet with CLIP-ResNet50, R-TPT achieves 47.7% adversarial accuracy versus 0.1% for CLIP and 40.1% for Ensemble.
• Ablation on fine-grained data shows the full model (Ensemble + weighted + EntMin) reaching 45.8% robust accuracy, compared with 41.3% for Ensemble alone.

Measured effects reported in the paper

Surprising findings

• Plain ensembling of augmented views gives strong adversarial robustness but reduces clean accuracy (e.g., 53.2% to 50.6% on fine-grained average), which the reliability weighting mitigates.

Limitations

• Robustness gains are weaker on adversarially pretrained TeCoA CLIP, where clean accuracy upper-bounds defense.
• Per-sample inference adds cost proportional to the number of augmented views, unlike one-time training-time defenses.
• Adversarial examples are generated against the original CLIP before adaptation, so the threat model assumes attackers do not target the adapted model.

Experimental setup

Datasets: Caltech101 Pets Cars Flower102 Aircraft DTD EuroSAT UCF101 ImageNet ImageNet-A ImageNet-V2 ImageNet-R

Benchmarks: Caltech101 Pets Cars Flower102 Aircraft DTD EuroSAT UCF101 ImageNet ImageNet-A ImageNet-V2 ImageNet-R

Metrics: clean accuracy adversarial accuracy under PGD

Baselines: CLIP zero-shot Ensemble TPT C-TPT MTA TeCoA APT

Builds on: TPT MEMO MTA AugMix PGD

Closest papers by meaning

• One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models (CVPR 2024 · similarity 0.93)
• TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models (CVPR 2025 · similarity 0.92)
• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.89)
• Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.89)
• Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models (ICML 2024 · similarity 0.89)
• Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices (ICML 2025 · similarity 0.89)