Research map / Adversarial robustness and security / Certified and empirical robustness
Topological Signatures of Adversaries in Multimodal Alignments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes topological-contrastive losses (total persistence and multi-scale kernel) on image-text embedding persistence diagrams, back-propagates them to per-sample features, and plugs them into SAMMD to improve MMD-based detection of adversarial inputs to CLIP and BLIP.
Problem
Detecting adversarial attacks on multimodal image-text alignment systems such as CLIP and BLIP, which remain under-defended compared with unimodal models.
Approach
Measure topological mismatch between image and text embedding point clouds via persistent homology (total persistence and multi-scale kernel losses), show these losses change monotonically with the adversarial fraction, and back-propagate them to sample-level features that enhance a semantic-aware MMD test (TPSAMMD, MKSAMMD).
Key results
- TPSAMMD improves test power over SAMMD on ImageNet, with gains of 14.3% for CLIP ViT-B/32 and 8.2% for CLIP ViT-L/14 under PGD at epsilon 4/255.
- TP and MK losses change monotonically with the adversarial fraction in most CLIP and BLIP settings (e.g., TP rises across nearly all CIFAR-10 and ImageNet attacks).
- Type-I error stays at or below 5% for the proposed tests on CIFAR-10 and ImageNet, except ME on ImageNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Total Persistence features to SAMMD | SAMMD adversarial detection · CIFAR10, PGD ε=1/255, CLIP ViT-B/32 | test power: 3.6% | Helped |
| Add Total Persistence features to SAMMD | SAMMD adversarial detection · CIFAR10, PGD ε=1/255, CLIP ViT-L/14 | test power: 8.6% | Helped |
| Add Total Persistence features to SAMMD | SAMMD adversarial detection · CIFAR10, PGD ε=1/255, BLIP ViT-B/14 | test power: 1.5% | Helped |
| Add Total Persistence features to SAMMD | SAMMD adversarial detection · CIFAR10, PGD ε=1/255, BLIP ViT-B/129 | test power: 16.7% | Helped |
| Add Total Persistence features to SAMMD | SAMMD adversarial detection · CIFAR10, PGD ε=4/255, CLIP ViT-B/32 | test power: 14.3% | Helped |
| Add Total Persistence features to SAMMD | SAMMD adversarial detection · CIFAR10, PGD ε=4/255, CLIP ViT-L/14 | test power: 8.2% | Helped |
| Add Total Persistence features to SAMMD | SAMMD adversarial detection · CIFAR10, PGD ε=4/255, BLIP ViT-B/14 and BLIP ViT-B/129 | test power: 0% | No clear effect |
| Replace clean samples in a batch with adversarial samples | CLIP and BLIP image-text alignments · CIFAR10, ImageNet, and CIFAR100 | normalized TP loss | Helped |
| Replace clean samples in a batch with adversarial samples | CLIP image-text alignments · CIFAR10 and ImageNet | normalized MK loss | Mixed |
| Add TP or MK topological features to SAMMD | Adversarial detection on ImageNet · ImageNet | test power | Mixed |
| Use TPSAMMD rather than SAMMD | Adversarial detection · CIFAR10, PGD and AutoAttack | test power | Helped |
Surprising findings
- FGSM breaks the monotonic pattern of the topological losses in most settings, which the authors attribute to unrealistic large distortions.
- On CIFAR-10, SAMMD-based methods already reach nearly 100% test power, so the topological gains are concentrated on ImageNet.
Limitations
- Batch-level gradient approximation does not guarantee independence of per-sample topological features.
- Monotonic behavior is not uniform: FGSM and some CLIP/BLIP settings are non-monotonic or reverse direction.
- Code is not yet released; the authors state it is under review.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet
Benchmarks: CIFAR-10 CIFAR-100 ImageNet
Metrics: test power Type-I error accuracy gain
Baselines: SAMMD Mean Embedding test Smooth Characteristic Functions test Classifier two-sample test
Builds on: SAMMD Total persistence Multi-scale kernel CLIP BLIP
Closest papers by meaning
- Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning (NeurIPS 2026 · similarity 0.85)
- Towards Scalable Topological Regularizers (ICLR 2025 · similarity 0.85)
- Visual Adversarial Examples Jailbreak Aligned Large Language Models (AAAI 2024 · similarity 0.85)
- Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (ICLR 2024 · similarity 0.84)
- When Witnesses Defend: A Witness Graph Topological Layer for Adversarial Graph Learning (AAAI 2025 · similarity 0.84)
- Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Prompting (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.