Research map / Adversarial robustness and security / Attacks on text and graphs
LipSim: A Provably Robust Perceptual Similarity Metric
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that DreamSim-style ViT-ensemble perceptual metrics are vulnerable to adversarial attacks, then distills DreamSim into a 1-Lipschitz SLL-based student and fine-tunes it on NIGHT to get LipSim, a perceptual metric with certified 2AFC scores.
Problem
Perceptual similarity metrics built on neural networks (e.g., DreamSim) are vulnerable to adversarial perturbations, and existing robust perceptual metrics rely on heuristic defenses or loose/expensive randomized smoothing certificates.
Approach
Distill DreamSim's ViT-ensemble embeddings into a 1-Lipschitz SDP-based Lipschitz network followed by an l2-ball projection and cosine distance, so the metric is provably 1-Lipschitz in the reference input; the 2AFC margin then yields closed-form certificates (margin / ||f(x0)-f(x1)||) with no Monte Carlo sampling.
Key results
- On NIGHT under l2-APGD at eps=2.0, LipSim retains 72.20% 2AFC score versus 0.93% for DreamSim, at a natural score of 85.58% vs 96.16%.
- LipSim-DreamSim with hinge margin 0.5 reports certified 2AFC scores of 66.39% / 44.90% / 23.46% at 36/255, 72/255, 108/255 on NIGHT.
- On BAPPS, DreamSim gets 0.0 certified score at all radii while LipSim gets 31.20% at 36/255 (margin 0.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DreamSim with a 1-Lipschitz feature extractor and unit-ball projection | Perceptual metric on NIGHT · NIGHT, ℓ2-APGD ε=0.5 to 3.0 | 2AFC score: DreamSim: 46.27, 16.66, 0.93, 0.93; LipSim: 82.89, 79.82, 72.20, 61.84 | Helped |
| Use a 1-Lipschitz feature extractor with unit-ball projection instead of DreamSim | Perceptual metric on NIGHT · NIGHT, natural images | natural 2AFC score: 96.16 -> 85.58 | Hurt |
| Increase hinge-loss margin from 0.2 to 0.5 (ablation) | LipSim -- DreamSim · NIGHT, certified at 36/255 | certified 2AFC score: 62.88 -> 66.39 | Helped |
| Increase hinge-loss margin from 0.2 to 0.5 (ablation) | LipSim -- DreamSim · NIGHT, natural images | natural 2AFC score: 85.58 -> 82.89 | Hurt |
| Use DreamSim rather than DINO or OpenCLIP as the distillation teacher (ablation) | LipSim, margin 0.5 · NIGHT, certified at 36/255 | certified 2AFC score: DINO: 66.28; OpenCLIP: 64.80; DreamSim: 66.39 | Helped |
| Use a 1-Lipschitz LipSim backbone rather than DreamSim | Perceptual metric on BAPPS · BAPPS, certified | certified 2AFC score: DreamSim: 0.0, 0.0, 0.0; LipSim (margin 0.4): 31.74, 15.19, 7.0 | Helped |
| Use LipSim rather than DreamSim for KNN classification | ImageNet-style KNN on Imagenette · Imagenette, 10-NN Top 1 | accuracy: 99.03 -> 85.32 | Hurt |
| Use LipSim rather than DreamSim for KNN classification | ImageNet-style KNN on Imagenette · Imagenette, 10-NN Top 5 | accuracy: 99.82 -> 97.20 | Hurt |
Surprising findings
- LipSim's natural 2AFC score (85.58%) is lower than DreamSim's (96.16%), with robustness gained at an accuracy cost.
- On the Imagenette 10-NN KNN task, LipSim's top-1 accuracy (85.32%) is far below DreamSim's (99.03%).
Limitations
- Natural score drops relative to DreamSim, and KNN top-1 accuracy is substantially lower.
- The NIGHT JND split was not released, so evaluation relies only on the 2AFC score.
- Certificates assume the embedding norm before projection is at least 1; the method abstains otherwise (reported as zero on NIGHT).
Experimental setup
Datasets: ImageNet-1k NIGHT BAPPS COCO image retrieval subset Imagenette
Benchmarks: NIGHT BAPPS Imagenette KNN
Metrics: 2AFC score certified 2AFC score adversarial 2AFC score under APGD/PGD/MIA top-1 and top-5 KNN accuracy
Baselines: DreamSim CLIP OpenCLIP DINO R-LPIPS LPIPS
Builds on: DreamSim SDP-based Lipschitz Layers APGD LPIPS
Closest papers by meaning
- The Unreasonable Effectiveness of Linear Prediction as a Perceptual Metric (ICLR 2024 · similarity 0.85)
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.85)
- ReSi: A Comprehensive Benchmark for Representational Similarity Measures (ICLR 2025 · similarity 0.84)
- Learning Bregman Divergences with Application to Robustness (NeurIPS 2024 · similarity 0.83)
- Comparing the Robustness of Modern No-Reference Image- and Video-Quality Metrics to Adversarial Attacks (AAAI 2024 · similarity 0.83)
- A Sober Look at the Robustness of CLIPs to Spurious Features (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.