Research map / Vision-language and multimodal / CLIP and zero-shot
CLIP in Mirror: Disentangling text from visual images through reflection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MirrorCLIP, a zero-shot, training-free method that disentangles textual and visual factors in CLIP image features by comparing features of original and horizontally flipped images to build sign-based masks and filters.
Problem
CLIP confuses textual and visual content in text-visual images (e.g., misleading words or typographic attacks), hurting both image recognition and text recognition.
Approach
CLIP image features of visual objects are approximately flip-invariant while text features are not; comparing sign changes of feature dimensions between original and flipped images yields disentangling masks, which are used to build soft textual and visual filters without any retraining.
Key results
- Visual features raise average typographic-attack accuracy by +15.49% on synthetic and +21.27% on real-world datasets over CLIP, and beat Defense Prefix on real-world (68.69 vs 64.52 avg).
- Textual features raise text recognition accuracy on typographic datasets from 39.32% to 73.95%.
- Mean cosine similarity between original and flipped image features drops from 0.9846 on clean data to 0.8225 on typographic data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MirrorCLIP visual features instead of original image features | CLIP ViT-B/32 · 10 synthetic typographic datasets | accuracy: 34.60 -> 50.09 | Helped |
| Use MirrorCLIP visual features instead of original image features | CLIP ViT-B/32 · 3 real-world typographic datasets | accuracy: 47.42 -> 68.69 | Helped |
| Use MirrorCLIP visual features instead of original image features | CLIP ViT-B/32 · 10 original image classification datasets | accuracy: 61.59 -> 61.95 | Helped |
| Use disentangled textual features instead of original image features | CLIP ViT-B/32 · Typographic attack datasets | accuracy: 39.32% to 73.95% | Helped |
| Use flipped image features instead of MirrorCLIP visual features (ablation) | CLIP image recognition · 3 real-world typographic datasets | accuracy: 68.69 -> 0.57 | Hurt |
| Use flipped image features instead of MirrorCLIP visual features (ablation) | CLIP image recognition · 10 original image classification datasets | accuracy: 61.59 -> 61.38 | No clear effect |
| Use textual features for image recognition (ablation) | CLIP · 10 original image classification datasets | accuracy: 61.59 -> 3.53 | Hurt |
| Use textual features for image recognition (ablation) | CLIP · 3 real-world typographic datasets | accuracy: 36.38 -> 72.51 | Helped |
| Replace the textual filter with hard masking (ablation) | MirrorCLIP text features · 3 real-world typographic datasets, text recognition | accuracy: 72.51 -> 61.03 | Hurt |
| Replace the visual filter with hard masking (ablation) | MirrorCLIP visual features · 10 synthetic typographic datasets | accuracy: 50.09 -> 41.57 | Hurt |
| Replace ViT-B/32 with RN50×4 as the CLIP image encoder (ablation) | MirrorCLIP · Image recognition on clean, synthetic typographic, and real-world typographic datasets | accuracy: 62.76; 48.51; 74.21 | Mixed |
| Apply MirrorCLIP to special palindromes (ablation) | CLIP image recognition · Special palindrome dataset (average) | accuracy: 52.44 -> 57.73 | Helped |
Surprising findings
- Directly using flipped image features as visual features gives the highest typographic defense accuracy but fails when inputs are themselves flipped (accuracy falls to 37.56%).
- Special palindromes (e.g., 'mom') remain largely undisentangled, improving only 5.29 points.
Limitations
- Cannot fully separate visual and textual features, which can affect image generation quality.
- Fails on special palindromes whose shape is unchanged by flipping.
- Evaluation limited to CLIP ViT-B/32 and RN50x4 encoders and classification-style tasks.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft DTD SUN397 EuroSAT RTA-100 PAINT typographic dataset
Benchmarks: ImageNet Caltech101 Food101 Flowers102 OxfordPets StanfordCars FGVCAircraft DTD SUN397 EuroSAT RTA-100 PAINT typographic dataset
Metrics: top-1 accuracy cosine similarity
Baselines: CLIP Materzynska et al. projection PAINT Defense Prefix
Builds on: CLIP Defense Prefix DecomCAM Stable unCLIP
Closest papers by meaning
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.89)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.88)
- Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP (NeurIPS 2024 · similarity 0.88)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.88)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.87)
- Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.