Research map / Vision-language and multimodal / Multimodal LLMs
Towards Robustness against Typographic Attack with Training-free Concept Localization
Bohan Liu, Wenqian Ye, Guangzhi Xiong, Zhenghao He, Sanchit Sinha, Aidong Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add lexical-circuit intervention (attention reweighting) to CLIP | ViT-H/14 · RTA-100 | object classification accuracy: 53.4 -> 76.2 | Improved |
| add lexical-circuit intervention to CLIP | ViT-H/14 · RTA-100 | text confusion rate: 42.0 -> 14.4 | Improved |
| add lexical-circuit intervention to CLIP | ViT-B/16 · Disentangling | object classification accuracy: 52.2 -> 88.3 | Improved |
| add lexical-circuit intervention to CLIP | ViT-B/16 · IN-100-Text | text confusion rate: 33.0 -> 7.1 | Improved |
| use nTAS circuit mining instead of Dyslexify | CLIP ViTs · RTA-100 (avg) | zero-shot accuracy: 67.0 -> 71.7 | Improved |
| use nTAS circuit mining instead of Dyslexify | CLIP ViTs · Disentangling (avg) | zero-shot accuracy: 67.6 -> 78.7 | Improved |
| use nTAS circuit mining instead of Defense-Prefix | CLIP ViTs · IN-100-Text (avg) | zero-shot accuracy: 70.1 -> 77.0 | Improved |
| use nTAS intervention instead of Defense-Prefix | CLIP ViTs · ImageNet-100 (avg) | clean accuracy: 81.4 -> 81.0 | No clear effect |
| add zero ablation of lexical circuit | Gemma3-12B · RIO-Bench obj-attack | VQA accuracy: 48.24 -> 49.78 | Improved |
| add zero ablation of lexical circuit | Qwen3-VL-8B · RIO-Bench obj-attack | VQA accuracy: 70.11 -> 71.69 | Improved |
| add zero ablation of lexical circuit | InternVL3.5-14B · RIO-Bench obj-attack | VQA accuracy: 53.07 -> 53.08 | No clear effect |
| add zero ablation of lexical circuit | vision encoders of LVLMs · RIO-Bench clean | VQA accuracy: -0.3% to +0.6% | No clear effect |
Closest papers by meaning
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.87)
- FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.87)
- Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning (ICML 2024 · similarity 0.87)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.86)
- SLADE: Shielding against Dual Exploits in Large Vision-Language Models (CVPR 2025 · similarity 0.86)
- Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
