Research map / Vision-language and multimodal / Human and embodied modeling
Textural or Textual: How Vision-Language Models Read Text in Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the ToT dataset of text-overlaid ImageNet images and paronym/synonym word pairs, uses layer-wise intrinsic dimension and linear probes to show CLIP reads text largely via texture until a late semantic stage, and fine-tunes only the last block to defend against typographic attacks.
Problem
Whether vision-language models genuinely encode the semantics of text in images or mainly treat it as texture, and at which depth textual semantics affect representations and typographic attack vulnerability.
Approach
Disentangles orthographic form from meaning using paronym/synonym word pairs and controlled text overlays, tracks intrinsic dimension across ViT layers alongside linear probe accuracy, and finds semantic differentiation only emerges in the final block, motivating last-block fine-tuning as a defense.
Key results
- CLIP accuracy drops from 98.4% on Cons 80 to 42.9% on Irr 80 at the same font size, while ViT-B/16 and DINOv2 vary little across semantic contexts.
- Fine-tuning only the last block (Ours) reaches 84.7% on Orig and 60.0% on Easy-Irr, and 35.0% Hard-Cons, 69.5% Hard-Irr, versus 54.8% when fine-tuning all blocks.
- In the cross-dataset easy-level evaluation on handwritten typographic datasets, the method averages 69.5% versus 46.8% for CLIP, 63.6% for Disentangle, and 64.5% for Prefix.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune only the last block on hard-level data | CLIP ViT-B/16 · Hard Nons | accuracy (%): 77.2 -> 81.7 | Helped |
| Fine-tune only the last block on hard-level data | CLIP ViT-B/16 · Hard Irr | accuracy (%): 59.9 -> 69.5 | Helped |
| Fine-tune only the last block on hard-level data | CLIP ViT-B/16 · Easy Orig | accuracy (%): 82.3 -> 84.7 | Helped |
| Fine-tune the swell blocks rather than the last block (ablation) | CLIP ViT-B/16 · Hard Nons | accuracy (%): 81.7 -> 56.2 | Hurt |
| Fine-tune the shrink blocks rather than the last block (ablation) | CLIP ViT-B/16 · Hard Nons | accuracy (%): 81.7 -> 81.1 | No clear effect |
| Fine-tune shrink blocks excluding the last block (ablation) | CLIP ViT-B/16 · Hard Nons | accuracy (%): 81.7 -> 81.7 | No clear effect |
| Fine-tune all blocks rather than only the last block (ablation) | CLIP ViT-B/16 · Hard Nons | accuracy (%): 81.7 -> 48.2 | Hurt |
| Fine-tune only the last block on hard-level rather than medium-level data (ablation) | CLIP ViT-B/16 · Hard Irr | accuracy (%): 22.0 -> 69.5 | Helped |
| Fine-tune only the last block on medium-level data | CLIP ViT-B/16 · Medium Irr | accuracy (%): 66.0 -> 82.4 | Helped |
| Use vision-only ViT rather than CLIP | ToT image classification · ToT Irr 80 | accuracy (%): 42.9 -> 87.1 | Helped |
| Increase irrelevant-text font size from 20 to 120 | CLIP · ToT Irr | accuracy (%): 78.8 -> 38.9 | Hurt |
| Add semantically consistent text overlay | CLIP · ToT classification | accuracy (%): 86.6 -> 98.4 | Helped |
Surprising findings
- Semantic differentiation (ID divergence across semantic categories) appears only in the final block, challenging the assumption that semantics are built progressively across depth.
- Early gains in semantic decoding accuracy largely reflect orthographic (texture) discrimination rather than semantic understanding.
Limitations
- Analysis focuses on CLIP ViT-B/16 vision encoder; other architectures only reported in appendix.
- Paronym/synonym subset is small (10 word pairs) and linear probe uses only 320 training images per pair.
Experimental setup
Datasets: ToT ImageNet-1k Disentangle PAINT Prefix
Benchmarks: ToT Disentangle PAINT Prefix
Metrics: accuracy intrinsic dimension
Baselines: CLIP Prefix Disentangle PAINT MetaCLIP ViT-B/16 DINOv2
Builds on: CLIP TwoNN intrinsic dimension estimator linear probe
Closest papers by meaning
- ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art (ICLR 2026 · similarity 0.89)
- Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (CVPR 2025 · similarity 0.88)
- Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective (ICLR 2025 · similarity 0.87)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.87)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.87)
- Visual Data-Type Understanding does not emerge from scaling Vision-Language Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.