Research map / Vision-language and multimodal / CLIP and zero-shot
T-MARS: Improving Visual Representations by Circumventing Text Feature Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T-MARS, a data filtering method that masks detected text in web images and keeps image-caption pairs whose masked image still has high CLIP similarity to the caption, improving CLIP-style pretraining on LAION and DataComp.
Problem
Web-crawled image-caption datasets like LAION contain many images whose only caption-correlated feature is overlaid text, which pushes contrastive models to learn OCR instead of visual features; naive removal of all text-bearing images also discards useful visual data.
Approach
Detect (not recognize) text regions with FAST, mask them by inpainting with neighboring pixel colors, then score the masked image against the original caption with CLIP and keep the top 50%; filtering targets images where text dominates visual content.
Key results
- At 64M LAION pool, T-MARS improves ImageNet zero-shot accuracy by 6.4% over no filtering and 3.7% over text-match (ViT-B-32 numbers in table: 25.78 vs 23.11).
- On DataComp medium scale, T-MARS outperforms CLIP score filtering by 6.5% on ImageNet, reported as top-ranked at that scale.
- Gains from T-MARS scale near-linearly with the exponential growth of pool size; T-MARS ∩ C-RHO at 32M gives 27.20% ImageNet, above the 64M LAION baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use T-MARS filtering instead of Text-Match filtering | ResNet-50 trained on LAION 64M · ImageNet | zero-shot accuracy: 32.47 vs 29.11; $3.7\%$ gains | Helped |
| Use T-MARS filtering instead of Text-Match filtering | ViT-B-32 trained on LAION 64M · ImageNet | zero-shot accuracy: 25.78 vs 23.11 | Helped |
| Use T-MARS filtering instead of CLIP score filtering | DataComp medium · ImageNet | zero-shot accuracy: $6.5\%$ | Helped |
| Intersect T-MARS retained data with C-SSFT retained data | DataComp medium · ImageNet | zero-shot accuracy: 33.0 -> 33.8 | Helped |
| Add OCR-only samples | Visual representation training · Imagenette | utility: $\mathcal{U}_t = -0.89$ | Hurt |
| Add mislabeled samples | Visual representation training · Imagenette | utility: $\mathcal{U}_r = -0.8$ | Hurt |
| Add images with useful visual features and correlated OCR | Visual representation training · Imagenette | utility: $\mathcal{U}_{it} = +0.27$ | Helped |
| Add images with useful visual features and random OCR | Visual representation training · Imagenette | utility: $\mathcal{U}_{irt} = +0.24$ | Helped |
| Add visual-only images | Visual representation training · Imagenette | utility: $\mathcal{U}_i = +0.23$ | Helped |
| Filter by CLIP score without masking | Visual representation training · LAION data curation | zero-shot accuracy | Hurt |
| Retain only images with visual features by removing all text-overlap images | Text-Match data filtering · LAION 64M | zero-shot accuracy: 29.11 vs T-MARS 32.47 | Hurt |
Surprising findings
- Pruning bad data has about 3x more utility than adding equivalent good samples, per synthetic utility analysis.
- Simply filtering by CLIP score without masking performs even worse than no filtering on LAION.
Limitations
- Filtering may introduce bias in the filtered subset; the approach is not analyzed for group bias.
- Text recognition failures limit Text-Match baseline; the CLIP score for masked images is a noisy, poorly calibrated metric.
Experimental setup
Datasets: LAION-400M DataComp small (12.8M) DataComp medium (128M) Conceptual Captions (CC3M) ImageNet-Captions PACS Imagenette
Benchmarks: ImageNet ImageNet distribution shifts VTAB MSCOCO retrieval Flickr retrieval WinoGAViL DataComp benchmark
Metrics: zero-shot accuracy average accuracy
Baselines: LAION filtering CLIP score filtering Text-Match C-SSFT C-RHO No filtering
Builds on: CLIP DataComp SSFT RHO loss Text Match filtering
Closest papers by meaning
- CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning (ICML 2025 · similarity 0.87)
- Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition (CVPR 2025 · similarity 0.87)
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.87)
- Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIP (AAAI 2025 · similarity 0.86)
- Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.86)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.