Research map / Vision-language and multimodal / CLIP and zero-shot
Visual Data-Type Understanding does not emerge from scaling Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Visual Data-Type Identification with two TypeIdent datasets (27 data-types on animal images), benchmarks 39 VLMs zero-shot, and shows that fine-tuning with data-type-aware captions (TeDaTy) improves identification.
Problem
Vision-language models excel at recognizing semantic content but it is unclear whether they can identify visual data-types (e.g., blur, rotation, cartoon style), which matters for data curation and autonomous vision.
Approach
Frame data-type identification as a separate perceptual task, build controlled datasets of animal images altered across 27 data-types, evaluate many VLMs zero-shot with informedness, analyze CLIP embeddings and LAION-2B caption frequencies, and show that fine-tuning on data-type-annotated captions (TeDaTy) helps where scaling does not.
Key results
- The best C-VLM reaches mean informedness 0.47 (SyntheticTypeIdent) and 0.50 (NaturalTypeIdent), while the best LMM reaches 0.22 and 0.25; IDEFICS-80B underperforms CLIP-RN50.
- Power-law fits suggest VLMs would need over a trillion parameters to reach mean informedness above 0.7.
- Fine-tuning CLIP ViT-B-32 on TeDaTy raises in-distribution informedness on SyntheticTypeIdent from 0.451 to 0.669 (full fine-tuning) and 0.777 (freeze-image).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CLIP ViT-B-32 on TeDaTy rather than zero-shot evaluation | CLIP ViT-B-32 · SyntheticTypeIdent ID | ID-I: 0.451 -> 0.669 | Helped |
| Fine-tune CLIP ViT-B-32 on TeDaTy rather than zero-shot evaluation | CLIP ViT-B-32 · SyntheticTypeIdent OOD | OOD-I: 0.457 -> 0.392 | Hurt |
| Fine-tune CLIP ViT-B-32 on TeDaTy rather than zero-shot evaluation | CLIP ViT-B-32 · NaturalTypeIdent ID | ID-I: 0.440 -> 0.691 | Helped |
| Fine-tune CLIP ViT-B-32 on TeDaTy rather than zero-shot evaluation | CLIP ViT-B-32 · NaturalTypeIdent OOD | OOD-I: 0.473 -> 0.412 | Hurt |
| Fine-tune CLIP ViT-B-32 on COCO only | CLIP ViT-B-32 · SyntheticTypeIdent ID | ID-I: 0.451 -> 0.451 | No clear effect |
| Add TeDaTy data-type examples to fine-tuning | CLIP ViT-B-32 · SyntheticTypeIdent ID | ID-I: 0.451 -> 0.669 | Helped |
| Freeze the image encoder during TeDaTy fine-tuning (ablation) | CLIP ViT-B-32 · SyntheticTypeIdent ID | ID-I: 0.669 -> 0.777 | Helped |
| Freeze the image encoder during TeDaTy fine-tuning (ablation) | CLIP ViT-B-32 · SyntheticTypeIdent OOD | OOD-I: 0.392 -> 0.469 | Helped |
| Freeze the text encoder during TeDaTy fine-tuning (ablation) | CLIP ViT-B-32 · SyntheticTypeIdent ID | ID-I: 0.669 -> 0.780 | Helped |
| Freeze the text encoder during TeDaTy fine-tuning (ablation) | CLIP ViT-B-32 · SyntheticTypeIdent OOD | OOD-I: 0.392 -> 0.370 | Hurt |
| Fine-tune Otter-LLaMA-7B on TeDaTy+COCO+IN100k rather than zero-shot evaluation | Otter-LLaMA-7B · SyntheticTypeIdent ID | ID-I: 0.051 -> 0.120 | Helped |
| Fine-tune Otter-LLaMA-7B on TeDaTy+COCO+IN100k rather than zero-shot evaluation | Otter-LLaMA-7B · SyntheticTypeIdent OOD | OOD-I: 0.180 -> 0.166 | Hurt |
Surprising findings
- Larger LMMs (e.g., IDEFICS-80B) substantially underperform much smaller CLIP-RN50 on data-type identification.
- Few-shot in-context learning (Otter) and training-free TIP-Adapter degrade or fail to improve data-type identification.
Limitations
- Datasets are built from only 10 animal classes and 50 reference images, limiting diversity.
- Many style and semantic data-types rely on generated images from Kandinsky-2.1 with manual curation.
- Causal hypotheses for LMM underperformance (weak alignment, discriminative-generative gap) are not tested in this work.
Experimental setup
Datasets: SyntheticTypeIdent NaturalTypeIdent TeDaTy COCO ImageNet PACS DomainNet LAION-2B-en KaggleAnimalImages
Benchmarks: SyntheticTypeIdent NaturalTypeIdent
Metrics: informedness mean informedness TPR FPR Spearman rank correlation
Baselines: CLIP BLIP-2-ITM CoCa Fromage GILL Multimodal-GPT OpenFlamingo Otter MPlugOwl LLaVA BLIP-2-LLM InstructBLIP
Builds on: CLIP TIP-Adapter Otter OpenFlamingo Kandinsky-2.1
Closest papers by meaning
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.89)
- Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.89)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.89)
- Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.