Research map / Vision-language and multimodal / Human and embodied modeling
A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Densely Captioned Images (DCI) dataset of ~8k SA-1B images with human mask-aligned dense descriptions, an LLM-summarized 77-token version (sDCI), a subcrop-caption matching benchmark, and shows LoRA fine-tuning CLIP on sDCI improves ARO/VL-Checklist results.
Problem
Curated vision-language captions are too short to capture rich image detail, and existing compositional benchmarks (ARO, VL-Checklist) rely on hard negatives that can be solved by language priors, so VLMs lack reliable dense, mask-aligned evaluation and training data.
Approach
Collect human-written, submask-aligned dense descriptions (averaging over 1000 words per image), summarize them with Llama2 to fit CLIP's 77-token limit, and use the multiple-crop structure to build a subcrop-caption matching test that checks whether models distinguish captions of different regions within the same image.
Key results
- CLIP baseline scores 40.06% on All SCM and 60.79% on All Neg; NegCLIP improves SCM to 43.35%, while DAC-SAM reaches 84.17% on All Neg but only 37.90% on All SCM.
- CLIP fine-tuned on sDCI with negatives loss (sDCI P1) reaches 81.17%→88.37% on VL-Checklist Object without negatives and 76.23% on ARO VG-R.
- Fine-tuned sDCI model outperforms DAC trained on matched small image counts (10k and 100k images) on ARO and VL-Checklist.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CLIP on sDCI with negatives loss | ViT/32B CLIP · ARO and VL-Checklist | accuracy | Mixed |
| Remove negatives loss (ablation) | sDCI Pick1 fine-tuning of ViT/32B CLIP · ARO and VL-Checklist | accuracy: 76.23% -> 57.34%; 67.56% -> 61.98%; 88.58% -> 39.36%; 91.30% -> 44.62%; 80.71% -> 88.37%; 68.69% -> 70.42%; 70.12% -> 61 | Mixed |
| Fine-tune CLIP on sDCI rather than DAC LLM captions using a sample-limited dataset | CLIP · ARO and VL-Checklist | accuracy | Mixed |
| Fine-tune CLIP on sDCI with negatives loss | ViT/32B CLIP · VL-Checklist Object | accuracy: 81.17% -> 80.71% | No clear effect |
Surprising findings
- Models trained with CLIP-style objectives do not improve subcrop-caption matching, and DAC, despite strong negatives scores, performs worst on SCM.
- Fine-tuning without negatives loss can degrade ARO and VL-Checklist performance relative to CLIP baseline, while still improving VL-Object.
Limitations
- LLM summarization to 77 tokens may introduce noise, drop details, and remove distinguishing information between captions.
- Pixel-level masks are not used; only crops around masks are used in experiments.
- The dataset is released under CC-BY-NC, and the dense captions exceed current CLIP text context length.
Experimental setup
Datasets: DCI sDCI SA-1B ARO VL-Checklist DAC captions
Benchmarks: sDCI Subcrop-Caption Matching sDCI negatives tests ARO VL-Checklist
Metrics: accuracy
Baselines: CLIP NegCLIP BLIP Flava X-VLM DAC-LLM DAC-SAM BLIP2 SVLC
Builds on: CLIP NegCLIP DAC (Densely Aligned Captions) Segment Anything Model Llama 2
Closest papers by meaning
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.91)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.90)
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.89)
- BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs (CVPR 2025 · similarity 0.89)
- Semantic and Expressive Variations in Image Captions Across Languages (CVPR 2025 · similarity 0.89)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.