Research map / Vision-language and multimodal / Human and embodied modeling
Alt-Text with Context: Improving Accessibility for Images on Twitter
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a multimodal alt-text generator that prefixes a GPT-2 decoder with both CLIP-projected image features and embedded tweet text, and releases a 371k-image Twitter alt-text dataset.
Problem
Generating accessible alt-text for images shared on social media (Twitter), where user-written tweet text can provide context beyond the image and gold alt-text is noisy and more descriptive than typical captions.
Approach
Concatenate a ClipCap-style projected CLIP image prefix with the embedded tweet text as a multimodal prompt to a finetuned GPT-2 decoder, so a unimodal LM conditions on both image and social media context; decoding uses beam search without repeats and optional ROUGE-L reranking against the tweet.
Key results
- Full model (Text + Image, beam search no repeats) reaches BLEU@4 1.826 vs 0.372 for frozen ClipCap and 0.111 for BLIP-2 (x100 scale), i.e. more than 2x over the best ClipCap baseline (0.696).
- Text-only ablation reaches BLEU@4 0.693 and CIDEr 1.738, while the full model reaches CIDEr 7.661, showing image and text signals stack.
- In human evaluation, annotators preferred the model over frozen ClipCap for fluency (61.4 vs 29.8) and descriptiveness (66.9 vs 26.2); over ClipCap from scratch the margin was tighter (descriptiveness 44.4 vs 37.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add tweet text conditioning to image conditioning | Ours (Text + Image) versus ClipCap (From Scratch) · Twitter alt-text test set | BLEU@4: 0.696 -> 1.826 | Helped |
| add image conditioning to tweet-text-only conditioning (ablation) | Ours (Text + Image) · Twitter alt-text test set | CIDEr: 1.738 -> 7.661 | Helped |
| replace the retrieved tweet with a randomly chosen tweet (ablation) | Ours (Text + Image) · Twitter alt-text test set | BLEU@4: 1.826 -> 0.546 | Hurt |
| use greedy decoding instead of beam search with no-repeat trigram blocking (ablation) | Ours (Text + Image) · Twitter alt-text test set | BLEU@4: 0.587 -> 1.826 | Hurt |
| add no-repeat trigram blocking to greedy decoding (ablation) | Ours (Text + Image) · Twitter alt-text test set | BLEU@4: 0.587 -> 0.469 | Hurt |
| add no-repeat trigram blocking to greedy decoding (ablation) | Ours (Text + Image) · Twitter alt-text test set | ROUGE-L: 7.166 -> 7.774 | Helped |
| use beam search without no-repeat trigram blocking instead of with it (ablation) | Ours (Text + Image) · Twitter alt-text test set | BLEU@4: 0.400 -> 1.826 | Hurt |
| rerank beam-search candidates by ROUGE-L similarity to tweet text (ablation) | Ours (Text + Image), BS (NR) · Twitter alt-text test set | BLEU@4: 1.826 -> 1.897 | Helped |
| rerank beam-search candidates by ROUGE-L similarity to tweet text (ablation) | Ours (Text + Image), BS (NR) · Twitter alt-text test set | CIDEr: 7.661 -> 6.791 | Hurt |
Surprising findings
- Copying the tweet text or a text-only model gets reasonable BLEU, and the text-only ablation is competitive with fine-tuned ClipCap on several metrics.
- Automatic metric scores are low relative to Conceptual Captions because user-written Twitter alt-text is noisy, so the authors rely on human evaluation.
Limitations
- Gold alt-text is user-written and noisy, so automatic metrics are low and may not reflect quality.
- Human evaluation used sighted annotators; no BLV user study, so downstream accessibility efficacy is untested.
- Not applicable where images lack textual context; the model has no OCR and struggles with counting.
Experimental setup
Datasets: Twitter alt-text dataset (371k images, scraped from Twitter API v2, 2022) Conceptual Captions
Benchmarks: Twitter alt-text test set (20,260 images)
Metrics: BLEU@4 METEOR ROUGE-L CIDEr human preference (fluency, descriptiveness)
Baselines: Nearest Neighbor Copy Tweet Text ClipCap (Frozen) ClipCap (Finetuned) ClipCap (From Scratch) BLIP-2 (Frozen) Ours (Text Only) Ours (Rand Text)
Builds on: ClipCap CLIP GPT-2 BLIP-2
Closest papers by meaning
- Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models (ICLR 2025 · similarity 0.86)
- ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.84)
- Image Captioning with Multi-Context Synthetic Data (AAAI 2024 · similarity 0.84)
- CaptionQA: Is Your Caption as Useful as the Image Itself? (CVPR 2026 · similarity 0.83)
- TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.83)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.