Research map / Vision-language and multimodal / CLIP and zero-shot
Explaining CLIP's Performance Disparities on Data from Blind/Low Vision Users
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates 25 CLIP variants zero-shot on BLV-user images (ORBIT, VizWiz-Classification) versus web-crawled images, finding ~15-point accuracy drops tied to disability content, image quality, and tactile/material language, with downstream checks on OWL-ViT, CLIPSeg, and DALL-E2.
Problem
CLIP, a widely used large multi-modal model likely underpinning assistive technologies, has not been systematically evaluated on images and text from blind and low vision (BLV) users, so its quality-of-service disparities for this group are unknown.
Approach
Standardized episodic zero-shot classification across 25 CLIP variants compares BLV-captured datasets against web-crawled ones, uses logistic regression to isolate effects of image quality and object type, and analyzes pre-training caption corpora to explain the disparities; few-shot ProtoNets is tested as a mitigation.
Key results
- CLIP variants (25) average 15 percentage points lower accuracy on BLV images (51.5%) than web-crawled images (66.5%).
- Disability objects are 16-17x less frequent in LAION-400M, LAION-2B and DataComp-1B captions than non-disability objects; materials ~4x less frequent than colors.
- DALL-E2 correctly generated disability objects in 18/80 images versus 74/80 for non-disability objects; 5-shot ProtoNets reduced disability-non-disability gap on ORBIT Clean to about 2 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evaluate CLIP on BLV-captured images rather than web-crawled images | 25 CLIP variants · ORBIT and VizWiz-Classification versus MS-COCO and Open Images | zero-shot accuracy: 15 percentage points lower on average | Hurt |
| classify disability objects rather than non-disability objects | CLIP · ORBIT Clean | accuracy: 58.9% ± 0.1% -> 41.8% ± 0.1% | Hurt |
| classify disability objects rather than non-disability objects | CLIP · ORBIT Clutter | accuracy: 50.9% ± 0.1% -> 25.8% ± 0.1% | Hurt |
| add 5 image examples per class with ProtoNets (ablation) | CLIP zero-shot classification · ORBIT Clean | accuracy: 41.8 -> 86.2 | Helped |
| add 5 image examples per class with ProtoNets (ablation) | CLIP zero-shot classification · ORBIT Clutter | accuracy: 25.8 -> 54.5 | Helped |
| use LLM-generated object descriptions instead of raw labels (ablation) | CLIP zero-shot classification · ORBIT Clean and ORBIT Clutter | accuracy: ORBIT Clean: 41.8 -> 48.3; ORBIT Clutter: 25.8 -> 32.1 | Helped |
| add a color adjective to the object name rather than a material adjective (ablation) | CLIP zero-shot classification · ORBIT | accuracy: 7.1 percentage points | Helped |
| add a material adjective to the object name (ablation) | CLIP image-text matching · ORBIT Clean | CLIP Score: 24.07 ± 0.02 -> 23.88 ± 0.02 | Hurt |
| add color and material adjectives rather than color alone (ablation) | CLIP image-text matching · ORBIT Clean | CLIP Score: 25.20 ± 0.02 -> 24.76 ± 0.02 | Hurt |
| introduce blur | CLIP · ORBIT | accuracy: 11 percentage points | Hurt |
| introduce blur | CLIP · VizWiz-Classification | accuracy: 1 percentage point | Hurt |
| introduce occlusion | CLIP · ORBIT and VizWiz-Classification | accuracy: 9 and 14 percentage points | Hurt |
Surprising findings
- Larger pre-training datasets or architectures do not close the BLV accuracy gap; the disability gap stays roughly constant with scale.
- Describing objects by material lowers CLIP accuracy relative to color prompts, and adding material to color lowers CLIP score.
Limitations
- Zero-shot classification is a worst-case proxy for downstream assistive applications.
- Disability object analysis relies on objects from ORBIT only, with 2-synonym or 15-synonym string matching for caption counts.
- Downstream model analysis uses only 180 images across 20 objects and manual annotation by three annotators.
Experimental setup
Datasets: ORBIT VizWiz-Classification MS-COCO Open Images LAION-400M LAION-2B DataComp-1B
Benchmarks: ORBIT VizWiz-Classification MS-COCO Open Images
Metrics: zero-shot accuracy accuracy gap (percentage points) marginal effect on accuracy CLIP score detection confidence generation correctness
Baselines: CLIP zero-shot with LLM-generated descriptions (Menon & Vondrick) CuPL (Pratt et al.) ProtoNets 5-shot OWL-ViT CLIPSeg DALL-E2
Builds on: CLIP ProtoNets OWL-ViT CLIPSeg DALL-E2
Closest papers by meaning
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.87)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.87)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.86)
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.86)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.86)
- Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.