Research map / Vision-language and multimodal / Multimodal LLMs
Same Person, Different Depiction: Counterfactual Evaluation of Vision-Language Models on Individuals with Limb Deficiencies
Heming Du, Jiaying Ying, Xin Chen, Sen Wang, Xue Li, Xin Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| show limb-deficiency cue in image (vs cue removed) | GPT-5 · InclusiveCFImageBias | mean rating shift: −0.089 | Improved |
| show limb-deficiency cue in image (vs cue removed) | Gemini-2.5 · InclusiveCFImageBias | mean rating shift: −0.404 | Improved |
| show limb-deficiency cue in image (vs cue removed) | Ministral-3 3B-Reasoning · InclusiveCFImageBias | mean rating shift: +0.071 | Hurt |
| show limb-deficiency cue in image (vs cue removed) | DeepSeek-VL2 · InclusiveCFImageBias | Yes/Unc. decision shift (pp): ∆Yes = −11.90 pp, ∆Unc. = +13.53 pp | Mixed |
| use Instruct instead of Thinking variants | Qwen3-VL · InclusiveCFImageBias | cue sensitivity (rating, regard) | Improved |
| keep limb cue visible in edit control (same-cue edit) (ablation) | Qwen-Image-Edit pipeline · InclusiveCFImageBias | rating shift: −0.151 -> −0.152 | No clear effect |
| same-cue edit control (ablation) | Qwen-Image-Edit pipeline · InclusiveCFImageBias | Yes-rate shift (pp): −3.58 pp -> −1.77 pp | Mixed |
| restrict to single-person pairs (ablation) | benchmark analysis · InclusiveCFImageBias (392 single-person pairs) | rating shift: −0.151 -> −0.169 | Improved |
| restrict to multi-person pairs (ablation) | benchmark analysis · InclusiveCFImageBias (70 multi-person pairs) | rating shift: −0.042 | No clear effect |
| compare original vs counterfactual instead of repeat of same input (ablation) | benchmark analysis · InclusiveCFImageBias | rating change: 0.122 -> 0.314 | Hurt |
| compare original vs counterfactual instead of repeat of same input (ablation) | benchmark analysis · InclusiveCFImageBias | decision TVD (pp): 1.81 pp -> 8.89 pp | Hurt |
| remove limb cue from image | VLMs · InclusiveCFImageBias | evaluative term frequency: 0.528 -> 0.396 | Improved |
Closest papers by meaning
- Explaining CLIP's Performance Disparities on Data from Blind/Low Vision Users (CVPR 2024 · similarity 0.86)
- Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (CVPR 2025 · similarity 0.85)
- CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models (NeurIPS 2026 · similarity 0.85)
- Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect (NeurIPS 2025 · similarity 0.85)
- Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models (NeurIPS 2024 · similarity 0.84)
- Understanding Task Transfer in Vision-Language Models (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
