Research map / Vision-language and multimodal / Multimodal LLMs

Culture Affordance Atlas: Reconciling Object Diversity Through Functional Mapping

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Re-annotates the Dollar Street dataset with function-centric labels (46 functions, 288 objects, 367 pairs) and shows CLIP/SigLIP2 retrieval with function prompts narrows high-low income performance gaps.

Problem

Mainstream vision-language datasets are object-centric and skewed toward higher-income Western contexts, so culturally distinct household objects from lower-income and non-Western regions are missing or mislabeled, degrading VL model performance for those groups.

Approach

Organize objects by the universal human functions they fulfill (grounded in Brown's Human Universals with ethnographic citations) rather than by Western object names, re-annotating Dollar Street with function descriptions and culturally grounded object labels, and use function prompts with CLIP to reduce income-based recall gaps.

Key results

• Function-based prompts reduce the per-topic high-low income recall gap by a median of 6 pp for CLIP (Wilcoxon p = 1.62e-17).
• Function prompts reduce the high-low income gap by 11% on SigLIP2 (p = 5.9e-11).
• Human validation of generated functional descriptions averages 90% across seven countries, all above 85%.
• 34 of 100 long-tail objects are absent from all seven popular VL datasets; only OpenImages covers more than half.

Surprising findings

• Function prompts lower overall retrieval recall relative to topic prompts, a trade-off for equity gains.
• Fleiss' kappa is low (e.g., 0.04 for Nigeria) despite high raw agreement, attributed to class imbalance.

Limitations

• Atlas citations are mostly single ethnographic sources (89% have one reference), and documentation is not exhaustive.
• Validation involved only 21 participants across seven countries.
• Function prompts trade off overall retrieval accuracy and can cause contextually incorrect retrievals due to CLIP's noun emphasis.

Experimental setup

Datasets: Dollar Street ImageNet COCO Open Images LVIS GeoDE YFCC100M Entity (Artifacts)

Benchmarks: Dollar Street (re-annotated)

Metrics: recall CLIP alignment score slope of income trend Wilcoxon signed-rank p-value human validation percentage Gwet's AC1 PABAK

Baselines: topic-image prompts (original Dollar Street topic labels)

Builds on: Dollar Street Human Universals (Brown) CLIP Gibson affordance theory

Closest papers by meaning

• No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models (NeurIPS 2024 · similarity 0.89)
• See It from My Perspective: How Language Affects Cultural Bias in Image Understanding (ICLR 2025 · similarity 0.89)
• World in a Frame: Understanding Culture Mixing as a New Challenge for Vision-Language Models (CVPR 2026 · similarity 0.88)
• Beyond Aesthetics: Cultural Competence in Text-to-Image Models (NeurIPS 2024 · similarity 0.88)
• CulturePark: Boosting Cross-cultural Understanding in Large Language Models (NeurIPS 2024 · similarity 0.88)
• Culture In a Frame: C$^3$B as a Comic-Based Benchmark for Multimodal Culturally Awareness (ICLR 2026 · similarity 0.87)