Research map / Vision-language and multimodal / Human and embodied modeling
Semantic and Expressive Variations in Image Captions Across Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures how image captions vary semantically (scene graphs) and expressively (LIWC measures, embeddings) across seven languages in human (Crossmodal) and model (LLaVA, Google Vertex) captions, and tests whether GIT finetuned on one language performs best on that language.
Problem
Vision-language models are mostly trained on English image-text pairs, possibly biasing them against perceptual modes and visual concepts expressed in other languages. The paper asks whether captions differ semantically and expressively across languages in datasets and model outputs.
Approach
Compare monolingual against multilingual unions of caption sets for the same images, after translating everything to English with GPT-4, to show that different languages surface distinct objects, relations, attributes and expressive tones; additionally show that finetuning on one language's captions yields the best performance on that language's test set.
Key results
- Multilingual unioned scene graphs of XM captions have 46.0% more objects, 66.1% more relations and 66.8% more attributes than English-only ones.
- LLaVA multilingual captions cover 21.9% more objects, 18.8% more relations and 20.1% more attributes than monolingual captions.
- GIT finetuned on Japanese captions scores SPICE .27 on Japanese references versus .23 on English references, and models finetuned on language X perform best on language X's test set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine human captions across languages instead of using one language | Crossmodal captions · Crossmodal | objects: 2.98 -> 4.35 | Helped |
| Combine human captions across languages instead of using one language | Crossmodal captions · Crossmodal | relations: 1.77 -> 2.94 | Helped |
| Combine human captions across languages instead of using one language | Crossmodal captions · Crossmodal | attributes: 1.78 -> 2.97 | Helped |
| Generate captions in multiple languages rather than one language | LLaVA · Crossmodal captions | objects: 4.78 -> 5.93 | Helped |
| Generate captions in multiple languages rather than one language | LLaVA · Crossmodal captions | relations: 3.95 -> 4.54 | Helped |
| Generate captions in multiple languages rather than one language | Vertex API · Crossmodal captions | objects: 3.48 -> 4.17 | Helped |
| Generate captions in multiple languages rather than one language | Vertex API · Crossmodal captions | relations: 2.77 -> 3.40 | Helped |
| Fine-tune on captions in the evaluation language rather than English captions | GIT · Crossmodal validation, Japanese | SPICE F-score: 0.23 -> 0.27 | Helped |
| Fine-tune on captions in the evaluation language | GIT · Crossmodal validation across languages | SPICE F-score | Helped |
Surprising findings
- Same-model across-language scene graphs are nearly as large as same-language across-model scene graphs (92.4% objects, 98.4% relations), suggesting differences are not only due to 'language modes'.
- Models finetuned on a language's captions do best on that language, and finetuning on multilingual data is consistently strong across evaluation sets.
Limitations
- Limited to seven languages and mid-sized scale of data
- Translation to English via GPT-4 may introduce minor artifacts, though human evaluation found it near perfect
Experimental setup
Datasets: Crossmodal (XM) Visual Genome FACTUAL-MR
Benchmarks: Crossmodal (XM) Visual Genome
Metrics: unique objects, relations, attributes in unioned scene graph concreteness authenticity tone embedding coverage SPICE F-score
Baselines: GIT BLIP LLaVA Google Vertex API imagetext-001
Builds on: Crossmodal dataset GIT LIWC
Closest papers by meaning
- Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.93)
- See It from My Perspective: How Language Affects Cultural Bias in Image Understanding (ICLR 2025 · similarity 0.91)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.89)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.89)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.89)
- AltDiffusion: A Multilingual Text-to-Image Diffusion Model (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.