Research map / Vision-language and multimodal / Human and embodied modeling
MASS: Overcoming Language Bias in Image-Text Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MASS, a training-free inference-time score that debiases language priors in autoregressive image captioning models by computing per-token PMI between image-conditioned and null-image (black image) likelihoods, improving image-text matching on color, counting, gender, Winoground and SVO-Probes benchmarks.
Problem
Image-text matching with pretrained visual-language models suffers from language bias: models rely on text priors and neglect visual content, and captioning-based likelihoods conflate linguistic plausibility with image-text association.
Approach
Score image-text pairs by the average token-level pointwise mutual information, approximating the marginal likelihood with a black null image instead of sampling random images, so the text prior is subtracted out at inference time with no retraining.
Key results
- On Winoground with OFA-large, MASS raises Groupscore from 9.5 (ITM) and 15.8 (TL) to 20.3, the only backbone-scoring variant above chance (16.7).
- On MS-COCO text-to-image retrieval, MASS reaches Recall@1 55.6 versus 36.5 for CLIP and 38.0 for OFA TL while keeping gender bias near the best debiasing baselines.
- On the VALSE counting adversarial set, MASS reaches 76.7 versus 72.0 for OFA-large ITM and 66.8 for TL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace token likelihood with MASS | OFA-large · VALSE balanced | pairwise ranking accuracy: 65.4 -> 70.0 | Helped |
| Replace token likelihood with MASS | OFA-large · VALSE Small | pairwise ranking accuracy: 71.1 -> 73.2 | Helped |
| Replace token likelihood with MASS | OFA-large · VALSE Adv. | pairwise ranking accuracy: 66.8 -> 76.7 | Helped |
| Replace token likelihood with MASS | OFA-large · MS-COCO test T2I | Recall@1: 38.0 -> 55.6 | Helped |
| Replace token likelihood with MASS | OFA-large · MS-COCO test I2T | Recall@1: 33.4 -> 57.3 | Helped |
| Replace token likelihood with MASS | OFA-large · Winoground Group | Groupscore: 15.8 -> 20.3 | Helped |
| Replace token likelihood with MASS | OFA-large · SVO-Probes Group | Groupscore: 12.5 -> 43.0 | Helped |
| Use MASS instead of ITM | OFA-large · SVO-Probes Object | Groupscore: 63.2 -> 55.3 | Hurt |
| Use MASS instead of ITM | OFA-large · Winoground Text | Textscore: 26.4 -> 32.0 | Helped |
| Use MASS instead of ITM | OFA-large · Winoground Image | Imagescore: 16.5 -> 31.5 | Helped |
| Use MASS instead of ITM | OFA-large · Winoground Group | Groupscore: 9.5 -> 20.3 | Helped |
| Use MASS instead of ITM | OFA-large · SVO-Probes Group | Groupscore: 39.7 -> 43.0 | Helped |
Surprising findings
- Raw token likelihood prefers its language bias over image context in about a third of color categories in the Natural Colors experiment.
- On Winoground, MASS underperforms ITM on Textscore for OFA-base and BLIP-2 (OPT) variants, and lags ITM in the Object subset of SVO-Probes.
Limitations
- Relies on null-image (black image) approximation of the marginal likelihood, whose accuracy is not verified against true marginals.
- Requires two forward passes per image-text pair, doubling inference cost.
- Addresses only language bias and social bias as language bias; does not handle visual bias, and underperforms ITM on some subsets (e.g., SVO-Probes Object, Winoground Textscore for some backbones).
Experimental setup
Datasets: Natural Colors Dataset VALSE MS-COCO Winoground SVO-Probes
Benchmarks: Natural Colors Dataset VALSE counting MS-COCO gender bias retrieval Winoground SVO-Probes
Metrics: pairwise ranking accuracy Bias@K Recall@K Textscore Imagescore Groupscore
Baselines: CLIP OFA ITM OFA token likelihood SCAN FairSample CLIP-clip LXMERT 12-in-1 VinVL BLIP-2
Builds on: OFA CLIP BLIP-2 Winoground VALSE
Closest papers by meaning
- CLIP the Bias: How Useful is Balancing Data in Multimodal Learning? (ICLR 2024 · similarity 0.89)
- The Alignment Illusion in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.88)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.88)
- MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? (NeurIPS 2025 · similarity 0.88)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.87)
- Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.