Research map / Vision-language and multimodal / Human and embodied modeling
Identification of Necessary Semantic Undertakers in the Causal View for Image-Text Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NUIF, a causal-inference framework that estimates the probability-of-necessity of image regions and text words for image-text matching and uses it to weight fragment alignments.
Problem
Image-text matching needs to measure visual-semantic relevance, but not all fragment alignments between image regions and text words contribute equally; existing methods mine vital fragments without a principled notion of their contribution.
Approach
Treats the degree of semantic sharing as an effect and fragments as possible causes, defines a counterfactual probability-of-necessity PNf = (P(m|od) - P(m|o'd))/P(m|od) from semantic dependencies, and implements it in two variants (difference-based PNf-d and ratio-based PNf-r) to weight cross-modal alignments.
Key results
- On Flickr30K with BUTD+BERT, NUIF reaches rSum 535.9 versus 524.2 for HREM (prior best reported in table).
- On MSCOCO 5K with BUTD+BERT, NUIF reaches rSum 466.9 versus 450.9 for HREM.
- Ablation on Flickr30K with Bi-GRU: removing PNf drops rSum from 515.1 to 504.1 (NUIF-full vs. w/o PNf).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NUIF instead of NAAF | BUTD + Bi-GRU · Flickr30K | rSum: 513.2 -> 515.1 | Helped |
| Use NUIF instead of HREM | BUTD + BERT · Flickr30K | rSum: 524.2 -> 535.9 | Helped |
| Use NUIF instead of CMCAN | BUTD + Bi-GRU · MSCOCO 1K | rSum: 529.3 -> 530.6 | Helped |
| Use NUIF instead of HREM | BUTD + BERT · MSCOCO 1K | rSum: 534.6 -> 542.3 | Helped |
| Use NUIF instead of Set-Based | BUTD + Bi-GRU · MSCOCO 5K | rSum: 437.8 -> 439.8 | Helped |
| Use NUIF instead of HREM | BUTD + BERT · MSCOCO 5K | rSum: 450.9 -> 466.9 | Helped |
| Add probability-of-necessity weighting (ablation) | NUIF, BUTD + Bi-GRU · Flickr30K | rSum: 504.1 -> 515.1 | Helped |
| Add probability-of-necessity weighting (ablation) | NUIF, BUTD + BERT · Flickr30K | rSum: 525.2 -> 535.9 | Helped |
| Add probability-of-necessity weighting (ablation) | NUIF, BUTD + Bi-GRU · MSCOCO 1K | rSum: 522.3 -> 530.6 | Helped |
| Add probability-of-necessity weighting (ablation) | NUIF, BUTD + BERT · MSCOCO 1K | rSum: 536.8 -> 542.3 | Helped |
| Add probability-of-necessity weighting (ablation) | NUIF, BUTD + Bi-GRU · MSCOCO 5K | rSum: 427.9 -> 439.8 | Helped |
| Add probability-of-necessity weighting (ablation) | NUIF, BUTD + BERT · MSCOCO 5K | rSum: 456.9 -> 466.9 | Helped |
Surprising findings
- MSCOCO gains from necessity identification are smaller than Flickr30K gains, which the authors attribute to MSCOCO's much smaller proportion of long texts and hence weaker causality.
Limitations
- Semantic dependency gathering relies on heuristics (spatial proximity for regions, noun phrases for words), which can wrongly assign necessity, as in the visualization example.
- Gains on MSCOCO are smaller because short texts give weak causality.
Experimental setup
Datasets: Flickr30K MSCOCO
Benchmarks: Flickr30K MSCOCO 1K MSCOCO 5K
Metrics: R@1 R@5 R@10 rSum
Baselines: GSMN GPO SGRAF CMCAN NAAF CHAN Set-Based VSRN++ MV-VSE HREM
Builds on: SCAN cross-attention alignment counterfactual probability-of-necessity (Pearl) VSE++ hardest-negative triplet loss
Closest papers by meaning
- ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.87)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.87)
- MASS: Overcoming Language Bias in Image-Text Matching (AAAI 2025 · similarity 0.86)
- ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.86)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.86)
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.