Research map / Vision-language and multimodal / Human and embodied modeling

Expanding the Scope of Negatives: Boosting Image-Text Matching with Negatives Distribution Guided Learning

AAAI 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Language-grounded scene text and structure · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Negatives Distribution Guided Learning (NDGL), which distills an ensemble's similarity matrix over all negatives into a margin regularization loss plus a hardest-negative rectification loss, applied on top of existing image-text matching models.

Problem

Hardest-negative mining in image-text matching ignores the semantic differences carried by the many other negatives, and hardest-negative selection is unstable early in training.

Approach

Build a target similarity matrix by averaging K models trained with hard negative mining (different initializations), then regularize the predicted pairwise similarity differences among all negatives toward this target distribution via a KL-based margin regularization, and rectify hardest negatives selected by the target similarity.

Key results

• NDGL applied to CHAN with BUTD+BERT raises Flickr30K RSUM from 519.3 to 533.8 (+14.5).
• With BUTD+BiGRU, CHAN+NDGL improves Flickr30K RSUM by 18.9% (505.3 to 524.2 as reported in Table 1).
• CHAN+NDGL reaches 538.3 RSUM on MS-COCO with BUTD+BERT, and GPO+NDGL reaches 552.3 RSUM on Flickr30K with ResNeXt-101+BERT, state-of-the-art in their tables.

Measured effects reported in the paper

Surprising findings

• A single model trained with NDGL is comparable to or surpasses the ensemble that serves as its reference distribution.
• NDGL with a small batch size matches the baseline trained with a much larger batch size, reducing dependence on batch size.

Limitations

• Requires training K models with hard negative mining to produce the target similarity matrix, adding training cost.
• Gains on ResNeXt-101 + BERT GPO are smaller on MS-COCO (+2.9 RSUM) than on weaker backbones.
• Evaluated only on MS-COCO and Flickr30K with image-text matching methods, not unified vision-language models.

Experimental setup

Datasets: MS-COCO Flickr30K

Benchmarks: MS-COCO 1K test Flickr30K test

Metrics: R@1 R@5 R@10 RSUM

Baselines: VSE++ GPO ESA HREM CHAN DivE RSUM CFM LSEH AOQ NAAF CGMN

Builds on: VSE++ triplet loss CHAN GPO label smoothing

Closest papers by meaning

• Generating Enhanced Negatives for Training Language-Based Object Detectors (CVPR 2024 · similarity 0.87)
• ReNeg: Learning Negative Embedding with Reward Guidance (CVPR 2025 · similarity 0.87)
• Noisy Correspondence Rectification via Asymmetric Similarity Learning (AAAI 2025 · similarity 0.87)
• Aligning Composed Query with Image via Discriminative Perception from Negative Correspondences (AAAI 2025 · similarity 0.86)
• TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.86)
• Achieving Ensemble-Like Performance in a Single Model: A Feature Diversification Framework for Image-Text Matching (AAAI 2025 · similarity 0.86)