Research map / Label-efficient and robust learning / Semi-supervised learning
Momentum Pseudo-Labeling for Weakly Supervised Phrase Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Momentum Pseudo Labeling (MPL), which uses an EMA momentum dual encoder to globally update phrase-region pseudo-labels for weakly supervised phrase grounding, plus false negative elimination/conversion in the contrastive loss.
Problem
Weakly supervised phrase grounding learns phrase-region alignment from image-caption pairs only; prior EM-based contrastive methods update pseudo-labels only locally within a batch (slow, sub-optimal) and treat regions from non-matching images as negatives, causing false negatives.
Approach
Use an EMA momentum copy of the dual encoder to compute pseudo-labels for all regions in the dataset on the fly (approximating a global E-step at low cost), and handle false negatives from non-matching image-caption pairs by eliminating or converting them to positives based on visual region similarity.
Key results
- MPL FNC reaches 64.15% on Flickr30k test, 70.19% on RefCOCO testA and 55.74% on RefCOCO+ testA, outperforming CLEM by 1.1 points on Flickr30k.
- On RefCOCO testA/testB, MPL improves over CLEM by 3.6/1.1 points; on RefCOCO+ testA/testB by 4.1/1.7 points.
- Momentum coefficient 0.99 gives the best Flickr30k validation accuracy (62.25%) versus 61.59% without momentum (gamma=0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MPL FNC rather than CPL | RN101 weakly supervised phrase grounding · Flickr30k Entities | IoU@0.5 accuracy: 63.87 -> 64.15 | Helped |
| Use MPL FNC rather than CPL | RN101 weakly supervised phrase grounding · RefCOCO testA | IoU@0.5 accuracy: 69.77 -> 70.19 | Helped |
| Use MPL FNC rather than CPL | RN101 weakly supervised phrase grounding · RefCOCO testB | IoU@0.5 accuracy: 63.44 -> 55.74 | Hurt |
| Use MPL FNC rather than CLEM | RN101 weakly supervised phrase grounding · RefCOCO+ testA | IoU@0.5 accuracy: 59.51 -> 63.59 | Helped |
| Use MPL FNC rather than CPL | RN101 weakly supervised phrase grounding · RefCOCO+ testB | IoU@0.5 accuracy: 45.52 -> 45.20 | Hurt |
| Add EMA (ablation) | MPL with FNC · Flickr30k val | accuracy: 62.09 -> 62.53 | No clear effect |
| Add FNC (ablation) | MPL with EMA · Flickr30k val | accuracy: 62.25 -> 62.53 | Helped |
| Add FNE (ablation) | MPL with EMA · Flickr30k val | accuracy: 62.25 -> 62.35 | Helped |
| Add FNE (ablation) | MPL with EMA · RefCOCO val | accuracy: 63.82 -> 63.45 | Hurt |
| Add FNC (ablation) | MPL with EMA · RefCOCO testA | accuracy: 69.77 -> 70.19 | Helped |
| Use EMA momentum coefficient γ=0.9 rather than γ=0 (ablation) | MPL · Flickr30k val | accuracy: 61.59 -> 62.22 | Helped |
| Increase momentum coefficient from γ=0.99 to γ=0.999 (ablation) | MPL · Flickr30k val | accuracy: 62.25 -> 62.20 | Hurt |
Surprising findings
- Global pseudo-label updates improve accuracy but are costly; MPL approaches global-update performance at near-local-update cost.
Limitations
- Relies on a frozen object detector (Faster R-CNN) for region features, limiting region quality.
- False negative detection uses hand-set similarity thresholds (0.85 and 0.95) that affect performance.
- Still below some methods that use extra pre-training or multimodal model knowledge on certain splits.
Experimental setup
Datasets: Flickr30k Entities RefCOCO RefCOCO+
Benchmarks: Flickr30k Entities RefCOCO RefCOCO+
Metrics: accuracy (IoU@0.5)
Baselines: ARN W-visualBERT Pseudo-Q CPL CCL InfoGround MAF KD+CL CLEM RefCLIP
Builds on: Expectation-Maximization MoCo CLEM MAF ALBEF
Closest papers by meaning
- Similarity Maps for Self-Training Weakly-Supervised Phrase Grounding (CVPR 2023 · similarity 0.90)
- Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding (CVPR 2024 · similarity 0.85)
- Improved Visual Grounding through Self-Consistent Explanations (CVPR 2024 · similarity 0.85)
- Cycle-Consistency Learning for Captioning and Grounding (AAAI 2024 · similarity 0.85)
- Pre-Trained Vision-Language Models as Noisy Partial Annotators (AAAI 2025 · similarity 0.85)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.