Research map / Vision-language and multimodal / Human and embodied modeling
Cycle-Consistency Learning for Captioning and Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CyCo, a cycle-consistency framework that jointly trains visual grounding and regional image captioning (built on BLIP) so that each task supervises the other, enabling semi-weakly supervised grounding and a region-aware captioner.
Problem
Visual grounding needs costly image-text-box triplets, and standard image captioners describe only the whole image rather than arbitrary regions; the two tasks are typically trained independently.
Approach
Treat grounding and regional captioning as mutually inverse processes forming two cycles (VG→IC with a text-consistency loss, IC→VG with a box-consistency loss), with shared cross-attention/FFN weights, so each task generates pseudo labels for the other and weakly labeled data can be used.
Key results
- Fully supervised CyCo reaches 89.47 / 91.87 / 85.33 on RefCOCO val/testA/testB and 81.31 / 81.04 on RefCOCOg val/test, outperforming the BLIP-B* baseline's 88.73/91.07/84.27 and 80.23/80.06 in the ablation table.
- Semi-weakly supervised CyCo (20% fully annotated) reaches 86.73 on RefCOCO val and 77.81 on RefCOCOg test, versus 66.65 for fully-annotated-only 20% data on RefCOCOg test.
- On COCO Karpathy test, CyCo achieves BLEU@4 40.6 and SPICE 24.4, slightly above BLIP-B's 39.7 BLEU@4, while CIDEr is 133.9 vs BLIP's 133.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add cycle-consistency losses to supervised grounding training (ablation) | CyCo visual grounding · RefCOCO testA | top-1 accuracy: 91.07 -> 91.87 | Helped |
| Add cycle-consistency losses to supervised grounding training (ablation) | CyCo visual grounding · RefCOCOg test | top-1 accuracy: 80.06 -> 81.04 | Helped |
| Use CyCo online labeling for the 80% weakly annotated training data instead of offline pseudo labels (ablation) | CyCo visual grounding · RefCOCOg test | top-1 accuracy: 75.10 -> 77.81 | Helped |
| Add COCO captioning data to grounding training (ablation) | CyCo visual grounding · RefCOCOg test | top-1 accuracy: 81.04 -> 82.20 | Helped |
| Train BLIP on COCO Caption and RefCOCOg instead of only COCO (ablation) | BLIP captioning model · COCO Caption | CIDEr: 133.3 -> 130.2 | Hurt |
| Use CyCo rather than BLIP with COCO and RefCOCOg training data | Image captioning · RefCOCOg | CIDEr: 96.2 -> 128.5 | Helped |
Surprising findings
- Naively adding COCO-caption data to BLIP fails to improve its RefCOCOg captioning (CIDEr drops from 74.6 to 96.2 only when combined; COCO CIDEr drops from 133.3 to 130.2), whereas CyCo benefits from both datasets.
Limitations
- Grounding gains are reported with ViT-B backbone at fixed resolution; the authors note stronger backbones and higher resolution are left for future work and FIBER-B outperforms it.
- Semi-weakly results still lag fully supervised training (e.g., 77.81 vs 80.06 on RefCOCOg test in ablation), and only 20% fully annotated data was tested.
- Pre-training is on large image-text corpora, and the semi-weakly setting was evaluated only with a single 20%/80% split.
Experimental setup
Datasets: Visual Genome COCO SBU Conceptual 3M LAION (115M filtered) RefCOCO RefCOCO+ RefCOCOg COCO Karpathy
Benchmarks: RefCOCO RefCOCO+ RefCOCOg COCO Karpathy test
Metrics: top-1 accuracy BLEU@4 METEOR CIDEr SPICE
Baselines: MCN QRNet VLTVG VG-LAW TransVG++ UNITER VILLA ERNIE-ViL BLIP MDETR OFA FIBER
Builds on: BLIP GIoU loss BERT ViT
Closest papers by meaning
- Consensus Graph Representation Learning for Better Grounded Image Captioning (AAAI 2021 · similarity 0.90)
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.88)
- Comprehensive Visual Grounding for Video Description (AAAI 2024 · similarity 0.87)
- Improved Visual Grounding through Self-Consistent Explanations (CVPR 2024 · similarity 0.87)
- GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.87)
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.