Research map / Vision-language and multimodal / Human and embodied modeling
Noise-Aware Image Captioning with Progressively Exploring Mismatched Words
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NIC, a word-level noise-aware training framework for image captioning that scores each word label's reliability via prediction entropy and subsequent-prediction cross-entropy, then fuses labels with model predictions into pseudo-labels over training.
Problem
Image captioning trained on web-crawled image-text pairs suffers from mismatched (noisy) text, and coarse example-level weighting fails to exploit the trustworthy words within partially matched captions.
Approach
Identify mismatched words rather than whole pairs by combining inter-modal representativeness (entropy of next-word prediction) and intra-modal discriminativeness (cross-entropy of subsequent predictions fed with the model's own predictions as mock labels), then build progressively trusted pseudo-word-labels modulated by training convergence.
Key results
- On MS-COCO uniform noise at 80%, NIC improves PureT's CIDEr by 6.2 (cross-entropy) and 5.3 (CIDEr optimization) and SPICE by 1.8/1.5.
- On CC152K, NIC outperforms all compared captioning methods on every metric, e.g., CIDEr 93.6 vs 89.8 for PureT under cross-entropy loss.
- Applying NIC to DLCT improves CIDEr by 6.5 (cross-entropy) on CC152K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply NIC to PureT | PureT · MS-COCO uniform noise 80% | CIDEr: 56.2 -> 62.4; 83.0 -> 88.3 | Helped |
| apply NIC to PureT | PureT · CC152K | CIDEr: 86.1 -> 93.6; 89.8 -> 95.5 | Helped |
| add prediction entropy evaluation (ablation) | NIC on PureT · MS-COCO uniform noise 80% | CIDEr: 87.0 -> 88.3 | Helped |
| add cumulative error evaluation (ablation) | NIC on PureT · MS-COCO uniform noise 80% | CIDEr: 84.7 -> 88.3 | Helped |
| add progressive coordination (ablation) | NIC on PureT · MS-COCO uniform noise 80% | CIDEr: 85.3 -> 88.3 | Helped |
Surprising findings
- Entropy-based and cross-entropy-based word reliability alone, without ever using clean/noisy labels for training, separate clean from noisy words well (accuracy and F1 reported across noise ratios).
Limitations
- Only validated with Transformer-based captioners and PureT as main baseline; applicability to other cross-modal models not tested.
- Requires additional decoding passes per word for subsequent prediction inspection, adding cost.
- Hyperparameters C and tau adapt via loss-curve curvature heuristics; sensitivity not fully analyzed.
Experimental setup
Datasets: MS-COCO Conceptual Captions CC152K
Benchmarks: MS-COCO CC152K
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L CIDEr SPICE
Baselines: AoANet ORT SGAE M2 Transformer X-Transformer RSTNet DLCT PureT SCD-Net
Builds on: PureT Self-Critical Sequence Training Transformer Wang et al. 2021 entropy-based label correction
Closest papers by meaning
- LEMoN: Label Error Detection using Multimodal Neighbors (ICML 2025 · similarity 0.89)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.88)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.88)
- Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning (AAAI 2024 · similarity 0.87)
- Panoptic Captioning: An Equivalence Bridge for Image and Text (NeurIPS 2025 · similarity 0.87)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.