Research map / Vision-language and multimodal / Human and embodied modeling
HAAV: Hierarchical Aggregation of Augmented Views for Image Captioning
Chia-Wen Kuo, Zsolt Kira
This paper tried a common change: Add a contrastive loss
Add contrastive loss to the shared encoder · MS-COCO Karpathy validation, cross-entropy only · CIDEr: 125.6 -> 127.6 Improved
What 1,317 accepted papers report for this change
- 1,253 improved
- 22 no clear effect
- 16 mixed
- 26 hurt
- Beyond Looking Up, Try Looking Around: Harmonizing Global Structure and Local Consistency in Optimal Transport for Short Text Clustering ICML 2026 · add contrastive learning · SearchSnippets · ACC: 56.65 -> 80.12Improved
- SleepLM: Natural-Language Intelligence for Human Sleep ICML 2026 · add contrastive objective to captioning (CoCa vs Cap) · sleep event IoU · IoU: 29.2 -> 27.4Hurt
- LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations ICML 2026 · add contrastive loss L_con · FS1000 average · Spearman correlation: 0.759 -> 0.789Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HAAV instead of Xmodal-Ctx | Image captioning · MS-COCO Karpathy test | CIDEr: 135.9 -> 141.5 | Improved |
| Use HAAV instead of ORT | Image captioning · Flickr30K Karpathy | CIDEr: 68.8 -> 81.7 | Improved |
| Add unlabeled MS-COCO images for semi-supervised training | HAAV · Flickr30K Karpathy + MS-COCO unlabeled | CIDEr: 81.7 -> 85.6 | Improved |
| Add contrastive loss to the shared encoder (ablation) | HAAV · MS-COCO Karpathy validation, cross-entropy only | CIDEr: 125.6 -> 127.6 | Improved |
| Use unshared rather than shared encoders without contrastive loss (ablation) | HAAV · MS-COCO Karpathy validation, cross-entropy only | CIDEr: 125.6 -> 125.4 | No clear effect |
| Use unshared encoders with contrastive loss instead of a shared encoder with contrastive loss (ablation) | HAAV · MS-COCO Karpathy validation, cross-entropy only | CIDEr: 127.6 -> 125.8 | Hurt |
| Replace hierarchical decoder with concatenation and one cross-attention (ablation) | HAAV · MS-COCO Karpathy validation, cross-entropy only | CIDEr: 127.6 -> 125.1 | Hurt |
| Replace view-level cross-attention with mean pooling (ablation) | HAAV · MS-COCO Karpathy validation, cross-entropy only | CIDEr: 127.6 -> 125.2 | Hurt |
| Replace view-level cross-attention with max pooling (ablation) | HAAV · MS-COCO Karpathy validation, cross-entropy only | CIDEr: 127.6 -> 122.9 | Hurt |
| Replace view-level cross-attention with sigmoid gating (ablation) | HAAV · MS-COCO Karpathy validation, cross-entropy only | CIDEr: 127.6 -> 125.5 | Hurt |
| Replace view-level cross-attention with tanh gating (ablation) | HAAV · MS-COCO Karpathy validation, cross-entropy only | CIDEr: 127.6 -> 124.8 | Hurt |
| Use HAAV rather than concatenated views (ablation) | Heterogeneous-view captioning · MS-COCO Karpathy validation | CIDEr: 122.8 -> 127.6 | Improved |
Closest papers by meaning
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (TPAMI 2022 · similarity 0.89)
- Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning (IJCV 2022 · similarity 0.87)
- Aligning Where to See and What to Tell: Image Captioning with Region-Based Attention and Scene-Specific Contexts (TPAMI 2016 · similarity 0.87)
- Hierarchy Parsing for Image Captioning (ICCV 2019 · similarity 0.87)
- Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.86)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
