Research map / Vision-language and multimodal / Human and embodied modeling
With a Little Help from Your Own Past: Prototypical Memory Networks for Image Captioning
Manuele Barraco, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add prototypical memory (prototype vectors from past keys/values) to the decoder (ablation) | Transformer captioner · COCO Karpathy test (cross-entropy) | CIDEr: 127.8 -> 131.5 | Improved |
| add prototypical memory under CIDEr optimization (SCST) | Transformer captioner · COCO Karpathy test (CIDEr optimization) | CIDEr: 140.3 -> 144.1 | Improved |
| add plain learnable memory vectors instead of prototypes (ablation) | Transformer captioner decoder · COCO Karpathy test (cross-entropy) | CIDEr: 127.8 -> 127.9 (64); 127.6 (1024) | No clear effect |
| increase number of prototypes from 256 to 1024 (ablation) | PMA-Net · COCO Karpathy test (cross-entropy) | CIDEr: 129.4 -> 131.5 | Improved |
| increase memory bank size T from 500 to 1500 (ablation) | PMA-Net · COCO Karpathy test (cross-entropy) | CIDEr: 128.6 -> 131.5 | Improved |
| add segment embeddings to distinguish prototypes from input keys (ablation) | PMA-Net · COCO Karpathy test (cross-entropy) | CIDEr: 130.1 -> 131.5 | Improved |
| add prototype memory in the first decoder layer too (ablation) | PMA-Net · COCO Karpathy test (cross-entropy) | CIDEr: 129.2 -> 131.5 | Improved |
| add prototype memory on robust COCO to reduce hallucination | Transformer captioner · Robust COCO test | CHs: 4.6 -> 4.3 | Improved |
| add prototype memory for novel object captioning | Transformer captioner · nocaps validation out-of-domain | CIDEr: 73.9 -> 75.9 | Improved |
Closest papers by meaning
- Meshed-Memory Transformer for Image Captioning (CVPR 2020 · similarity 0.90)
- Partially-Supervised Image Captioning (NeurIPS 2018 · similarity 0.89)
- An Empirical Study of Language CNN for Image Captioning (ICCV 2017 · similarity 0.89)
- Reflective Decoding Network for Image Captioning (ICCV 2019 · similarity 0.89)
- Convolutional Image Captioning (CVPR 2018 · similarity 0.89)
- Text-Guided Attention Model for Image Captioning (AAAI 2017 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
