Research map / Image generation / Autoregressive and latent generation
Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning
Jyoti Aneja, Harsh Agrawal, Dhruv Batra, Alexander Schwing
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ELMo-based backward LSTM in the encoder (ablation) | Seq-CVAE · MSCOCO ablation (100 samples, latent 512) | CIDEr (C@100): 1.089 -> 1.568 | Improved |
| use a per-word sequential latent instead of a single latent variable (ablation) | Seq-CVAE with ELMo · MSCOCO ablation (20 samples) | CIDEr: 0.952 -> 1.448 | Improved |
| learn the intention model (LSTM) instead of a constant Gaussian (ablation) | Seq-CVAE with ELMo · MSCOCO ablation (20 samples) | CIDEr: 1.244 -> 1.448 | Improved |
| use separate LSTMs for encoder, decoder and transition network (ablation) | Seq-CVAE · MSCOCO ablation (20 samples) | CIDEr: 1.14 -> 1.448 | Improved |
| replace pretrained ELMo with a backward RNN trained on COCO (ablation) | Seq-CVAE · MSCOCO ablation (20 samples) | CIDEr: 1.448 -> 1.348 | No clear effect |
| condition the intention model on previous words x_<t (ablation) | Seq-CVAE with ELMo · MSCOCO ablation (20 samples, latent 512) | CIDEr (C@20): 1.332 -> 1.332 | No clear effect |
| condition the intention model on previous words x_<t (ablation) | Seq-CVAE with ELMo · MSCOCO ablation (100 samples, latent 256) | CIDEr (C@100): 1.559 -> 1.575 | Improved |
| learned LSTM intention instead of constant Gaussian, without ELMo (ablation) | Seq-CVAE without ELMo · MSCOCO ablation (100 samples, latent 512) | CIDEr (C@100): 1.082 -> 1.089 | No clear effect |
| reduce latent size from 512 to 128 (ablation) | Seq-CVAE with ELMo · MSCOCO ablation (100 samples) | CIDEr (C@100): 1.573 -> 1.571 | No clear effect |
| use Seq-CVAE instead of POS for diversity | Seq-CVAE vs POS · MSCOCO M-RNN test (20 samples, best-5) | Div-2: 0.35 -> 0.54 | Improved |
| use Seq-CVAE instead of POS for novel sentences | Seq-CVAE vs POS · MSCOCO M-RNN test (100 samples, best-5) | # novel sentences: 3446 -> 4215 | Improved |
| sample from Seq-CVAE instead of beam search (20 samples) | Seq-CVAE vs beam search · MSCOCO M-RNN test (20 samples, best-1 oracle) | CIDEr: 1.595 -> 1.448 | Hurt |
Closest papers by meaning
- Diverse Image Captioning with Context-Object Split Latent Spaces (NeurIPS 2020 · similarity 0.90)
- On Diversity in Image Captioning: Metrics and Methods (TPAMI 2020 · similarity 0.88)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.87)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.85)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.85)
- Causal Graphical Models for Vision-Language Compositional Understanding (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
