Research map / Video / Video-language understanding
End-to-End Generative Pretraining for Multimodal Video Captioning
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use bidirectional future/present utterance generation for multimodal pretraining | MV-GPT · YouCook2 | CIDEr: 1.98 -> 2.14 | Helped |
| Add backward generation to forward generation pretraining (ablation) | MV-GPT · YouCook2 | CIDEr: 2.05 -> 2.10 | Helped |
| Apply MLM to decoder outputs rather than encoder outputs (ablation) | MV-GPT with forward generation · YouCook2 | CIDEr: 2.05 -> 2.09 | Helped |
| Add weight decay (ablation) | MV-GPT · YouCook2 | CIDEr: 2.11 -> 2.14 | Helped |
| Jointly pretrain encoder and decoder instead of pretraining the encoder only (ablation) | video captioning pretraining · YouCook2 | CIDEr: 1.46 -> 1.88 | Helped |
| Fine-tune the visual encoder end-to-end from the start (ablation) | MV-GPT · YouCook2 | CIDEr: 2.20 -> 2.14 | Hurt |
| Initially freeze the visual encoder, then fine-tune end-to-end after convergence (ablation) | MV-GPT · YouCook2 | CIDEr: 2.14 -> 2.21 | Helped |
| Fine-tune the visual encoder end-to-end (ablation) | MV-GPT · MSR-VTT | CIDEr: 0.55 -> 0.60 | Helped |
| Pretrain with MV-GPT from random initialization rather than use public weights without MV-GPT pretraining (ablation) | MV-GPT · YouCook2 | CIDEr: 1.03 -> 2.09 | Helped |
| Include visual input during multimodal pretraining rather than pretrain with text only (ablation) | MV-GPT · YouCook2 | CIDEr: 1.43 -> 2.14 | Helped |
| Add generative losses to Bi-NCE retrieval pretraining (ablation) | video retrieval · MSR-VTT | R@1: 31.6 -> 33.6 | Helped |
| Add ASR inputs to retrieval pretraining (ablation) | MV-GPT retrieval · MSR-VTT | R@1: 33.6 -> 37.3 | Helped |
Closest papers by meaning
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.89)
- Emu: Generative Pretraining in Multimodality (ICLR 2024 · similarity 0.88)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.87)
- VideoPoet: A Large Language Model for Zero-Shot Video Generation (ICML 2024 · similarity 0.87)
- OmniViD: A Generative Framework for Universal Video Understanding (CVPR 2024 · similarity 0.87)
- JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.