Research map / Vision-language and multimodal / Cross-modal learning
Vx2Text: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs
Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, Lorenzo Torresani
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Vx2Text instead of Simple | AVSD captioning without caption input · AVSD test | CIDEr: 0.905 -> 1.357 | Improved |
| use Vx2Text instead of MTN-TMT | AVSD captioning with caption input · AVSD test | CIDEr: 1.357 -> 1.605 | Improved |
| use Vx2Text instead of BERT QA | TVQA · TVQA validation | Top-1 accuracy: 72.4 -> 74.9 | Improved |
| use Vx2Text instead of HERO with multimodal pretraining | TVQA · TVQA test | Top-1 accuracy: 73.6 -> 75.0 | Improved |
| use Vx2Text instead of MMT | TVC video captioning · TVC validation | CIDEr: 0.444 -> 0.482 | Improved |
| add video input (ablation) | Vx2Text · AVSD validation | performance | Improved |
| add each available modality (ablation) | Vx2Text · AVSD and TVQA validation | performance | Improved |
| add previous question-answer history (ablation) | Vx2Text · AVSD validation | performance | Improved |
| use differentiable tokenization instead of frozen tokenization (ablation) | Vx2Text · AVSD and TVQA validation | performance | Improved |
| use frozen tokenization instead of multimodal feature embedding (ablation) | Vx2Text · AVSD and TVQA validation | performance | Improved |
| use generative decoding instead of a discriminative classification head (ablation) | Vx2Text · TVQA with 10% of training data | Top-1 accuracy: 34.2% -> 64.1% | Improved |
| add cycle-consistency training (ablation) | Generative Vx2Text · TVQA with 10% of training data | Top-1 accuracy: 64.1% -> 66.1% | Improved |
Closest papers by meaning
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.89)
- Mimir: Improving Video Diffusion Models for Precise Text Understanding (CVPR 2025 · similarity 0.89)
- InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (ICLR 2024 · similarity 0.88)
- Emu: Generative Pretraining in Multimodality (ICLR 2024 · similarity 0.88)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.88)
- CoDi-2: In-Context Interleaved and Interactive Any-to-Any Generation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
