Research map / Vision-language and multimodal / Cross-modal learning
TVLT: Textless Vision-Language Transformer
Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use raw audio rather than ASR text with YTT-S pretraining | TVLT · MSRVTT test | R@1: 19.3 -> 23.8 | Improved |
| Use raw audio rather than ASR text with YTT-S pretraining | TVLT · YouCook validation | R@1: 26.3 -> 32.8 | Improved |
| Use raw audio rather than ASR text with YTT-S pretraining | TVLT · CrossTask validation | R@1: 12.2 -> 15.3 | Improved |
| Use raw audio rather than ASR text with YTT-S pretraining | TVLT · PlacesAudio | R@1: 54.3 -> 49.0 | Hurt |
| Use raw audio rather than ASR text with YTT-S pretraining | TVLT · VQAv2 test-dev | accuracy: 63.2 -> 61.0 | Hurt |
| Use raw audio rather than ASR text with YTT-S pretraining | TVLT · CMU-MOSEI test | A2: 76.6 -> 76.8 | Improved |
| Use raw audio queries rather than caption text | TVLT · MSRVTT | R@1: 22.0 -> 20.1 | Hurt |
| Remove ASR from the inference pipeline | TVLT · 20s video / 8 frames | latency: 2916 -> 103 | Improved |
| Add speech-span-focused audio masking (ablation) | TVLT with 16×16 audio patches · MSRVTT | R@1: 21.7 -> 22.3 | Improved |
| Add speech-span-focused audio masking (ablation) | TVLT with 16×16 audio patches · VQAv2 | accuracy: 57.8 -> 58.6 | Improved |
| Use 2×128 rather than 16×16 audio patches (ablation) | TVLT with speech-focused masking · MSRVTT | R@1: 22.3 -> 21.2 | Hurt |
| Use 2×128 rather than 16×16 audio patches (ablation) | TVLT with speech-focused masking · VQAv2 | accuracy: 58.6 -> 59.2 | Improved |
Closest papers by meaning
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (ICML 2021 · similarity 0.89)
- MAGVLT: Masked Generative Vision-and-Language Transformer (CVPR 2023 · similarity 0.88)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.87)
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training (AAAI 2020 · similarity 0.87)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.86)
- EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
