Research map / Video / Video-language understanding
SwinBERT: End-to-End Transformers With Sparse Attention for Video Captioning
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the input from 2 to 64 video frames (ablation) | SwinBERT without sparse attention · MSRVTT validation | CIDEr: 36.6 -> 55.3 | Helped |
| Increase the input from 2 to 64 video frames (ablation) | SwinBERT without sparse attention · VATEX validation | CIDEr: 47.4 -> 72.7 | Helped |
| Add a sparsity constraint to the learnable attention mask (ablation) | SwinBERT, 32 frames · MSRVTT validation | CIDEr: +2.8 | Helped |
| Add a sparsity constraint to the learnable attention mask (ablation) | SwinBERT, 32 frames · VATEX validation | CIDEr: +0.5 | Helped |
| Replace full attention with a fixed spatial-window pattern (ablation) | SwinBERT, 32 frames · MSRVTT validation | CIDEr: 52.3 -> 51.9 | Hurt |
| Replace full attention with a fixed temporal-window pattern (ablation) | SwinBERT, 32 frames · MSRVTT validation | CIDEr: 52.3 -> 51.0 | Hurt |
| Replace full attention with a fixed spatial-window pattern (ablation) | SwinBERT, 32 frames · VATEX validation | CIDEr: 71.1 -> 71.0 | Hurt |
| Replace full attention with a fixed temporal-window pattern (ablation) | SwinBERT, 32 frames · VATEX validation | CIDEr: 71.1 -> 70.2 | Hurt |
| Use learnable sparse attention instead of full attention (ablation) | SwinBERT, 32 frames · MSRVTT validation | CIDEr: 52.3 -> 55.1 | Helped |
| Use learnable sparse attention instead of full attention (ablation) | SwinBERT, 32 frames · VATEX validation | CIDEr: 71.1 -> 71.6 | Helped |
| Replace VidSwin with TimeSformer (ablation) | SwinBERT with full attention · MSRVTT | CIDEr: 52.3 -> 49.9 | Hurt |
| Add sparse attention (ablation) | TimeSformer-based captioner · MSRVTT | CIDEr: 49.9 -> 51.9 | Helped |
Closest papers by meaning
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.89)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.89)
- VideoPrism: A Foundational Visual Encoder for Video Understanding (ICML 2024 · similarity 0.88)
- Progress-Aware Video Frame Captioning (CVPR 2025 · similarity 0.88)
- SEAL: Semantic Attention Learning for Long Video Representation (CVPR 2025 · similarity 0.88)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.