Research map / Video / Video segmentation and tracking
CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects
Gabriel Fiastre, Antoine Yang, Cordelia Schmid
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LVISCap image pretraining to LV-VISCap training (ablation) | CaptionFormer · LV-VISCap val | CapA: 37.9 -> 43.6 | Improved |
| add LVISCap and LV-VISCap pretraining (vs COCO only) | CaptionFormer · VidSTG DVOC val | CapA: 44.3 -> 51.0 | Improved |
| add query temporal aggregation (32 clips) (ablation) | CaptionFormer · VidSTG DVOC val | CapA: 51.0 -> 55.4 | Improved |
| add temporal aggregation (ablation) | CaptionFormer · VLN DVOC val | CapA: 22.9 -> 23.4 | No clear effect |
| add temporal aggregation (ablation) | CaptionFormer · BenSMOT val | CIDEr: 39.9 -> 42.6 | Improved |
| add mask loss in detection loss (ablation) | CaptionFormer · VLN DVOC val | CHOTA: 45.4 -> 47.4 | Improved |
| weighted mean instead of arithmetic mean for aggregation (ablation) | CaptionFormer · VidSTG DVOC val | CapA: 49.1 -> 51.6 | Improved |
| add captioning loss (ablation) | CaptionFormer · LV-VISCap val | VIS mAP: 31.7 -> 34.2 | Improved |
| add bounding-box visual prompt plus category labels in text prompt (ablation) | CaptionFormer · caption generation (Gemini) | avg rating: 29.5 -> 80.7 | Improved |
| use mask boundaries instead of bounding boxes as visual prompt (ablation) | CaptionFormer · caption generation (Gemini) | avg rating: 85.1 -> 77.1 | Hurt |
| add LVISCap captions (ablation) | CaptionFormer · LV-VISCap manual subset | CapA: 22.9 -> 31.3 | Improved |
| use best-score single clip vs middle frame (ablation) | CaptionFormer · VidSTG DVOC val | CapA: 46.9 -> 51.0 | Improved |
Closest papers by meaning
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.92)
- OW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioning (NeurIPS 2024 · similarity 0.88)
- Segment and Caption Anything (CVPR 2024 · similarity 0.87)
- Event-Equalized Dense Video Captioning (CVPR 2025 · similarity 0.87)
- VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning (AAAI 2025 · similarity 0.87)
- DiffDVC: Accurate Event Detection for Dense Video Captioning via Diffusion Models (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
