Research map / Video / Video-language understanding
Dense Video Object Captioning from Disjoint Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes dense video object captioning (detect, track and caption all object trajectories) with an end-to-end GRiT-based model trained on disjoint pretraining datasets, evaluated with a new CHOTA metric on VidSTG and VLN.
Problem
Generating captions for all object trajectories in a video requires joint spatial, temporal and language understanding, but full captioned-trajectory annotations are scarce and per-frame captioning plus offline tracking yields temporally inconsistent captions.
Approach
An end-to-end model that adds a differentiable greedy identity-assignment tracker and trajectory feature aggregation on top of GRiT, trained with a mixture of disjoint datasets (COCO detection, Visual Genome dense captions, SMiT video captions, Aug-COCO tracking) so each loss is supervised by whichever annotations exist.
Key results
- On VidSTG validation, the end-to-end model (hard aggregation) reaches 54.9 CHOTA vs 52.3 for the best per-frame captioner plus ByteTrack baseline.
- Zero-shot (no full Dense VOC training) the full disjoint-pretrained model achieves 31.1 CHOTA on VidSTG and 29.2 on VLN; after finetuning it reaches 56.9 CHOTA on VidSTG, 2.0 above the best single-dataset pretraining.
- On BenSMOT the model reaches 90.19 HOTA vs 71.98 for the prior BeyondMOT baseline, and it achieves 65.1 on VLN Location-QA vs 62.1 for GRiT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace per-frame captioning plus ByteTrack with end-to-end tracking and hard aggregation | Dense VOC on VidSTG validation · VidSTG validation | CHOTA: 52.3 -> 54.9 | Helped |
| Replace per-frame captioning plus ByteTrack with end-to-end tracking and hard aggregation | Dense VOC on VidSTG validation · VidSTG validation | AssA: 60.2 -> 65.9 | Helped |
| Use hard rather than soft feature aggregation (ablation) | Dense VOC on VidSTG validation · VidSTG validation | CHOTA: 54.6 -> 54.9 | Helped |
| Add COCO detection pretraining to VG and SMiT pretraining (ablation) | Dense VOC zero-shot · VidSTG | DetA: 19.1 -> 51.3 | Helped |
| Add SMiT video-captioning pretraining to COCO and VG pretraining (ablation) | Dense VOC zero-shot · VidSTG | CapA: 8.1 -> 9.1 | Helped |
| Add Aug-COCO tracking pretraining to COCO, VG, and SMiT pretraining (ablation) | Dense VOC zero-shot · VidSTG | CHOTA: - -> 31.1 | Helped |
| Add all four disjoint pretraining datasets rather than VG-only pretraining (ablation) | Dense VOC finetuned · VidSTG | CHOTA: 54.9 -> 56.9 | Helped |
| Add all four disjoint pretraining datasets rather than VG-only pretraining (ablation) | Dense VOC finetuned · VLN | CHOTA: 40.6 -> 41.3 | Helped |
| Increase hard-aggregation sampling from m=1 to m=6 (ablation) | Dense VOC on VidSTG validation · VidSTG validation | CHOTA: 53.6 -> 54.9 | Helped |
| Increase hard-aggregation sampling from m=6 to m=8 (ablation) | Dense VOC on VidSTG validation · VidSTG validation | CHOTA: 54.9 -> 54.6 | Hurt |
| Replace UVO with Aug-COCO for tracking pretraining (ablation) | Dense VOC zero-shot · VidSTG | CHOTA: 24.8 -> 31.1 | Helped |
| Replace UVO with Aug-COCO for tracking pretraining (ablation) | Dense VOC finetuned · VidSTG | CHOTA: 50.9 -> 56.9 | Helped |
Surprising findings
- Zero-shot model trained on COCO only is poor at captioning, but adding Aug-COCO tracking supervision raises captioning accuracy (CapA) without any extra caption data.
- A generative captioner used for grounding by likelihood ranking outperforms dedicated spatial grounding models on VidSTG sIoU and VLN, despite not being trained for grounding.
Limitations
- The model produces only a single caption per trajectory and cannot caption multiple action segments within a trajectory.
- Existing grounding datasets were repurposed rather than a new dense captioning dataset being annotated.
- Temporal localization is weak because the model generates trajectories without conditioning on the query.
Experimental setup
Datasets: COCO Visual Genome SMiT Aug-COCO VidSTG VLN UVO BenSMOT
Benchmarks: VidSTG VLN BenSMOT VLN Location-QA
Metrics: CHOTA DetA AssA CapA AP_M HOTA CIDEr sIoU tIoU vIoU
Baselines: Per-frame captioning with IOU tracker Per-frame captioning with FairMOT Per-frame captioning with ByteTrack OW-VisCap BeyondMOT ReferFormer TubeDETR STCAT
Builds on: GRiT GIT HOTA GTR
Closest papers by meaning
- Segment and Caption Anything (CVPR 2024 · similarity 0.89)
- Comprehensive Visual Grounding for Video Description (AAAI 2024 · similarity 0.88)
- OW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioning (NeurIPS 2024 · similarity 0.88)
- Event-Equalized Dense Video Captioning (CVPR 2025 · similarity 0.88)
- VideoGEM: Training-free Action Grounding in Videos (CVPR 2025 · similarity 0.88)
- Implicit Location-Caption Alignment via Complementary Masking for Weakly-Supervised Dense Video Captioning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.