Research map / Video / Video-language understanding
Video Repurposing from User Generated Content: A Large-scale Dataset and Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Repurpose-10K, a dataset of ~11K long UGC videos with 120K+ user-curated short clip annotations, plus a transformer baseline fusing audio, visual, and caption features with cross-modal alignment losses for long-to-short video repurposing.
Problem
Automatically selecting engaging, coherent short clips from long user-generated videos (video repurposing) lacks large-scale annotated data and suitable end-to-end methods, since existing highlight, summarization, and event localization datasets are domain-specific and do not match this task.
Approach
Annotations are collected from real UGC users who select and refine clips produced by a repurposing tool, and a multi-modal model jointly learns classification and 1-D IoU regression with a caption-enhanced encoder and uni-modal-to-multimodal KL alignment guidance.
Key results
- The proposed model reaches 11.57 average tIoU recall on Repurpose-10K, versus 10.16 for the best baseline (QD-DETR with V&A).
- Human study: Repurpose-10K clips score 7.286 total versus 7.676 for professional editors and 3.131 for random clips.
- Caption features alone give 9.59 average, and adding caption enhancement lifts audio+visual from 10.31 to 10.75 (without alignment).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use audio and visual modalities instead of audio alone (ablation) | Repurpose-10K model · Repurpose-10K test | Avg. tIoU: 9.66 -> 10.31 | Helped |
| use audio and visual modalities instead of visual alone (ablation) | Repurpose-10K model · Repurpose-10K test | Avg. tIoU: 9.52 -> 10.31 | Helped |
| add captions to audio and visual modalities (ablation) | Repurpose-10K model without auxiliary losses · Repurpose-10K test | Avg. tIoU: 10.31 -> 10.75 | Helped |
| add uni-modal focal loss (ablation) | Audio & visual model · Repurpose-10K test | Avg. tIoU: 10.31 -> 10.71 | Helped |
| add alignment loss after uni-modal focal loss (ablation) | Audio & visual model · Repurpose-10K test | Avg. tIoU: 10.71 -> 10.96 | Helped |
| add uni-modal focal loss (ablation) | Audio, visual & captions model · Repurpose-10K test | Avg. tIoU: 10.75 -> 11.28 | Helped |
| add alignment loss after uni-modal focal loss (ablation) | Audio, visual & captions model · Repurpose-10K test | Avg. tIoU: 11.28 -> 11.57 | Helped |
| use the 3/3/3 layer configuration instead of 1/4/4 (ablation) | Repurpose-10K model · Repurpose-10K test | Avg. tIoU: 10.48 -> 11.57 | Helped |
| use the 3/3/3 layer configuration instead of 5/2/2 (ablation) | Repurpose-10K model · Repurpose-10K test | Avg. tIoU: 9.48 -> 11.57 | Helped |
| use the 3/3/3 layer configuration instead of 7/1/1 (ablation) | Repurpose-10K model · Repurpose-10K test | Avg. tIoU: 8.57 -> 11.57 | Helped |
| use the proposed multimodal model instead of QD-DETR with audio and visual inputs | Repurpose-10K test models · Repurpose-10K test | Avg. tIoU: 10.16 -> 11.57 | Helped |
Surprising findings
- Caption features alone outperform the visual-only model at lower IoU thresholds.
- Highly imbalanced fusion depth (7/1/1) performs worse than single-modality models.
Limitations
- Annotations depend on clips that users chose to re-edit, so the dataset is curated toward user-preferred content.
- Evaluation relies on a single test split and a small human study (10 participants, 462 clips).
Experimental setup
Datasets: Repurpose-10K
Benchmarks: Repurpose-10K test set
Metrics: tIoU recall at 0.5-0.9 average tIoU recall human engagement and completeness scores
Baselines: SL-Module Moment-DETR UMT QD-DETR
Builds on: Moment-DETR QD-DETR focal loss WhisperX
Closest papers by meaning
- VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation (NeurIPS 2025 · similarity 0.88)
- Video ReCap: Recursive Captioning of Hour-Long Videos (CVPR 2024 · similarity 0.86)
- MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation (CVPR 2025 · similarity 0.85)
- LVD-2M: A Long-take Video Dataset with Temporally Dense Captions (NeurIPS 2024 · similarity 0.85)
- Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos (ICLR 2025 · similarity 0.85)
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.