Research map / Video / Video-language understanding
REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes REGen, a hybrid long-to-short video editing framework in which a finetuned LLaMA writes a teaser script with quote placeholders and a BART-based multitask quote retriever fills them with interview clips from the source documentary.
Problem
Abstractive video teaser generation cannot quote footage from the source video, while extractive summaries lack a coherent narrative; there is also no dataset annotating quoted footage versus narration.
Approach
Two-stage retrieval-embedded generation: an LLM emits <QUOTE> placeholders (or direct-quote spans) in a script, and a multitask BART retriever (masked infilling plus contrastive retrieval over a sentence/visual embedding of candidate clips) selects the interview clip that best supports the surrounding narration.
Key results
- QuoteRetriever-TV reaches Recall@10 of 30.00% vs 22.50% for GPT-4o infilling on quote retrieval.
- In the subjective study, REGen-IDQ-TV scores highest on coherence (3.29), alignment (3.30) and realness (3.05) versus baselines including TeaserGen and GPT-4o-SP-DQ.
- In a lecture-video A/B preference study, participants preferred TeaserGen 62% vs 38% for REGen-IDQ-TV, so REGen did not win on that domain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaMA with <SOQ> and <EOQ> direct-quote markers | LLaMA · Direct quotation from long content | Overlap ratio: 0 -> 0.07 | Helped |
| Use QuoteRetriever-T instead of GPT-4o infilling | Quote retrieval · Quote retrieval | Recall@5: 13.89 ± 1.27 -> 17.50 | Helped |
| Use the learned quote retriever instead of GPT-4o infilling | Quote retrieval · Quote retrieval | Insertion effectiveness: 2.48 ± 0.31 -> 3.56 ± 0.22 | Helped |
| Add multimodal visual information to text-only retrieval | QuoteRetriever · Quote retrieval | Recall@5: 17.50 -> 15.00 | Hurt |
| Use 256 instead of 128 maximum context tokens (ablation) | QuoteRetriever-TV · Quote retrieval | Recall@5: 17.50 -> 17.50 | No clear effect |
| Set α to 1 rather than 0 (ablation) | QuoteRetriever · Quote retrieval | Recall@5: 6.67 -> 20.00 | Helped |
| Use 30% same-document interviews as hard negatives rather than all or none (ablation) | QuoteRetriever · Quote retrieval | Recall@5: 17.50 -> 20.00 | Helped |
| Use contrastive loss instead of L2 loss (ablation) | QuoteRetriever · Quote retrieval | Recall@5: 6.67 -> 20.00 | Helped |
| Move the <SUM> retrieval token from the start to the end (ablation) | QuoteRetriever · Quote retrieval | Recall@5: 10.00 -> 20.00 | Helped |
| Use REGen-IDQ-TV instead of TeaserGen | Documentary teaser generation · Documentary teaser generation | Coherence: 3.22 ± 0.23 -> 3.29 ± 0.24 | Helped |
| Use REGen-IDQ-TV instead of TeaserGen | Documentary teaser generation · Documentary teaser generation | Alignment: 2.92 ± 0.24 -> 3.30 ± 0.26 | Helped |
| Use REGen-IDQ-TV instead of TeaserGen | Documentary teaser generation · Documentary teaser generation | Realness: 2.86 ± 0.23 -> 3.05 ± 0.25 | Helped |
Surprising findings
- Vanilla LLaMA could not produce meaningful quotes, and GPT-4o given the full transcript reached only 0.13 overlap ratio on direct quoting from long context.
- In the lecture-video A/B study, the abstractive TeaserGen was preferred over REGen, showing the extractive-grounded approach does not generalize uniformly.
Limitations
- Risk of misplacing a quote in the wrong narrative context.
- Relies on successful speaker diarization for segmentation, which may not transfer to domains like lecture recordings.
- LLaMA's limited context window prevents giving all quote candidates to the script generator.
Experimental setup
Datasets: DocumentaryNet WhisperX transcripts with speaker diarization
Benchmarks: DocumentaryNet teaser test set
Metrics: QDI QCR overlap ratio ROUGE-1 ROUGE-2 ROUGE-L G-Eval Recall@1 Recall@5 Recall@10
Baselines: Random extraction Extractive-then-Smoothing A2Summ TeaserGen GPT-4o-DQ GPT-4o-SP-DQ GPT-4o infilling LLaMA-DQ
Builds on: retrieval-augmented generation TeaserGen A2Summ BART UniVTG
Closest papers by meaning
- TeaserGen: Generating Teasers for Long Documentaries (ICLR 2025 · similarity 0.87)
- Video Summarization with Large Language Models (CVPR 2025 · similarity 0.84)
- Video Repurposing from User Generated Content: A Large-scale Dataset and Benchmark (AAAI 2025 · similarity 0.84)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.84)
- Video ReCap: Recursive Captioning of Hour-Long Videos (CVPR 2024 · similarity 0.84)
- Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.