Research map / Video / Video-language understanding
Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FVE, a fine-grained video editing diffusion framework that turns seen source-domain video moments into synthetic moments with unseen novel concepts (from text only) and selects them with a hybrid metric to augment video moment retrieval training.
Problem
Video moment retrieval models generalise poorly to queries containing novel semantic concepts unseen in both video and text of the source domain, and existing approaches need target-domain videos or produce noisy/repetitive generated data.
Approach
Generate VMR training data by editing source-domain moments with target-domain sentences via an instance-preserving two-stage video diffusion (DreamBooth-style instance token then frozen-spatial temporal layers), and select the synthetic moments with a hybrid score combining CLIP cross-modal relevance, DINO structure similarity, and VMR performance disparity.
Key results
- On Charades-STA with Slowfast features, FVE raises MESM's R1@0.7 from 29.78 to 31.94 and mIoU from 44.16 to 44.59 under novel-word testing.
- On QVHighlights, FVE (Slowfast, UVCOM baseline) reaches mAP avg 41.33 vs 40.30 for UVCOM† without video-text target data.
- On action editing over 10 videos, FVE achieves the best harmonic score s_cu of 0.3933 versus 0.3862 for Video-P2P.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FVE-generated training data (ablation) | VDI · Charades-STA (I3D) | R1@0.5: 46.19 -> 48.51 | Helped |
| Add FVE-generated training data | VDI · Charades-STA (I3D) | R1@0.7: 26.19 -> 28.48 | Helped |
| Replace MESM with FVE using text-only target data | Charades-STA (Slowfast) · Charades-STA novel-word | R1@0.7: 29.78 -> 31.94 | Helped |
| Use FVE with text-only target data instead of UVCOM† | QVHighlights · QVHighlights | R1@0.5: 61.39 -> 63.35 | Helped |
| Use FVE with text-only target data instead of MESM† | TaCoS · TaCoS | R1@0.5: 29.39 -> 31.92 | Helped |
| Increase training-data volume using replicated samples (ablation) | VDI · Charades-STA (I3D) | R1@0.5: 46.19 -> 46.23 | No clear effect |
| Replace generated data with random concatenations of existing videos (ablation) | VDI · Charades-STA (I3D) | R1@0.5: 46.19 -> 46.23 | No clear effect |
| Use FVE selection instead of random sampling of generated videos (ablation) | VDI · Charades-STA (I3D) | R1@0.5: 47.19 -> 48.51 | Helped |
| Remove the combined cross-modal relevance and uni-modal structure selection (s_cu) (ablation) | FVE · Charades-STA (I3D) | R1@0.5: 43.88 -> 48.51 | Hurt |
| Remove model performance disparity selection (s_mpd), keeping k=500 (ablation) | FVE · Charades-STA (I3D) | R1@0.5: 46.64 -> 48.51 | Hurt |
| Set hybrid-selection parameters to k=500 and l=100 (ablation) | FVE · Charades-STA (I3D) | R1@0.5: 46.64 -> 48.51 | Helped |
| Use FVE rather than Video-P2P for video editing | Video action editing · 10-video action editing evaluation | s_cu: 0.3862 -> 0.3933 | Helped |
Surprising findings
- Adding replicated or concatenated samples to increase data volume does not improve VMR performance, while selective hybrid selection does.
Limitations
- Cross-modal relevance score for FVE is lower than Tune-A-Video on action editing (0.2722 vs 0.2910).
- Evaluation on only a 10-video action-editing set with qualitative examples limited by space.
- Relies on source-domain VMR annotations and a VMR model in the selection loop, so selection depends on the base model.
Experimental setup
Datasets: Charades-STA QVHighlights TaCoS
Benchmarks: Charades-STA QVHighlights TaCoS
Metrics: R1@0.5 R1@0.7 mIoU mAP mAPavg cross-modal relevance s_c uni-modal structure s_u harmonic s_cu
Baselines: VDI MESM UVCOM M-DETR QD-DETR Tune-A-Video Video-P2P FateZero Dreamix VISA
Builds on: DreamBooth DDIM inversion Dreamix Tune-A-Video CLIP
Closest papers by meaning
- VERIFIED: A Video Corpus Moment Retrieval Benchmark for Fine-Grained Video Understanding (NeurIPS 2024 · similarity 0.90)
- Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language (AAAI 2024 · similarity 0.90)
- Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models (AAAI 2025 · similarity 0.90)
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.89)
- CDTR: Semantic Alignment for Video Moment Retrieval Using Concept Decomposition Transformer (AAAI 2025 · similarity 0.89)
- MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.