Research map / Video / Video-language understanding
D&M: Enriching E-commerce Videos with Sound Effects by Key Moment Detection and SFX Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D&M, a DETR-based model that jointly detects key moments in e-commerce videos and matches sound effects to them, trained with moment-SFX matching pre-training and tag-aware negative sampling, plus a new SFX-Moment dataset.
Problem
Adding sound effects to key moments of e-commerce videos requires deciding where to insert SFX and which SFX to use, but prior work only does holistic video-to-SFX matching or moment localization without moment-to-SFX matching.
Approach
A DETR-decoder with learnable queries predicts moment boundaries and moment embeddings that are matched by cosine similarity to SFX embeddings, so localization and SFX selection are learned jointly; training uses MSM pre-training and tag-aware negative sampling that down-weights likely false negatives.
Key results
- D&M reaches 12.8 mAP_SFX@0.5 and 34.3 mAP_vid@0.5 (mean 23.6), beating the best baseline R^2-Tuning+ at 9.6 and 32.3 (mean 21.0).
- On key moment detection, D&M achieves 40.0 mAP_key@0.5 versus 38.2 for R^2-Tuning.
- Removing MSM pre-training drops the mean score from 23.6 to 17.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use D&M instead of the strongest baseline, R^2-Tuning+ | VDSFX · SFX-Moment test | mAP_SFX @0.5: 9.6 -> 12.8 | Helped |
| Use D&M instead of the strongest baseline, R^2-Tuning+ | VDSFX · SFX-Moment test | mAP_vid @0.5: 32.3 -> 34.3 | Helped |
| Use D&M instead of the strongest baseline, UniVTG | Key moment detection · SFX-Moment test | mAP_vid @0.5: 55.8 -> 59.9 | Helped |
| Use D&M instead of the strongest baseline, R^2-Tuning | Key moment detection · SFX-Moment test | mAP_key @0.5: 38.2 -> 40.0 | Helped |
| Add MSM-based pre-training (ablation) | D&M · SFX-Moment test | Mean @0.5: 17.6 -> 23.6 | Helped |
| Use MSM pre-training instead of SSP pre-training (ablation) | D&M · SFX-Moment test | Mean @0.5: 17.6 -> 23.6 | Helped |
| Use MSM pre-training instead of SFX-TP pre-training (ablation) | D&M · SFX-Moment test | Mean @0.5: 18.0 -> 23.6 | Helped |
| Use MSM for pre-training instead of only as an end-to-end auxiliary task (ablation) | D&M · SFX-Moment test | Mean @0.5: 20.4 -> 23.6 | Helped |
| Do not reuse MSM as an auxiliary task after pre-training (ablation) | D&M · SFX-Moment test | Mean @0.5: 23.1 -> 23.6 | Helped |
| Use TaNS instead of uniform sampling (ablation) | D&M · SFX-Moment test | Mean @0.5: 21.3 -> 23.6 | Helped |
| Use TaNS instead of HardNS (ablation) | D&M · SFX-Moment test | Mean @0.5: 22.1 -> 23.6 | Helped |
| Use full TaNS instead of one-side TaNS (ablation) | D&M · SFX-Moment test | Mean @0.5: 22.3 -> 23.6 | Helped |
Surprising findings
- The DETR-based Moment-DETR+ and QD-DETR+ baselines perform clearly worse than encoder-only methods like UniVTG+ and R^2-Tuning+.
- Sliding-window baseline is relatively competitive compared with several other baselines.
Limitations
- Each frame is represented by a single CLS token, which may miss small-scale regions of interest.
- Does not support interactive VDSFX with real-time user guidance.
Experimental setup
Datasets: SFX-Moment
Benchmarks: SFX-Moment
Metrics: mAP@0.5 mAP@0.75 AP_SFX AP_vid AP_key
Baselines: Sliding-Win Moment-DETR+ QD-DETR+ UniVTG+ R^2-Tuning+
Builds on: Moment-DETR QD-DETR UniVTG R^2-Tuning ImageBind
Closest papers by meaning
- Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection (CVPR 2024 · similarity 0.84)
- MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval (NeurIPS 2025 · similarity 0.82)
- Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection (CVPR 2024 · similarity 0.82)
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.82)
- Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection (AAAI 2025 · similarity 0.81)
- TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.