Research map / Vision-language and multimodal / Cross-modal learning
Modality-Aware Shot Relating and Comparing for Video Scene Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MASRC, which models long-term entity-based and short-term place-based shot relations via GCNs and detects scene-ending shots by convolving similarity matrices between fore-and-aft shots.
Problem
Video scene detection: deciding whether each shot is a scene-ending shot, which requires correlating multi-modal (entity and place) shot cues and comparing context on both sides of a shot; prior methods treat modalities uniformly and do not explicitly compare the two sides.
Approach
Modality-aware shot relating builds an entity jumping graph for long-range links between similar distant shots and a place continuity graph (wide/detail shot affiliation) for short-range links, then multi-shot comparison detection convolves a cross-side cosine similarity matrix to predict ending shots.
Key results
- On MovieNet under fully supervised training, MASRC (entity+place) reaches 67.4 AP, 65.8 mIoU and 63.8 F1, versus 64.0 AP for NeighborNet and 54.8 AP for MHRT.
- In self-supervised transfer on MovieNet, MASRC reaches 73.2 AP versus 71.9 for NeighborNet.
- Zero-shot transfer from MovieNet yields 48.3 AP on OVSD and 53.2 AP on BBC, above NeighborNet's 47.3 and 50.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MASRC instead of NeighborNet | MovieNet self-supervised video scene detection · MovieNet test, self-supervised | AP: 51.2 -> 52.8 | Helped |
| Use MASRC instead of NeighborNet | MovieNet fully supervised video scene detection · MovieNet test, fully supervised | AP: 64.0 -> 67.4 | Helped |
| Use MASRC instead of NeighborNet | MovieNet self-supervised transfer · MovieNet test, self-supervised transfer | AP: 71.9 -> 73.2 | Helped |
| Use MASRC instead of NeighborNet | Cross-dataset scene detection · OVSD | AP: 47.3 -> 48.3 | Helped |
| Use MASRC instead of NeighborNet | Cross-dataset scene detection · BBC | AP: 50.6 -> 53.2 | Helped |
| Add MCD to the baseline detector (ablation) | MASRC · MovieNet test, fully supervised | AP: 16.4 -> 47.1 | Helped |
| Add ELD to MCD (ablation) | MASRC · MovieNet test, fully supervised | AP: 47.1 -> 59.2 | Helped |
| Add PSD to MCD (ablation) | MASRC · MovieNet test, fully supervised | AP: 47.1 -> 59.6 | Helped |
| Add PSD alongside ELD and MCD (ablation) | MASRC · MovieNet test, fully supervised | AP: 59.2 -> 67.4 | Helped |
| Use long-term rather than short-term relations for entity features (ablation) | MASRC entity modality · MovieNet test, fully supervised | AP: 56.0 -> 59.2 | Helped |
| Use short-term rather than long-term relations for place features (ablation) | MASRC place modality · MovieNet test, fully supervised | AP: 57.3 -> 59.6 | Helped |
| Use both entity long-term and place short-term relations (ablation) | MASRC · MovieNet test, fully supervised | AP: 59.2 -> 67.4 | Helped |
Limitations
- Fails on scenes with frequent motion transitions that weaken entity and place coherence.
- Does not use audio cues, which the authors suggest could help.
- Wide-shot identification relies on similarity and can differ from true camera focal-length wide shots.
Experimental setup
Datasets: MovieNet MovieScenes OVSD BBC
Benchmarks: MovieNet OVSD BBC
Metrics: AP mIoU F1
Baselines: BaSSL SSM TranS4mer VSMBD NeighborNet Temporal Perceiver MHRT CANet ShotCoL SCRL CAT LGSS
Builds on: graph convolutional network modified dynamic warping VGG ResNet-50
Closest papers by meaning
- Neighbor Relations Matter in Video Scene Detection (CVPR 2024 · similarity 0.86)
- Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation (AAAI 2024 · similarity 0.85)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.85)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.84)
- Revisiting Few-Shot Object Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.84)
- Multi-Modal Prompting for Open-Vocabulary Video Visual Relationship Detection (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.