Research map / Video / Video segmentation and tracking
Neighbor Relations Matter in Video Scene Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeighborNet, which refines shot-to-shot affinities using semantic and temporal neighbor relations within a local time window and propagates them via two cascaded GCNs to learn scene-discriminative shot features for video scene boundary detection.
Problem
Video scene detection requires scene-distinguishable shot affinities, but ordinary shot-to-shot similarity links visually similar shots from different scenes and fails to connect dissimilar shots within the same scene.
Approach
Replace raw shot-to-shot similarity with neighbor-aware similarities: semantic neighbor-node and neighbor-neighbor relations build a feature graph, and temporal-context similarity over an intervening-shot temporal graph links dissimilar same-scene shots; GCN message passing on both graphs produces context-embedded shot features.
Key results
- On MovieNet, Ours reaches 64.0 AP fully supervised and 51.2 AP self-supervised, and 71.9 AP with self-supervised transfer learning, surpassing TranS4mer by at least 11 points in all metrics.
- Transfer to OVSD and BBC without fine-tuning yields 47.3 AP on OVSD and 50.6 AP on BBC, beating TranS4mer (36.0 and 43.6).
- Plugging NeighborNet into ViT-S/16 raises AP from 34.1 to 65.5 on MovieNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add NeighborNet's neighbor-aware graph reasoning (ablation) | ResNet-50 shot features · MovieNet (fully supervised) | AP: 40.0 -> 64.0 | Helped |
| Add NeighborNet's neighbor-aware graph reasoning (ablation) | ViT-S/16 shot features · MovieNet (fully supervised) | AP: 34.1 -> 65.5 | Helped |
| Add IRS to the feature graph (ablation) | NeighborNet · MovieNet (fully supervised) | AP: 40.0 -> 52.5 | Helped |
| Add RNS to the feature graph (ablation) | NeighborNet with IRS · MovieNet (fully supervised) | AP: 52.5 -> 56.9 | Helped |
| Add RRS to the feature graph (ablation) | NeighborNet with IRS and RNS · MovieNet (fully supervised) | AP: 56.9 -> 60.0 | Helped |
| Add TCS to the temporal graph (ablation) | NeighborNet with ICS · MovieNet (fully supervised) | AP: 50.4 -> 58.8 | Helped |
| Combine IRS, RNS, RRS, ICS, and TCS (ablation) | NeighborNet · MovieNet (fully supervised) | AP: 40.0 -> 64.0 | Helped |
| Set sliding-window scale l to 11 (ablation) | NeighborNet · MovieNet (fully supervised) | AP | Helped |
| Set temporal-neighbor scale r to 5 (ablation) | NeighborNet · MovieNet (fully supervised) | AP | Helped |
| Set feature-neighbor scale k to 5 (ablation) | NeighborNet · MovieNet (fully supervised) | AP | Helped |
| Vary the number of input shots N (ablation) | NeighborNet · MovieNet (fully supervised) | AP | No clear effect |
| Use different shot encoders with NeighborNet (ablation) | NeighborNet · MovieNet (fully supervised) | AP: 40.0 -> 64.0; 34.1 -> 65.5 | Helped |
Surprising findings
- Transformer baseline with raw shot-to-shot attention scored lower AP (23.0 self-supervised) than several prior methods, suggesting raw similarity is weak for this task.
Limitations
- Relies on pseudo-boundaries from Modified Dynamic Warping for self-supervised pre-training.
- Evaluated only on movie and TV-style datasets (MovieNet, OVSD, BBC) with ResNet/ViT shot encoders.
Experimental setup
Datasets: MovieNet MovieScenes BBC OVSD
Benchmarks: MovieScenes (MovieNet) OVSD BBC
Metrics: AP mIoU F1
Baselines: TimeSformer BaSSL TranS4mer Transformer GATv2 HopGNN ShotCoL Movies2Scenes SCRL SSM Temporal Perceiver MHRT
Builds on: TranS4mer BaSSL GATv2 HopGNN Modified Dynamic Warping
Closest papers by meaning
- Modality-Aware Shot Relating and Comparing for Video Scene Detection (AAAI 2025 · similarity 0.86)
- Scene Graph Embeddings Using Relative Similarity Supervision (AAAI 2021 · similarity 0.85)
- Shot Contrastive Self-Supervised Learning for Scene Boundary Detection (CVPR 2021 · similarity 0.85)
- G-TAD: Sub-Graph Localization for Temporal Action Detection (CVPR 2020 · similarity 0.85)
- SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval (ICML 2025 · similarity 0.84)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.