Research map / Video / Video segmentation and tracking
DiffVsgg: Diffusion-Driven Online Video Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffVsgg, an online video scene graph generation method that treats scene graph prediction as step-wise latent-diffusion denoising over pairwise object embeddings, with previous-frame graphs as conditions.
Problem
Video scene graph generation (VSGG) top solutions are offline, requiring full video sequences, consuming large GPU memory, and not handling real-time streams; they also aggregate frame-level predictions poorly for temporal reasoning.
Approach
Frames VSGG as iterative scene graph updating via a latent diffusion denoising U-Net over a shared adjacency-matrix embedding of object pairs, conditioned on the previous frame's denoised graph and motion cues (approach speed), so that object classification, box regression and predicate prediction share one embedding.
Key results
- Under the w constraint SGDET setting, DiffVsgg reaches R@10/R@20/R@50 of 32.8/39.9/45.5, versus 30.3/34.8/36.1 for DSG-DETR.
- Under the w/o constraint setting, DiffVsgg achieves 35.4/42.5/51.0 versus 32.1/40.9/48.3 for DSG-DETR in SGDET.
- Ablation shows LDM denoising, condition-based temporal reasoning, and motion enhancement progressively raise w-constraint SGDET R@10 from 26.9 to 32.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LDM relationship modeling (ablation) | DIFFVSGG · Action Genome test, SGDET, w constraint | R@10: 26.9 -> 29.7 | Helped |
| Add condition-based temporal reasoning (ablation) | DIFFVSGG · Action Genome test, SGDET, w constraint | R@10: 29.7 -> 32.0 | Helped |
| Add motion-enhanced denoising (ablation) | DIFFVSGG · Action Genome test, SGDET, w constraint | R@10: 32.0 -> 32.8 | Helped |
| Add subject features to union box features (ablation) | DIFFVSGG adjacency matrix · Action Genome test, SGDET, w constraint | R@10: 29.5 -> 31.0 | Helped |
| Add subject location features (ablation) | DIFFVSGG adjacency matrix · Action Genome test, SGDET, w constraint | R@10: 31.0 -> 32.8 | Helped |
| Use 50 diffusion steps instead of 20 (ablation) | DIFFVSGG · Action Genome test, SGDET, w constraint | R@10: 32.8 -> 33.2 | Helped |
| Increase diffusion steps from 50 to 100 (ablation) | DIFFVSGG · Action Genome test, SGDET, w constraint | R@10: 33.2 -> 33.1 | Hurt |
| Use 3 U-Net layers instead of 2 (ablation) | DIFFVSGG · Action Genome test, SGDET, w constraint | R@10: 31.9 -> 32.8 | Helped |
| Increase U-Net depth from 3 to 5 layers (ablation) | DIFFVSGG · Action Genome test, SGDET, w constraint | R@10: 32.8 -> 32.4 | Hurt |
| Increase U-Net depth from 3 to 6 layers (ablation) | DIFFVSGG · Action Genome test, SGDET, w constraint | R@10: 32.8 -> 32.6 | Hurt |
| Use the full subject row instead of one adjacency element for graph readout (ablation) | DIFFVSGG · Action Genome test, SGDET, w constraint | R@10: 31.7 -> 32.8 | Helped |
Surprising findings
- Inference speed (8.7 FPS) is slightly slower than some existing methods due to step-wise denoising, though it has the fewest trainable parameters (46.7M).
Limitations
- Inference is slower than some prior methods due to iterative denoising steps.
- Relies on an off-the-shelf frozen object detector whose features are static.
- Hardware and compute details are not reported.
Experimental setup
Datasets: Action Genome
Benchmarks: Action Genome
Metrics: Recall@10 Recall@20 Recall@50 mean Recall@10 mean Recall@20 mean Recall@50
Baselines: RelDN TRACE VRD Motif Freq MSDN VCTREE GPS-Net STTran APT TR2 TEMPURA DSG-DETR
Builds on: Latent Diffusion Models STTran DSG-DETR
Closest papers by meaning
- TD²-Net: Toward Denoising and Debiasing for Video Scene Graph Generation (AAAI 2024 · similarity 0.90)
- OED: Towards One-stage End-to-End Dynamic Scene Graph Generation (CVPR 2024 · similarity 0.89)
- Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.89)
- Noise-Guided Predicate Representation Extraction and Diffusion-Enhanced Discretization for Scene Graph Generation (ICML 2025 · similarity 0.89)
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.88)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.