Research map / Video / Video segmentation and tracking
Unbiased Video Scene Graph Generation via Visual and Semantic Dual Debiasing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VISA, a dual-debiasing framework for unbiased video scene graph generation that uses memory-enhanced temporal integration to reduce visual bias and an iterative hierarchical semantic extractor to reduce semantic (predicate prior) bias.
Problem
Video scene graph generation is skewed toward frequent predicate and object classes, due to visual degradation (blur, occlusion) and reliance on visual-only predicate prediction, leaving rare relations underrepresented.
Approach
Separates scene-graph bias into visual bias and semantic bias; a Memory Guided Sequence Modeling module fuses past-frame memory to lower feature variance, and an Iterative Relation Generator fuses triplet semantics (GloVe embeddings, predicate embeddings) hierarchically to increase KL divergence from the biased prior.
Key results
- Under Semi Constraint on AG SGCLS, VISA reaches mR@20 52.6 vs TEMPURA 39.5 (+13.1 points).
- Under No Constraint PREDCLS, improvements over the best baseline are as small as +1.2% mR@50.
- On PVSG, MGSM+IRG reaches mR@20 8.82 vs 2.03-level Transformer baselines, and on PSG4D-GTA mR@20 10.83 vs 3.31 for PSG4DFormer (R).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VISA instead of the strongest reported baseline, FloCoDe | VidSGG on AG, With Constraint · AG PREDCLS, With Constraint | mR@20: 49.2 -> 52.0 | Helped |
| use VISA instead of the strongest reported baseline, TEMPURA | VidSGG on AG, Semi Constraint · AG SGCLS, Semi Constraint | mR@20: 39.5 -> 52.6 | Helped |
| use VISA instead of the strongest reported baseline, FloCoDe | VidSGG on AG, No Constraint · AG SGDET, No Constraint | mR@20: 35.4 -> 36.7 | Helped |
| add MGSM to VISA (ablation) | VISA on AG SGCLS, Semi Constraint · AG SGCLS, Semi Constraint | mR@10: 45.6 -> 47.8 | Helped |
| add MGSM and IRG to the model without both modules (ablation) | VidSGG on AG SGCLS, No Constraint · AG SGCLS, No Constraint | mR@10: 48.3 -> 52.0 | Helped |
| increase IRG iterations from N=1 to N=2 (ablation) | VISA on AG SGCLS, Semi Constraint · AG SGCLS, Semi Constraint | mR@20: 52.6 -> 53.6 | Helped |
| increase IRG iterations from N=2 to N=3 (ablation) | VISA on AG SGCLS, Semi Constraint · AG SGCLS, Semi Constraint | mR@20: 53.6 -> 53.9 | Helped |
| increase IRG iterations from N=1 to N=2 (ablation) | VISA on AG SGDET, Semi Constraint · AG SGDET, Semi Constraint | mR@20: 31.7 -> 31.9 | Helped |
| increase IRG iterations from N=2 to N=3 (ablation) | VISA on AG SGDET, Semi Constraint · AG SGDET, Semi Constraint | mR@20: 31.9 -> 32.1 | Helped |
| set memory update λ to 0.04 rather than 0.06 (ablation) | VISA on AG SGCLS, Semi Constraint · AG SGCLS, Semi Constraint | mR@20: 45.8 -> 52.6 | Helped |
| set memory update λ to 0.06 rather than 0.04 (ablation) | VISA on AG SGDET, Semi Constraint · AG SGDET, Semi Constraint | mR@20: 29.5 -> 31.7 | Helped |
| apply MGSM + IRG instead of IPS+T | PVSG · PVSG | mR@20: 1.75 -> 8.82 | Helped |
Limitations
- Gains from more iterations are diminishing and training cost grows with N.
- Removing MGSM is not applicable to PREDCLS since it uses ground-truth visual inputs.
- lambda must be tuned per task (0.04 for SGCLS, 0.06 for SGDET).
Experimental setup
Datasets: Action Genome
Benchmarks: Action Genome PVSG PSG4D-GTA PSG4D-HOI
Metrics: mR@10 mR@20 mR@50
Baselines: RelDN HCRD TRACE ISGG STTran STTran-TPI TEMPURA FloCoDe
Builds on: STTran TEMPURA Faster R-CNN GloVe
Closest papers by meaning
- TD²-Net: Toward Denoising and Debiasing for Video Scene Graph Generation (AAAI 2024 · similarity 0.85)
- DiffVsgg: Diffusion-Driven Online Video Scene Graph Generation (CVPR 2025 · similarity 0.83)
- Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.83)
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.81)
- HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding (CVPR 2024 · similarity 0.81)
- Conformal Prediction and MLLM aided Uncertainty Quantification in Scene Graph Generation (CVPR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.