Research map / Video / Video segmentation and tracking
TD²-Net: Toward Denoising and Debiasing for Video Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TD2-Net, which combines a differentiable Top-K object selector (Gumbel-softmax) based denoising spatio-temporal transformer with an asymmetrical reweighting loss to tackle contextual noise and label bias in dynamic scene graph generation on Action Genome.
Problem
Dynamic scene graph generation in videos suffers from contextual noise (occluded or blurred objects in frames) and label bias (positive-negative imbalance and long-tailed predicate distribution).
Approach
A differentiable Top-K object selector using gumbel-softmax sampling with replacement picks relevant neighboring objects across frames to denoise object features, and an asymmetric focusing loss with class-balanced effective-number weights addresses positive-negative and head-tail imbalance.
Key results
- TD2-Net outperforms TEMP by 12.7% at mR@10 on PredCLS under the With Constraint setting.
- TD2-Net (P) outperforms GPS-Net by 3.3%, 5.8%, and 4.0% at R@10 on PredCLS, SGCLS, and SGDET respectively.
- Ablation: AR-Loss mainly raises PredCLS mR@10 (37.8 to 53.1 relative to STTran baseline), while D-Trans mainly raises SGCLS R@10 (46.4 to 49.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TD2-Net instead of TEMP | Dynamic scene graph generation · Action Genome, PredCLS, with constraint | mR@10: 42.9 -> 54.2 | Helped |
| use TD2-Net instead of TEMP | Dynamic scene graph generation · Action Genome, SGCLS, with constraint | mR@10: 34.0 -> 40.9 | Helped |
| use TD2-Net instead of TEMP | Dynamic scene graph generation · Action Genome, SGDET, with constraint | mR@10: 18.5 -> 20.4 | Helped |
| use TD2-Net instead of TEMP | Dynamic scene graph generation · Action Genome, PredCLS, with constraint | R@10: 68.8 -> 67.8 | Hurt |
| add D-Trans (ablation) | TD2-Net · Action Genome, SGCLS, with constraint | R@10: 46.4 -> 49.7 | Helped |
| add D-Trans (ablation) | TD2-Net · Action Genome, PredCLS, with constraint | mR@10: 37.8 -> 40.4 | Helped |
| add AR-Loss (ablation) | TD2-Net · Action Genome, PredCLS, with constraint | mR@10: 37.8 -> 53.1 | Helped |
| add AR-Loss with class-balanced weights (ablation) | TD2-Net · Action Genome, PredCLS, with constraint | mR@10: 41.9 -> 54.2 | Helped |
| set Top-K to 8 (ablation) | D-Trans in TD2-Net (P) · Action Genome, SGCLS, with constraint | R@10: 49.6 -> 51.1 | Helped |
| increase Top-K from 8 to 10 (ablation) | D-Trans in TD2-Net (P) · Action Genome, SGCLS, with constraint | R@10: 51.1 -> 50.9 | Hurt |
| add D-Trans (ablation) | TD2-Net (P) · Action Genome, SGCLS, with constraint | R@10: 47.3 -> 51.1 | Helped |
| use both Eq. (1) and Eq. (2) object-matching strategies (ablation) | D-Trans in TD2-Net (P) · Action Genome, SGCLS, with constraint | R@10: 49.1 -> 51.1 | Helped |
Surprising findings
- The variant without class-balanced weights (TD2-Net (P)) achieves higher R@K while the full model with class-balanced weights achieves higher mR@K, indicating a recall vs. mean-recall trade-off.
Limitations
- Evaluated only on the Action Genome benchmark.
- Some mR values for baselines were obtained via email with authors because source code was not public.
- Top-K increases memory usage beyond K=8 with diminishing benefit.
Experimental setup
Datasets: Action Genome
Benchmarks: Action Genome
Metrics: R@10 R@50 mR@10 mR@50
Baselines: VRD Motif Freq MSDN VCTREE RelDN GPS-Net STTran TPI TEMP TRACE
Builds on: STTran focal loss class-balanced loss Gumbel-softmax
Closest papers by meaning
- DiffVsgg: Diffusion-Driven Online Video Scene Graph Generation (CVPR 2025 · similarity 0.90)
- Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.90)
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.88)
- Semi-Supervised Clustering Framework for Fine-grained Scene Graph Generation (AAAI 2025 · similarity 0.87)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.87)
- OED: Towards One-stage End-to-End Dynamic Scene Graph Generation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.