Research map / Detection and segmentation / CNN features
OED: Towards One-stage End-to-End Dynamic Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OED, a one-stage end-to-end DETR-style set-prediction framework for dynamic scene graph generation with pair-wise features and a Progressively Refined Module for temporal context aggregation, evaluated on Action Genome.
Problem
Dynamic scene graph generation (DSGG) in videos is usually solved with multi-stage pipelines (detection, tracking, relation classification) that enumerate many subject-object pairs and optimize sub-problems separately, and capturing temporal dependencies typically relies on trackers, trajectories, o
Approach
Reformulates DSGG as one-stage set prediction over pair-wise queries (instance and relation decoders cascaded), and aggregates temporal context from reference frames via a Progressively Refined Module that iteratively keeps top-K reference pair features, avoiding explicit trackers so the whole network trains end-to-end.
Key results
- On Action Genome SGDET with constraint, OED reaches R@10 33.5, R@20 40.9, R@50 48.9, outperforming prior methods such as TPT and DSG-DETR across all SGDET metrics.
- Under the No Constraint SGDET setting, OED achieves R@10 35.3, R@20 44.0, R@50 51.8, versus TPT's 32.0/39.6/51.5 at R@10/20/50 (TPT only reports R@50 at 51.5 and is a second-best; OED's R@50 is slightly higher).
- Ablations show the cascaded decoders and matched predicate loss add gains over the DETR baseline (R@50 with constraint 32.1 to 37.7/43.7 with spatial aggregation), and progressive refinement improves over non-progressive temporal aggregation (R@50 47.8 to 48.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the multi-stage pipeline with one-stage end-to-end OED | Dynamic scene graph generation · Action Genome SGDET, With Constraint | R@50: 38.3 -> 48.9 | Helped |
| Replace the multi-stage pipeline with one-stage end-to-end OED | Dynamic scene graph generation · Action Genome SGDET, No Constraints | R@50: 51.5 -> 51.8 | Helped |
| Replace the multi-stage pipeline with one-stage end-to-end OED | Dynamic scene graph generation · Action Genome PredCLS, With Constraint | R@50: 73.8 -> 76.1 | Helped |
| Replace the multi-stage pipeline with one-stage end-to-end OED | Dynamic scene graph generation · Action Genome PredCLS, No Constraints | R@10: 85.6 -> 83.3 | Hurt |
| Add Spatial Context Aggregation (ablation) | OED baseline · Action Genome SGDET, With Constraint | R@50: 32.1 -> 43.7 | Helped |
| Add Temporal Context Aggregation (ablation) | OED with Spatial Context Aggregation · Action Genome SGDET, With Constraint | R@50: 43.7 -> 48.9 | Helped |
| Add Cascaded Decoders (ablation) | OED baseline · Action Genome SGDET, With Constraint | R@50: 32.1 -> 33.7 | Helped |
| Add Matched Predicate Loss (ablation) | OED with Cascaded Decoders · Action Genome SGDET, With Constraint | R@50: 33.7 -> 43.7 | Helped |
| Add temporal interaction with all reference pair-wise features (ablation) | OED with Spatial Context Aggregation · Action Genome SGDET, With Constraint | R@50: 43.7 -> 47.8 | Helped |
| Add Progressive Refinement to temporal interaction (ablation) | OED with temporal interaction · Action Genome SGDET, With Constraint | R@50: 47.8 -> 48.9 | Helped |
| Use oracle reference-query selection instead of Progressive Refinement (ablation) | OED temporal aggregation · Action Genome SGDET | R@50: 48.9 -> 51.9 | Helped |
Surprising findings
- In PredCLS under the No Constraints setting, OED is marginally lower than TPT and TR2, which the authors attribute to not explicitly using oracle tracks in a one-stage design.
- Oracle selection of reference pair features (upper bound) substantially outperforms the PRM selection (R@50 with constraint 51.9 vs 48.9), indicating room for improvement in reference selection.
Limitations
- PredCLS under No Constraints is marginally below TPT and TR2 because the one-stage design cannot explicitly exploit oracle tracks.
- Performance gap to oracle reference selection indicates the PRM selection of true-positive reference pairs is imperfect.
- Evaluation limited to the Action Genome benchmark.
Experimental setup
Datasets: Action Genome MS-COCO
Benchmarks: Action Genome
Metrics: Recall@10 Recall@20 Recall@50
Baselines: VRD MSDN M-FREQ VCTree RelDN GPS-Net TRACE RelTR STTran APT STTran-TPI TR2
Builds on: DETR TPT STTran Focal loss
Closest papers by meaning
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
- DiffVsgg: Diffusion-Driven Online Video Scene Graph Generation (CVPR 2025 · similarity 0.89)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.88)
- Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.88)
- Adaptive Visual Scene Understanding: Incremental Scene Graph Generation (NeurIPS 2024 · similarity 0.87)
- TD²-Net: Toward Denoising and Debiasing for Video Scene Graph Generation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.