Research map / Graph learning / Knowledge and temporal graphs

A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models

NeurIPS 2026 · Graph learning › Knowledge and temporal graphs · approach: Scene graph generation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Revisits dynamic scene graph generation by adding precision, ranking, retrieval and QA metrics, and builds an MLLM-based generator (reason-then-locate, temporal relation set, importance-aware finetuning) that reports SOTA on Action Genome, VidVRD and PVSG.

Problem

Dynamic scene graph generation is evaluated only with recall-based metrics, which hide precision loss and redundant, uninformative relations, and existing bottom-up detector-based models lag behind MLLM capabilities.

Approach

Use MLLMs directly as video scene graph generators with a top-down reason-then-locate strategy, a compact temporal relation set output format (5-tuples with start/end times), and importance-ordered finetuning, grounded afterwards by SAM 3; also proposes five extra metrics (MLLM-as-judge precision, SGRet, SGQA, importance-aware nDCG, SGCls-to-SGDet-style variant).

Key results

• On Action Genome SGDet with constraint, Ours (Qwen3VL) reaches R@10/20/50 of 49.6/57.9/62.2, +10.3/+12.7/+8.4 over UNO.
• On SGQA, Ours (Qwen3-VL) reaches 48.9% accuracy vs 36.2% for OED.
• TRS training speed improves 2.3x on AG and inference latency drops up to 8.6x.

Surprising findings

• Standard MLLM finetuning without importance-aware training yields ranking quality comparable to bottom-up OED, indicating it does not inherently prioritize informative triplets.
• Prior DSGG models show a concave SGRet trend, performing poorly at top-1 and degrading at large k, because highest-confidence predictions are often trivial relations.

Limitations

• Limited to close-vocabulary setting.
• Evaluated on short clips (8 frames), not long videos.
• Relies on MLLM-as-a-judge and an off-the-shelf grounding model (SAM 3) for precision and localization.

Experimental setup

Datasets: Action Genome ImageNet-VidVRD PVSG

Benchmarks: Action Genome VidVRD PVSG SGQA

Metrics: recall@k precision@k (MLLM-as-a-judge) nDCG@k SGRet R@k SGQA accuracy mean recall@k

Baselines: STTran TEMPURA TRACE OED VrdONE UNO DSG-DETR VISA MCL VPS IPS+T

Builds on: LoRA SAM 3 STTran OED

Closest papers by meaning

• OED: Towards One-stage End-to-End Dynamic Scene Graph Generation (CVPR 2024 · similarity 0.90)
• From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.90)
• Scene Graph Generation with Role-Playing Large Language Models (NeurIPS 2024 · similarity 0.90)
• Weakly Supervised Video Scene Graph Generation via Natural Language Supervision (ICLR 2025 · similarity 0.89)
• Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation (AAAI 2025 · similarity 0.89)
• HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.89)