Research map / Detection and segmentation / Open-set and few-shot detection
SGDiff: Scene Graph Guided Diffusion Model for Image Collaborative SegCaptioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SegCaptioning, where a single box prompt yields multiple parallel (caption, mask) pairs, and proposes SGDiff, a scene-graph-guided diffusion model with a prompt-centric scene graph adaptor, a bimodal transformer, and multi-entity contrastive loss.
Problem
Controllable image understanding (captioning, segmentation) requires costly prompts and yields unique outputs, and sequential caption-then-mask pipelines suffer error propagation; the paper targets generating multiple aligned caption-mask pairs from a minimal box prompt.
Approach
A Prompt-Centric Scene Graph Adaptor maps a box prompt to a subgraph (with node filtering and ordering), and a discrete-to-continuous bit diffusion model with a scene-graph-guided bimodal transformer jointly denoises caption and mask features, aligned by a multi-entity contrastive loss at intra- and inter-sample levels.
Key results
- On MSCOCO, SGDiff reaches SPICE 26.1, CIDEr 137.4, mIoU 66.3 and mAP 47.2, outperforming CAT, SCA and GLaMM on joint captioning and segmentation.
- On Flickr30k Entities, SGDiff achieves BLEU-4 38.2, METEOR 27.4, SPICE 24.5 and CIDEr 98.6, above GROUNDHOG (36.7 BLEU-4, 91.3 CIDEr).
- Ablation shows the PSGA filtering and ranking components together raise CIDEr from 127.4 to 137.4 and mIoU from 62.3 to 66.3 on MSCOCO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add node filtering (ablation) | SGDiff PSGA · MSCOCO | SPICE: 24.2 -> 24.9 | Helped |
| add node ranking (ablation) | SGDiff PSGA · MSCOCO | SPICE: 24.2 -> 25.4 | Helped |
| add both node filtering and ranking (ablation) | SGDiff PSGA · MSCOCO | SPICE: 24.2 -> 26.1 | Helped |
| add cross-attention (ablation) | SGDiff SGBTrans · MSCOCO | SPICE: 24.5 -> 26.1 | Helped |
| jointly train caption and mask generation (ablation) | SGDiff · MSCOCO | SPICE: 25.2 -> 26.1 | Helped |
| add intra-sample and inter-sample MECL (ablation) | SGDiff · MSCOCO | SPICE: 24.8 -> 26.1 | Helped |
| add intra-sample MECL (ablation) | SGDiff · MSCOCO | SPICE: 24.8 -> 25.4 | Helped |
| add inter-sample MECL (ablation) | SGDiff · MSCOCO | SPICE: 24.8 -> 25.6 | Helped |
| increase PSGA loss weight λ1 from 0 to 2 (ablation) | SGDiff · MSCOCO | SPICE: 24.2 -> 26.1 | Helped |
| increase PSGA loss weight λ1 from 2 to 5 (ablation) | SGDiff · MSCOCO | SPICE: 26.1 -> 25.8 | Hurt |
| increase PSGA loss weight λ1 from 2 to 10 (ablation) | SGDiff · MSCOCO | SPICE: 26.1 -> 25.3 | Hurt |
| add MECL with λ2=1 (ablation) | SGDiff · MSCOCO | SPICE: 24.8 -> 26.1 | Helped |
Surprising findings
- SCA's captioning CIDEr (85.0 on COCO) and Flickr30k BLEU-4 is lower than some older captioning baselines despite its large-model design, while SGDiff is reported to outperform all.
Limitations
- Severely occluded objects in crowds are not fully segmented.
- Overlapping masks occur when segmenting both whole objects and their parts.
Experimental setup
Datasets: MSCOCO Flickr30k Entities
Benchmarks: MSCOCO Flickr30k Entities
Metrics: BLEU-4 METEOR SPICE CIDEr mIoU mAP
Baselines: Sub-GC SATIC X-Transformer CMAL SCD-Net OpenSeg OpenSeeD ODISE X-Decoder CAT SCA GLaMM
Builds on: Bit Diffusion Mask2Former InfoNCE
Closest papers by meaning
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.88)
- Noise-Guided Predicate Representation Extraction and Diffusion-Enhanced Discretization for Scene Graph Generation (ICML 2025 · similarity 0.87)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.87)
- Universal Scene Graph Generation (CVPR 2025 · similarity 0.87)
- GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.