Research map / Vision-language and multimodal / Human and embodied modeling
Improving Panoptic Narrative Grounding by Harnessing Semantic Relationships and Visual Confirmation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes XPNG, a one-stage Panoptic Narrative Grounding model with Semantic Context Convolution and Visual Context Verification modules that model inter-phrase semantic and geometric relationships, reaching 63.3 average recall on PNG.
Problem
Single-stage Panoptic Narrative Grounding methods match pixels to phrases independently and neglect semantic and visual relationships among phrase-level instances, limiting multi-modal reasoning for pixel-level grounding of long narrative captions.
Approach
A "differentiation-refinement-localization" paradigm: Semantic Context Convolution uses text-conditioned dynamic channel-wise and pixel-wise convolutions to make visual kernels distinctive and model inter-instance relations; Visual Context Verification adds mask-derived geometric (size and centroid) relations into geometry-aware attention to correct semantic-only biases.
Key results
- XPNG reaches 63.3 overall average recall on PNG, versus 59.4 for PPMN (+3.9).
- SCC kernels outperform text-only kernels by 2.9 points and visual-only kernels by 2.2 points in overall recall (62.3 vs 59.4/60.1 with frozen BERT).
- Zero-shot RES on RefCOCO testA p@0.5 improves from 13.3 (PPMN) to 23.0 (XPNG).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add XPNG's semantic context convolution and visual context verification | PPMN · PNG test | segmentation average recall (All): 59.4 -> 63.3 | Helped |
| use SCC kernels instead of text kernels (ablation) | XPNG · PNG test | segmentation average recall (Overall): 59.4 -> 62.3 | Helped |
| use SCC kernels instead of visual kernels without pixel-wise and channel-wise convolution (ablation) | XPNG · PNG test | segmentation average recall (Overall): 60.1 -> 62.3 | Helped |
| add self-attention instead of removing both self-attention and VCV (ablation) | XPNG · PNG test | segmentation average recall (Overall): 58.2 -> 61.3 | Helped |
| add geometric-aware attention and VCV instead of self-attention without VCV (ablation) | XPNG · PNG test | segmentation average recall (Overall): 61.3 -> 62.3 | Helped |
| increase the iteration stages from 1 to 3 (ablation) | XPNG · PNG test | segmentation average recall (Overall): 60.1 -> 62.3 | Helped |
| increase the iteration stages from 3 to 4 (ablation) | XPNG · PNG test | segmentation average recall (Overall): 62.3 -> 62.2 | Hurt |
Surprising findings
- Zero-shot transfer to RES without RES training data substantially improves over PPMN on RefCOCO, RefCOCO+ and RefCOCOg.
Limitations
- Performance declines when the number of stages increases to 4, attributed to overfitting.
- Gains over baseline depend on pretrained FPN frozen from COCO panoptic segmentation; the paper does not analyze other backbones.
Experimental setup
Datasets: Panoptic Narrative Grounding (PNG) dataset MS COCO
Benchmarks: Panoptic Narrative Grounding (PNG) benchmark RefCOCO RefCOCO+ RefCOCOg
Metrics: average recall (overall, thing, stuff, singular, plural) precision at IoU 0.3/0.4/0.5 (zero-shot RES) mIoU
Baselines: MCN PNG EPNG PPMN
Builds on: PPMN Panoptic Narrative Grounding (PNG) Panoptic Narrative Grounding baseline (Gonzalez et al. 2021) Dynamic convolution (Hu et al. 2023) Geometry-aware attention relation module (Hu et al. 2018) Kernel Update / cascaded iterative kernels (Zhang et al. 2021)
Closest papers by meaning
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.88)
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.87)
- AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.87)
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.87)
- Investigating Compositional Challenges in Vision-Language Models for Visual Grounding (CVPR 2024 · similarity 0.86)
- GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.