Research map / Graph learning / Knowledge and temporal graphs
Conformal Prediction and MLLM aided Uncertainty Quantification in Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PC-SGG, a post-hoc, model-agnostic conformal prediction framework that builds class-conditional prediction sets over scene graph triplets and uses BLIP-2 MCQ-style plausibility filtering to shrink them.
Problem
Scene graph generation models produce single, often noisy predictions with no uncertainty quantification or coverage guarantees, and long-tailed classes make marginal coverage insufficient for tail classes.
Approach
Apply class-conditional conformal prediction separately to object and predicate classifier heads, combine the sets into triplet prediction sets, and prune implausible triplets by asking an MLLM (BLIP-2 with FLAN-T5-XL) multiple-choice questions over cropped union-box images.
Key results
- BGNN+PC-SGG reaches cR@50 of 46.32 and cmR@50 of 32.52, versus R@50 of 30.07 and mR@50 of 9.63 for standalone BGNN.
- MLLM post-processing shrinks BGNN triplet prediction set average size from 971.69 to 464.11 while keeping triplet coverage at 80.45%.
- Average recall-hit rate improves by 15.70 points (cR@50 vs R@50) and class-conditional hit rate by 21.57 points (cmR@50 vs mR@50) across SGG methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MLLM-based plausibility post-processing to conformal triplet sets (ablation) | BGNN, MOTIFS, SQUAT, VCTREE, MOTIFS-D · VG150 SGDET | AvgSize (triplet set): roughly halves set size (e.g., BGNN 971.69 to 464.11) | Helped |
| Add MLLM-based plausibility post-processing to conformal triplet sets (ablation) | BGNN, MOTIFS, SQUAT, VCTREE, MOTIFS-D · VG150 SGDET | CovT: unchanged for BGNN, MOTIFS, VCTREE; drops of 0.18 (SQUAT) and 0.26 (MOTIFS-D) | No clear effect |
| Raise MLLM token threshold tau from 0.1 to 0.5 (ablation) | BGNN+PC-SGG · VG150 SGDET | CovT: 80.45 at tau=0.1 to 47.79 at tau=0.5 | Hurt |
| Remove image cropping (use full image in MLLM prompt) (ablation) | BGNN+PC-SGG · VG150 SGDET | CovT: 80.45 to 79.96 | Hurt |
| Remove system prompt and one-shot example prompt from MLLM (ablation) | BGNN+PC-SGG · VG150 SGDET | CovT: 80.45 to 77.96 without both prompts | Hurt |
| Add conformal prediction sets (PC-SGG) on top of standard SGG | BGNN, MOTIFS, MOTIFS-D, VCTREE, SQUAT · VG150 SGDET | cR@50 vs R@50 (recall-hit rate): about 15.70 points on average | Helped |
| Add conformal prediction sets (PC-SGG) on top of standard SGG | BGNN, MOTIFS, MOTIFS-D, VCTREE, SQUAT · VG150 SGDET | cmR@50 vs mR@50 (tail-class recall): about 21.57 points on average | Helped |
| Use class-conditional conformal prediction over object and predicate heads | BGNN, MOTIFS, MOTIFS-D, VCTREE, SQUAT · VG150 SGDET | Object coverage: Object Cov 88.94 to 90.35 across models, all below 95% target | No clear effect |
| Add conformal prediction sets on top of SGG methods | all SGG models · VG150 SGDET | Triplet coverage CovT: CovT 74.97 to 80.45 across models, none reach the 90% target | No clear effect |
Surprising findings
- No evaluated SGG method achieves the marginal triplet coverage guarantee of (1-α_o)(1-α_r), suggesting distribution shift between train and test splits.
Limitations
- Triplet coverage falls short of the nominal guarantee for all evaluated SGG models.
- Higher MLLM token threshold tau reduces set size but sharply lowers triplet coverage, so a sensitive hyperparameter.
- Uses an 8-bit quantized BLIP-2 due to compute constraints.
Experimental setup
Datasets: Visual Genome VG150
Benchmarks: Visual Genome VG150
Metrics: empirical coverage average class coverage gap average set size triplet coverage cR@K cmR@K R@K mR@K
Baselines: MOTIFS MOTIFS-D VCTREE SQUAT BGNN
Builds on: conformal prediction class-conditional conformal prediction BLIP-2
Closest papers by meaning
- Noise-Guided Predicate Representation Extraction and Diffusion-Enhanced Discretization for Scene Graph Generation (ICML 2025 · similarity 0.88)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.87)
- Adaptive Visual Scene Understanding: Incremental Scene Graph Generation (NeurIPS 2024 · similarity 0.87)
- HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation (CVPR 2024 · similarity 0.87)
- Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.86)
- TD²-Net: Toward Denoising and Debiasing for Video Scene Graph Generation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.