Research map / Large language models / Language, safety and interpretability
SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SocialGPT, a zero-shot modular pipeline where VFMs (SAM+BLIP-2) turn images into textual social stories that LLMs reason over, plus Greedy Segment Prompt Optimization (GSPO) for segment-level prompt tuning.
Problem
Social relation recognition from images is limited in generalization and interpretability when using dedicated end-to-end trained networks; prompt design for LLM-based reasoning is manual and long prompts make automatic optimization hard.
Approach
Convert the image into a textual social story with symbol-based object referencing using VFMs, then perform zero-shot relation reasoning with LLMs using a four-segment SocialPrompt; optimize prompts with GSPO, a greedy segment-level search guided by gradients of segment indicators.
Key results
- SocialGPT with Vicuna-13B reaches 66.7% zero-shot accuracy on PIPA, surpassing the supervised TRGAT (65.3%).
- GSPO raises Vicuna-13B on PISC from 65.12% to 66.19% and on PIPA from 66.70% to 69.23%.
- SocialGPT (Vicuna-13B) outperforms GPT-4V by 7.03 points on PIPA (66.70 vs 59.67).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply GSPO prompt optimization | SocialGPT with Vicuna-7B · PIPA test | accuracy: 61.58 -> 62.99; +1.41 | Helped |
| apply GSPO prompt optimization | SocialGPT with Vicuna-7B · PISC test | accuracy: 45.13 -> 49.79; +4.66 | Helped |
| apply GSPO prompt optimization | SocialGPT with Vicuna-13B · PIPA test | accuracy: 66.70 -> 69.23; +2.53 | Helped |
| apply GSPO prompt optimization | SocialGPT with Vicuna-13B · PISC test | accuracy: 65.12 -> 66.19; +1.07 | Helped |
| apply GSPO prompt optimization | SocialGPT with Llama2-7B · PIPA test | accuracy: 31.91 -> 34.07; +2.16 | Helped |
| apply GSPO prompt optimization | SocialGPT with Llama2-7B · PISC test | accuracy: 36.71 -> 38.04; +1.33 | Helped |
| apply GSPO prompt optimization | SocialGPT with Llama2-13B · PIPA test | accuracy: 37.86 -> 41.27; +3.41 | Helped |
| apply GSPO prompt optimization | SocialGPT with Llama2-13B · PISC test | accuracy: 42.74 -> 48.39; +5.65 | Helped |
| add dense captions (ablation) | SocialGPT with Vicuna-7B · PIPA test | accuracy: 61.58 -> 52.63 | Helped |
| add task-oriented captions (ablation) | SocialGPT with Vicuna-7B · PIPA test | accuracy: 61.58 -> 59.89 | Helped |
| use symbol-based object references instead of coordinates (ablation) | SocialGPT with Vicuna-7B · PIPA test | accuracy: 61.58 -> 57.68 | Helped |
| use symbol-based object references instead of object captions (ablation) | SocialGPT with Vicuna-7B · PIPA test | accuracy: 61.58 -> 59.83 | Helped |
Surprising findings
- With GPT-3.5 as reasoning LLM, SocialGPT scores only 53.43% on PISC, well below the open-source Vicuna-13B (65.12%).
- Removing the guidance (in-context example) segment drops accuracy by 18.02 points, while removing the context segment with category definitions costs only about 0.4 points.
Limitations
- Performance is bounded by the errors of the underlying segmentation model, BLIP-2 captions, and LLM reasoning.
- Mainly a modular pipeline with dependence on foundation model quality; computational efficiency not analyzed.
- Reported results on PISC use accuracy instead of the mAP used by prior work, limiting direct comparability.
Experimental setup
Datasets: PIPA PISC
Benchmarks: PIPA PISC
Metrics: accuracy
Baselines: GR$^2$N TRGAT GRM MGR Dual-Glance Pair CNN SRG-GN BLIP-2 LLaVA GPT-4V
Builds on: AutoPrompt Segment Anything Model BLIP-2 Vicuna
Closest papers by meaning
- Exploiting the Social-Like Prior in Transformer for Visual Reasoning (AAAI 2024 · similarity 0.87)
- VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligence (AAAI 2025 · similarity 0.85)
- Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception (NeurIPS 2026 · similarity 0.84)
- Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection (CVPR 2024 · similarity 0.84)
- SAM-Aware Graph Prompt Reasoning Network for Cross-Domain Few-Shot Segmentation (AAAI 2025 · similarity 0.83)
- SSTAG: Structure-Aware Self-Supervised Learning Method for Text-Attributed Graphs (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.