Research map / Vision-language and multimodal / Visual question answering
Exploiting the Social-Like Prior in Transformer for Visual Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Social-like Self-Attention (SSA), with Dynamic Structured Interaction and Discriminative Pillar Verification modules inspired by social network theory, plugged into transformer VQA and REC models to improve visual reasoning.
Problem
Self-attention in transformers suffers from rank collapse and representation degradation with depth, causing redundant aggregation and homogenized features in visual reasoning tasks like VQA and REC.
Approach
Borrows structural hole and degree centrality concepts from social network theory to make SA interactions structured via a learned binary mask over the interaction matrix, and re-weights regions by a learned "pillar" score derived from their focus and transitivity, yielding two plug-and-play modules.
Key results
- Integrating SSA into TRAR gives 72.5% overall accuracy on VQA 2.0 test-dev, a new SOTA reported by the authors.
- On RefCOCOg val-u, the VLTVG+Ours variant reaches 72.24% versus 71.08% for VLTVG, a 1.53-point gain over prior SOTA (as stated in the text).
- On VQA 2.0 val, MCAN improves from 67.27 to 68.01 overall accuracy with DSI+DPV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SSA to VLTVG* | VLTVG* · RefCOCO val | Acc@0.5: 82.45 -> 83.09 | Helped |
| Add SSA to VLTVG* | VLTVG* · RefCOCO testA | Acc@0.5: 84.98 -> 84.87 | Hurt |
| Add SSA to VLTVG* | VLTVG* · RefCOCO testB | Acc@0.5: 76.45 -> 76.98 | Helped |
| Add SSA to VLTVG* | VLTVG* · RefCOCO+ val | Acc@0.5: 71.08 -> 71.52 | Helped |
| Add SSA to VLTVG* | VLTVG* · RefCOCO+ testA | Acc@0.5: 75.71 -> 76.46 | Helped |
| Add SSA to VLTVG* | VLTVG* · RefCOCO+ testB | Acc@0.5: 60.45 -> 61.46 | Helped |
| Add SSA to VLTVG* | VLTVG* · RefCOCOg val-g | Acc@0.5: 69.20 -> 70.73 | Helped |
| Add SSA to VLTVG* | VLTVG* · RefCOCOg val-u | Acc@0.5: 71.08 -> 72.24 | Helped |
| Add SSA to VLTVG* | VLTVG* · RefCOCOg test-u | Acc@0.5: 70.64 -> 70.79 | Helped |
| Add DSI only (ablation) | MCAN · VQA 2.0 val | overall accuracy: 67.27 -> 67.58 | Helped |
| Add DPV only (ablation) | MCAN · VQA 2.0 val | overall accuracy: 67.27 -> 67.69 | Helped |
| Add DSI and DPV (ablation) | MCAN · VQA 2.0 val | overall accuracy: 67.27 -> 68.01 | Helped |
Limitations
- REC baselines TransVG and VLTVG were re-implemented with a bootstrapped MCAN-based transformer, which may cause about 2% drop versus official results.
- Gains on some benchmarks are modest (e.g., +0.1 to +0.6 points on several settings).
Experimental setup
Datasets: VQA 2.0 CLEVR RefCOCO RefCOCO+ RefCOCOg
Benchmarks: VQA 2.0 CLEVR RefCOCO RefCOCO+ RefCOCOg
Metrics: accuracy Acc@0.5
Baselines: MCAN TRAR BAN ReGAT AGAN MMNAS VLTVG TransVG NS-VQA OCCAM
Builds on: MCAN TRAR TransVG VLTVG BERT
Closest papers by meaning
- VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligence (AAAI 2025 · similarity 0.87)
- Semantic-Enriched Latent Visual Reasoning (ICML 2026 · similarity 0.87)
- SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization (NeurIPS 2024 · similarity 0.87)
- SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.86)
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.86)
- Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.