Research map / Multi-agent systems and games / LLM agents

Schema-Guided Scene-Graph Reasoning Based on Multi-Agent Large Language Model System

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SG-RwR, a schema-guided multi-agent LLM framework where a Reasoner and a Retriever iteratively reason over scene graphs, with the Retriever answering queries by generating executable graph code.

Problem

LLMs struggle to reason over large textualized scene graphs due to distraction and hallucination from irrelevant graph information, and existing API-based ReAct or retrieve-then-reason methods are limited by fixed tool capacity and context accumulation.

Approach

Two multi-agent modules (Reasoner: Task Planner + Tool Caller; Retriever: Code Writer + Verifier) interact iteratively via natural-language queries; both are prompted only with a scene graph schema, not the graph itself, and the Retriever executes LLM-written code on the graph to fetch only relevant information.

Key results

• On BabyAI NumQ&A, SG-RwR reaches 98% success vs 86% for ReAct and 67% for the best direct prompting baseline (Alg).
• On BabyAI Trv-2 (extrapolation), SG-RwR achieves 96% vs 95% for ReAct and at most 20% for graph-prompting baselines.
• In VirtualHome, SG-RwR completes 8/8 tasks in both VH-1 and VH-2, while the best baselines complete 7/8 and 6/8.

Surprising findings

• Few-shot CoT underperforms zero-shot CoT on BabyAI NumQ&A (56% vs 60%) and few-shot graph prompting drops to 0% on Trv-2.
• With GPT-4o-limited APIs, ReAct-limit drops sharply (e.g., Trv-2 from 95% to 11%), while SG-RwR-limit retains 70%.

Limitations

• Experiments limited to static settings with simulated environments (BabyAI, VirtualHome).
• Relies on hand-annotated few-shot examples and manually curated reasoning tools and APIs.
• Small language models perform poorly with the approach except Phi-4 (60% on NumQ&A), and the method is more costly in iterations and tokens on complex tasks.

Experimental setup

Datasets: BabyAI (simulated 2D grid world scene graphs) VirtualHome household tasks (ProgPrompt tasks)

Benchmarks: BabyAI NumQ&A BabyAI Trv-1 BabyAI Trv-2 VirtualHome VH-1 VirtualHome VH-2

Metrics: success rate

Baselines: zero-shot prompting Zero-Shot Chain-of-Thought Least-to-Most Chain-of-Thought Build-a-Graph Algorithmic Prompting SayPlan ReAct ReAct-limit

Builds on: ReAct AutoGen SayPlan NLGraph Build-a-Graph

Closest papers by meaning

• Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.89)
• DiGraphHal-Bench: Evaluating Multimodal Large Language Models on Complex Directed Graphs (CVPR 2026 · similarity 0.87)
• SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models (CVPR 2026 · similarity 0.87)
• SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.87)
• PixelCraft: A Multi-Agent system for High-Fidelity Visual Reasoning on Structured Images (ICLR 2026 · similarity 0.86)
• VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context (ICML 2024 · similarity 0.86)