Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Learning Visually Grounded Domain Ontologies via Embodied Conversation and Explanation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an explanatory interactive learning framework where a neurosymbolic agent explains its wrong toy-truck predictions and a teacher's corrective feedback and generic part-whole rules refine both its ontology and visual recognition, improving data efficiency.
Problem
Fine-grained visual classification of novel toy-truck types when the learner starts with no domain ontology and a deficient part recognizer; existing neurosymbolic systems assume symbolic knowledge is fully encoded or batch-learned before deployment.
Approach
Agent answers why-questions with part-based sufficient-reason explanations; teacher corrections of wrong parts (deictic statements) and generic whole-part rules are used to update a visual exemplar base, a symbolic KB, and a factor-graph probabilistic reasoner online.
Key results
- In single_4way (LQ start), Vis+Genr+Expl is most data-efficient, followed by Vis+Genr and Vis-Only, with all differences significant (p<1e-6).
- In double_5way, Vis+Genr performs worse than Vis-Only, while Vis+Genr+Expl regains the advantage.
- Vis+Genr+Expl vs Vis+Genr final regret p-values are 0.015/0.005/0.187 for LQ/MQ/HQ starting part models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add generic part-whole rules and feedback correcting explanations | Vis+Genr+Expl · single_4way, LQ | cumulative regret | Helped |
| Add generic part-whole rules (ablation) | Vis-Only · single_4way, LQ | cumulative regret | Helped |
| Add feedback correcting explanations (ablation) | Vis+Genr · single_4way, LQ | cumulative regret | Helped |
| Add generic part-whole rules without explanation correction (ablation) | Vis-Only · double_5way, LQ | cumulative regret | Hurt |
| Add feedback correcting explanations to generic part-whole rules (ablation) | Vis+Genr · double_5way, LQ | cumulative regret | Helped |
| Add generic part-whole rules (ablation) | Vis-Only · double_5way, MQ and HQ | cumulative regret | Helped |
| Add feedback correcting explanations (ablation) | Vis+Genr · double_5way, LQ | cumulative regret: p=0.015 | Helped |
| Add feedback correcting explanations (ablation) | Vis+Genr · double_5way, MQ | cumulative regret: p=0.005 | Helped |
| Add feedback correcting explanations (ablation) | Vis+Genr · double_5way, HQ | cumulative regret: p=0.187 | No clear effect |
Surprising findings
- Generic part-whole rules without explanation-based correction can hurt data efficiency (Vis+Genr worse than Vis-Only in double_5way with low-quality part recognition).
Limitations
- Evaluated only in a simulated Unity toy-truck domain with simulated teachers.
- Benefit of explanation-based correction depends on the quality of the initial part recognition model.
- Assumes teacher-provided generic rules are always correct.
Experimental setup
Datasets: simulated Unity toy truck scenes
Benchmarks: single_4way double_5way
Metrics: cumulative regret classification accuracy
Baselines: Vis-Only Vis+Genr
Builds on: interactive task learning neurosymbolic architecture CAIPI Matcher DINOv2
Closest papers by meaning
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.85)
- ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both (NeurIPS 2026 · similarity 0.85)
- Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understanding (CVPR 2026 · similarity 0.85)
- Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.85)
- ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.85)
- SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.