Research map / Vision-language and multimodal / Multimodal LLMs

Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Concept-RuleNet, a three-agent neurosymbolic system that grounds LLM-generated symbols and first-order rules in visual concepts mined from training images, and combines them with VLM predictions for zero-shot image classification.

Problem

Label-conditioned neurosymbolic rule generation with LLMs produces hallucinated and non-representative symbols, especially on out-of-distribution datasets such as medical and remote-sensing images.

Approach

Condition LLM symbol exploration and rule formation on visual concepts extracted by a VLM from representative training images, and verify symbols with a VLM via yes/no token probabilities, fusing the resulting rule score with System-1 predictions; an extension adds counterfactual (negated) symbols.

Key results

• Concept-RuleNet beats Symbol-LLM by about 5 percentage points on average across datasets and models (paired t-test, p=0.019 vs S-LLM).
• With InstructBLIP as verifier, UCMerced accuracy rises from 48.0 (Symbol-LLM) to 57.33 (Concept-RuleNet); DermaMNIST with LLaVA-1.6 rises from 38.19 to 48.74 (CRN column, LLaVA-1.6 Derma 48.74).
• Concept-RuleNet++ with counterfactual symbols improves over Concept-RuleNet by about 1-2 points on average (e.g., DermaMNIST 8.54 to 14.23 on InstructBLIP).

Surprising findings

• LLaVA-Med used as both System-1 model and verifier performs poorly, suggesting strong verification does not imply strong reasoning.
• Symbol-LLM's symbols have occurrence likelihood below 0.5 on the Satellite and WHU datasets.

Limitations

• Gains on some datasets are modest (e.g., WHU 2-4%) and the abstract's 50% hallucination reduction is not clearly isolated in tables.
• Relies on GPT-4o-mini API and VLM verifiers; results depend on prompts and hyperparameters such as lambda and entailment threshold tuned on validation.
• Concept-RuleNet++ is not directly comparable to Symbol-LLM and is evaluated only on InstructBLIP.

Experimental setup

Datasets: BloodMNIST DermaMNIST UC-Merced Satellite Land Use WHU iNaturalist-21

Benchmarks: BloodMNIST DermaMNIST UCMerced-Satellite WHU iNaturalist-21

Metrics: accuracy symbol grounding likelihood (VLM-estimated presence) representativeness score (GPT-o1 rated)

Baselines: Symbol-LLM System-1 VLM only (InstructBLIP, LLaVA-1.5, LLaVA-1.6)

Builds on: Symbol-LLM VQAScore

Closest papers by meaning

• NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning (NeurIPS 2026 · similarity 0.90)
• Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.90)
• NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language (ICLR 2026 · similarity 0.89)
• Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens (CVPR 2026 · similarity 0.88)
• DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning (ICLR 2026 · similarity 0.88)
• LINE: LLM-based Iterative Neuron Explanations for Vision Models (NeurIPS 2026 · similarity 0.88)