Research map / Multi-agent systems and games / LLM agents

Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies visual hallucination snowballing in VLM multi-agent systems and proposes ViF, which relays unimodal vision tokens as a visual flow between agents plus attention reallocation, training-free beyond a small added transformer block.

Problem

Hallucinations seeded in one VLM agent get amplified by later agents in multi-agent systems because visual information is relayed only through text, which reduces attention to vision tokens over agent turns.

Approach

Select vision tokens with a unimodal attention peak in middle layers as "visual relay tokens", contextualize them with instruction tokens via a lightweight transformer block, pass them alongside text to the next agent as a visual flow, and reallocate attention (temperature scaling and redistribution) in middle/deep layers to amplify this pattern; a Key-Norm alternative handles FlashAttention model

Key results

• ViF raises average performance of six base VLMs across linear, layered, random, and circular MAS structures by 2.4-3.8%.
• On circular MAS with LLaVA-NeXT-7B, ViF reduces the proposed hallucination snowballing score (HS) by 39.8% on average.
• Relative to single-model hallucination mitigation methods (MemVR, VISTA, FarSight, DeCo, TAME), ViF attains at least 4.2% gains on original metrics and 34.4% HS reduction on average.

Surprising findings

• Existing single-model hallucination mitigation methods (MemVR, VISTA, FarSight, DeCo, TAME) often perform worse than the baseline when applied in multi-agent settings, attributed to retained textual relay of visual information.
• Dropping unimodal vision tokens in middle layers degrades performance far more than dropping other token subsets, while shallow and deep layers show near-uniform importance.

Limitations

• Evaluated only on models 7B-34B; smaller (3B) and larger (72B) VLMs and other families such as InternVL, Llama 3, Ovis, MiniCPM not tested.
• Combination with other hallucination mitigation strategies and more optimal vision token selection left for exploration.
• HS metric depends on a judge model's severity scores (GPT-5), introducing judge dependence.

Experimental setup

Datasets: MME MMBench MM-Vet CHAIR POPE AMBER MMHal-Bench HallBench MMIU MuirBench MVBench Video-MME

Benchmarks: MME MMBench MM-Vet CHAIR POPE AMBER MMHal-Bench HallBench MMIU MuirBench MVBench Video-MME

Metrics: accuracy MME score CHAIR_S hallucination snowballing score (HS)

Baselines: MemVR VISTA FarSight DeCo TAME

Builds on: LLaVA-NeXT FlashAttention MemVR VISTA FarSight DeCo TAME

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.91)
• MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue (ICML 2026 · similarity 0.90)
• InEx: Hallucination Mitigation via Introspection and Cross-Modal Multi-Agent Collaboration (AAAI 2026 · similarity 0.90)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.90)
• Dual-Pathway Circuits of Object Hallucination in Vision-Language Models (NeurIPS 2026 · similarity 0.90)
• Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.89)