Research map / Vision-language and multimodal / Multimodal LLMs

Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses steering-vector-based activation patching to locate emotional circuits in LVLMs (Qwen3-VL, LLaVA-OV-1.5) and proposes VEENA, a training-free inference-time intervention that scales attention and MLP neurons to improve descriptive emotion recognition.

Problem

LVLMs produce emotional hallucinations when describing visual stimuli, but the internal mechanisms by which they translate visual cues into emotional narratives are unexplored, and standard token-level causal metrics do not fit open-ended descriptive generation.

Approach

Extract emotion steering vectors from contrastive emotional vs. neutral image pairs, use a hidden-space Latent Restoration Metric to localize critical layers, attention heads and MLP neurons, then intervene with flow-aware attention scaling (VEE) and neuron amplification (ENA) at inference time.

Key results

• VEENA raises average MER-UniBench performance on Qwen3-VL-4B from 58.1% to 64.8% (+6.7%).
• On MER2023, VEENA reaches 66.03 hit rate vs. 53.92 baseline and 59.61 for PAI on Qwen3-VL-4B.
• VEENA improves POPE average accuracy only slightly (87.38 to 87.62) and does not yield universal gains on general benchmarks, with a MME drop of 33.1 for LLaVA-OV-1.5-4B.

Surprising findings

• Visual-to-query attention shows a counter-intuitive dip during early layers despite high saliency, interpreted as modality realignment.
• VEENA does not yield universal improvements on general benchmarks; LLaVA-OV-1.5-4B loses 33.1 MME points.

Limitations

• Mechanism analysis relies on a small set of 250 contrastive pairs and a few models (Qwen3-VL and LLaVA-OV-1.5).
• Hit rate and F_s depend on an LLM-as-judge keyword extraction and emotion wheel mapping.
• Intervention does not improve general capabilities uniformly and shows trade-offs on some metrics.

Experimental setup

Datasets: MER-UniBench POPE CHAIR MME MSCOCO

Benchmarks: MER-UniBench POPE CHAIR MME

Metrics: hit rate weighted average F-score F_s accuracy F1 CHAIR_S CHAIR_I MME score

Baselines: VISTA PAI

Builds on: activation patching attribution patching activation steering AffectGPT

Closest papers by meaning

• Pathways of Visual Information Flow in Vision-Language Models (NeurIPS 2026 · similarity 0.87)
• EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
• AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimization (ICLR 2026 · similarity 0.87)
• Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models (ICML 2025 · similarity 0.87)
• Dual-Pathway Circuits of Object Hallucination in Vision-Language Models (NeurIPS 2026 · similarity 0.86)
• Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding (ICML 2026 · similarity 0.86)