Research map / Vision-language and multimodal / Multimodal LLMs

NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces NeuronEye, a plug-in that builds a sparse SAE-based concept-level neuron vocabulary from VLM vision tokens and uses query-guided cluster activation plus suppression of dominant directions to improve vision-centric reasoning on a frozen backbone.

Problem

Vision-language models store visual information in dense hidden states where object identity, layout, and attributes are entangled, limiting their ability to isolate and modulate query-relevant visual evidence.

Approach

Decompose intermediate vision-token states with a sparse autoencoder into concept-level neuron clusters, use the language query to route to relevant clusters and localized patches, inject the focused evidence back into vision tokens, and attenuate dominant perceptual directions at a later layer, all in one forward pass over a frozen VLM.

Key results

• On Qwen2.5-VL-7B, NeuronEye raises CV-Bench overall from 78.5 to 81.6 (+3.1) and Distance by +9.5 (76.2 to 85.7).
• BLINK Multi-view improves by +8.3 (55.6 to 63.9) on Qwen2.5-VL-7B.
• On OmniMedVQA-Mini, NeuronEye reduces overall accuracy from 65.30% to 63.10%, showing limited domain transfer.

Surprising findings

• Direct fine-tuning of the lightweight modules without sparse intervention reduces CV-Bench Distance from 76.2 to 69.8.
• NeuronEye lowers LLaVA-1.6-7B CV-Bench Count (-8.3) and Depth (-3.3) despite gains elsewhere.

Limitations

• SAE and neuron clusters trained on general-domain VQAv2 lack coverage of specialized domains, reducing OmniMedVQA-Mini accuracy.
• Routing supervision relies on LLM-generated cluster labels, only ~20% manually verified.
• Inference overhead of about 20% latency and 42% peak memory increase.

Experimental setup

Datasets: VQAv2 CV-Bench BLINK RealWorldQA MMStar OmniMedVQA-Mini

Benchmarks: CV-Bench BLINK RealWorldQA MMStar OmniMedVQA-Mini

Metrics: accuracy

Baselines: FastV SparseVLM PruMerge MustDrop PyramidDrop SSL SAVE VISTA Baseline + FT Random SNS + NVF

Builds on: sparse autoencoder FastV VISTA

Closest papers by meaning

• VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.91)
• Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models (AAAI 2026 · similarity 0.90)
• LINE: LLM-based Iterative Neuron Explanations for Vision Models (NeurIPS 2026 · similarity 0.90)
• Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.90)
• LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models (CVPR 2026 · similarity 0.89)
• DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning (ICLR 2026 · similarity 0.89)