Research map / Vision-language and multimodal / Multimodal LLMs
Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks 14 VLMs against their LLM backbones on factual recall, finds most degrade, and uses attribution patching, activation patching, and probing to attribute the drop to late entity resolution that bypasses early-layer factual-recall MLPs.
Problem
Vision language models often recall facts from images worse than their LLM backbones; it is unclear why multimodal alignment fails to reuse the backbone's factual recall circuit.
Approach
Frames visual factual recall as a two-hop problem: the VLM must form entity representations from images before the LLM's early-layer MLPs can retrieve facts. Uses mechanistic tools (attribution patching, heuristic activation patching, linear probes) to show degraded VLMs resolve entities too late, and tests heuristic patching and chain-of-thought as mitigations.
Key results
- 11 of 14 VLMs show factual recall degradation versus their LLM backbones; e.g., LLaVA-1.5-7B drops from 30% to 19% (36.7% relative).
- Heuristic patching of early-layer LLM MLP outputs recovers on average 35% of the VLM accuracy gap, versus 13% for back-patching and 16% for random patching.
- Chain-of-thought closes the full performance gap for Pixtral-12B and Pixtral-Large-124B and raises Pixtral-Large-124B VLM accuracy from 56% to 71%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the VLM for image-based factual recall instead of its text-only LLM backbone | 14 VLM/backbone pairs · Factual-recall benchmark | accuracy: 11 of 14 | Hurt |
| Add chain-of-thought prompting (ablation) | LLaVA-style VLMs · Factual-recall benchmark | accuracy: LLaVA-MORE-8B: +5.0; LLaVA-NEXT-8B: +6.0; LLaVA-1.5-7B: –1.0; LLaVA-1.5-13B: +9.0 | Mixed |
| Add chain-of-thought prompting (ablation) | Pixtral-12B and Pixtral-Large-124B · Factual-recall benchmark | accuracy: Pixtral-12B: +12.0; Pixtral-Large-124B: +15.0 | Helped |
| Add chain-of-thought prompting (ablation) | Qwen2.5-VL-7B-it and Qwen2.5-VL-72B-it · Factual-recall benchmark | accuracy: Qwen2.5-VL-7B-it: +15.0; Qwen2.5-VL-72B-it: +3.0 | Helped |
| Add chain-of-thought prompting (ablation) | Native VLMs · Factual-recall benchmark | accuracy: Llama-4-Maverick: +6.0; Gemini-2.0-Flash: +7.0; Gemma-3-27B-it: +11.0; Gemma-3-12B-it: +6.0; GPT-4o: +6.0 | Helped |
| Add chain-of-thought prompting (ablation) | Llama-3.2-Vision-11B · Factual-recall benchmark | accuracy: +9.0 | Helped |
| Use heuristic patching of backbone LLM early-layer MLP outputs | LLaVA-style VLMs · Incorrectly answered factual-recall examples | recovered accuracy gap: 35% | Helped |
| Use back-patching of layer outputs | LLaVA-style VLMs · Incorrectly answered factual-recall examples | recovered accuracy gap: 13% | Helped |
| Use random patching positions | LLaVA-style VLMs · Incorrectly answered factual-recall examples | recovered accuracy gap: 16% | Helped |
Surprising findings
- Qwen2.5-VL-72B slightly outperforms its LLM backbone (53% vs 49%).
- Chain-of-thought lowers accuracy for LLaVA-1.5-7B (-1.0 points) despite improving its backbone.
Limitations
- Focuses only on factual recall; other tasks not examined.
- Mechanistic analysis limited to a small subset of models (three degraded, two well-performing).
- Entity-misidentified samples are discarded, and factual question generation relies on GPT-4.1 which may introduce errors.
Experimental setup
Datasets: WIT (Wikipedia-based Image-Text) ImageNet-100 (50-class subset) WildHallucinations (for model selection)
Benchmarks: Multimodal factual recall benchmark (15000 generated questions, 1000 valid per model) ImageNet-100 entity probing
Metrics: factual recall accuracy accuracy drop relative to LLM backbone KL divergence probe accuracy recovered performance gap
Baselines: LLaVA-1.5-7B LLaVA-1.5-13B LLaVA-MORE-8B LLaVA-NEXT-8B Pixtral-12B Pixtral-Large-124B Qwen2.5-VL-7B-it Qwen2.5-VL-72B-it Llama-4-Maverick Gemini-2.0-Flash Gemma-3-12B-it Gemma-3-27B-it
Builds on: Factual recall localization in early-layer MLPs Attribution patching Back-patching for two-hop problems Logit lens analysis of VLM information flow
Closest papers by meaning
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.90)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.89)
- Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models (ICML 2026 · similarity 0.89)
- Dual-Latent Memory Routing for Vision-Language Reasoning (ICML 2026 · similarity 0.89)
- Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.