Research map / Vision-language and multimodal / Visual question answering
SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SearchLVLMs, a plug-and-play framework where a LoRA-tuned hierarchical filter (website then content) selects up-to-date web content from search results to prompt existing LVLMs, and introduces the UDK-VQA news-based VQA benchmark.
Problem
LVLMs lack up-to-date knowledge because they cannot be retrained frequently; existing internet-augmented generation (IAG) in commercial LVLMs is opaque and long contexts are costly for LVLMs.
Approach
A hierarchical filtering model (website filter on title/snippet, then content filter on segments, plus CLIP-based diversity selection) trained on pseudo-labels from multi-LVLM voting selects helpful web content to prompt any frozen LVLM without fine-tuning the LVLM.
Key results
- Framework with LLaVA-1.6 reaches 90.2% overall accuracy on UDK-VQA versus 24.2% for raw GPT-4V and 64.5% for GPT-4V IAG, i.e. outperforming GPT-4V by roughly 25 points.
- InternVL-1.5 with SearchLVLMs reaches 92.9% overall on UDK-VQA, the best result among the compared variants.
- On GQA, InfoSeek and A-OKVQA, the framework gives small gains (e.g. LLaVA-1.6 InfoSeek 37.86 to 41.25).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SearchLVLMs retrieval and filtering | GPT-4V · UDK-VQA test | accuracy: 64.5 -> 87.2 | Helped |
| Add SearchLVLMs retrieval and filtering | GPT-4o · UDK-VQA test | accuracy: 57.8 -> 91.8 | Helped |
| Add SearchLVLMs retrieval and filtering | LLaVA-1.6 · UDK-VQA test | accuracy: 31.8 -> 90.2 | Helped |
| Add SearchLVLMs retrieval and filtering | InternVL-1.5 · UDK-VQA test | accuracy: 31.8 -> 92.9 | Helped |
| Use question and image similarity for segment retrieval instead of no retrieval (ablation) | LLaVA-1.6 · UDK-VQA test | accuracy: 31.8 -> 46.1 | Helped |
| Use question and image query generators together rather than one generator (ablation) | SearchLVLMs with LLaVA-1.6 · UDK-VQA test | accuracy: 65.9 -> 87.6 | Helped |
| Use all question query generators together with image query generation (ablation) | SearchLVLMs with LLaVA-1.6 · UDK-VQA test | accuracy: 87.6 -> 90.2 | Helped |
| Replace the LLaVA-1.5 filtering backbone with Qwen-VL (ablation) | SearchLVLMs with LLaVA-1.6 · UDK-VQA test | accuracy: 90.2 -> 89.6 | No clear effect |
| Train the filtering model separately while freezing the LVLM instead of joint training (ablation) | Qwen-VL · UDK-VQA test | accuracy: 68.5 -> 84.8 | Helped |
| Invoke SearchLVLMs when the model responds with E | LLaVA-1.6 · GQA | accuracy: 61.66 -> 62.33 | Helped |
| Invoke SearchLVLMs when the model responds with E | LLaVA-1.6 · InfoSeek | accuracy: 37.86 -> 41.25 | Helped |
| Invoke SearchLVLMs when the model responds with E | InternVL-1.5 · InfoSeek | accuracy: 51.13 -> 53.10 | Helped |
Surprising findings
- Jointly training the filter with the LVLM performed much worse (68.0%) than separate training with frozen LVLMs (88.9%) for LLaVA-1.5.
- Top-K selection accuracy drops as K increases because LVLMs repeat inputs instead of answering when given many duplicate segments.
Limitations
- Training labels are pseudo-scores from LVLM voting rather than human annotation, which can bias the filter.
- Gains on general datasets (GQA, InfoSeek, A-OKVQA) are small, and the framework targets up-to-date knowledge only.
- Commercial LVLM IAG mechanisms are compared via web interfaces, which are not reproducible.
Experimental setup
Datasets: UDK-VQA GQA InfoSeek A-OKVQA
Benchmarks: UDK-VQA GQA InfoSeek A-OKVQA
Metrics: accuracy
Baselines: GPT-4V GPT-4o Gemini 1.5 Pro LLaVA-1.6 InternVL-1.5 Chameleon-style IAG (Cham.) CLIP→FID (C→F) CFR Oracle→FID Omni-SMoLA
Builds on: retrieval-augmented generation LoRA CLIP LLaVA-1.5
Closest papers by meaning
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.92)
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.90)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.90)
- Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models (ICML 2026 · similarity 0.89)
- Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines (AAAI 2025 · similarity 0.89)
- Seeking and Updating with Live Visual Knowledge (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.