Research map / Vision-language and multimodal / Multimodal LLMs
Seeking and Updating with Live Visual Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LiveVQA, a post-cutoff visual QA dataset built from news, YouTube, and arXiv (Apr 2024-May 2025), benchmarks 17 MLLMs with and without search, and tests LoRA/DoRA updating of Qwen2.5-VL with new visual knowledge.
Problem
MLLMs have fixed knowledge cutoffs and struggle with visual entities and events that appear after training, and there is no benchmark for seeking and updating live visual knowledge.
Approach
An automated, LLM/MLLM-in-the-loop data engine collects visual content published after model cutoffs from news, YouTube, and arXiv, generates Level-1 entity and Level-2 multi-hop questions, and filters them with a model pool so the test set stays hard for current models; it then evaluates search-augmented seeking and PEFT-based updating.
Key results
- GPT-4.1 with MM-Search raises average Level-1 accuracy from 16.5% (no search) to 33.4% and Level-2 from 3.0% to 30.4%.
- GPT-o3 with native image search reaches 23.3% Level-1 and 15.8% Level-2 average accuracy, versus 3.0% for GPT-4.1 without search on Level 2.
- Fine-tuning with LoRA/DoRA on LiveVQA improves MMMU by 4.2%, while MMStar drops from 65.80% to 58.16% with the simple-answer format.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MM-Search multimodal search | GPT-4.1 · LiveVQA Level 1 Avg. | accuracy (%): 16.5% to 33.4% | Helped |
| Add MM-Search multimodal search | GPT-4.1 · LiveVQA Level 2 Avg. | accuracy (%): 3.0% to 30.4% | Helped |
| Add native image search | GPT-o3 · LiveVQA Level 2 Avg. | accuracy (%): 3.0% to 15.8% | Helped |
| Add text search | GPT-4.1 · LiveVQA Level 1 Avg. | accuracy (%): 16.5 -> 15.6 | Hurt |
| Add text search | GPT-4.1 · LiveVQA Level 2 Avg. | accuracy (%): 3.0 -> 4.3 | Helped |
| Fine-tune on LiveVQA using concise multiple-choice answers (ablation) | Qwen2.5-VL · MMMU | performance: 4.2% | Helped |
| Fine-tune with simple-answer format (ablation) | Qwen2.5-VL · MMStar | performance: 65.80% to 58.16% | Hurt |
| Increase LoRA rank (ablation) | Qwen2.5-VL visual knowledge updating · LiveVQA validation subset | performance: 5.4% | Helped |
| Use direct multiple-choice questions with concise answers instead of QA or QAR formats (ablation) | Qwen2.5-VL visual knowledge updating · LiveVQA validation set | performance over training | Mixed |
| Continue training beyond the first epoch with QA or QAR formats (ablation) | Qwen2.5-VL visual knowledge updating · LiveVQA validation set | performance | No clear effect |
| Vary LoRA rank, training steps, or training format while fine-tuning (ablation) | Qwen2.5-VL · MMStar | performance | Hurt |
Surprising findings
- Qwen2.5-VL-7B outperforms the larger Qwen2.5-VL-72B on the visual factuality seeking benchmark.
- Visual knowledge updating degrades general visual perception (MMStar), an apparent conflict between knowledge injection and perception.
Limitations
- Older news and video data (over two months) were hard to crawl, causing date-category imbalance.
- Automated GPT-4.1 pipeline may still introduce QA errors despite human validation.
- Update experiments limited to LoRA/DoRA and the Qwen2.5-VL family, without full fine-tuning.
Experimental setup
Datasets: LiveVQA training split MMMU MMStar
Benchmarks: LiveVQA MMMU MMStar
Metrics: accuracy F-score calibration
Baselines: GPT-4.1 Gemini-2.5-Pro Claude-3.7-Sonnet GPT-o3 Qwen2.5-VL-72B MM-Search
Builds on: SimpleQA MM-Search LoRA DoRA
Closest papers by meaning
- SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge (NeurIPS 2024 · similarity 0.89)
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.89)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.88)
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.88)
- VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark (NeurIPS 2024 · similarity 0.87)
- Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.