Research map / Vision-language and multimodal / Multimodal LLMs
DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DyFo, a training-free visual search method where an LMM and a visual expert (Lang-Segment-Anything) interact via Monte Carlo Tree Search over focus nodes, improving fine-grained understanding and reducing hallucination.
Problem
LMMs struggle with fine-grained visual understanding and hallucinate when irrelevant content in high-resolution or cluttered images dilutes their attention; existing visual search methods need vocabulary expansion, localization modules, or extra training data.
Approach
A training-free focus adjuster in which the LMM refines a textual focus and a visual expert (Lang-SAM) extracts the corresponding image region, with two human-inspired actions (semantic focus and semantic scatter) explored via MCTS; the reward is LMM-expert consistency weighted by the region's area ratio, and final answers are a reward-weighted vote over the focus tree.
Key results
- On V* Bench, DyFo-Q (Qwen2-VL based) reaches 81.15% overall, versus 75.39% for SEAL and 72.25% for Qwen2-VL alone.
- On POPE-COCO random setting, DyFo with Qwen2-VL raises F1 from 89.22 to 91.71 and with LLaVA-1.5 from 89.58 to 91.46 (DyFo F1 for LLaVA-1.5 is 91.46 vs regular 89.58).
- On POPE GQA adversarial, DyFo with Qwen2-VL improves accuracy from 75.90 to 81.97.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DyFo dynamic focus search | Qwen2-VL · V* Bench | accuracy: 72.25 -> 81.15 | Helped |
| Apply DyFo dynamic focus search | LLaVA-1.5 · V* Bench | accuracy: 48.68 -> 59.16 | Helped |
| Apply DyFo with Qwen2-VL | Qwen2-VL · POPE-COCO Random and Popular | accuracy: 90.07 -> 92.13; 88.57 -> 89.97 | Helped |
| Use both A1 Semantic Focus and A2 Semantic Scatter rather than either action alone (ablation) | DyFo with Qwen2-VL · V* Bench | accuracy: 76.78 -> 80.00; 75.67 -> 80.00; 70.25 -> 82.89; 69.50 -> 82.89 | Helped |
| Use both A1 Semantic Focus and A2 Semantic Scatter rather than either action alone (ablation) | DyFo with LLaVA-v1.5 · V* Bench | accuracy: 57.50 -> 62.61; 45.46 -> 62.61; 55.50 -> 53.95; 52.61 -> 53.95 | Helped |
| Combine the LMM with a visual expert instead of using the visual expert alone (ablation) | DyFo · POPE-COCO Random | accuracy: 88.87 -> 91.40; 88.87 -> 92.13 | Helped |
| Use Dynamic Focus rather than Uniform Dividing Policy (ablation) | V* Bench search · V* Bench | search length: 4.90 -> 3.20 | Helped |
| Use MCTS rather than A*, BFS, or DFS (ablation) | V* Bench search · V* Bench | search length: 4.33 -> 3.20; 5.65 -> 3.20; 6.15 -> 3.20 | Helped |
Surprising findings
- Lang-SAM visual expert alone scores below DyFo on POPE-COCO, and the search step count for MCTS (3.20) is lower than A* (4.33), BFS, and DFS in the reported efficiency study.
Limitations
- Relies on the quality of the visual expert (Lang-SAM) and its consistency with the LMM for rewards.
- Inference cost from multi-step tree search and repeated LMM and expert calls is not fully quantified in the main text.
Experimental setup
Datasets: POPE V* Bench
Benchmarks: POPE V* Bench
Metrics: Accuracy Precision Recall F1 score
Baselines: SEAL LLaVA-1.5 Qwen2-VL Osprey-7B Ferret-7B Shikra-7B GPT-4V Gemini Pro MM-React Visual ChatGPT Visprog
Builds on: SEAL Monte Carlo Tree Search Upper Confidence bounds applied to Trees self-consistency Lang-Segment-Anything
Closest papers by meaning
- CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception (ICML 2026 · similarity 0.88)
- Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding (CVPR 2026 · similarity 0.88)
- V?: Guided Visual Search as a Core Mechanism in Multimodal LLMs (CVPR 2024 · similarity 0.87)
- SIFThinker: Spatially-Aware Image Focus for Visual Reasoning (AAAI 2026 · similarity 0.87)
- CoFFT: Chain of Foresight-Focus Thought for Visual Language Models (NeurIPS 2025 · similarity 0.87)
- See What You Are Told: Visual Attention Sink in Large Multimodal Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.