Research map / Vision-language and multimodal / Multimodal LLMs
HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiRes-LLaVA, which uses a SliceRestore Adapter and Self-Mining Sampler to reduce context fragmentation from sliced high-resolution inputs, and introduces the EntityGrid-QA benchmark to measure edge-vs-center fragmentation.
Problem
Slicing high-resolution images into fixed-size patches for LVLMs fragments objects at slice boundaries, disrupting cross-patch context and hurting performance; existing Q-Former-style resamplers also lack positional information and need high training overhead.
Approach
Restore sliced patch features into their original spatial layout inside ViT attention layers via parallel local (depth-wise conv) and global (attention on downsampled view) fusion, and compress tokens with a sampler whose queries are pooled from the visual tokens themselves rather than learned queries.
Key results
- On EntityGrid-QA, HiRes-LLaVA reaches 70.20 mean accuracy with PD1 94.34%, versus 63.78 for IXC-4KHD and 65.22 for LLaVA-NeXT.
- HiRes-LLaVA (Llama3.1-8B, 1344px) reports 74.2 on VQA-Text and 84.9 on DocVQA, above listed slicing and high-resolution baselines in the paper's table.
- Ablation: SMS outperforms Resampler by 6.9% average on four benchmarks; SRA with global and local fusion raises average from 52.9 to 55.4 (16 slices) in the ablation setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HiRes-LLaVA rather than slicing-based baselines | LVLMs · EntityGrid-QA | Acc_mean: 65.22 -> 70.20 | Helped |
| Add SliceRestore Adapter (SRA) | HiRes-LLaVA · EntityGrid-QA edge positions | Acc_e: 58.00% -> 68.40% | Helped |
| Add SRA global and local fusion to SMS (ablation) | HiRes-LLaVA · VQA-Text | score: 60.0 -> 61.5 | Helped |
| Add separated separators (ablation) | HiRes-LLaVA with SMS and SRA · VQA-Text | score: 61.5 -> 61.8 | Helped |
| Replace Resampler with SMS (ablation) | HiRes-LLaVA without SRA · VQA-Text | score: 58.8 -> 60.0 | Helped |
| Increase maximum slice count from 4 to 16 (ablation) | HiRes-LLaVA · DocQA | score: 37.0 -> 59.7 | Helped |
| Increase SMS pooling kernel size (fewer visual tokens) (ablation) | HiRes-LLaVA · DocVQA | score: 59.7 -> 46.3 -> 35.3 | Hurt |
| Replace Vicuna-1.5-7B with Llama3.1-8B (ablation) | HiRes-LLaVA with CLIP-ViT-224px · DocQA | score: 59.7 -> 67.2 | Helped |
| Replace CLIP-ViT-224px with InternViT-448px (ablation) | HiRes-LLaVA with Llama3.1 · DocQA | score: 67.2 -> 74.4 | Helped |
| Use same separators instead of separated separators (ablation) | HiRes-LLaVA · VQA-Text | score: 61.8 -> 57.2 | Hurt |
| Increase training data mixture to the full dataset (ablation) | HiRes-LLaVA · seven evaluated benchmarks | performance | Helped |
| Use 60% rather than 100% of training data (ablation) | HiRes-LLaVA · seven evaluated benchmarks | performance: above 90% | No clear effect |
Surprising findings
- Q-Former-like and ConcatChannel downsamplers degrade ablation results relative to SMS, and LLaVA-NeXT runs out of memory at 1344px batch size 2 under the paper's setup.
Limitations
- EntityGrid-QA is synthetic and restricted to non-overlapping 2x2 slicing with entity identification tasks, so it may not fully reflect real-world fragmentation.
- Ablations use LoRA and a different data mixture from the main training, so main-table numbers are not directly comparable to ablation numbers.
- Some reported comparisons use different input resolutions and baseline configurations across tables.
Experimental setup
Datasets: LLaVA-558k ShareGPT4V ALLaVA ShareGPT-4o SynthDoG MMC-Alignment UReader Magpie Pro LLaVA-665k DocVQA ChartQA InfoVQA
Benchmarks: VQA-Text ChartQA DocVQA InfoVQA AI2D ScienceQA MME MMBench MM-Vet EntityGrid-QA
Metrics: accuracy precision discrepancy PD1 precision discrepancy PD2 accuracy standard deviation training time FLOPs
Baselines: LLaVA-1.5 LLaVA-NeXT IXC-4KHD Monkey Cambrian-1 Mini-Gemini-HD LLaVA-UHD mPLUG-DocOwl2 Q-Former Resampler C-Abstractor ConcatChannel
Builds on: LLaVA-NeXT Q-Former CLIP-ViT LoRA
Closest papers by meaning
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.90)
- Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.89)
- Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models (ICLR 2025 · similarity 0.89)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.89)
- Matryoshka Query Transformer for Large Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.