Research map / Vision-language and multimodal / Multimodal LLMs
Scaling Vision Pre-Training to 4K Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales CLIP-style vision pre-training to 4K resolution by contrasting local high-res regions with local captions and learning top-down/bottom-up patch selection, yielding PS3 and the VILA-HD MLLM plus the 4KPro benchmark.
Problem
Vision pre-training (CLIP/SigLIP) is limited to low resolution (378x378) because compute grows steeply with image resolution, leaving high-resolution perception weak in vision encoders and MLLMs.
Approach
Replace global image-caption contrastive learning with contrastive learning between selected high-res local regions and local captions, using learned top-down (prompt-conditioned) and bottom-up (saliency) patch selection so that cost is decoupled from image size and scales near-constantly to 4K.
Key results
- PS3 at 4K (VILA-HD) reaches 63.9% average over seven resolution-sensitive benchmarks versus 55.2% for S^2 at 756 and 60.8% for S^2 at 1512, with 4K pre-training at ~79x less cost than SigLIP global contrast.
- On 4KPro, VILA-HD-8B-4K reaches 71.0% accuracy, above Qwen2-VL-7B (71.0% in the 4KPro column of the comparison table is tied; GPT-4o and Gemini-1.5-Pro reach 59.7%), with 4K patch selection at 18%.
- PS3 at 1512 with 20% of high-res patches selected improves over AnyRes at 1512 by about 2% while using 5x fewer tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale PS3 pre-training and inference from 1512 to 3780 resolution at a constant patch budget | PS3/VILA-HD · 7 resolution-sensitive benchmarks | average accuracy: 60.8 -> 63.9 | Helped |
| Scale PS3 resolution from 1512 to 3780 for high-resolution perception | PS3/VILA-HD · 4KPro | accuracy: +4.8% | Helped |
| Increase the number of high-resolution patches at test time | PS3/VILA-HD · 4KPro | accuracy: +8.2% | Helped |
| Increase the selected patches at 1512 resolution from 729 to 1600 | PS3/VILA-HD · 7 resolution-sensitive benchmarks | average accuracy: 58.2 -> 61.0 | Helped |
| Increase selected patches at 1512 resolution from 1600 to 3645 | PS3/VILA-HD · 7 resolution-sensitive benchmarks | average accuracy: +2.2% | Helped |
| Use fewer high-resolution patches at 1512 resolution | PS3/VILA-HD · TextVQA and RealWorldQA | performance: 2.0% and 1.2% | Hurt |
| Train with 20% patches and select 44% at test time | PS3/VILA-HD · 7 resolution-sensitive benchmarks | accuracy: +1.2% | Helped |
| Train with 40% patches and select 100% at test time | PS3/VILA-HD · 7 resolution-sensitive benchmarks | accuracy: +0.9% | Helped |
| Use PS3 high-resolution pre-training instead of low-resolution SigLIP pre-training | VILA-HD · 4KPro | accuracy: +7.3% on average over NVILA-8B | Helped |
| Align PS3 with an LLM | PS3 · HRBench 4K | score: 58.5 -> 62.9 | Helped |
| Use ground-truth selection scores and pool only tokens in ground-truth boxes during localized contrastive pre-training (ablation) | PS3 · 7 resolution-sensitive benchmarks | average accuracy: over 8% | Helped |
| Avoid contrast between local regions from the same image in a batch (ablation) | PS3 · 7 resolution-sensitive benchmarks | average accuracy: 5.3% | Helped |
Surprising findings
- Most existing high-resolution benchmarks (DocVQA, V*Bench, etc.) have a minimum recognizable resolution around 1K despite containing 4K images.
- Global-only contrastive pre-training degrades low-res representation quality, requiring mixing of global and local contrast.
Limitations
- VILA-HD underperforms NVILA and Qwen2-VL on OCR-heavy benchmarks such as ChartQA, DocVQA, TextVQA, InfoVQA and OCRBench.
- Test-time scaling of selected patches still performs worse than training-time selection.
- 4KPro is small (curated from 5 frames per video) and built with GPT-4o-assisted QA plus manual review.
Experimental setup
Datasets: DataComp SA-1B IDL PDFA TextVQA ChartQA DocVQA InfoVQA OCRBench V*Bench RealWorldQA ShareGPT4V
Benchmarks: 4KPro TextVQA ChartQA DocVQA InfoVQA OCRBench V*Bench RealWorldQA HRBench-4K HRBench-8K MME-RealWorld MathVista
Metrics: accuracy latency
Baselines: SigLIP AnyRes S^2 SigLIP2 C-RADIOv2 PE-Core AIMv2 InternViT2.5 PE-Lang ToMe FastV VisionZip
Builds on: CLIP SigLIP NVILA S^2 AnyRes
Closest papers by meaning
- InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD (NeurIPS 2024 · similarity 0.87)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.86)
- Chain-of-Zoom: Extreme Super-Resolution via Scale Autoregression and Preference Alignment (NeurIPS 2025 · similarity 0.85)
- VITRIX-UniViTAR: Unified Vision Transformer with Native Resolution (NeurIPS 2025 · similarity 0.84)
- ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models (ICLR 2024 · similarity 0.84)
- QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.