Research map / Vision-language and multimodal / Visual question answering
Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Monkey raises LMM input resolution to 1344x896 by splitting images into 448x448 patches processed by a shared ViT with per-patch LoRA adapters and a resampler, and adds multi-level LLM-generated detailed captions, training on Qwen-VL.
Problem
Large multimodal models are limited to low input resolution (about 448x448), which hurts dense text and detailed scene understanding, and existing high-resolution methods need costly curriculum pretraining or suffer from positional-embedding interpolation issues.
Approach
Use a sliding window to split a high-resolution image into 448x448 patches, each encoded by a shared ViT with its own LoRA adapter plus a global downscaled view, combined with a multi-level caption pipeline (BLIP2, PPOCR, GRIT, SAM, ChatGPT) that produces richer image-text labels, avoiding pretraining from scratch.
Key results
- Monkey reaches 86.1 on Flickr30K captioning and 93.2 on VQAv2 in Table 2, outperforming Qwen-VL on most tasks.
- On Doc-oriented VQA, Monkey improves over Qwen-VL by an average of 9.77%, e.g. DeepForm 40.6 vs 4.1.
- Ablation on LLaVA1.5 shows the multi-level descriptions raise TextVQA by up to 1.8 points at 448 resolution and MMVet by 2.6 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace conventional positional interpolation with patch-based high-resolution processing (ablation) | Monkey / Qwen-VL · DeepForm | score: 11.4 -> 40.6 | Helped |
| Add LoRA adapters rather than using no LoRA (ablation) | Monkey / Qwen-VL · STVQA | score: 65.1 -> 67.7 | Helped |
| Increase input resolution from 672×672 to 784×784 (ablation) | Monkey / Qwen-VL · DeepForm | score: 31.3 -> 38.9 | Helped |
| Increase input resolution to 896×1344 (ablation) | Monkey / Qwen-VL · DeepForm | score: 40.6 -> 42.3 | Helped |
| Increase input resolution to 1344×896 (ablation) | Monkey / Qwen-VL · TextVQA | score: 67.6 -> 67.3 | Hurt |
| Add one LoRA adapter instead of none at 896×896 (ablation) | Monkey / Qwen-VL · STVQA | score: 65.1 -> 67.0 | Helped |
| Increase LoRA adapter count from one to four (ablation) | Monkey / Qwen-VL · STVQA | score: 67.0 -> 67.7 | Helped |
| Use LoRA with traditional positional interpolation (ablation) | Monkey / Qwen-VL · DeepForm | score: 32.7 -> 7.2 | Hurt |
| Train with generated multi-level descriptions instead of original CC3M annotations (ablation) | LLaVA1.5 · TextVQA | score: 60.2 -> 62.0 | Helped |
| Train with generated multi-level descriptions instead of original CC3M annotations (ablation) | LLaVA1.5 · MMVet | score: 33.6 -> 36.2 | Helped |
| Train with generated multi-level descriptions instead of original CC3M annotations (ablation) | LLaVA1.5 · GQA | score: 64.3 -> 64.6 | Helped |
| Increase image resolution from 224 to 448 with generated descriptions (ablation) | LLaVA1.5 · MMVet | score: 33.7 -> 36.2 | Helped |
Surprising findings
- Traditional positional-embedding interpolation to 896 resolution performed worse than the patch-based method, and adding LoRA to the interpolated encoder hurt performance.
Limitations
- Input is limited to a maximum of six patches due to LLM input length, which caps resolution expansion.
- Multi-level descriptions are bounded by the world knowledge of BLIP2 and original CC3M annotations, so they cannot identify specifics such as country.
- Most results are reported at 896x896 because of training time limits.
Experimental setup
Datasets: COCO Caption TextCaps VQAv2 OKVQA GQA ScienceQA VizWiz TextVQA OCRVQA AI2D DocVQA ChartQA
Benchmarks: Flickr30K TextCaps VQAv2 OKVQA GQA ScienceQA VizWiz TextVQA AI2D STVQA ESTVQA DocVQA
Metrics: CIDEr accuracy MME perception score throughput and FLOPs
Baselines: Qwen-VL Qwen-VL-Chat LLaVA1.5 mPLUG-Owl2 InstructBLIP GPT4V Flamingo-80B BLIP-2
Builds on: Qwen-VL Flamingo LoRA LLaVA-1.5
Closest papers by meaning
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.88)
- VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks (NeurIPS 2024 · similarity 0.88)
- Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models (ICLR 2025 · similarity 0.88)
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.88)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.88)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.