Research map / Vision-language and multimodal / Multimodal LLMs

InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

NeurIPS 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends the InternLM-XComposer2 LVLM to handle images from 336 pixels up to 4K HD using dynamic patch partitioning with a learnable newline token, reaching SOTA on 6 of 16 benchmarks at 7B scale.

Problem

Existing LVLMs are limited to roughly 1500x1500 input resolution and narrow resolution ranges, which hurts fine-grained understanding of OCR-heavy content such as documents, charts, and infographics.

Approach

Dynamic image partitioning keeps the image aspect ratio and splits it into 336x336 ViT patches (up to a maximum patch count), adds a global view, and inserts a learnable newline token after each patch row so the LLM can recover 2D layout; training resolution scales to 4K HD with a frozen-ViT-resolution design.

Key results

• IXC2-4KHD scores 90.0 on DocVQA and 81.0 on ChartQA, surpassing GPT-4V and Gemini Pro by a non-trivial margin.
• On InfographicVQA, IXC2-4KHD reaches 68.6, +17.9 over DocOwl 1.5 among open-source models.
• Performance on InfoVQA rises from 50.5 (HD-9) to 63.6 (HD-25) with training resolution; no saturation observed up to 4KHD.

Measured effects reported in the paper

Surprising findings

• Inferring at a slightly higher resolution than training (e.g., HD-16 on an HD-9-trained model) improves text-related tasks without extra training, though ChartQA degrades.
• A learnable Re-Sampler token compressor performed noticeably worse than concatenation or C-Abstractor under the same compression rate.

Limitations

• Computational cost grows with input resolution; upper bound of resolution benefit not explored.
• Re-Sampler underperforms due to limited pre-training data; ChartQA degrades when inference resolution differs from training.
• Mixed results: lower on MMMU (39.7 vs 41.4 for IXC2-VL) and does not beat closed-source APIs on all benchmarks.

Experimental setup

Datasets: ShareGPT4V COCO Nocaps TextCaps LAION400M SBU CC 3M Concept Data WanJuan Flickr MMC-Inst RCTW-17

Benchmarks: DocVQA ChartQA InfographicVQA TextVQA OCRBench MMStar MathVista MMMU AI2D MME MMBench MMBench-CN

Metrics: accuracy score

Baselines: GPT-4V Gemini Pro TextMonkey CogAgent UReader DocOwl 1.5 Monkey LLaVA-Next Qwen-VL-Chat InternLM-XComposer2-VL

Builds on: InternLM-XComposer2 Monkey LLaVA-UHD CLIP ViT-L/14

Closest papers by meaning

• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.88)
• iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.87)
• FastVLM: Efficient Vision Encoding for Vision Language Models (CVPR 2025 · similarity 0.87)
• Scaling Vision Pre-Training to 4K Resolution (CVPR 2025 · similarity 0.87)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (CVPR 2024 · similarity 0.87)