Research map / Vision-language and multimodal / CLIP and zero-shot
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales a vision transformer (InternViT-6B) to 6B parameters and progressively aligns it with an LLM-initialized middleware (QLLaMA) via contrastive, generative, and SFT stages for generic visual-linguistic tasks.
Problem
Vision and vision-language foundation models lag behind LLMs in scale, and the small "glue" layers used to connect vision encoders to LLMs leave representations inconsistent and under-utilize LLM capacity.
Approach
Scale the vision encoder to 6B parameters, initialize a large 8B cross-attention middleware (QLLaMA) from multilingual LLaMA-7B, and align them with a progressive pipeline: contrastive training on noisy web image-text pairs, then frozen-backbone generative training on filtered data, then SFT with an LLM decoder such as Vicuna.
Key results
- InternViT-6B linear probing reaches 88.2% top-1 on ImageNet-1K (best reported without JFT data) and 47.2 mIoU linear probing on ADE20K, +12.6 over ViT-22B.
- InternVL-C achieves 80.6 on ObjectNet and 82.4 average over ImageNet variants zero-shot; InternVL-G reaches 95.7 Flickr30K I2T R@1.
- InternVL-Chat with QLLaMA and Vicuna-13B reaches MME 1586.4 and POPE 87.6 when compared to LLaVA-1.5 (MME 1531.3, POPE 85.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InternVL-G instead of InternVL-C for image-text retrieval | InternVL · English and Chinese image-text retrieval | retrieval performance | Helped |
| Use 8 sampled video frames instead of 1 | InternVL-C · Kinetics-400 | top-1 accuracy: 64.8 -> 69.1 | Helped |
| Use width 3968 and depth 32 (variant 1) instead of the selected variant 3 (ablation) | InternViT-6B · zero-shot ImageNet variants | accuracy: 66.2 -> 65.8 | Hurt |
| Use width 3200, depth 48, and 50 heads (variant 2) instead of variant 3 (ablation) | InternViT-6B · zero-shot ImageNet variants | accuracy: 66.2 -> 66.1 | No clear effect |
| Use width 2496, depth 48, and MLP size 19968 (variant 4) instead of variant 3 (ablation) | InternViT-6B · zero-shot ImageNet variants | accuracy: 66.2 -> 65.9 | Hurt |
| Use width 2816 and depth 64 (variant 5) instead of variant 3 (ablation) | InternViT-6B · zero-shot ImageNet variants | accuracy: 66.2 -> 66.2 | No clear effect |
| Use width 2496 and depth 80 (variant 6) instead of variant 3 (ablation) | InternViT-6B · zero-shot ImageNet variants | accuracy: 66.2 -> 66.2 | No clear effect |
| Use InternViT-6B instead of EVA-E as the visual encoder, keeping the MLP connector and Vicuna-7B (ablation) | InternVL-Chat · MME | score: 970.5 -> 1022.3 | Helped |
| Add QLLaMA as the connector instead of using the MLP connector (ablation) | InternVL-Chat with InternViT-6B and Vicuna-7B · MME | score: 1022.3 -> 1227.5 | Helped |
| Train on the paper's instruction data instead of LLaVA-665K (ablation) | InternVL-Chat with InternViT-6B, QLLaMA, and Vicuna-7B · NoCaps | captioning score: 94.5 -> 120.5 | Helped |
| Use Vicuna-13B instead of Vicuna-7B with InternViT-6B, QLLaMA, and the paper's instruction data (ablation) | InternVL-Chat · MME | score: 1298.5 -> 1317.2 | Helped |
Surprising findings
- With only MLP layers trainable and a frozen LLM, the model still achieves strong multimodal dialogue results, which the authors attribute to the feature-space alignment of QLLaMA.
Limitations
- Stage 1 and stage 2 data cleaning and training details are deferred to supplementary material
- Compute and training cost are not reported in the main text
Experimental setup
Datasets: LAION-en LAION-multi LAION-COCO COYO Wukong CC12M CC3M SBU COCO Caption TextCaps VQAv2 OKVQA
Benchmarks: ImageNet-1K ImageNet variants ObjectNet ADE20K Kinetics-400 Kinetics-600 Kinetics-700 Flickr30K COCO Flickr30K-CN COCO-CN NoCaps
Metrics: top-1 accuracy mIoU recall@K CIDEr accuracy MME score
Baselines: EVA-02-CLIP-E+ OpenCLIP-G ViT-22B BLIP-2 InstructBLIP Qwen-VL-Chat LLaVA-1.5 ViCLIP
Builds on: CLIP BLIP-2 LLaVA-1.5 ViT-22B LLaMA
Closest papers by meaning
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
- MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (ICLR 2024 · similarity 0.90)
- Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training (CVPR 2025 · similarity 0.89)
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.89)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.89)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.