Research map / Vision-language and multimodal / Visual question answering
On Scaling Up a Multilingual Vision and Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales the PaLI multilingual vision-language model to PaLI-X (55B) by pairing a 22B ViT with a 32B UL2 encoder-decoder, trained on a mixed objective mixture, reaching SOTA on 20+ V&L benchmarks.
Problem
How far scaling both the vision and language components of a multilingual vision-language model improves performance, and how the training mixture affects fine-tuning versus few-shot trade-offs.
Approach
Scale both visual (ViT-22B, OCR-tuned) and language (32B UL2 variant, 50 layers) components and train on a broad mixture of ~13 pretraining objectives (prefix-completion, masked-token, OCR, detection pseudo-labels, image-token prediction), with an episodic WebLI dataset for few-shot ability, yielding emergent counting and multilingual detection.
Key results
- PaLI-X (55B) reaches 86.1 on VQAv2 test-std and 66.1 on OKVQA, and improves TallyQA complex counting by +18.8 over MoVie (86.0 vs 74.9? reported 86.0 complex vs prior SOTA).
- PaLI-X achieves SOTA on AI2D with a +42.9 point jump when answer choices are given as input.
- On LVIS, the detection-tuned PaLI-X reaches 30.64 AP overall and 31.42 AP on rare classes; zero-shot is about 12 AP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale the vision and language components together from PaLI to PaLI-X | PaLI · VQAv2 test-std | accuracy: 84.3 -> 86.1 | Helped |
| Scale the vision and language components together from PaLI to PaLI-X | PaLI · TallyQA simple test | accuracy: 81.7 -> 86.0 | Helped |
| Scale the vision and language components together from PaLI to PaLI-X | PaLI · TallyQA complex test | accuracy: 70.9 -> 75.6 | Helped |
| Use PaLI-X rather than the previous model | TextCaps · TextCaps | CIDEr: 160.4 -> 163.7 | Helped |
| Use PaLI-X rather than the previous model | TextVQA with OCR pipeline input · TextVQA | accuracy: 73.67 -> 80.78 | Helped |
| Use PaLI-X rather than the previous model | DocVQA with OCR pipeline input · DocVQA | ANLS: 87.8 -> 86.8 | Hurt |
| Use PaLI-X rather than the previous model | AI2D · AI2D | accuracy: 38.5 -> 81.4 | Helped |
| Increase video input from 8 to 16 frames | PaLI-X · ActivityNet Captions | CIDEr: 49.0 -> 54.9 | Helped |
| Increase video input from 8 to 16 frames | PaLI-X · MSR-VTT Captions | CIDEr: 74.6 -> 76.8 | Helped |
| Increase video input from 8 to 16 frames | PaLI-X · MSR-VTT QA | full-string accuracy: 46.9 -> 47.1 | No clear effect |
| Fine-tune PaLI-X on detection data | PaLI-X · LVIS | bounding box mean AP: 12.36 -> 30.64 | Helped |
| Fine-tune PaLI-X on detection data | PaLI-X · LVIS rare | bounding box mean AP: 12.16 -> 31.42 | Helped |
Surprising findings
- Multitask fine-tuning across captioning benchmarks incurs only a 1.4-point average drop (0.2 excluding Screen2Words), matching single-task fine-tuning performance.
- Emergent multilingual object detection and complex counting arise without being explicitly in the training mix.
Limitations
- Fairness analysis shows lower detection performance for Hispanic and Black subgroups in gender prediction, possibly due to underrepresentation in data.
- Few-shot VQAv2 gap versus Flamingo may result from fine-tuning the language backbone, losing some few-shot text-only QA capability.
- Zero-shot LVIS performance (about 12 AP) is far below fine-tuned performance.
Experimental setup
Datasets: WebLI Episodic WebLI CC3M XM3600 CoCo-Captions VTP OWL-ViT pseudo detection labels COCO NoCaps VQAv2 OKVQA TallyQA
Benchmarks: COCO Captions NoCaps VQAv2 OKVQA TallyQA TextCaps TextVQA VizWiz-VQA ST-VQA OCR-VQA DocVQA InfographicVQA
Metrics: CIDEr accuracy AP WUPS precision recall toxicity
Baselines: PaLI Flamingo GIT2 Pix2Struct MoVie BEiT-3 PaLM-E OwL-ViT mPLUG2 VINDLU
Builds on: PaLI UL2 ViT-22B Flamingo pix2struct
Closest papers by meaning
- LRM-LLaVA: Overcoming the Modality Gap of Multilingual Large Language-Vision Model for Low-Resource Languages (AAAI 2025 · similarity 0.89)
- Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.89)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.89)
- Semantic and Expressive Variations in Image Captions Across Languages (CVPR 2025 · similarity 0.88)
- ROD-MLLM: Towards More Reliable Object Detection in Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.