AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

PaLI: A Jointly-Scaled Multilingual Language-Image Model

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver and 17 more

ICLR 2023 · Vision-language and multimodal › Human and embodied modeling · approach: Vision-language pretraining · 193 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Scale vision encoder from ViT-G to ViT-ePaLI · COCO Captions and OKVQAperformance per 1B added parameters: +2.2 per 1B more ViT params; +0.4 per 1B more mT5 paramsImproved
Remove split-captioning objective (ablation)PaLI-3B · COCO Captions Karpathy-testCIDEr: 141.4 -> 140.4 (-1.0)Hurt
Remove split-captioning objective (ablation)PaLI-3B · XM-3600 ENCIDEr: 93.8 -> 87.5 (-6.3)Hurt
Remove CC3M-35L captioning objective (ablation)PaLI-3B · XM-3600CIDEr: 93.8 -> 94.9 (+1.1) EN; 42.5 -> 39.9 (-2.6) 6LMixed
Remove OCR objective (ablation)PaLI-3B · TextVQAscore: 41.6 -> 39.9 (-1.7)Hurt
Remove OCR objective (ablation)PaLI-3B · COCO CaptionsCIDEr: 141.4 -> 142.3 (+0.9)Improved
Remove VQA objective (ablation)PaLI-3B · TextVQAscore: 41.6 -> 40.0 (-1.6)Hurt
Remove VQG objective (ablation)PaLI-3B · VQG zero-shotscore: 96.7 -> 17.9 (-78.8)Hurt
Remove object-aware QA and object detection objectives (ablation)PaLI-3B · XM-3600 ENCIDEr: 93.8 -> 90.9 (-2.9)Hurt
Initialize from scratch instead of pretrained unimodal checkpoints (ablation)PaLI-3B · COCO Captions Karpathy-testCIDEr: 141.4 -> 72.8Hurt
Freeze ViT during pretraining instead of fine-tuning it (ablation)PaLI-3B · COCO Captions Karpathy-testCIDEr: 138.8 -> 139.3Improved
Include multilingual WebLI data rather than only WebLI English (ablation)1.5B PaLI · XM-3600 6LCIDEr: 8.2 -> 30.0Improved

Later papers that compared with this method

9 later accepted papers report PaLI-17B as a baseline in their results tables; 4 of them report a better number on most of the results they share with it.

1 more not listed.

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy