Research map / Vision-language and multimodal / Multimodal LLMs
Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Chain-of-Sight, a multi-scale visual resampler bridge for MLLMs that uses few visual tokens during pre-training and scales tokens up to 16x at fine-tuning, cutting pre-training wall-clock time ~73% with comparable performance.
Problem
Pre-training MLLMs is slow because the many visual tokens fed to the LLM dominate compute, while reducing visual tokens during pre-training typically hurts performance.
Approach
A chain of visual resamplers operating on windowed features at multiple scales (global-to-local token order) produces multi-scale visual tokens; a compound token scaling strategy (resolution and window-size scaling) lets the token count grow up to 16x after pre-training, so pre-training can use few tokens.
Key results
- Pre-training with 32 visual tokens takes 0.27x wall-clock time (about 3.7x acceleration) and matches or beats the standard 336-token resampler after fine-tuning with 528 tokens.
- CoS-7B reaches 82.9 VQAv2, 64.0 GQA, 93.9 ScienceQA-IMG and 72.8 MMBench with 80 pre-training tokens and 532M trainable parameters.
- On RefCOCO-family grounding, CoS-7B (LoRA) averages 87.68 across the 8 splits, above the compared 7B models listed in the table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reduce Chain-of-Sight pre-training tokens from 336 to 32 (ablation) | Chain-of-Sight · 12 benchmarks | average performance | No clear effect |
| Reduce pre-training tokens from 336 to 32 (ablation) | Chain-of-Sight · pre-training | wall-clock time: 0.27×; 73% | Helped |
| Increase fine-tuning tokens from 80 to 336 (ablation) | Chain-of-Sight · RefCOCO / RefCOCO+ / RefCOCOg | average accuracy: 83.40 -> 86.19 | Helped |
| Increase fine-tuning tokens from 80 to 1296 (ablation) | Chain-of-Sight · VQA | VQA performance: 58.4 -> 59.4 | Helped |
| Scale up global tokens only (ablation) | Chain-of-Sight · Captioning / VQA / Text | average performance: 111.1 / 66.2 / 84.5 -> 111.3 / 66.0 / 84.8 | No clear effect |
| Scale up local tokens only (ablation) | Chain-of-Sight · Captioning / VQA / Text | average performance: 111.1 / 66.2 / 84.5 -> 112.4 / 67.0 / 86.5 | Helped |
| Scale up both global and local tokens (ablation) | Chain-of-Sight · Captioning / VQA / Text | average performance: 112.4 / 67.0 / 86.5 -> 112.7 / 66.9 / 86.4 | Helped |
| Use Chain-of-Sight instead of a standard resampler with 80 pre-training and 80 fine-tuning tokens | Vision-language bridge · Captioning | CIDEr: 138.8 -> 140.7 | Helped |
| Use Chain-of-Sight instead of a standard resampler with 80 pre-training and 80 fine-tuning tokens | Vision-language bridge · TextVQA | VQA Score: 59.0 -> 60.4 | Helped |
| Use Chain-of-Sight instead of a standard resampler with 80 pre-training and 80 fine-tuning tokens | Vision-language bridge · RefCOCO / RefCOCO+ / RefCOCOg | average accuracy: 79.58 -> 83.40 | Helped |
| Reduce standard-resampler pre-training tokens from 336 to 80 while keeping 80 fine-tuning tokens (ablation) | Standard visual resampler · 12 benchmarks | average performance: 86.8 -> 84.4 | Hurt |
| Increase fine-tuning tokens from 80 to 336 (ablation) | Standard visual resampler · RefCOCO / RefCOCO+ / RefCOCOg | average accuracy: 79.58 -> 84.76 | Helped |
Surprising findings
- Scaling fine-tuning visual tokens matters more than pre-training tokens for vision-language benchmarks, per the authors.
- Scaling only global tokens has negligible effect on average performance, while scaling local tokens gives the main gain.
Limitations
- Uses LoRA on the language model rather than full fine-tuning, which the authors say limits generality.
- No instruction-tuning stage, so MME and VizWiz results are weak.
- Acceleration percentages are measured on a 65M-pair pre-training set; gains at larger scale are expected but not shown.
Experimental setup
Datasets: COYO CC3M CC12M COCO Visual Genome captions SBU VQAv2 GQA OK-VQA AOK-VQA ScienceQA TextVQA
Benchmarks: NoCaps Flickr30k COCO Caption VQAv2 GQA OK-VQA ScienceQA-IMG TextVQA TextCaps SEED-Bench (image) MMBench POPE
Metrics: CIDEr VQA score accuracy F1 wall-clock training time
Baselines: Linear projection (LLaVA-style) Standard visual resampler (Qwen-VL-style) LLaVA-1.5 Qwen-VL VILA Honeybee Mini-Gemini InstructBLIP Shikra Ferret MiniGPT-v2 CogVLM
Builds on: Perceiver Flamingo-style visual resampler LLaVA LoRA
Closest papers by meaning
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.91)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.90)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.90)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.90)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.