Research map / Vision-language and multimodal / Human and embodied modeling
Sequential Modeling Enables Scalable Learning for Large Vision Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a 300M-3B autoregressive transformer (LLaMA-style) on 420B tokens of purely visual "visual sentences" (1.64B images/frames, UVDv1) using VQGAN tokens and next-token prediction, and shows scaling and visual-prompting generalization without language data.
Problem
Whether a large vision model can scale and be prompted for many vision tasks learned purely from pixels and sequential modeling, without any linguistic data.
Approach
Represent diverse visual data (raw images, videos, 3D renderings, and annotations such as segmentation, depth, and pose rendered as images) as unified sequences of images ("visual sentences"), then train a causal transformer with next-token cross-entropy over VQGAN tokens so that tasks are specified by visual prompts at test time.
Key results
- Training loss and downstream perplexity decrease consistently as model size grows from 300M to 3B on 420B tokens.
- LVM reaches 81.2 PCK on Pascal 3D+ keypoint detection without training on that dataset, above StarMap's 78.6.
- On the Visual Prompting comparison, the sequential LVM beats MAE-VQGAN (CVF) on foreground segmentation and single object detection, e.g. 48.94 vs 27.83 mIOU on Split 0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sequential LVM visual prompting instead of the compared prompting baselines | LVM · Foreground Segmentation and Single Object Detection | mIOU | Helped |
| Use sequential LVM visual prompting instead of the best compared baseline | LVM · Colorization | MSE: 0.43 -> 0.51 | Hurt |
| Use sequential LVM visual prompting instead of the best compared baseline | LVM · Colorization | LPIPS: 0.40 -> 0.46 | Hurt |
| Apply LVM without training on Pascal 3D+ | LVM · Pascal 3D+ | PCK: 78.6 -> 81.2 | Helped |
| Use LVM rather than chance-level selection | LVM · Raven's Progressive Matrices | accuracy: 20% -> 30% | Helped |
| Increase model size | LVM · ImageNet validation set, four prompted tasks | perplexity | Helped |
| Add single-image, video, and annotation data components (ablation) | LVM trained on UVDv1 · ImageNet validation set, four prompted tasks | perplexity | Helped |
Surprising findings
- Raven matrices accuracy is above chance but far from solved, and the authors describe the 'Sparks of AGI' reasoning as only hinted at.
Limitations
- Visual prompts for tasks are often under-constrained, leading to task confusion.
- Tokenizer artifacts and poor handling of digits and out-of-distribution synthetic data.
- Results lag behind bespoke task-specific models; lack of high-quality video training data.
Experimental setup
Datasets: LAION-5B (filtered 1.49B subset) Objaverse ImageNet COCO Kinetics-700 VIPSeg AVA UVDv1
Benchmarks: ImageNet validation perplexity Kinetics-700 val Pascal 3D+ keypoint detection Raven's Progressive Matrices Visual Prompting segmentation/detection/colorization
Metrics: perplexity PCK mIOU MSE LPIPS accuracy
Baselines: Visual Prompting (MAE-VQGAN) MAE StackedHourglass MSS-Net StarMap
Builds on: LLaMA VQGAN Visual Prompting (Bar et al.) MaskGIT
Closest papers by meaning
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.92)
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.88)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.88)
- Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (CVPR 2024 · similarity 0.88)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.88)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.