Research map / Vision-language and multimodal / Multimodal LLMs
Data-efficient Large Vision Models through Sequential Autoregression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies data-efficient autoregressive large vision models (LVMs) by using data augmentation to balance multi-task visual data and knowledge distillation from a LLaMA-1B teacher to compact LLaMA-300M/80M students.
Problem
Autoregressive large vision models (LVMs) rely on billions-parameter models and hundreds of billions of visual tokens, and multi-task training on imbalanced task datasets hurts performance on low-data tasks.
Approach
Apply random crop/flip augmentation to small task datasets to balance multi-task data (outperforming re-sampling), and use knowledge distillation from a larger LLaMA-1B LVM to train compact LLaMA-300M and LLaMA-80M models on limited data.
Key results
- Augmentation-balanced multi-task training gives perplexity 107.77 on segmentation vs 151.66 with re-sampling, and 140.39 vs collapsed result on pose estimation.
- KD from LLaMA-1B to LLaMA-300M reduces segmentation validation loss from 4.64 to 4.59 in the single-task setting.
- LLaMA-80M with KD reaches 83.04% ImageNet top-1 accuracy with a patch embedding and linear probe setup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Balance task data with augmentation instead of leaving the mixture unbalanced (ablation) | LLaMA-300M · Pose estimation (COCO-Pose) | loss: 4.95 -> 4.94 | Helped |
| Balance task data with augmentation instead of leaving the mixture unbalanced (ablation) | LLaMA-300M · Image deraining (Rain13K) | loss: 5.69 -> 5.64 | Helped |
| Balance task data by repeating minority-task samples (ablation) | LLaMA-300M · Pose estimation (COCO-Pose) | loss: 4.95 -> 16.11 | Hurt |
| Balance task data by repeating minority-task samples (ablation) | LLaMA-300M · Image deraining (Rain13K) | loss: 5.69 -> 21.73 | Hurt |
| Distill the student from LLaMA-1B (ablation) | LLaMA-300M · Image segmentation (10% SA-1B) | loss: 4.64 -> 4.59 | Helped |
| Distill the student from LLaMA-1B (ablation) | LLaMA-300M · Pose estimation (COCO-Pose) | loss: 4.97 -> 4.91 | Helped |
| Distill the student from LLaMA-1B (ablation) | LLaMA-300M · Image deraining (Rain13K) | loss: 5.63 -> 5.62 | Helped |
| Distill the student from LLaMA-1B (ablation) | LLaMA-80M · Image segmentation (10% SA-1B) | perplexity: 156.19 -> 147.78 | Helped |
| Distill the student from LLaMA-1B (ablation) | LLaMA-80M · Pose estimation (COCO-Pose) | perplexity: 164.57 -> 159.78 | Helped |
| Distill the student from LLaMA-1B (ablation) | LLaMA-80M · Image deraining (Rain13K) | perplexity: 264.56 -> 256.65 | Helped |
| Increase SA-1B training data from 1% to 10% (ablation) | LLaMA-300M · SA-1B validation subset | loss: decreases by 0.19 | Helped |
| Increase SA-1B training data from 1% to 10% (ablation) | LLaMA-300M · SA-1B validation subset | perplexity: decreases by 22.4 | Helped |
Surprising findings
- Sequential multi-task training without shuffling causes catastrophic forgetting, with the model attending only to the last-trained task.
- Re-sampling to balance tasks performs worse than leaving data unbalanced, even on the minority tasks.
Limitations
- Outputs are images, making quantifiable results hard to derive for complex tasks.
- Foreground segmentation uses a VQGAN encoder trained on LAION rather than a task-tuned one, limiting mIoU.
- Experiments are on small models and limited data scales, with ImageNet accuracy below MIM methods.
Experimental setup
Datasets: SA-1B COCO-Pose Rain13K HDvila-100m LAION-400M MPII Pascal-5i ImageNet
Benchmarks: Pascal-5i ImageNet SA-1B held-out COCO-Pose Rain13K test MPII
Metrics: validation loss accuracy perplexity mIoU top-1 accuracy
Baselines: LVM-3B BEiT VQGAN MAE-VQGAN re-sampling
Builds on: LVM VQGAN Hinton knowledge distillation InternLM
Closest papers by meaning
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.92)
- Sequential Modeling Enables Scalable Learning for Large Vision Models (CVPR 2024 · similarity 0.92)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.89)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- Multimodal Autoregressive Pre-training of Large Vision Encoders (CVPR 2025 · similarity 0.89)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.