Research map / Continual learning / Continual fine-tuning of LLMs
Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adapt-∞, a dynamic multi-way data selection method for lifelong multimodal instruction tuning that clusters gradient vectors into pseudo-skills, picks an entropy-best scorer per cluster (including a new Image Grounding score), and permanently prunes redundant samples.
Problem
Continually instruction-tuning multimodal LLMs on a growing pool of redundant, multi-task visual instruction datasets causes catastrophic forgetting and is computationally costly; static score-based data selection performs poorly on evolving multi-task distributions.
Approach
Cluster samples into pseudo-skill clusters using gradient vectors, select per cluster the importance scorer with highest score entropy from a pool of experts (perplexity, image grounding, EL2N, entropy), sample a balanced subset with CCS, and permanently prune redundant samples in hidden-state space at each timestep.
Key results
- Adapt-∞ at 25k samples reaches 102.3% relative gain and 0.9% forgetting, versus 68.0% relative gain and 26.0% forgetting for sequential training.
- Adapt-∞ at 100k samples reaches 109.7% relative gain, 0.4% forgetting and 52.5 average accuracy.
- Random 25k selection gets 95.3% relative gain and 2.1% forgetting; the Image Grounding score alone gets 92.3% relative gain and 5.6% forgetting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Adapt-$\infty$ multi-way data selection instead of random pruning | LLaVA-1.5-7B lifelong multimodal instruction tuning · Lifelong multimodal instruction tuning | Avg. Acc.: 47.2 -> 50.5 | Helped |
| Use Adapt-$\infty$ multi-way data selection instead of sequential training | LLaVA-1.5-7B lifelong multimodal instruction tuning · Lifelong multimodal instruction tuning | Forgetting Rate %: 26.0 -> 0.9 | Helped |
| Increase Adapt-$\infty$ training data from 25k to 50k | Adapt-$\infty$ · Lifelong multimodal instruction tuning | Relative Gain %: 102.3 -> 107.2 | Helped |
| Add permanent semantic deduplication | Adapt-$\infty$ · Lifelong multimodal instruction tuning | Avg. Acc.: 50.5 -> 49.6 | Hurt |
| Use a single image grounding score instead of multi-way pruning (ablation) | Adapt-$\infty$ · Lifelong multimodal instruction tuning | Avg. Acc.: 50.5 -> 48.8 | Hurt |
| Use EL2N as the only within-cluster pruner instead of multi-way pruning (ablation) | Adapt-$\infty$ · Lifelong multimodal instruction tuning | Avg. Acc.: 50.5 -> 49.1 | Hurt |
| Use middle-layer gradients for clustering instead of first-layer gradients (ablation) | Adapt-$\infty$ · Lifelong multimodal instruction tuning | Avg. Acc.: 47.9 -> 50.5 | Helped |
| Use middle-layer gradients for clustering instead of last-layer gradients (ablation) | Adapt-$\infty$ · Lifelong multimodal instruction tuning | Avg. Acc.: 50.1 -> 50.5 | Helped |
| Use middle-layer gradients rather than gradients from all layers for clustering (ablation) | Adapt-$\infty$ · Lifelong multimodal instruction tuning | Avg. Acc.: 49.1 -> 50.5 | Helped |
| Use middle-layer gradients instead of hidden-layer outputs for clustering (ablation) | Adapt-$\infty$ · Lifelong multimodal instruction tuning | Avg. Acc.: 47.4 -> 50.5 | Helped |
| Allocate cluster budgets uniformly instead of by density (ablation) | Adapt-$\infty$ · Lifelong multimodal instruction tuning | Avg. Acc.: 49.5 -> 50.5 | Helped |
| Replace full gradients with zero-order gradients (ablation) | Adapt-$\infty$ · Lifelong multimodal instruction tuning | Avg. Acc.: 50.5 -> 48.5 | Hurt |
Surprising findings
- Random subset selection from the combined past and new data beats most score-based selectors (EL2N, entropy, perplexity) and embedding-based pruning baselines on relative gain.
- Language-only (MMLU) skill recovers during sequential training even though no training dataset contains MMLU-like samples.
Limitations
- Scoring-based and gradient-based selection is expensive: data scoring takes 92 hours on 8 A100 GPUs for Adapt-∞, comparable to the full training time saved.
- Multilingual skills remain hardest to retain over time.
- Visual multi-turn chat is evaluated only qualitatively with a single example.
Experimental setup
Datasets: LLaVA-665K M3IT MiniGPT4 MANTIS LAMM VisionFLAN Natural Instructions Alpaca-CoT xP3 UltraChat Firefly
Benchmarks: GQA LLaVA-Bench ScienceQA OK-VQA A-OKVQA COCO-CN Flickr8K-CN RefCOCO RefCOCO+ RefCOCOg MMMU MMLU
Metrics: accuracy relative gain forgetting rate BLEU IOU
Baselines: Sequential training Multi-task training Random experience replay EL2N Entropy Perplexity SemDeDup COINCIDE
Builds on: LESS COINCIDE SemDeDup CCS coverage-based coreset selection LoRA
Closest papers by meaning
- Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning (ICML 2025 · similarity 0.91)
- Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning (NeurIPS 2026 · similarity 0.91)
- CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization (NeurIPS 2025 · similarity 0.90)
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.90)
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.90)
- TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.