Research map / Vision-language and multimodal / Multimodal LLMs
Diffusion Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Lavender, an SFT method that aligns VLM text-to-image attention maps with those from Stable Diffusion via an attention alignment loss, improving VLM benchmark and OOD medical QA performance with small datasets.
Problem
VLMs are data-hungry and their text-vision attention is less precisely aligned than that of diffusion models; standard autoregressive SFT on limited image-text data can overfit and forget, especially OOD.
Approach
Extract per-word cross-attention maps from Stable Diffusion offline and add an MSE attention alignment loss (through a learned Aligner network over parallel attention) to the VLM's next-token loss during SFT, so diffusion models act as a visual attention teacher without extra annotations.
Key results
- Lavender-Llama-3.2-11B with LoRA improves over autoregressive LoRA fine-tuning by up to 30% on 19 of 20 benchmarks in the reported setting.
- On the OOD WorldMedQA-V benchmark, Lavender improves Llama-3.2-11B by 68%.
- Lavender surpasses autoregressive fine-tuning by up to 72% in zero-shot accuracy across seven OpenFlamingo benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add diffusion attention alignment with LoRA fine-tuning | MiniCPM-V-2.5 · 20 multimodal benchmarks | benchmark performance: improves performance on 16 out of 20 tasks by up to 4%; drops to a maximum of -1% | Mixed |
| add diffusion attention alignment with LoRA fine-tuning | Llama-3.2-11B-Vision-Instruct · 20 multimodal benchmarks | benchmark performance: up to 30% on 19 out of 20 benchmarks | Helped |
| add diffusion attention alignment with full fine-tuning | Llama-3.2-11B-Vision-Instruct · 20 multimodal benchmarks | benchmark performance: up to 25% on 17 out of 20 benchmarks | Helped |
| add diffusion attention alignment | Llama-3.2-11B · WorldMedQA-V | accuracy: 68% | Helped |
| use learned aggregation instead of manual aggregation methods (ablation) | Lavender-Llama 3.2-11B · eight benchmarks | exact-match performance | Helped |
| pretrain the Aligner Network before jointly updating model parameters (ablation) | Lavender-Llama 3.2-11B · eight benchmarks | exact-match performance | Helped |
| use LoRA instead of full fine-tuning (ablation) | Lavender-Llama 3.2-11B · majority of benchmarks | benchmark performance | Mixed |
| align all eight cross-attention layers rather than a subset (ablation) | Lavender-Llama 3.2-11B · eight VLM benchmarks | exact-match performance | Helped |
| use exact word match instead of root word match sampling (ablation) | OpenFlamingo with Lavender · Flickr30k attention alignment | attention alignment | Helped |
| use a convolutional Aligner instead of an MLP (ablation) | OpenFlamingo with Lavender · Flickr30k attention alignment | attention alignment | Helped |
| use moderate learning rates rather than a lower learning rate (ablation) | OpenFlamingo with Lavender · Flickr30k attention alignment | attention alignment: 1e-4 instead of 1e-5 | Helped |
| mix OCRVQA data into the training data (ablation) | Lavender · overall performance | performance | Hurt |
Surprising findings
- Fully fine-tuning without aligner pretraining causes catastrophic forgetting, while short aligner-only pretraining prevents it.
- Learned attention aggregation starts as second-worst at epoch 5 but becomes best by epoch 20.
Limitations
- Trained only on up to 0.13M samples, far smaller than SOTA data scales, so results are not comparable to large-scale SOTA models.
- Self-attention-only models (MiniCPM-V-2.5) show modest gains (up to 4%) and the parallel attention mechanism is underexplored for them.
- Uses Stable Diffusion v1.4 at low resolution (32x32 attention maps) and may be inaccurate for infrequent words.
Experimental setup
Datasets: Flickr30k Laion50k RLAIF-V 83k OCRVQA 30k WorldMedQA-V
Benchmarks: AI2D CCBench DocVQA InfoVQA MMBench MME MMMU MMStar OCRBench OCRVQA POPE RealWorldQA
Metrics: zero-shot accuracy relative improvement attention entropy MSE alignment loss CIDEr
Baselines: autoregressive LoRA fine-tuning autoregressive full fine-tuning MiniCPM-V-2.5 LLaVA-1.5-7B LLaVA-NeXt-7B Qwen2-VL-7B GPT-4o
Builds on: Flamingo Stable Diffusion LoRA attention flow Squeeze-and-Excitation
Closest papers by meaning
- Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.88)
- Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer (NeurIPS 2025 · similarity 0.88)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.88)
- Reconstructive Visual Instruction Tuning (ICLR 2025 · similarity 0.88)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.88)
- SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.