Research map / Vision-language and multimodal / Multimodal LLMs
Reconstructive Visual Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ROSS, which adds a denoising reconstruction loss over frozen VAE latent tokens of the input image to LMM visual outputs, so that a single SigLIP encoder LMM preserves visual detail and reduces hallucination.
Problem
Standard visual instruction tuning supervises only text outputs, leaving visual token outputs unsupervised, which leads to fine-grained visual shortcomings and hallucinations in LMMs.
Approach
Supervise visual outputs by having a small conditional denoiser (conditioned on LMM visual-token outputs) recover clean latent tokens of the input image from a frozen KL-16 VAE, using a per-token diffusion denoising objective instead of raw-pixel regression, as an intrinsic alternative to extrinsic multi-encoder ensembles.
Key results
- ROSS-7B with a single SigLIP encoder reaches 57.1 HallusionBench, 88.3 POPE, 79.1 MMBench-EN and 56.7 MMVP, surpassing Cambrian-1-8B on most listed benchmarks.
- Adding the vision-centric denoising loss to Qwen2-7B with SigLIP raises MMVP from 40.7 to 49.3 and ChartQA from 44.4 to 46.3 over the LLaVA-style baseline.
- On SpatialBench, ROSS improves the average from 57.7 (RGB) to 66.1 (RGB+D), while LLaVA and GPT-4o do not benefit consistently from depth-map inputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reconstructive visual denoising supervision | Qwen2-7B-Instruct LLaVA baseline with CLIP-ViT-L/14@336 · MMVP | accuracy: 29.3 -> 42.0 | Helped |
| replace latent-token regression with denoising (ablation) | ROSS regression using KL-16 · MMVP | accuracy | Helped |
| reconstruct latent tokens rather than raw RGB pixels (ablation) | ROSS regression · hallucination and fine-grained comprehension benchmarks | benchmark scores | Helped |
| use KL-16 continuous tokenizer instead of VQ-16 (ablation) | ROSS denoising · ImageNet 256×256 validation | rFID: 2.19 -> 0.87 | Helped |
| use a GLIDE Softmax denoising schedule (ablation) | ROSS denoising · HallusionBench | average accuracy: 55.0 -> 59.1 | Helped |
| add a self-attention module to the denoiser (ablation) | ROSS denoiser · hallucination and fine-grained comprehension benchmarks | benchmark scores | Helped |
| use reconstructive rather than generative supervision (ablation) | LMM with 102K caption-derived extra data · MMVP | accuracy: 30.0 -> 37.3 | Helped |
| test with RGB + depth instead of RGB only | ROSS on SpatialBench · SpatialBench Average | accuracy: 57.7 -> 66.1 | Helped |
| add reconstructive supervision | Vicuna-7B-v1.5 with CLIP-ViT-L/14@336 · MMVP | accuracy: 28.0 -> 36.0 | Helped |
| add reconstructive supervision | Qwen2-7B-Instruct with SigLIP-ViT-SO400M/14@384 · MMVP | accuracy: 40.7 -> 49.3 | Helped |
| add reconstructive supervision | Qwen2.5-7B with CLIP-ViT-L/14@336 · HallusionBench | average accuracy: 48.7 -> 53.7 | Helped |
| use RGB + depth rather than RGB only (ablation) | LLaVA baseline on SpatialBench · SpatialBench Average | accuracy: 55.0 -> 53.6 | Hurt |
Surprising findings
- Regressing raw RGB pixels provides no meaningful supervision, while latent regression and denoising do; the paper argues pixel regression fails due to spatial redundancy.
- Generative objectives (query-based denoising for image creation) hurt comprehension relative to the baseline, while reconstructive objectives help.
Limitations
- Relies on high-level LMM features being able to reconstruct latent tokens; reconstruction quality beyond the probe is only qualitative.
- High-resolution results require the anyres technique and are reported in the appendix only.
- Training data overlap between Cambrian-737K and SMR-473K is acknowledged but not removed.
Experimental setup
Datasets: LLaVA-558K Cambrian-737K SMR-473K ShareGPT4V ALLaVA SpatialQA-853K ImageNet-1K LLaVA-665K
Benchmarks: POPE HallusionBench MMVP ChartQA MMBench MMBench-CN SEED-Bench MMMU GQA AI2D OCRBench DocVQA
Metrics: accuracy mean attention score T-test p-value Mann-Whitney U test p-value rFID
Baselines: LLaVA-v1.5 LLaVA-v1.6 Cambrian-1 Mini-Gemini DeepSeek-VL GPT-4V-1106 Gemini-1.5 Pro MM-1
Builds on: LLaVA-v1.5 Cambrian-1 latent diffusion models masked autoencoder denoising autoencoder
Closest papers by meaning
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.91)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.88)
- Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.88)
- Diffusion Instruction Tuning (ICML 2025 · similarity 0.88)
- Instruction-Guided Visual Masking (NeurIPS 2024 · similarity 0.87)
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.