Research map / Vision-language and multimodal / Multimodal LLMs

Diffusion Guided Chain-of-Vision for Large Autoregressive Vision Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Diffusion Guided Chain-of-Vision, which inserts diffusion-generated intermediate images (DDIM-inversion latent slerp interpolation) as chain-of-vision steps to supervise pure-vision autoregressive LVMs for in-context visual tasks.

Problem

Pure-vision autoregressive models lack an explicit multi-step output pathway (chain-of-thought analog) for visual in-context learning, limiting performance on complex visual tasks.

Approach

Use DDIM inversion of source and target images with slerp interpolation in the diffusion noise space, then denoise to get visually coherent intermediate images that serve as task-free chain-of-vision supervision, interleaved in the autoregressive visual sentence.

Key results

• On LVM-7B, diffusion guided interpolation raises MS-COCO segmentation IoU from 0.463 (fine-tune without interpolation) to 0.500, and linear interpolation reaches 0.487.
• With LVM-300M, diffusion interpolation improves LoL light enhancement PSNR from 17.12 to 18.97.
• On LVM-7B with 8-length interpolation, image segmentation IoU reaches 0.533 versus 0.475 for the single-pair baseline; diffusion length 4 already exceeds linear length 8.

Surprising findings

• Diffusion interpolation with only 4 intermediate steps surpasses linear interpolation at 8 steps on all metrics, suggesting diffusion-guided steps are more sample-efficient.

Limitations

• Gains are most pronounced at smaller model sizes, shrinking for LVM-7B.
• Evaluated only on LVM models with VQGAN tokenization and fixed 256x256 resolution in the main text.
• Requires per-task fine-tuning for some analyses (Tables 2-3 fine-tuned per task).

Experimental setup

Datasets: MS-COCO COCO-Pose ImageNet-1K LoL Prismer annotations LVM task annotations

Benchmarks: MS-COCO COCO-Pose ImageNet-1K LoL

Metrics: IoU P-ACC LPIPS RMSE delta E A.Rel S.Rel PSNR SSIM

Baselines: LVM Chain-of-Focus (CoF) linear interpolation (RGB) fine-tuning without interpolation

Builds on: Large Vision Models (LVM) DDIM DiffMorpher Chain-of-Focus

Closest papers by meaning

• What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.87)
• Scaling Properties of Diffusion Models For Perceptual Tasks (CVPR 2025 · similarity 0.87)
• Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.87)
• TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.87)
• Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.87)
• Latent Chain-of-Thought for Visual Reasoning (NeurIPS 2025 · similarity 0.87)