Research map / Vision-language and multimodal / Multimodal LLMs

MiVE: Multiscale Vision-language features for reference-guided video Editing

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MiVE, which repurposes Qwen3-VL as a multiscale feature extractor (first and last layers) feeding a unified self-attention DiT for reference-guided video editing, initialized from Wan2.1-T2V-14B.

Problem

Reference-guided video editing must apply edits from a reference image while preserving source motion and unedited content; decoupled encoders suffer modality gaps and VLM-based unified encoders using only final-layer features lose fine spatial detail.

Approach

Extract hidden states from both the first and last layers of a VLM (Qwen3-VL), project them to condition tokens, and process them jointly with VAE video latents via self-attention in a DiT rather than cross-attention, with reference-frame latents concatenated for appearance anchoring.

Key results

• On the Complex subset, MiVE scores highest on all six Gemini dimensions (e.g., IA 9.23 vs. 8.87 for Wan-Animate) and ranks first in human preference (3.75 vs. Kling O1 3.61, not significant at p=0.247).
• On the Simple subset, MiVE has human preference 4.18 vs. Kling O1 3.69 (p=0.0087) and leads IA (9.30) and VA (8.83).
• Ablation: first+last VLM layers gives IA 9.23 and TS 8.27 vs. last-layer only IA 9.04 and TS 8.03; self-attention outperforms cross-attention variants on IA and TS.

Surprising findings

• Kling O1 is not significantly different from MiVE on the Complex subset in human preference (p=0.247), and MiVE trails Kling O1 on some VLM dimensions on the Simple subset.
• SSIM and LPIPS show inconsistent trends with VLM and human judgments, so the paper omits them as primary metrics.

Limitations

• Benchmark is not publicly released due to licensing, limiting reproducibility.
• Human study shows moderate inter-rater agreement (Krippendorff alpha 0.506-0.634) and the advantage over Kling O1 on Complex is not significant.
• Inference is slow (about 6.5 minutes per clip) and depends on an external image editor for the reference frame.

Experimental setup

Datasets: OpenVE-3M synthetic human-centric composited data RoseBench VPBench

Benchmarks: MiVE simple subset (30 videos from RoseBench and VPBench) MiVE complex subset (30 portrait videos)

Metrics: IA CC TS PR VA SC human preference (1-5) mean score

Baselines: VACE LucyEdit VideoCof Kling O1 Wan-Animate VideoPainter ICVE Ditto

Builds on: Wan2.1 Z-Image VACE OpenVE-3M flow matching

Closest papers by meaning

• Bernini: Latent Semantic Planning for Video Diffusion (NeurIPS 2026 · similarity 0.87)
• Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.87)
• VicEdit: Learning to Edit Videos from Visual In-Context Examples (NeurIPS 2026 · similarity 0.87)
• VINCIE-NExT: Unlocking Video Editing from Images via In-Context Modeling (NeurIPS 2026 · similarity 0.87)
• Text-Vision Co-Instructed Image Editing (NeurIPS 2026 · similarity 0.86)
• FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model (CVPR 2025 · similarity 0.86)