Research map / Vision-language and multimodal / Multimodal LLMs

Bernini: Latent Semantic Planning for Video Diffusion

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Bernini unifies an MLLM planner (predicting target ViT embeddings via masked generative modeling) with a Wan2.2-based DiT renderer for text-to-video, reference-to-video, and video editing, and introduces Bernini-Bench.

Problem

Unifying multimodal understanding with high-fidelity diffusion-based video generation and editing, where a compact semantic interface must carry intent from an MLLM to a video diffusion renderer.

Approach

Use the MLLM's own ViT embedding space as the semantic bridge: the MLLM plans target ViT embeddings via masked generative modeling with a lightweight flow-matching decoder, and a DiT renderer conditions on these semantic features plus VAE latents of source inputs; SA-3D RoPE disambiguates multiple visual segments.

Key results

• On Bernini-Bench V2V, overall score 3.49 vs 3.30 for Wan2.7 and 3.05 for Kling O3 (GPT-5.4 judge).
• On OpenS2V-Eval, Total score 62.94 (reported in text) vs 60.42 for RefAlign-14B; FaceSim 78.20 vs 57.20 for Kling O3.
• On VBench, Bernini reaches Total 84.46 (text reports 84.64), essentially matching the base Wan2.2-A14B at 84.79.

Surprising findings

• Bernini's text-to-video VBench total (84.46 in table, 84.64 in text) is essentially on par with its base Wan2.2-A14B, so unification does not improve T2V quality.
• The paper notes MLLM judges cannot reliably assess small distortions or artifacts, so generation-quality scores are reference-only.

Limitations

• Generation quality is judged by MLLM scorers that the paper admits are unreliable for small distortions and artifacts.
• Bernini-Bench is small (300 cases) and evaluated mainly by a single MLLM judge plus human SBS comparisons.
• Text-to-video quality is not improved over the base Wan2.2-A14B.

Experimental setup

Datasets: OpenS2V-Top200K UniREdit-100K Pico-Banana-400K Senorita-Controllable SynCamVideo-Dataset CamCloneMaster Inhouse T2V Inhouse T2I Inhouse I2I OmniCorpus Bernini-Bench

Benchmarks: Bernini-Bench OpenVE-Bench EditVerse FiVE VBench OpenS2V-Eval

Metrics: overall score instruction following video consistency reference image consistency generation quality BT score win rate PSNR LPIPS SSIM CLIP score

Baselines: Kling O3 Wan2.7 UniVideo VINO VACE-14B EditVerse Lucy-Edit Wan2.2-A14B RefAlign-14B

Builds on: Qwen2.5-VL Wan2.2 MAR SD3 classifier-free guidance distillation

Closest papers by meaning

• Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents (NeurIPS 2025 · similarity 0.88)
• Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
• I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (ICML 2025 · similarity 0.88)
• Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · similarity 0.88)
• DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.88)
• Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.88)