Research map / Vision-language and multimodal / Multimodal LLMs

UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts CLIP (InternViT) into a continuous unified visual encoder via two-stage reconstruction training with self-distillation, and pairs it with a dual-condition (query + hidden states) MetaQuery-style diffusion head for generation and editing.

Problem

CLIP-based unified tokenizers either degrade CLIP's understanding (quantization or freezing) or reconstruct inconsistently, and query-based MLLM-to-DiT bridges lose reference-image detail needed for editing.

Approach

Two-stage training makes CLIP reconstructive: first train only a pixel decoder on frozen CLIP, then unfreeze CLIP with an MSE distillation loss to the original features (CLIP LR 0.1x). The generator conditions the DiT on both MLLM multimodal hidden states and learnable query embeddings, avoiding the fixed-length query bottleneck for editing.

Key results

• UniLIP-3B reaches 0.90 GenEval and 0.63 WISE, matching or exceeding similarly sized unified models.
• UniLIP-3B reaches 3.94 on ImgEdit, above UniWorld-V1 (3.26) and BAGEL (3.20).
• At 256x256, UniLIP achieves rFID 0.79 at 32x downsampling, versus 1.37 for DualViTok at 16x; at 448x448 it reaches rFID 0.31 vs 3.27 for Emu2.

Surprising findings

• Reconstruction training slightly improves several understanding benchmarks (e.g., MME-P 1492 to 1499, AI2D 69.4 to 70.7) over frozen InternViT.
• Removing self-distillation drops MMBench from 72.6 to 18.4 (direct reconstruction finetuning nearly destroys understanding).

Limitations

• Scale limited to 3B parameters and 448x448 resolution.
• Subpar text rendering and editing performance, as acknowledged by the authors.
• Frozen LLM means understanding and generation are not jointly tuned.

Experimental setup

Datasets: BLIP3-o pretraining data CC12M JourneyDB GPT-Image-Edit-1.5M ShareGPT-4o-Image ImageNet

Benchmarks: ImageNet 50k reconstruction MME-P MMBench MMMU MM-Vet SEED AI2D MMVP GenEval WISE ImgEdit RISE

Metrics: rFID PSNR SSIM accuracy overall score

Baselines: Emu2 VILA-U TokenFlow DualViTok BAGEL UniWorld-V1 BLIP3-o MetaQuery

Builds on: MetaQuery InternVL3 DC-AE SANA BLIP3-o

Closest papers by meaning

• un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.90)
• VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.89)
• UNICOM: Unified Multimodal Modeling via Compressed Continuous Semantic Representations (NeurIPS 2026 · similarity 0.89)
• WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.89)
• Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.89)
• Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions (NeurIPS 2025 · similarity 0.89)