Research map / Vision-language and multimodal / Multimodal LLMs

UNICOM: Unified Multimodal Modeling via Compressed Continuous Semantic Representations

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UniCom, a unified understanding-and-generation model that compresses SigLIP2 continuous features along the channel dimension with an attention-based compressor and generates them with a Transfusion-style flow-matching pathway, avoiding VAE latents.

Problem

Unified multimodal models rely on discrete tokenizers that lose fine-grained semantics, while modeling high-dimensional continuous ViT features (CLIP/SigLIP) for generation is unstable and slow to converge.

Approach

Compress continuous SigLIP2 features along the channel dimension (not the token sequence) with a lightweight multi-head self-attention compressor, jointly train it with a diffusion decoder to form a compact semantic latent space, and then generate those latents with a Transfusion-style unified transformer rather than a query-based MLLM pathway.

Key results

• Channel-compressed d64 tokenizer reaches rFID 0.42 and PSNR 22.28 on ImageNet, close to uncompressed d1152 (rFID 0.38) and far from token-reduced variants.
• UniCom reports GenEval 0.87 overall and WISE 0.58 overall, the latter the best among compared models in the paper's table.
• UniCom achieves KRIS-Bench overall 70.11 and ImgEdit overall 4.22 (ImgEdit ranks behind TUNA at 4.31), claiming best open-source performance on KRIS-Bench and WorldEdit (4.12).

Surprising findings

• Compressing channels to d=64 converges about 5x faster than using uncompressed d=1152 features in generation, the opposite of the intuition that full-dimensional features are better.
• Reducing the token count to 256 hurts reconstruction and later-stage generation more than channel compression does.

Limitations

• Information loss from compression may hurt very fine-grained details in complex images.
• Training requires substantial compute and large-scale data; training data is internal and not released.
• Evaluation of WorldEdit-Test relies on LLM/human-style judging whose details are unclear; some reported numbers inconsistent across tables.

Experimental setup

Datasets: ImageNet ImageNet50k LLaVA-Pretrain-557k Cambrian-737k internal image-text data

Benchmarks: ImageNet validation GenEval DPG-Bench WISE ImgEdit-Bench GEdit-Bench KRIS-Bench WorldEdit GQA RealWorldQA SEED MMMU

Metrics: rFID PSNR SSIM GenEval overall DPG overall WISE overall ImgEdit overall GEdit G-SC/G-PQ/G-Overall KRIS overall WorldEdit overall

Baselines: FLUX.1 [Dev] BAGEL OmniGen2 Show-o2 Janus-Pro MetaQuery-XL BLIP3-o TUNA Qwen-Image FLUX.1 Kontext [Pro] SD-VAE VA-VAE

Builds on: Transfusion MetaQuery UniTok VUGEN FLUX.1-dev

Closest papers by meaning

• UniCompress: Token Compression for Unified Vision-Language Understanding and Generation (CVPR 2026 · similarity 0.91)
• UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing (ICLR 2026 · similarity 0.89)
• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.89)
• MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.89)
• G$^2$TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models (NeurIPS 2026 · similarity 0.88)
• EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling (CVPR 2026 · similarity 0.88)