Research map / Vision-language and multimodal / Multimodal LLMs
Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Vitron is a unified pixel-level vision LLM (Vicuna-7B backbone) that understands, segments, generates and edits both images and videos by routing LLM outputs to backend specialists via hybrid text-plus-embedding instructions and an adversarial cross-task synergy module.
Problem
Existing vision LLMs treat images and videos separately, cover few vision functions (mostly understanding), and lose precision when passing LLM decisions to downstream decoders.
Approach
Combine discrete textual invocation instructions with continuous signal embeddings passed to backend specialists (GLIGEN, SEEM, ZeroScope, I2VGen-XL, StableVideo), plus pixel-level spatiotemporal grounding tuning and an adversarial task-discriminator that makes shared fine-grained features task-invariant.
Key results
- Vitron reaches 75.5 cIoU on RefCOCO val, slightly above NExT-Chat's 74.7, and 84.2 J&F on DAVIS 17 test-dev, above ISVOS's 82.8.
- On MSR-VTT text-to-video, Vitron reports FID 10.11 and CLIPSIM 0.3682, versus NExT-GPT's 13.04 and 0.3085.
- Ablating the synergy module lowers referring-segmentation cIoU by about 1.5 to 2.4 points across RefCOCO splits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-task synergy learning (ablation) | Vitron · RefCOCO referring image segmentation | cIoU: -2.4 | Helped |
| add cross-task synergy learning (ablation) | Vitron · VidSTG video spatial grounding | mIoU: -4.3 | Helped |
| add cross-task synergy learning (ablation) | Vitron · COCO-caption text-to-image generation | FID: +4.4 | Helped |
| add cross-task synergy learning (ablation) | Vitron · MSR-VTT text-to-video generation | FID: +3.17 | Helped |
| add cross-task synergy learning (ablation) | Vitron · MSR-VTT text-to-video generation | CLIPSIM: -0.5672 | Helped |
| add cross-task synergy learning (ablation) | Vitron · MagicBrush image editing | CLIP_dir: -0.012 | Helped |
| combine textual instructions with continuous signal embeddings | Vitron · six vision tasks | task performance | Helped |
| use text instructions rather than continuous signal embeddings | Vitron · backend module execution | successful execution rate | Helped |
| use continuous signal embeddings rather than text instructions | Vitron · six vision tasks | task performance | Helped |
| remove fine-grained visual grounding learning strategies (ablation) | Vitron · downstream vision tasks | performance | Hurt |
Surprising findings
- Text instructions help backend modules execute successfully, while soft embeddings help task-specific quality, suggesting the two channels play complementary roles.
Limitations
- Video editing is evaluated only by a manual study with five raters, with no standardized benchmark.
- Backend modules are off-the-shelf specialists, so the system is a composition rather than a single end-to-end model.
Experimental setup
Datasets: CC3M WebVid RefCOCO RefCOCO+ RefCOCOg gRefCOCO MagicBrush COCO2017 LAION-400M Flickr30K Entities LaSOT GOT10K
Benchmarks: RefCOCO RefCOCO+ RefCOCOg VidSTG HC-STVG DAVIS 17 OKVQA GQA ActivityNet-QA MSR-VTT COCO UCF101
Metrics: cIoU mIoU J&F METEOR CIDEr accuracy FID CLIPSIM FVD IS
Baselines: NExT-Chat PG-Video-LLaVA NExT-GPT GLaMM Osprey MiniGPT-v2 Video-LLaVA DreamLLM
Builds on: GLIGEN SEEM Vicuna LoRA NExT-GPT
Closest papers by meaning
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.88)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.88)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.88)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.88)
- Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion (CVPR 2025 · similarity 0.88)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.