Research map / Vision-language and multimodal / Multimodal LLMs
FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FOCUS, a unified LVLM (Qwen2.5-3B based) with dual-branch visual encoder, MoVQGAN-style dual-codebook tokenizer, Mask2Former-style segmentation decoder, and SDXL-initialized diffusion decoder conditioned on predicted masks, trained with a four-stage progressive pipeline for understanding, referring segmentation, and mask-guided editing.
Problem
Existing approaches either perform isolated segmentation or use masks only as conditions for editing, relying on disjointed multi-model pipelines; unified models lack fine-grained object-level, region-controllable generation.
Approach
An end-to-end LVLM that jointly optimizes segmentation masks and uses them as spatial condition prompts for a diffusion decoder, with a dual-branch (global semantic + high-resolution ConvNeXt) visual encoder and dual-branch generative tokenizer, trained progressively from low to high resolution.
Key results
- On RefCOCO val, FOCUS reaches 84.1 mIoU, comparable to HyperSeg (84.8); on RefCOCOg val it reaches 79.3 vs HyperSeg 79.4 (per table).
- On Emu Edit, FOCUS achieves CLIP-T 0.278 and DINO 0.831, the best among compared methods, with CLIP-I 0.876 second to ILLUME's 0.879.
- On the 3B-scale understanding benchmarks, FOCUS scores 88.0 on POPE and 81.5 on MMBench, slightly above ILLUME+ (3B) on those benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FOCUS instead of ILLUME+ | Qwen2.5-3B multimodal understanding · MM-Vet | score: 40.3 -> 50.2 | Helped |
| Use FOCUS instead of ILLUME+ | Qwen2.5-3B multimodal understanding · MME-P | score: 1414.0 -> 1570.3 | Helped |
| Use FOCUS instead of ILLUME+ | Qwen2.5-3B image generation · MJHQ-30K | FID: 6.00 -> 6.05 | Hurt |
| Use FOCUS instead of ILLUME+ | Qwen2.5-3B image generation · GenAI-bench Basic | score: 0.72 -> 0.83 | Helped |
| Use FOCUS instead of ILLUME+ | Qwen2.5-3B image generation · GenEval Overall | score: 0.72 -> 0.75 | Helped |
| Use FOCUS instead of ILLUME+ | Qwen2.5-3B image generation · GenEval Two Obj. | score: 0.99 -> 0.98 | Hurt |
| Use explicit spatial masks rather than mask-token embeddings as diffusion conditioning | FOCUS diffusion decoder · controllable image editing | localization and editing faithfulness | Helped |
| Use FOCUS instead of HyperSeg | RefCOCO referring segmentation · RefCOCO testA | mIoU: 85.7 -> 86.3 | Helped |
Limitations
- Compute hardware details are not stated beyond training durations (about 3 days for tokenizer and diffusion, 13 days for the 3B MLLM).
- Several table comparisons are mixed; FOCUS is not best on all metrics (e.g., MJHQ-30K, Emu Edit CLIP-I and RefCOCO val).
Experimental setup
Datasets: COYO EMOVA LAION-2B LLaVA-150K UltraEdit AnyEdit SEED-Edit RefCOCO RefClef Paco-LVIS PartImageNet DAVIS-2017
Benchmarks: POPE MMBench SEED MME-P MM-Vet MMMU AI2D VQA-text ChartQA DocVQA InfoVQA OCRBench
Metrics: accuracy FID GenEval overall CLIP-I CLIP-T CLIP-DIR DINO mIoU gIoU cIoU
Baselines: LISA-7B PSALM HyperSeg ILLUME+ Janus-Pro-7B Emu3 VILA-U Show-o Emu Edit OmniGen
Builds on: MoVQGAN SDXL Mask2Former SimVQ Qwen2.5
Closest papers by meaning
- Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation (NeurIPS 2026 · similarity 0.89)
- FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model (CVPR 2025 · similarity 0.88)
- Click2Mask: Local Editing with Dynamic Mask Generation (AAAI 2025 · similarity 0.88)
- ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.88)
- Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing (NeurIPS 2024 · similarity 0.88)
- UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.