Research map / Vision-language and multimodal / Multimodal LLMs
Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Lumen, an LMM (LLaVA-1.5-7B) that decouples vision-centric learning into a task-agnostic heatmap-matching stage (LoRA-tuned LLM plus transformer V-L dense aligner) and a task-specific decoding stage using peak selection, a box decoder and SAM, achieving competitive detection, pose, grounding and referring segmentation while keeping VQA performance.
Problem
Existing LMMs add vision-centric abilities (detection, grounding, pose) by forcing outputs into language-style coordinate sequences, which handles multiple simultaneous objects poorly and overlooks the distinct structure of vision tasks, while degrading general VQA ability.
Approach
Decouple vision-centric learning: a task-agnostic stage produces a shared instruction-to-region matching heatmap from a special [LOC] token and high-resolution CLIP features via a lightweight transformer aligner; a task-specific stage routes this heatmap through peak selection, a learned box decoder and SAM's promptable mask decoder to produce task-specific outputs.
Key results
- Lumen-7B reaches 35.3 AP / 53.2 AP50 on COCO object detection, surpassing Griffon-13B (AP50 37.6) and InstructCV (AP50 48.5) among LMM generalists.
- On RefCOCOg val grounding, Lumen-7B gets AP50 83.6 vs Shikra-7B 82.3 and Griffon-13B 81.5; referring segmentation cIoU 65.1 on RefCOCOg val.
- On VQA, Lumen (7B) scores 64.9 on MMBench and 1426 on MME, comparable to LLaVA-v1.5 (64.3 and 1511).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a transformer-based rather than convolutional V-L dense aligner (ablation) | Lumen · COCO val | AP: 18.4 -> 28.4 | Helped |
| Use pretrained SAM rather than a TransDec. mask decoder (ablation) | Lumen · RefCOCO val | cIoU: 65.8 -> 66.6 | Helped |
| Use pretrained SAM rather than a TransDec. mask decoder (ablation) | Lumen · RefCOCOg val | cIoU: 62.0 -> 61.5 | Hurt |
| Replace LLaVA-v1.0 with LLaVA-v1.0* using the stronger CLIP ViT-L/14-336 vision backbone (ablation) | Lumen · COCO val | AP: 24.0 -> 26.8 | Helped |
| Increase input size from 336×336 to 448×448 (ablation) | Lumen · COCO val | AP: 26.1 -> 28.4 | Helped |
| Increase input size from 448×448 to 896×896 (ablation) | Lumen · COCO val | AP: 28.4 -> 24.5 | Hurt |
| Replace CLIP ViT-L/14-336 with SAM ViT-H/16-1024 (ablation) | Lumen · COCO val | AP: 28.4 -> 14.5 | Hurt |
| Increase K from 20 to 100 (ablation) | Lumen dense prediction · COCO val | AP: 26.3 -> 28.4 | Helped |
| Increase K from 100 to 200 (ablation) | Lumen dense prediction · COCO val | AP: 28.4 -> 28.5 | No clear effect |
| Add visual grounding training data (ablation) | Lumen multitask training · COCO val | AP50: 41.2 -> 45.7 | Helped |
| Add object detection training data (ablation) | Lumen multitask training · RefCOCOg val | AP50: 72.3 -> 75.8 | Helped |
| Remove pose estimation data from the all-task training mixture (ablation) | Lumen multitask training · COCO val | AP50: 45.0 -> 45.7 | Helped |
Surprising findings
- Removing pose estimation data from multi-task training improved grounding and detection (RefCOCOg and COCO AP50), which the authors attribute to data conflict.
- Scaling the CLIP input from 448 to 896 pixels hurt detection (AP 28.4 to 24.5), and a SAM ViT-H encoder performed far worse (AP 14.5).
Limitations
- Detection performance lags task-specific specialists and generalists such as VisionLLM and Pix2Seq-v2, attributed to 448 input size and no DETR-style decoding.
- A single [LOC] token must query the whole image, which slows convergence; the model was trained only briefly due to compute limits.
- Pixel-level mask quality is bounded by the heatmap's dense alignment, and SAM's pretrained encoder did not help when used as the vision backbone.
Experimental setup
Datasets: MSCOCO Objects365 OpenImages RefCOCO RefCOCO+ RefCOCOg COCO keypoints AIC ShareGPT4V
Benchmarks: COCO val COCO keypoints val RefCOCO RefCOCO+ RefCOCOg PASCAL VOC2007 MMBench SEEDBench MME MMMU MathVista
Metrics: AP AP50 AP75 cIoU OKS-based mAP accuracy
Baselines: Shikra-7B MiniGPT-v2-7B Griffon-13B InstructCV VisionLLM Pix2Seq-v2 Uni-Perceiver-v2 mPLUG-2 LISA Qwen-VL-Chat LLaVA-v1.5 InstructBLIP
Builds on: LLaVA-1.5 CLIP ViT-L/14-336 SAM CenterNet-style Gaussian heatmap regression Pix2Seq
Closest papers by meaning
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.89)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
- VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks (NeurIPS 2024 · similarity 0.88)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.88)
- DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs (NeurIPS 2024 · similarity 0.88)
- Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.