Research map / Vision-language and multimodal / Human and embodied modeling
Towards More Unified In-context Visual Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a unified in-context learning framework for visual understanding that quantizes text and images into a shared token space and trains a GPT-2-style sparse MoE transformer from scratch to produce image or text outputs on class-aware segmentation and captioning.
Problem
Existing visual in-context learning frameworks are confined to a single output modality (image-only for SegGPT-style inpainting, text-only for Flamingo-style models), so they cannot produce content across multiple modalities in one unified model.
Approach
Quantize images with a VQGAN tokenizer and text with GPT-2 BPE, embed both into a unified space as interleaved in-context sequences, and train a decoder-only transformer with attribute-routing sparse MoE layers using next-token prediction, so the same model can output either image tokens or text tokens.
Key results
- On CA-ICL segmentation at 256 resolution, the model reaches 58.04 MIoU versus 51.12 for SegGPT evaluated at the same resolution.
- On CA-ICL captioning, the model reaches 86.9 CIDEr and 5.3 BLEU4, beating OpenFlamingo (61.3 CIDEr, 4.6 BLEU4) with about 10x fewer parameters.
- Adding AG MoE with multi-task learning raises segmentation MIoU from 21.74 (all-tasks baseline) to 49.91 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SegGPT at 256 resolution instead of the proposed method | CA-ICL segmentation · CA-ICL segmentation | MIoU: 51.12 -> 58.04 | Hurt |
| Include small objects and use large object scale (ablation) | CA-ICL segmentation · CA-ICL segmentation | MIoU: 21.74 -> 45.04 | Helped |
| Include small objects while using small object scale (ablation) | CA-ICL segmentation · CA-ICL segmentation | MIoU: 31.82 -> 33.54 | Helped |
| Use large rather than small object scale with small objects excluded (ablation) | CA-ICL segmentation · CA-ICL segmentation | MIoU: 31.82 -> 42.87 | Helped |
| Use bbox information encoded as an image (ablation) | CA-ICL captioning · CA-ICL captioning | CIDEr: 104.4 -> 2.7 | Hurt |
| Encode bbox information as text (ablation) | CA-ICL captioning · CA-ICL captioning | CIDEr: 104.4 -> 112.0 | Helped |
| Use sparse MoE layers (ablation) | GPT-2 small co-training CA-ICL tasks · CA-ICL segmentation | MIoU: 21.74 -> 40.48 | Helped |
| Use AG MoE layers (ablation) | GPT-2 small co-training CA-ICL tasks · CA-ICL segmentation | MIoU: 21.74 -> 42.02 | Helped |
| Use multi-task learning strategy (ablation) | GPT-2 small co-training CA-ICL tasks · CA-ICL segmentation | MIoU: 21.74 -> 33.72 | Helped |
| Combine AG MoE with multi-task learning (ablation) | GPT-2 small co-training CA-ICL tasks · CA-ICL segmentation | MIoU: 21.74 -> 49.91 | Helped |
| Combine AG MoE with multi-task learning (ablation) | GPT-2 small co-training CA-ICL tasks · CA-ICL captioning | CIDEr: 77.3 -> 78.3 | Helped |
Surprising findings
- Outputting bounding boxes as image tokens hurt captioning (CIDEr 104.4 to 2.7 with bbox image), while text-formatted bboxes improved CIDEr to 112.0.
Limitations
- Segmentation still trails SegGPT at its native 448 resolution (58.04 vs 62.83 MIoU).
- Captioning mAP lags GRiT, which uses a foreground object extractor.
- Evaluated only on two class-aware in-context tasks (segmentation and captioning).
Experimental setup
Datasets: MS-COCO Visual Genome ImageNet
Benchmarks: MS-COCO Visual Genome
Metrics: MIoU MAE BLEU4 METEOR CIDEr mAP
Baselines: SegGPT FPTrans VAT DCAMA GRiT OpenFlamingo
Builds on: Flamingo SegGPT VQGAN GShard Uni-Perceiver-MoE
Closest papers by meaning
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.91)
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.90)
- Improving Context Understanding in Multimodal Large Language Models via Multimodal Composition Learning (ICML 2024 · similarity 0.89)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.89)
- Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning (AAAI 2026 · similarity 0.89)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.