Research map / Vision-language and multimodal / Visual question answering
FlexCap: Describe Anything in Images in Controllable Detail
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FlexCap, a length-conditioned region captioner trained on box-caption triplets mined from web image-text data with OWL-ViT, and uses its localized descriptions as text input to an LLM (PaLM2-S) for zero-shot VQA and dense captioning.
Problem
Region-level image description typically lacks control over description detail; existing dense captioners are limited by small annotated datasets, and VLMs struggle to exhaustively describe all objects in an image.
Approach
Condition a region captioner on a word-length token so the same box yields descriptions from one-word labels to detailed captions, training on ~32B box-caption triplets auto-generated by matching n-grams of alt-text to OWL-ViT boxes; then describe many boxes as text for an LLM.
Key results
- Zero-shot FlexCap-LLM reaches 65.6% on VQAv2 test-dev, above BLIP-2 (65.2) and Flamingo (56.3) in the zero-shot table.
- On Visual Genome dense captioning with GRiT boxes, FlexCap achieves 16.2 mAP versus GRiT's 15.52.
- On MS-COCO region classification, Top-20 captions reach 85.0 mAP versus 62.8 for RegionCLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate 20 captions instead of one caption per box (ablation) | FlexCap region classification · MS-COCO Region Classification | mAP: 72.0 -> 85.0 | Helped |
| add contrastive pre-training (ablation) | ViT-B/16 FlexCap · Visual Genome GT boxes | mAP: 43.6 -> 45.1 | Helped |
| add contrastive pre-training (ablation) | ViT-B/16 FlexCap · Visual Genome dense captioning with GRiT boxes | mAP: 15.6 -> 15.8 | Helped |
| scale region-caption training data from YFCC100M (0.2B) to WebLI (32B) (ablation) | ViT-B/16 FlexCap · Visual Genome GT boxes | mAP: 38.5 -> 45.1 | Helped |
| scale region-caption training data from YFCC100M (0.2B) to WebLI (32B) (ablation) | ViT-B/16 FlexCap · Visual Genome dense captioning with GRiT boxes | mAP: 14.2 -> 15.8 | Helped |
| increase the backbone from ViT-B/16 (86M) to SO-ViT/14 (428M) (ablation) | FlexCap · Visual Genome GT boxes | mAP: 45.1 -> 46.9 | Helped |
| increase the backbone from ViT-B/16 (86M) to SO-ViT/14 (428M) (ablation) | FlexCap · Visual Genome dense captioning with GRiT boxes | mAP: 15.8 -> 16.2 | Helped |
| increase maximum caption length from 1 to 8 (ablation) | FlexCap-LLM · VQAv2 val | accuracy: 57.4 -> 67.8 | Helped |
| increase maximum caption length from 1 to 8 (ablation) | FlexCap-LLM · GQA testdev-balanced | accuracy: 43.6 -> 48.8 | Helped |
| use FlexCap-LLM rather than BLIPv2 | zero-shot VQAv2 · VQAv2 test-dev | accuracy (%): 65.2 -> 65.6 | Helped |
| use FlexCap-LLM rather than InstructBLIP | zero-shot GQA · GQA test-dev balanced | accuracy (%): 49.5 -> 48.8 | Hurt |
| use FlexCap-LLM rather than ViperGPT | zero-shot OKVQA · OKVQA val | accuracy (%): 51.9 -> 52.1 | Helped |
Surprising findings
- Using the length prefix instead of a plain start token reduces caption prefix-matching probability from 30.8% to 11.1%.
- FlexCap-LLM is strong on VizWiz (41.8%) despite out-of-distribution atypical images.
Limitations
- FlexCap-LLM is a composition of a VLM and an LLM and is not end-to-end trainable.
- Training data inherits biases from OWL-ViT and alt-text.
- Pipeline relies on noisy web alt-text and automatic box labels rather than human annotations.
Experimental setup
Datasets: WebLI YFCC100M MS-COCO Visual Genome VQAv2 GQA OK-VQA VizWiz MSRVTT-QA MSVD-QA COCO Captions VOC
Benchmarks: MS-COCO region classification VQAv2 GQA OK-VQA VizWiz MSRVTT-QA MSVD-QA Visual Genome dense captioning
Metrics: mAP accuracy mean length compliance
Baselines: CLIP CLIM RegionCLIP FCLN CAG-Net GRiT BLIP-2 Flamingo ViperGPT InstructBLIP LLaVA 1.5
Builds on: SigLIP OWL-ViT SimVLM GRiT
Closest papers by meaning
- Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning (CVPR 2022 · similarity 0.88)
- Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations (NeurIPS 2019 · similarity 0.88)
- EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension (CVPR 2024 · similarity 0.88)
- FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · similarity 0.88)
- Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.87)
- BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.