Research map / Vision-language and multimodal / Multimodal LLMs
ControlCap: Controllable Region-level Captioning
Yuzhong Zhao, Liu Yue, Zonghao Guo, weijia wu, Chen Gong, Qixiang Ye, Fang Wan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add contextual visual embedding (region plus global RoI features) (ablation) | BLIP2 region captioner · VG V1.2 dense captioning | mAP (%): 37.9 -> 42.4 | Improved |
| add class embeddings to contextual visual embedding (ablation) | ControlCap CVE · VG V1.2 dense captioning | mAP (%): 39.7 -> 42.4 | Improved |
| add detail-rich region features to contextual visual embedding (ablation) | ControlCap CVE · VG V1.2 dense captioning | mAP (%): 35.9 -> 42.4 | Improved |
| add control embedding without bridging (ablation) | ControlCap with CVE · VG V1.2 dense captioning (self control) | mAP (%): 42.4 -> 42.0 | Hurt |
| add control embedding (enables interactive control) (ablation) | ControlCap with CVE · VG V1.2 dense captioning (interactive control) | mAP (%): 65.1 with interactive control | Improved |
| add region tagging module to generate self controls (ablation) | ControlCap · VG V1.2 dense captioning (self control) | mAP (%): 42.4 -> 42.8 | Improved |
| add region tagging module under interactive control (ablation) | ControlCap · VG V1.2 dense captioning (interactive control) | mAP (%): 65.8 -> 69.0 | Improved |
| add bidirectional bridging module (ablation) | ControlCap with CE · VG V1.2 dense captioning (interactive control) | mAP (%): 65.1 -> 65.8 | Improved |
| add second bridging branch Fr to Fc after Fc to Fr (ablation) | BiB module · VG V1.2 dense captioning (self control) | mAP (%): 42.3 -> 42.4 | No clear effect |
| use interactive control words (first GT noun) at inference | ControlCap · VG referring expression generation | CIDEr: 181.9 -> 302.3 | Improved |
| give interactive control words versus no control | ControlCap · GPT-4v preference on Object365 | GPT-4v preference (better/worse/equal): (54 / 5 / 41) | Improved |
Later papers that compared with this method
2 later accepted papers report ControlCap as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.
- DynRefer: Delving into Region-level Multimodal Tasks via Dynamic Resolution (CVPR 2025 · better on 3 of 3 shared results)
- FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · better on 1 of 1 shared result)
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- BLIP2 — BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (ICML 2022) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · similarity 0.86)
- FlexCap: Describe Anything in Images in Controllable Detail (NeurIPS 2024 · similarity 0.86)
- Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.86)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.86)
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.85)
- CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
