AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

Controllable Contextualized Image Captioning: Directing the Visual Narrative through User-Defined Highlights

Shunqi Mao, Chaoyi Zhang, Hang Su, Hwanjun Song, Igor Shalyminov, Weidong Cai

ECCV 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Concept-guided image captioning · 5 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use highlight-aware controllers (P-Ctrl or R-Ctrl) instead of extracting highlight-centered contextLongT5-base captioner · Wiki-Web2M full, single highlighthighlight recall: 18.45 -> 57.07 (P-Ctrl), 64.88 (R-Ctrl)Improved
truncate context to highlight-centered sentencesLongT5 CIC model · Wiki-Web2M fullhighlight control and caption quality: R 17.62-18.45 vs 57.07 (P-Ctrl)Hurt
use R-Ctrl instead of P-CtrlCtrl-CIC controllers · Wiki-Web2M full, single highlighthighlight recall vs Div-2: R 57.07 -> 64.88; Div-2 10.54 -> 10.30Mixed
fine-tune controllers for highlights instead of zero-shot LLaVA-1.5Ctrl-CIC · Wiki-Web2M full, single highlightCLIPScore: 65.56 -> 60.91 (P-Ctrl)Hurt
use R-Ctrl instead of CIC ground truth captionCtrl-CIC · GPT-4V evaluationContext Relevance: 1 -> 1.01No clear effect
use Ctrl-CIC controllers on standard CIC task instead of LongT5 finetuneP-Ctrl · Wiki-Web2M test CICBLEU-4: 10.31 -> 10.88No clear effect
use P-Ctrl on standard CIC task instead of LongT5 finetuneP-Ctrl · Wiki-Web2M test CICCIDEr: 142.77 -> 130.80Hurt
remove visual input (text-only Llama-2) (ablation)Ctrl-CIC · Wiki-Web2M fullImage Consistency (GPT-4V): 1 -> 0.61Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy