Research map / Vision-language and multimodal / Multimodal LLMs
Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
Li Li, Yongliang Wu, Jingze Zhu, Jiawei Peng, Jianfei Cai, and Xu Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the number of shots | OpenFlamingo and IDEFICS · MSCOCO captioning | CIDEr and CLIPScore | Mixed |
| use similarity-based image retrieval instead of random sampling | OpenFlamingo and IDEFICS · MSCOCO captioning | CIDEr and CLIPScore | Mixed |
| use low-quality ICE captions (TF-60) with more shots | OpenFlamingo and IDEFICS · MSCOCO captioning | CIDEr and CLIPScore | Hurt |
| use high-quality ICE captions | OpenFlamingo · MSCOCO captioning | CIDEr | Improved |
| use inverse MHL-TF (CIDEr-unbiased) ICE captions | OpenFlamingo · MSCOCO captioning | CIDEr and CLIPScore | Mixed |
| use model-generated captions (MHL) matched to human captions | OpenFlamingo and IDEFICS · MSCOCO captioning | CIDEr | Improved |
| anchor-centric masking at layers 10-30 (4-shot) (ablation) | IDEFICS · MSCOCO | CIDEr: 97.3 -> 74.1 | Hurt |
| anchor-centric masking at layers 15-30 (4-shot) (ablation) | IDEFICS · MSCOCO | CIDEr: 97.3 -> 94.8 | No clear effect |
| context-centric (local window) masking at layers 10-30 (32-shot) (ablation) | IDEFICS · MSCOCO | CIDEr: 104.8 -> 105.1 | No clear effect |
| prune non-anchor tokens from layer 20 (32-shot) (ablation) | IDEFICS · MSCOCO RS 32-shot | CIDEr: 108.1 -> 108.0 | No clear effect |
| prune non-anchor tokens from layer 10 (32-shot), with recovery (ablation) | IDEFICS · MSCOCO RS 32-shot | CIDEr: 108.1 -> 69.95 | Hurt |
| recover pruned token states before prediction (Fast_10, 4-shot) (ablation) | IDEFICS · MSCOCO RS 4-shot | CIDEr: 55.5 -> 61.1 | Improved |
Closest papers by meaning
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.92)
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.92)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.90)
- LIVE: Learnable In-Context Vector for Visual Question Answering (NeurIPS 2024 · similarity 0.90)
- Improving Context Understanding in Multimodal Large Language Models via Multimodal Composition Learning (ICML 2024 · similarity 0.89)
- Link-Context Learning for Multimodal LLMs (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
