AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

Removing Distributional Discrepancies in Captions Improves Image-Text Alignment

Mu Cai, Haotian Liu, Yuheng Li, Yijun Li, Eli Shechtman, Zhe Lin, Yong Jae Lee, Krishna Kumar Singh

ECCV 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Concept-guided image captioning · 5 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add text-only-classifier data filtering (remove top 30%) (ablation)LLaVA-score · Winogroundgroup score: 46.75 -> 47.25Improved
add data filtering (ablation)LLaVA-score · MagicBrushgroup score: 82.99 -> 87.28Improved
add data filtering (ablation)LLaVA-score · SeeTRUE COCO-T2IROC-AUC: 81.1 -> 84.9Improved
filter biased data instead of random subsampling to same size (ablation)LLaVA-score · MagicBrushgroup score: 83.12 -> 87.28Improved
combine replace and swap negatives (ablation)LLaVA-score · SugarCrepe addaccuracy: 95.3 -> 97.5 (vs only replace); 91.3 -> 97.5 (vs only swap)Improved
train on swap-only negatives (ablation)LLaVA-1.5 · MagicBrushgroup score: 82.61 -> 76.63Hurt
fine-tune LLaVA-1.5 on curated dataLLaVA-1.5 · Winogroundgroup score: 34.25 -> 47.25Improved
fine-tune LLaVA-1.5 on curated dataLLaVA-1.5 · SeeTRUE EditBenchROC-AUC: 78.3 -> 77.7Mixed
fine-tune BLIP2 ITM head on curated dataBLIP2-ITM · SeeTRUE DrawBenchROC-AUC: 60.8 -> 87.5Improved
use contrastive loss instead of cross-entropy for fine-tuning (ablation)LLaVA-score · MagicBrushgroup score: 87.28 -> 86.98No clear effect
increase filtered percentage k beyond 30-40% (ablation)LLaVA-score · four test datasetsperformanceHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy