AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

FineMatch: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction

Hang Hua, Jing Shi, Kushal Kafle, Simon Jenni, Daoan Zhang, John Collomosse, Scott Cohen, Jiebo Luo

ECCV 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Self-distillation for vision-language pretraining · 20 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
scale language model from 7B to 13B (ablation)LLaVA-1.5 fine-tuned on FineMatch · FineMatch test, mismatch detectionITM-IoU: 62.25 -> 66.02Improved
upgrade from LLaVA-1.5 to LLaVA-1.613B VLM fine-tuned on FineMatch · FineMatch test, mismatch detectionITM-IoU: 66.02 -> 66.10Improved
use LoRA instead of full fine-tuningLLaVA-1.5 13B · FineMatch test, mismatch detectionITM-IoU: 66.02 -> 65.51No clear effect
supervised tuning on FineMatch versus in-context learningVLMs · FineMatch testITM-IoU: 21.92 (GPT-4V ICL) vs 66.10 (tuned LLaVA-1.6)Improved
use strong in-context models (GPT-4V, Gemini Pro Vision) for fine-grained mismatch analysisFineMatch · FineMatch test subsetITM-IoU: GPT-4V 21.92, Gemini 9.07 vs human 88.32Hurt
add correction output (MD&C) on top of detectionfine-tuned VLMs · FineMatch testITM-IoU: 62.18 -> 63.80 (LLaVA-1.5-LoRA 7B)No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy