Research map / Vision-language and multimodal / CLIP and zero-shot
FineMatch: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction
Hang Hua, Jing Shi, Kushal Kafle, Simon Jenni, Daoan Zhang, John Collomosse, Scott Cohen, Jiebo Luo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale language model from 7B to 13B (ablation) | LLaVA-1.5 fine-tuned on FineMatch · FineMatch test, mismatch detection | ITM-IoU: 62.25 -> 66.02 | Improved |
| upgrade from LLaVA-1.5 to LLaVA-1.6 | 13B VLM fine-tuned on FineMatch · FineMatch test, mismatch detection | ITM-IoU: 66.02 -> 66.10 | Improved |
| use LoRA instead of full fine-tuning | LLaVA-1.5 13B · FineMatch test, mismatch detection | ITM-IoU: 66.02 -> 65.51 | No clear effect |
| supervised tuning on FineMatch versus in-context learning | VLMs · FineMatch test | ITM-IoU: 21.92 (GPT-4V ICL) vs 66.10 (tuned LLaVA-1.6) | Improved |
| use strong in-context models (GPT-4V, Gemini Pro Vision) for fine-grained mismatch analysis | FineMatch · FineMatch test subset | ITM-IoU: GPT-4V 21.92, Gemini 9.07 vs human 88.32 | Hurt |
| add correction output (MD&C) on top of detection | fine-tuned VLMs · FineMatch test | ITM-IoU: 62.18 -> 63.80 (LLaVA-1.5-LoRA 7B) | No clear effect |
Closest papers by meaning
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.89)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.88)
- FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · similarity 0.88)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.88)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.87)
- Remodeling Semantic Relationships in Vision-Language Fine-Tuning (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
