AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibits

Ada-Astrid Balauca, Danda Pani Paudel, Kristina Toutanova, Luc Van Gool

ECCV 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Fine-grained improvements to CLIP · 5 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add parseNet context-aware structured prediction on top of fine-tuned CLIPCLIP-FA · A-MUZEMean Avg Accuracy: 15.60 -> 23.51Improved
add parseNet context-aware structured prediction on top of fine-tuned CLIPCLIP-FA · B-MUZEMean Acc @ 1: 20.29 -> 30.47Improved
fine-tune a separate CLIP image encoder per attribute (CLIP-FA) instead of full-caption fine-tuning (ablation)CLIP · A-MUZEMean Avg Precision: 1.73 -> 3.20Improved
fine-tune CLIP with full caption (CLIP-FC) instead of zero-shot CLIP (ablation)CLIP · A-MUZEMean Avg Precision: 1.69 -> 1.73No clear effect
choose CLIP backbone variant (CFC vs CFA vs C) under parseNet (ablation)MUZE · A-MUZE and B-MUZEMean Avg Precision: 4.80 / 4.87 / 4.36 on A-MUZENo clear effect
add text context of known attribute values to CLIP at inference (CLIP-CTX) (ablation)CLIP · A-MUZE categoriesMean Acc @ 1: 8.71 -> 0.54Hurt
also fine-tune the CLIP text encoder (ablation)CLIP-FA · A-MUZEMean Avg Precision: 3.20 -> 2.45Hurt
use phrase-like input format during fine-tuning (ablation)CLIP-FA · A-MUZEMean Avg Precision: 3.20 -> 2.61Hurt
increase number of attribute-value pairs given as context to MUZE (ablation)MUZE · A-MUZEperformanceImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy