Research map / Vision-language and multimodal / CLIP and zero-shot
Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibits
Ada-Astrid Balauca, Danda Pani Paudel, Kristina Toutanova, Luc Van Gool
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add parseNet context-aware structured prediction on top of fine-tuned CLIP | CLIP-FA · A-MUZE | Mean Avg Accuracy: 15.60 -> 23.51 | Improved |
| add parseNet context-aware structured prediction on top of fine-tuned CLIP | CLIP-FA · B-MUZE | Mean Acc @ 1: 20.29 -> 30.47 | Improved |
| fine-tune a separate CLIP image encoder per attribute (CLIP-FA) instead of full-caption fine-tuning (ablation) | CLIP · A-MUZE | Mean Avg Precision: 1.73 -> 3.20 | Improved |
| fine-tune CLIP with full caption (CLIP-FC) instead of zero-shot CLIP (ablation) | CLIP · A-MUZE | Mean Avg Precision: 1.69 -> 1.73 | No clear effect |
| choose CLIP backbone variant (CFC vs CFA vs C) under parseNet (ablation) | MUZE · A-MUZE and B-MUZE | Mean Avg Precision: 4.80 / 4.87 / 4.36 on A-MUZE | No clear effect |
| add text context of known attribute values to CLIP at inference (CLIP-CTX) (ablation) | CLIP · A-MUZE categories | Mean Acc @ 1: 8.71 -> 0.54 | Hurt |
| also fine-tune the CLIP text encoder (ablation) | CLIP-FA · A-MUZE | Mean Avg Precision: 3.20 -> 2.45 | Hurt |
| use phrase-like input format during fine-tuning (ablation) | CLIP-FA · A-MUZE | Mean Avg Precision: 3.20 -> 2.61 | Hurt |
| increase number of attribute-value pairs given as context to MUZE (ablation) | MUZE · A-MUZE | performance | Improved |
Closest papers by meaning
- Demystifying CLIP Data (ICLR 2024 · similarity 0.85)
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.84)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.84)
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.84)
- PoSh: Using Scene Graphs to Guide LLMs-as-a-Judge for Detailed Image Descriptions (ICLR 2026 · similarity 0.84)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
