Research map / Vision-language and multimodal / Human and embodied modeling
ConCon-Chi: Concept-Context Chimera Benchmark for Personalized Vision-Language Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ConCon-Chi, a concept-context benchmark with chimeric and common objects photographed in many contexts, to evaluate personalized text-to-image retrieval and generation, showing current methods struggle with new-meaning learning and compositionality.
Problem
Existing personalized text-to-image retrieval and generation benchmarks lack novel concepts and compositional structure (many contexts per concept, multiple concepts per context), so they cannot properly test learning new meanings and composing them with known contexts.
Approach
Model novel concepts as chimeric objects (union of two unrelated concepts), photograph them in a large varied set of real contexts arranged in a concept-context matrix, and add real ground-truth images per query so retrieval and generation can be evaluated with distribution-based metrics.
Key results
- On ConCon-Chi TIR, SEARLE (5 shots) reaches 30.74 mAP and 43.83 mRR, above the Discriminative text-description oracle (30.16 mAP, 43.16 mRR) and PALAVRA (23.59 mAP).
- With context-only queries (alpha=1), ConCon-Chi retrieval drops 74% from peak performance, versus 29% for PerVL DF2 and much less for ZS-CIR datasets, indicating reduced context bias.
- PALAVRA achieves 91.52 F1 on concept recognition but only 28.56 F1 on context recognition, suggesting poor compositionality of learned tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SEARLE instead of PALAVRA for 5-shot personalized retrieval | ConCon-Chi TIR · ConCon-Chi test | mAP: 23.59 -> 30.74 | Helped |
| Use SEARLE instead of Pic2Word for 5-shot personalized retrieval | ConCon-Chi TIR · ConCon-Chi test | mAP: 26.39 -> 30.74 | Helped |
| Increase training images from 1 to 5 per concept | SEARLE · ConCon-Chi test | mAP: 28.16 ± 0.55 -> 30.74 | Helped |
| Retrieve queries containing chimeric rather than common concepts | Personalized TIR methods and baselines · ConCon-Chi test | mAP | Hurt |
| Replace the concept identifier with a Rich description rather than a Discriminative description | CLIP retrieval baseline · ConCon-Chi test | mAP: 30.16 -> 27.65 | Hurt |
| Train DreamBooth on concepts rather than use the pretrained SDM baseline | Personalized TIG · ConCon-Chi test, common concepts | Density (k=3): 2.66 -> 6.55 | Helped |
| Train DreamBooth on concepts rather than use the pretrained SDM baseline | Personalized TIG · ConCon-Chi test, chimeric concepts | Density (k=3): 0.78 -> 4.81 | Helped |
| Use DreamBooth rather than Textual Inversion with SDM | Personalized TIG · ConCon-Chi test, common concepts | Density (k=3): 3.43 -> 6.55 | Helped |
| Use DreamBooth rather than Textual Inversion with SDM | Personalized TIG · ConCon-Chi test, chimeric concepts | Density (k=3): 2.71 -> 4.81 | Helped |
Surprising findings
- Adding visual details to text descriptions (Rich) degrades retrieval performance relative to Discriminative descriptions.
- Personalization methods achieve equal or better concept recognition on chimeric than on common concepts, yet struggle more at retrieving chimeric concepts in composition with contexts.
Limitations
- Only 20 concepts (6 chimeric, 14 common), a fairly small concept set.
- Personalized methods are evaluated with the authors' released code and default settings, which may limit fair comparison.
- Chimeric concepts are hand-crafted physical objects, so the benchmark's realism of new meanings is limited.
Experimental setup
Datasets: ConCon-Chi
Benchmarks: ConCon-Chi
Metrics: mAP mRR R@1 F1 FID-concept FID-context Density Coverage
Baselines: PALAVRA Pic2Word SEARLE Textual Inversion DreamBooth CLIP zero-shot descriptions
Builds on: PerVL Textual Inversion DreamBooth Pic2Word SEARLE
Closest papers by meaning
- Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.87)
- Improving Personalized Search with Regularized Low-Rank Parameter Updates (CVPR 2025 · similarity 0.87)
- ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty (NeurIPS 2024 · similarity 0.87)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.86)
- Contextualized Visual Personalization in Vision-Language Models (ICML 2026 · similarity 0.86)
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.