Research map / Vision-language and multimodal / CLIP and zero-shot

Self-Evolving Visual Concept Library using Vision-Language Critics

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Open-vocabulary alignment from VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ESCHER, a self-evolving visual concept library where a VLM critic's contrastive feedback guides an LLM, with history of past concept evolutions, to iteratively refine concepts for concept-bottleneck classifiers in zero-shot, few-shot, and fine-tuned settings.

Problem

Manually defined visual concepts are labor-intensive, and LLM-generated concepts can lack discriminative power or ignore interactions between concepts, limiting concept-bottleneck visual recognition.

Approach

Treat concept discovery as library learning: alternate between fitting a concept-bottleneck adapter and using confusion between frequently confused class pairs to prompt an LLM, conditioned on a history bank of past concept evolutions and their VLM-measured scores, to propose disambiguating concepts.

Key results

• LM4CV+ESCHER raises CUB-200-2011 top-1 accuracy from 63.26 to 83.17 and Stanford Cars from 86.84 to 93.76 with no human annotations.
• CbD+ESCHER improves zero-shot CIFAR-100 from 76.20 to 77.80 and Oxford Flowers from 79.41 to 81.37.
• In few-shot LaBO experiments results are mixed: ESCHER gains on Stanford Cars (75.07 to 75.99 at 8-shot) but slightly hurts CIFAR100 at 8-shot (74.23 to 73.62).

Measured effects reported in the paper

Surprising findings

• Sampling three times more LLM concepts without VLM feedback (LM4CV + Many Concepts) gives little improvement, suggesting VLM critic feedback matters more than concept count.
• In few-shot settings ESCHER gives mixed results, with slight drops on some datasets such as CIFAR100 at 8-shot.

Limitations

• Few-shot gains are mixed and sometimes negative, attributed to poor LaBO calibration and overfitting.
• Gains are smaller for zero-shot CbD because CLIP scores are less calibrated, producing noisier iterations.
• Relies on the backbone VLM's ability to score fine-grained concepts; weaker backbones (ViT-B/16) reduce absolute performance.

Experimental setup

Datasets: CIFAR-100 CUB-200-2011 Food-101 NABirds Stanford Cars Flowers-102 Oxford-IIIT Pets

Benchmarks: CIFAR-100 CUB-200-2011 Food-101 NABirds Oxford Flowers Oxford IIIT Pets Stanford Cars

Metrics: Top-1 accuracy

Baselines: LM4CV LaBO Classify by Descriptions CLIP ViT-L/14 LM4CV with many concepts

Builds on: LM4CV LaBO Classify by Descriptions library learning (LILO-style program synthesis) chain-of-thought prompting

Closest papers by meaning

• Bridging the gap to real-world language-grounded visual concept learning (NeurIPS 2025 · similarity 0.86)
• V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer (AAAI 2025 · similarity 0.86)
• Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.86)
• A Vision Check-up for Language Models (CVPR 2024 · similarity 0.86)
• Concept-Aware Batch Sampling Improves Language-Image Pretraining (CVPR 2026 · similarity 0.85)
• Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.85)