Research map / Vision-language and multimodal / Human and embodied modeling
Bridging the gap to real-world language-grounded visual concept learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a scalable language-grounded visual concept learning framework that uses a VLM with a universal prompt to discover image-specific concept axes, a universal concept encoder (Q-Former-style) to bind visual features to axes, and a compositional anchoring loss to disentangle them on real-world ImageNet/CelebA-HQ/AFHQ images.
Problem
Language-grounded visual concept learning is limited to a few predefined primitive axes (color, shape) on synthetic data, and does not scale to real-world images with diverse, image-specific concept axes.
Approach
Adaptively discover image-dependent concept axes with a VLM via a single universal prompt, bind visual features to axes using one Q-Former-style encoder whose queries are axis text embeddings, and disentangle axes by swapping concept embeddings across images and regularizing composites to match composed text descriptions (compositional anchoring) instead of forcing embeddings to equal text embeddi
Key results
- On CelebA-HQ, the method reaches 23.88 CLIP and 49.58 BLIP, above LIVCL and the text-based editing baselines.
- In human evaluation, the method scores 0.636 on CelebA-HQ, 0.589 on AFHQ-Dog and 0.623 on AFHQ-Cat, the best among all compared methods.
- In the multi-image composition task, the method shows only a marginal drop as the number of edited axes grows from 1 to 4, while all baselines drop clearly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Qwen2.5-VL instead of InternVL2-5 for concept-axis discovery (ablation) | Ours · CelebA-HQ visual concept editing | CLIP: 23.88 -> 23.72 | Hurt |
| use Ovis2 instead of InternVL2-5 for concept-axis discovery (ablation) | Ours · CelebA-HQ visual concept editing | CLIP: 23.88 -> 23.48 | Hurt |
| drop 10% of discovered axes (ablation) | Ours · CelebA-HQ visual concept editing | CLIP: 23.88 -> 23.52 | Hurt |
| drop 20% of discovered axes (ablation) | Ours · CelebA-HQ visual concept editing | CLIP: 23.88 -> 23.65 | Hurt |
| LoRA-finetune the T2I decoder (ablation) | Ours · CelebA-HQ visual concept editing | CLIP: 23.88 -> 23.67 | No clear effect |
| replace DINOv2 with CLIP as the vision encoder (ablation) | Ours · CelebA-HQ visual concept editing | CLIP: 23.88 -> 22.03 | Hurt |
| replace UCE with a shared MLP concept encoder (ablation) | Ours · CelebA-HQ visual concept editing | CLIP: 23.88 -> 21.63 | Hurt |
| add g_phi and avoid direct regression of concept representations to text embeddings (ablation) | Ours · CelebA-HQ visual concept editing | CLIP: 21.1 -> 23.88 | Helped |
| add the compositional anchoring loss (ablation) | Ours · CelebA-HQ visual concept editing | CLIP: 22.89 -> 23.88 | Helped |
| use Qwen2.5-VL instead of InternVL2-5 for concept-axis discovery (ablation) | Ours · CelebA-HQ visual concept editing | BLIP: 49.58 -> 48.64 | Hurt |
| use Ovis2 instead of InternVL2-5 for concept-axis discovery (ablation) | Ours · CelebA-HQ visual concept editing | BLIP: 49.58 -> 48.35 | Hurt |
| drop 10% of discovered axes (ablation) | Ours · CelebA-HQ visual concept editing | BLIP: 49.58 -> 48.69 | Hurt |
Surprising findings
- Replacing the decoder with a LoRA-finetuned T2I model does not improve results (CLIP 23.67 vs 23.88 frozen), suggesting the frozen T2I prior is not a bottleneck.
Limitations
- Can only discover concept axes that the pretrained VLM recognizes.
- Subtle or rare attributes may fall outside the discovered axes, so not all ground-truth factors are recovered.
- No error bars or repeated runs were reported due to compute cost.
Experimental setup
Datasets: ImageNet (20-class subset, ImageNet-S20) CelebA-HQ AFHQ-Dog AFHQ-Cat
Benchmarks: ImageNet-S20 CelebA-HQ AFHQ-Dog AFHQ-Cat
Metrics: CLIP-Score BLIP-Score human preference (Borda score)
Baselines: LIVCL SDEdit InstructPix2Pix Null-text Inversion DDPM-Inversion
Builds on: LIVCL Q-Former Textual Inversion DINOv2
Closest papers by meaning
- Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.93)
- Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners (CVPR 2024 · similarity 0.89)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.89)
- Concept-Aware Batch Sampling Improves Language-Image Pretraining (CVPR 2026 · similarity 0.89)
- Concept-Guided Prompt Learning for Generalization in Vision-Language Models (AAAI 2024 · similarity 0.88)
- Learning to Infer Generative Template Programs for Visual Concepts (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.