Research map / Vision-language and multimodal / Human and embodied modeling
Language-Informed Visual Concept Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a pre-trained text-to-image model and VQA model into per-axis concept encoders that extract disentangled, language-anchored visual concept embeddings from images for remixing and editing.
Problem
Visual nuances along language-specified concept axes (e.g., a particular painting style or shade of color) exceed what text can describe, and per-instance textual inversion does not share or disentangle concepts across images.
Approach
Train concept encoders on CLIP features to predict axis-specific embeddings that reconstruct images through a frozen DeepFloyd T2I model via an axis-informed template, with a small-weight anchor loss pulling embeddings toward BLIP-2 VQA answer text embeddings for disentanglement; test-time finetuning generalizes to unseen concepts.
Key results
- On category editing, Ours reaches a human-evaluation score of 0.968 versus 0.448 for the BLIP-2 baseline and 0.287 for Null-text Inversion.
- On color editing, Ours reaches CLIP Cat.&Clr. 0.302 vs 0.287 for the BLIP-2 baseline and 0.259 for Null-text Inversion; human color score 0.840 vs 0.379 for the BLIP-2 baseline.
- Removing the anchor loss and encoder drops color CIELAB delta E errors substantially (e.g., Cherry 35.50 for Ours vs 101.34 without anchor loss).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned concept encoders with text anchor loss instead of BLIP-2 text prompting | Visual concept editing · Category edit, category-preserving composite CLIP score | CLIP-Score: 0.313 vs 0.308 | No clear effect |
| Use learned concept encoders with text anchor loss instead of BLIP-2 text prompting | Visual concept editing · Color edit, category-preserving composite CLIP score | CLIP-Score: 0.287 -> 0.302 | Helped |
| Use learned concept encoders with text anchor loss instead of InstructPix2Pix | Visual concept editing · Human evaluation, category editing | Score: 0.233 -> 0.968 | Helped |
| Use learned concept encoders with text anchor loss instead of InstructPix2Pix | Visual concept editing · Human evaluation, color editing | Score: 0.648 -> 0.840 | Helped |
| Add the text anchor loss (ablation) | Language-Informed Visual Concept Learning · Visual concept editing, category edit | CLIP-Score: 0.268 -> 0.308 | Helped |
| Add the text anchor loss (ablation) | Language-Informed Visual Concept Learning · Visual concept editing, color edit | CLIP-Score: 0.263 -> 0.302 | Helped |
| Add the concept encoder and text anchor loss (ablation) | Language-Informed Visual Concept Learning · Visual concept editing, color edit | CLIP-Score: 0.242 -> 0.302 | Helped |
| Add the text anchor loss (ablation) | Language-Informed Visual Concept Learning · Color editing, Cherry, Mango, Strawberry, and Apple | CIELAB ΔE*: 101.34 -> 35.50; 86.34 -> 4.76; 85.31 -> 16.12; 122.01 -> 15.64 | Helped |
| Add the concept encoder and text anchor loss (ablation) | Language-Informed Visual Concept Learning · Color editing, Cherry, Mango, Strawberry, and Apple | CIELAB ΔE*: 85.86 -> 35.50; 82.46 -> 4.76; 82.07 -> 16.12; 127.80 -> 15.64 | Helped |
| Use learned concept encoders instead of BLIP-2 text prompting | Color editing · Cherry, Mango, Strawberry, and Apple | CIELAB ΔE*: 79.90 -> 35.50; 25.20 -> 4.76; 47.30 -> 16.12; 73.62 -> 15.64 | Helped |
Surprising findings
- The BLIP-2 text-based baseline matches or exceeds the proposed method on some CLIP-score metrics despite visibly worse color fidelity, which the authors attribute to CLIP metric limitations.
- InstructPix2Pix scores highest on changing color in CLIP score but performs poorly at preserving category.
Limitations
- Concept axes must be specified a priori by language, limiting the generality of the learned concept space.
- A separate encoder is trained for each concept axis, not exploiting hierarchical structure among axes.
- Training uses only small synthetic datasets (about 669 images per domain), and unseen concepts require test-time finetuning.
Experimental setup
Datasets: synthetic DeepFloyd-generated fruit, art, figurine, clothing, furniture images
Benchmarks: in-house synthetic concept editing set (CLIP-score) human evaluation with 20 participants
Metrics: CLIP score Borda count human ranking CIELAB delta E
Baselines: Null-text Inversion InstructPix2Pix Baseline w/ BLIP-2
Builds on: Textual Inversion DeepFloyd IF BLIP-2 CLIP
Closest papers by meaning
- Bridging the gap to real-world language-grounded visual concept learning (NeurIPS 2025 · similarity 0.93)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.90)
- Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.89)
- A Vision Check-up for Language Models (CVPR 2024 · similarity 0.89)
- Concept-Guided Prompt Learning for Generalization in Vision-Language Models (AAAI 2024 · similarity 0.88)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.