Research map / Vision-language and multimodal / Human and embodied modeling
How can embedding models bind concepts?
Arnas Uselis, Darina Koishigarina, Seong Joon Oh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add concept components to scene embedding (removing them eliminates concept decoding) (ablation) | CLIP-B/32 text probe · PUG:SPARE | concept decoding accuracy: 1.00 → 0.06 | Improved |
| add concept components to scene embedding (removing them eliminates concept decoding) (ablation) | CLIP-B/32 image probe · PUG:SPARE | concept decoding accuracy: 0.94 → 0.05 | Improved |
| keep object components (removing concepts leaves object decoding largely intact) (ablation) | CLIP-B/32 text probe · PUG:SPARE | object decoding accuracy: 1.00 → 0.99 | No clear effect |
| keep concept components (removing concepts barely changes image object decoding) (ablation) | CLIP-B/32 image probe · PUG:SPARE | object decoding accuracy: 0.96 → 0.85 | Mixed |
| add object components to scene embedding (removing them collapses object decoding) (ablation) | CLIP-B/32 text probe · PUG:SPARE | object decoding accuracy: 1.00 → 0.04 | Improved |
| add object components to scene embedding (removing them collapses object decoding) (ablation) | CLIP-B/32 image probe · PUG:SPARE | object decoding accuracy: 0.96 → 0.01 | Improved |
| use position-aware (AVG+POS) object embeddings instead of AVG (ablation) | CLIP image encoder · PUG:SPARE | R^2 of scene reconstruction: 0.75 / 0.84 | Improved |
| use position-aware (AVG+POS) object embeddings instead of AVG (ablation) | DINO · PUG:SPARE object replacement edit | retrieval accuracy: 0.48 / 0.76 | Improved |
| use single-object scene embeddings for object edits instead of averaged (ablation) | CLIP · CLEVR object replacement edit | retrieval accuracy: 1.00 → 0.97 | Hurt |
| add object-level components on top of bag-of-concepts (scenes from objects vs concepts) (ablation) | CLIP-B/32 image · PUG:SPARE (AVG) | R^2: 0.30 → 0.75 | Improved |
| add multiplicative interactions to additive bag-of-concepts probe (ablation) | probe on transformer embeddings · held-out objects | object recognition accuracy | Improved |
| increase training coverage of objects from 30% to 40% | transformer dual-encoder, |O|=125,000 · held-out objects | object recognition accuracy: 30% to 40% | Improved |
Closest papers by meaning
- Object-centric binding in Contrastive Language-Image Pretraining (NeurIPS 2025 · similarity 0.89)
- Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers? (NeurIPS 2025 · similarity 0.89)
- Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.89)
- Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem (NeurIPS 2024 · similarity 0.89)
- Seeing to Generalize: How Visual Data Corrects Binding Shortcuts (ICML 2026 · similarity 0.88)
- Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
