Research map / Vision-language and multimodal / Visual question answering
CTRL-O: Language-Controllable Object-Centric Visual Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CTRL-O, an object-centric model whose slots are conditioned on language (and center-of-mass) queries via a contrastive grounding loss, enabling user-directed slot extraction used for instance-controllable image generation and VQA.
Problem
Existing unsupervised object-centric models decompose scenes into slots without letting users specify which objects should be represented, lacking controllability over slot semantics.
Approach
Condition slot initialization and the broadcast decoder on language query embeddings (LLM2Vec LLaMA-3-8B), and train with a control contrastive loss that aligns attention-weighted DINOv2 slot features with their queries, so slots bind to the queried object without mask supervision.
Key results
- CTRL-O achieves FG-ARI 47.5 on COCO versus 40.5 for DINOSAUR (MLP decoder) while its mBO (27.2) trails DINOSAUR (TF decoder, 31.6).
- On zero-shot referring expression segmentation (RefCOCO val), CTRL-O with language and point queries reaches 28.2 mIoU, above Shatter & Gather's 21.80.
- CTRL-O-SD improves FID to 25.20 vs 26.20 for Stable-LSD and CLIP-I to 0.78 vs 0.71 for SD; on VQAv2 CTRL-O with coupling reaches 60.25 accuracy vs 58.64 for CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add language and point conditioning instead of language-only conditioning | CTRL-O · RefCOCO testA | mIoU (%): 20.10 -> 33.13 | Helped |
| Add language and point conditioning instead of language-only conditioning | CTRL-O · Gref val | mIoU (%): 25.32 -> 30.50 | Helped |
| Use language and point conditioning instead of language-only conditioning | CTRL-O · RefCOCO+ testB | mIoU (%): 21.36 -> 22.58 | Helped |
| Use language and point conditioning instead of language-only conditioning | CTRL-O · RefCOCO+ val | mIoU (%): 21.90 -> 25.87 | Helped |
| Use language and point conditioning instead of language-only conditioning | CTRL-O · RefCOCO+ testA | mIoU (%): 21.54 -> 30.58 | Helped |
| Use language and point conditioning instead of language-only conditioning | CTRL-O · RefCOCO testB | mIoU (%): 21.57 -> 27.05 | Helped |
| Use language and point conditioning instead of language-only conditioning | CTRL-O · RefCOCO val | mIoU (%): 21.80 -> 28.2 | Helped |
| Add contrastive loss (ablation) | CTRL-O · COCO val grounding | Binding Hits: 10.11 -> 56.3 | Helped |
| Add decoder conditioning (ablation) | CTRL-O with contrastive loss · COCO val grounding | Binding Hits: 56.3 -> 61.3 | Helped |
| Add contrastive loss and decoder conditioning (ablation) | CTRL-O · COCO val grounding | Binding Hits: 8.1 -> 61.3 | Helped |
| Add coupling by inserting slots into the question | CTRL-O on VQAv2 · VQAv2 | accuracy: 59.18 -> 60.25 | Helped |
| Use CTRL-O-SD rather than LSD | COCO image generation · COCO image generation | FID: 26.20 -> 25.20 | Helped |
Surprising findings
- Conditioning slots on queries alone does not produce correct grounding; the control contrastive loss is the most crucial component for binding.
Limitations
- Mask quality (mBO) lags behind DINOSAUR with transformer decoder because of the MLP decoder.
- VQA performance remains below state-of-the-art models (>80%) that use LLMs and web-scale data.
- Grounding is imperfect even with full mask supervision (Binding Hits well below 100%).
Experimental setup
Datasets: COCO Visual Genome VQAv2 RefCOCO RefCOCO+ G-Ref
Benchmarks: COCO Visual Genome RefCOCO RefCOCO+ G-Ref VQAv2
Metrics: FG-ARI mBO Binding Hits mIoU FID CLIP-I accuracy
Baselines: DINOSAUR Stable-LSD SlotDiffusion Stable-LSD (bbox supervision) GroupViT MaskCLIP Shatter & Gather SD CLIP DINOv2
Builds on: DINOSAUR Slot Attention DINOv2 LLM2Vec Stable Diffusion
Closest papers by meaning
- Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models (ICLR 2025 · similarity 0.89)
- GLASS: Guided Latent Slot Diffusion for Object-Centric Learning (CVPR 2025 · similarity 0.88)
- Grounded Object-Centric Learning (ICLR 2024 · similarity 0.88)
- OBJECT-UNI: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation (NeurIPS 2026 · similarity 0.87)
- MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning (NeurIPS 2025 · similarity 0.87)
- Adaptive Slot Attention: Object Discovery with Dynamic Slot Number (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.