Research map / Vision-language and multimodal / Visual question answering
Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Large empirical study (684 downstream VQA transformers, 15 upstream representations) comparing object-centric models, foundation models, and VAEs on synthetic and real-world VQA, finding foundation models competitive but costlier and OC bias on foundation models (DINOSAURv2) offering the best trade-off.
Problem
Whether object-centric representations are actually useful for compositional reasoning in VQA, and how they compare to large pre-trained foundation models in accuracy and compute.
Approach
Evaluates OC, fixed-region, and global upstream representations by freezing them and training a common transformer VQA downstream model on synthetic (Multi-dSprites, CLEVR6, CLEVRTex) and real-world (GQA, VQA-v2) data, and introduces DINOSAURv2 (DINOv2 backbone with Slot Attention) to test adding an OC bias to a foundation model.
Key results
- Large foundation models (DINOv2, CLIP, MAE) without fine-tuning perform comparably or best across synthetic datasets, though not by a large margin, with T-15 downstream.
- DINOSAURv2 outperforms DINOv2 on CLEVRTex, Multi-dSprites and GQA with T-2 while requiring significantly less downstream compute; DINOv2 catches up at T-5 and beyond.
- Upstream MSE and ARI are not good predictors of downstream VQA accuracy, while property prediction accuracy correlates strongly with VQA performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply the object-centric Slot Attention bias to DINOv2 | DINOv2 (DINOSAURv2) · CLEVRTex | VQA accuracy | Helped |
| apply the object-centric Slot Attention bias to DINOv2 | DINOv2 (DINOSAURv2) · Multi-dSprites | VQA accuracy | Helped |
| apply the object-centric Slot Attention bias to DINOv2 | DINOv2 (DINOSAURv2) · CLEVR | VQA accuracy | Hurt |
| use DINOSAURv2 rather than DINOv2 | T-2 VQA model · CLEVRTex, Multi-dSprites, GQA | VQA accuracy | Helped |
| scale the downstream model from T-2 to T-5 or larger | DINOv2 VQA model · CLEVRTex, Multi-dSprites, GQA | VQA accuracy relative to DINOSAURv2 | No clear effect |
| increase training images from 40k to 320k | Most upstream models · Multi-dSprites | VQA error rate: typically around 20--40% decrease | Helped |
| increase training images from 40k to 320k | CLIP · Multi-dSprites | overall error rate: up to 50% | Hurt |
| increase training images from 40k to 320k | CNN, MultiCNN, and VQ-AE · Multi-dSprites | VQA performance | No clear effect |
| split the global VAE vector into K vectors (ablation) | Global VAE representation · VQA downstream task | VQA performance | Helped |
| apply a 2-layer MLP to the global vector before splitting it (ablation) | Global VAE representation · VQA downstream task | VQA performance | Hurt |
| use the global vector as one token (ablation) | Global VAE representation · VQA downstream task | VQA performance | Hurt |
| split the global vector into D_global sequences of size 1 (ablation) | Global VAE representation · VQA downstream task | VQA performance | Hurt |
Surprising findings
- Upstream reconstruction MSE and ARI do not predict downstream VQA performance; STEVE has the worst MSE among OC models yet second-best VQA accuracy on CLEVRTex.
- Even with a 20-layer downstream model, 3M steps and 320k images, a VAE's global representation cannot match OC models trained on 40k images.
Limitations
- Foundation models differ in pre-training objectives and data scale from OC models, complicating direct comparison.
- Study limited to static images; video not evaluated.
- DINOSAURv2 performance is highly sensitive to hyperparameters; STEVE training is unstable across seeds.
Experimental setup
Datasets: Multi-dSprites CLEVR CLEVR6 CLEVRTex VQA-v2 GQA MS COCO 2014 ImageNet-1K OpenImages v4
Benchmarks: CLEVR CLEVRTex Multi-dSprites GQA VQA-v2
Metrics: average VQA accuracy MSE ARI segmentation covering mean segmentation covering GFLOPs
Baselines: MONet SPACE Slot Attention ResNet Slot Attention STEVE DINOv2 MAE CLIP
Builds on: Slot Attention DINOSAUR STEVE DINOv2
Closest papers by meaning
- Are Object-Centric Representations Better at Compositional Generalization? (ICML 2026 · similarity 0.93)
- Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (CVPR 2017 · similarity 0.92)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.91)
- REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (NeurIPS 2022 · similarity 0.91)
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (CVPR 2019 · similarity 0.91)
- Object Attribute Matters in Visual Question Answering (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.