Research map / Vision-language and multimodal / Visual question answering
Detection-Based Intermediate Supervision for Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIS, a detection-based intermediate supervision method for neural module network VQA that serializes multiple grounded objects, logical outcomes and answers into sequences decoded by a generative head, improving GQA accuracy and reasoning consistency.
Problem
Intermediate supervision in neural module networks for compositional VQA is sketchy: it assumes one grounded object per module and uses IoU-based supervision that introduces noisy signals from bounding-box overlap, hurting answer accuracy and reasoning consistency.
Approach
Replace IoU-based intermediate supervision with a generative detection framework: intermediate results obtained by symbolic execution on ground-truth scene graphs are serialized into token sequences (quantized box coordinates, True/False, answers) and decoded by a Transformer decoder with auto-regressive loss, providing supervision for multiple grounded objects; the decoder is discarded at test ti
Key results
- On GQA test2019 leaderboard, DIS single model reaches 61.31% overall accuracy vs. 60.83% for MMN.
- On GQA-Sub with data augmentation, DIS + DA reaches RC(1)=73.11%, RC(2)=64.20%, RC(3)=55.28%, above RCVQA + DA (71.47%, 61.94%, 52.80%).
- Without augmentation, DIS improves val accuracy from 65.05% (MMN) to 69.71% and val-sub from 64.46% to 70.31%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MMN's IoU-based intermediate supervision with detection-based sequence supervision | MMN-style VQA · GQA blind test2019 | Overall accuracy: 60.83 -> 61.31 | Helped |
| replace MMN's IoU-based intermediate supervision with detection-based sequence supervision | MMN-style VQA · GQA blind test2019 | Binary accuracy: 78.90 -> 79.36 | Helped |
| replace MMN's IoU-based intermediate supervision with detection-based sequence supervision | MMN-style VQA · GQA-Sub val | Acc: 65.05 -> 69.71 | Helped |
| replace MMN's IoU-based intermediate supervision with detection-based sequence supervision | MMN-style VQA · GQA-Sub | RC(1): 58.79 -> 64.98 | Helped |
| add sub-question data augmentation to DIS | DIS · GQA-Sub | RC(1): 64.98 -> 73.11 | Helped |
| add object names to bounding-box coordinate supervision (ablation) | DIS · GQA testdev | Accuracy: 59.97 -> 59.40 | Hurt |
| set DIS loss weight α to 0.50 instead of 0.00 (ablation) | DIS · GQA testdev | Accuracy: 58.75 -> 59.97 | Helped |
| increase bootstrapping epochs from 1 to 5 (ablation) | DIS · GQA testdev | Accuracy: 61.31 -> 60.59 | Hurt |
| quantize coordinates into 256 bins rather than 64 bins (ablation) | DIS · GQA testdev | Accuracy: 59.35 -> 59.97 | Helped |
| increase maximum generated objects from 1 to 4 (ablation) | DIS · GQA testdev | Accuracy: 59.16 -> 59.97 | Helped |
| shuffle the order of intermediate objects (ablation) | DIS · GQA testdev | Accuracy: 59.57 -> 59.97 | Helped |
Surprising findings
- Adding object names to the intermediate supervision format decreased accuracy (59.40% vs 59.97% with box coordinates only).
- Bootstrapping for more than 1 epoch on all-split reduced final accuracy (61.31% at 1 epoch vs 60.59% at 5 epochs).
Limitations
- Intermediate supervision requires ground-truth scene graphs, available only for GQA train and val splits, so DIS is used only in training.
- Symbolic reasoning rules are manually designed per function type.
- Gains on the blind test leaderboard over MMN are small (+0.46% binary, +0.47% open).
Experimental setup
Datasets: GQA GQA-Sub Visual Genome
Benchmarks: GQA GQA-Sub
Metrics: accuracy binary accuracy open accuracy reasoning consistency RC(k)
Baselines: BUTD MAC GRN LCGN BAN PVR LXMERT MCAN MMN RPR RCVQA
Builds on: MMN Pix2Seq MCAN T5 RCVQA
Closest papers by meaning
- Visual Question Reasoning on General Dependency Tree (CVPR 2018 · similarity 0.89)
- Separating Skills and Concepts for Novel Visual Question Answering (CVPR 2021 · similarity 0.87)
- Divide and Conquer: Answering Questions With Object Factorization and Compositional Reasoning (CVPR 2023 · similarity 0.87)
- Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models (ICLR 2025 · similarity 0.87)
- Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules (CVPR 2021 · similarity 0.87)
- SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.