Research map / Vision-language and multimodal / Visual question answering
Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IPRM, a lightweight, fully differentiable neural reasoning module that combines iterative multi-step computation with parallel operation states for complex image and video VQA.
Problem
Complex visual reasoning and VQA requiring compositional multi-step processing and handling of independent parallel operations, which purely iterative or purely transformer-attention methods handle poorly.
Approach
A memory of parallel operation states and result states, updated over T iterative steps via operation formation, operation execution (feature-modulated visual attention), and operation composition (inter-operation attention over a look-back window), usable as a drop-in block on transformer or non-transformer VL backbones.
Key results
- IPRM reaches 69.9% average accuracy on STAR val (vs. 64.9% SeViLA-BLIP2) and 60.4% overall on AGQAv2, about 5% above prior video QA methods.
- On CLEVR-Humans IPRM achieves 63.8% zero-shot and 85.5% finetuned accuracy, improving finetuned accuracy over MDETR (81.7%) by 3.8%.
- On GQA with CLIP ViT-L/14, IPRM reaches 59.2% test-dev vs. 58.7% for Concat-Att (4L), with 5.2M added parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine iterative reasoning steps with parallel operations using IPRM | video VQA · STAR val | Avg. accuracy: 65.6 -> 69.9 | Helped |
| use IPRM instead of the strongest listed task-specific baseline | video VQA · AGQAv2 | Overall accuracy: 55.1 -> 60.4 | Helped |
| use IPRM rather than NS-DR | CLEVRER-Humans zero-shot · CLEVRER-Humans | Per-question accuracy: 32.0 -> 38.9 | Helped |
| use IPRM rather than ALOE | CLEVRER-Humans finetune · CLEVRER-Humans | Per-question accuracy: 31.7 -> 53.0 | Helped |
| use IPRM rather than MDETR | image reasoning · CLEVR-Humans finetune | Accuracy: 81.7 -> 85.5 | Helped |
| use IPRM rather than MDETR | compositional image reasoning · CLEVR-CoGenT ValB | Accuracy: 76.7 -> 80.3 | Helped |
| use IPRM rather than OSCAR | GQA · GQA | Accuracy: 61.6 -> 60.3 | Hurt |
| use reduction ratio r=1 instead of r=2 (ablation) | IPRM on CLEVR-Humans · CLEVR-Humans | Zero-shot accuracy: 63.3% -> 65.6% | Helped |
| use reduction ratio r=1 instead of r=2 (ablation) | IPRM on CLEVR-Humans · CLEVR-Humans | Finetuned accuracy: 85.4% -> 86.3% | Helped |
| add the operation composition block (OPC) (ablation) | IPRM with Nop=6 and T=9 · CLEVR-Humans reduced-resolution ablation | Accuracy: ~74% -> ~82% | Helped |
| increase reasoning steps from 6 to 9 with Nop=1 (ablation) | IPRM on STAR · STAR val | Avg. accuracy: 63.8 -> 68.4 | Helped |
| increase parallel operations from 1 to 6 at T=1 (ablation) | IPRM on STAR · STAR val | Avg. accuracy: 63.8 -> 67.7 | Helped |
Surprising findings
- Ablation shows the number of parallel operations and reasoning steps are co-dependent; setting T=1 gives about 75% regardless of N_op.
- IPRM performs less well than SeViLA-BLIP2 on STAR feasibility questions, which the authors attribute to needing commonsense knowledge.
Limitations
- May reflect biases from VQA benchmark training distributions and pretrained backbones.
- Performance drops on STAR feasibility questions relative to SeViLA-BLIP2, which needs commonsense knowledge.
- Predicted-visual-input setup on STAR shows a drop of about 9% relative to ground-truth vision.
Experimental setup
Datasets: STAR AGQAv2 CLEVRER-Humans CLEVR-Humans CLEVR CLEVR-CoGenT CLOSURE GQA NLVRv1 NLVRv2
Benchmarks: STAR AGQAv2 CLEVRER-Humans CLEVR-Humans CLEVR-CoGenT CLOSURE GQA NLVRv1 NLVRv2
Metrics: accuracy per-question accuracy per-option accuracy
Baselines: SeViLA-BLIP2 MIST InternVideo Concat-Att Cross-Att MDETR MAC FiLM NS-VQA ALOE NS-DR VRDP
Builds on: MAC FiLM MDETR SeViLA CLIP
Closest papers by meaning
- Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning" (ICML 2020 · similarity 0.90)
- V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices (AAAI 2020 · similarity 0.89)
- SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.88)
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.88)
- VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.88)
- Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.