Research map / Vision-language and multimodal / Human and embodied modeling
Learning Visual Abstract Reasoning through Dual-Stream Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DRNet, a dual-stream (CNN + ViT) encoder with a learnable fusion and rule extractor for solving Raven's Progressive Matrices, achieving state-of-the-art average accuracy across PGM, RAVEN, I-RAVEN and RAVEN-FAIR.
Problem
Visual abstract reasoning on Raven's Progressive Matrices (RPM), where deep networks struggle with relational rule inference and out-of-distribution generalization.
Approach
Inspired by the two-stream hypothesis, a parallel CNN (local/what) and ViT (spatial/where) encoder is fused by a learnable linear layer, and a 1D-conv rule extractor infers abstract rules from each context-candidate combination before an MLP scores candidates.
Key results
- DRNet reaches 97.79% average accuracy across PGM-Neutral, RAVEN, I-RAVEN and RAVEN-FAIR, versus 96.7% for PredRNet.
- On PGM out-of-distribution regimes DRNet averages 58.33% versus 47.1% for PredRNet, with up to 11.23% gains in OOD settings.
- Dual-stream DRNet scores 93.74% on PGM HO AP, while a single CNN stream with DA gets 62.87% and a single ViT stream with DA gets 72.26%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the DRNet dual-stream architecture instead of PredRNet | RPM reasoning · PGM Neutral | accuracy: 97.4 -> 99.06 | Helped |
| Use the DRNet dual-stream architecture instead of PredRNet | RPM reasoning · I-RAVEN | accuracy: 96.5 -> 97.62 | Helped |
| Use the DRNet dual-stream architecture instead of PredRNet | RPM reasoning · RAVEN-FAIR | accuracy: 97.1 -> 97.58 | Helped |
| Use DRNet rather than ARII | PGM Extrapolation · PGM Extrapolation | accuracy: 29.0 -> 22.22 | Hurt |
| Use DRNet rather than MRNet | PGM Held-Out Line Type · PGM Held-Out Line Type | accuracy: 30.1 -> 27.92 | Hurt |
| Use DRNet rather than MRNet | PGM Held-Out Shape Color · PGM Held-Out Shape Color | accuracy: 16.9 -> 13.09 | Hurt |
| Increase ViT depth from 12 to 16 (ablation) | DRNet · PGM Held-Out Attribute Pairs | accuracy: 93.7% to 89.7% | Hurt |
| Increase ViT depth from 12 to 16 (ablation) | DRNet · I-RAVEN | accuracy: 97.6 -> 96.2 | Hurt |
| Increase convolution kernel size from 3 to 7 (ablation) | DRNet · I-RAVEN | accuracy: 90.9 -> 97.6 | Helped |
| Increase convolution kernel size from 3 to 7 (ablation) | DRNet · PGM Held-Out Attribute Pairs | accuracy: 84.6 -> 93.7 | Helped |
| Add the CNN branch to ViT+DA (ablation) | DRNet · I-RAVEN | accuracy: 87.52 -> 97.62 | Helped |
| Add the CNN branch to ViT+DA (ablation) | DRNet · PGM Held-Out Attribute Pairs | accuracy: 72.26 -> 93.74 | Helped |
Surprising findings
- A dual-CNN network with 24M parameters reached higher I-RAVEN accuracy (98.24%) than DRNet but dropped to 73.78% on PGM HO AP.
- Increasing ViT depth to 16 decreased HO AP accuracy from 93.7% to 89.7%.
Limitations
- Performs poorly on RAVEN-style 3x3 grid configuration, attributed to convolution kernel size and large ViT patch size.
- Larger parameter count than baselines (24.6M), and more parameters do not necessarily improve performance.
- Single result reported for PGM due to computational demands; RAVEN results are median of 5 runs.
Experimental setup
Datasets: PGM RAVEN I-RAVEN RAVEN-FAIR
Benchmarks: PGM RAVEN I-RAVEN RAVEN-FAIR
Metrics: recognition accuracy
Baselines: WReN CoPINet MRNet SCL MLRN Rel-Base ARII STSN PredRNet NVSA MXGNet
Builds on: two-stream hypothesis Vision Transformer WReN Rel-Base
Closest papers by meaning
- ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both (NeurIPS 2026 · similarity 0.86)
- Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection (ICLR 2024 · similarity 0.85)
- Monet: Reasoning in Latent Visual Space Beyond Image and Language (CVPR 2026 · similarity 0.85)
- Beyond Task-Specific Reasoning: A Unified Conditional Generative Framework for Abstract Visual Reasoning (ICML 2025 · similarity 0.85)
- Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoning (AAAI 2024 · similarity 0.85)
- DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.