Research map / Vision-language and multimodal / Multimodal LLMs
On the generalization capacity of neural networks during generic multimodal reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces gCOG, a configurable synthetic multimodal QA benchmark, and trains six encoder-only architectures from scratch to test distractor, systematic, and productive compositional generalization.
Problem
Unclear whether neural architectures (RNNs, GRUs, Transformers, Perceivers) generalize OOD on controlled multimodal reasoning tasks, and which architectural components drive distractor, systematic, and productive generalization.
Approach
A generic, modality-agnostic extension of the COG task with categorical tokens and arbitrarily deep task trees, with splits for distractor, systematic, and productive OOD generalization; models trained from scratch to isolate architecture effects.
Key results
- Models with cross-attention (CrossAttn and Perceiver) achieved the best OOD distractor generalization, with the other models degrading as distractors increased from 10 to 20.
- On depth-3 systematicity, the Perceiver reached 75.4% IID and 65.7% OOD accuracy, versus next best 59.2% IID (CrossAttn) and 54.1% OOD (RNN).
- All models failed productive generalization to task trees of depth 5 and 7, performing at or below chance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cross-attention to integrate task rules and stimuli | CrossAttn and Perceiver · gCOG distractor generalization, OOD | accuracy | Helped |
| Use cross-attention to integrate task rules and stimuli | CrossAttn and Perceiver · gCOG systematic generalization, individual operators | accuracy: >97% | Helped |
| Use a latent Transformer with cross-attention | Perceiver · gCOG systematic generalization, depth 3, IID | accuracy: 75.4% | Helped |
| Use a latent Transformer with cross-attention | Perceiver · gCOG systematic generalization, depth 3, OOD | accuracy: 65.7% | Helped |
| Scale the model by increasing encoder layers (ablation) | BERT-like SSTfmr · gCOG distractor generalization | accuracy | Helped |
| Increase attention heads per layer (ablation) | BERT-like SSTfmr · gCOG distractor generalization | accuracy | Helped |
| Scale the model by increasing encoder layers (ablation) | BERT-like SSTfmr · gCOG systematic generalization, depth 1 and depth 3 | accuracy | Helped |
| Increase attention heads per layer (ablation) | BERT-like SSTfmr · gCOG systematic generalization, depth 1 and depth 3 | accuracy | Helped |
| Scale the model by increasing encoder depth and attention heads (ablation) | BERT-like SSTfmr · gCOG productive generalization, depth 5 and 7 | accuracy | No clear effect |
| Replace absolute positional encoding with relative positional encoding (ablation) | Transformer models · gCOG productive generalization, depth 5 and 7 | accuracy | No clear effect |
| Use relative positional encoding (ablation) | Transformer models · gCOG systematic generalization, depth 3 | accuracy | Mixed |
| Use cross-attention or deeper attention layers | Neural architectures · gCOG productive compositional generalization | accuracy | No clear effect |
Surprising findings
- Scaling encoder depth and heads improved distractor and systematic generalization but gave no gain in productive generalization.
- Relative positional encodings did not improve productive generalization on gCOG, contrary to prior work.
Limitations
- All evaluations use categorical task encodings rather than image pixels and natural language, limiting direct multimodal realism.
- Models are trained from scratch and small-scale; findings may not transfer to large pretrained models.
- No productive generalization was achieved by any architecture, so the benchmark's productivity split remains unsolved by the tested models.
Experimental setup
Datasets: gCOG
Benchmarks: gCOG distractor split gCOG systematicity split gCOG productivity split
Metrics: accuracy chance-normalized accuracy
Baselines: RNN GRU SSTfmr DSTfmr CrossAttn Perceiver
Builds on: COG gSCAN CLOSURE Perceiver BERT
Closest papers by meaning
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.90)
- Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts (CVPR 2024 · similarity 0.87)
- Head Pursuit: Probing Attention Specialization in Multimodal Transformers (NeurIPS 2025 · similarity 0.87)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.87)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.87)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.