Research map / Large language models / Agents, code and math
GENOME: Generative Neuro-Symbolic Visual Reasoning by Growing and Reusing Modules
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GENOME, an LLM-driven neuro-symbolic visual reasoning framework that initializes, generates, tests, and reuses new visual modules from a few training examples.
Problem
LLM-based neuro-symbolic visual reasoning models (VisProg, ViperGPT) generate full code for every new instance without reusing or verifying modules, and require hand-defined modules for new tasks.
Approach
Use an LLM to decide whether existing modules suffice, propose new module signatures when needed, generate module code validated against few-shot training examples used as test cases, and add passing modules to a reusable library for program execution.
Key results
- On GQA, GENOME reaches 45.6 accuracy (Ours-Instruct) vs 45.4 for VisProg-Instruct and 38.2 for ViperGPT-Instruct; on RefCOCO it reaches 69.2 vs 62.4 for ViperGPT-Instruct.
- On transfer to image editing, GENOME scores 55.3 accuracy vs 16.7 for VisProg.
- On Raven, GENOME scores 80.1 on Center vs 58.1 for ResNet+DRT, but below ALANS-V (98.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add module learning rather than relying only on predefined modules (ablation) | GENOME on RefCOCO · RefCOCO testA | accuracy: 62.3 -> 69.2 | Helped |
| Replace gpt-3.5-turbo-instruct with WizardCoder-Python-34B-V1.0 (ablation) | GENOME on RefCOCO · RefCOCO testA | accuracy: 69.2 -> 64.4 | Hurt |
| Use 10 rather than 50 training examples (ablation) | GENOME on RefCOCO · RefCOCO testA | accuracy: 49.4 -> 67.0 | Hurt |
| Increase training examples from 10 to 50 (ablation) | GENOME on RefCOCO · RefCOCO testA | accuracy: 49.4 -> 67.0 | Helped |
| Increase training examples from 50 to 100 (ablation) | GENOME on RefCOCO · RefCOCO testA | accuracy: 67.0 -> 67.1 | No clear effect |
| Transfer modules learned on GQA and RefCOCO to image editing | GENOME versus VisProg · Image editing | accuracy: 16.7 -> 55.3 | Helped |
| Transfer modules learned on GQA and RefCOCO to knowledge tagging | GENOME versus VisProg · Knowledge tagging | F1: 19.9 -> 58.8 | Helped |
| Use GENOME instead of ViperGPT-CodeX | GENOME on GQA · GQA test-dev | accuracy: 48.1 -> 45.6 | Hurt |
| Use GENOME instead of ViperGPT-CodeX | GENOME on RefCOCO · RefCOCO testA | accuracy: 72.0 -> 69.2 | Hurt |
| Use GENOME for Raven relational reasoning | GENOME versus ALANS-V · Raven Center | accuracy: 98.4 -> 80.1 | Hurt |
| Use GENOME for MEWL color prediction | GENOME versus Flamingo-1.1B · MEWL color | accuracy: 35.3 -> 45.3 | Helped |
Surprising findings
- GENOME trails the fully supervised ALANS-V substantially on Raven, and trails ViperGPT-CodeX on GQA and RefCOCO despite its LLM swap.
- Ablation shows 10 training examples on RefCOCO yield 49.4, below the version without module learning (62.3), suggesting overfitting with few examples.
Limitations
- Requires task-specific prompts for each reasoning task.
- Relies on the capability of the underlying LLM; results depend on GPT-3.5-turbo-instruct or WizardCoder, and GPT-4 was not tested due to cost.
- Few-shot module learning can overfit when training examples are scarce (e.g., 10 RefCOCO examples).
Experimental setup
Datasets: GQA RefCOCO Raven MEWL
Benchmarks: GQA RefCOCO Raven MEWL image editing (new dataset) knowledge tagging (new dataset)
Metrics: accuracy precision recall F1 IoU-based localization accuracy BERT matching
Baselines: BLIP-2 KOSMOS-2 ViperGPT VisProg ResNet+DRT ALANS-V Aloe Flamingo
Builds on: VisProg ViperGPT Toolformer
Closest papers by meaning
- Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.89)
- Hint2Gen: Bridging Understanding and Generation via Code-structured Hints (CVPR 2026 · similarity 0.89)
- NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning (NeurIPS 2026 · similarity 0.89)
- NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language (ICLR 2026 · similarity 0.88)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.88)
- Thinking with Programming Vision: Towards a Unified View for Thinking with Images (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.