Research map / Vision-language and multimodal / Multimodal LLMs

NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes NePTune, a training-free neuro-symbolic framework where an LLM writes Python programs that execute soft fuzzy-logic operations over VLM yes/no probability scores, with optional LoRA neuro-symbolic fine-tuning, evaluated on CLEVR variants and referring-expression benchmarks.

Problem

VLMs struggle with compositional visual reasoning on novel combinations, while existing neuro-symbolic methods use crisp execution or are limited to predicates learned from training data, hurting zero-shot and domain-adaptation performance.

Approach

An LLM generates executable Python programs in which predicates are scored by a VLM (visual prompting with bounding boxes, P(Yes) from logits) and combined via differentiable fuzzy-logic operators (min/max, softmax iota, soft equality/inequality), mixing imperative control flow with soft composition; the same operations allow neuro-symbolic fine-tuning.

Key results

• On CLEVR, NePTune (zero-shot, InternVL2.5 backbone) reaches 92.65% vs 90.25% for the backbone and 36.05% for ViperGPT.
• On Ref-GTA, NePTune with InternVL2.5-8B reaches 69.69% vs 6.95% for the end-to-end InternVL2.5-8B and 58.73% for NAVER.
• On CLEVR-Humans, NePTune reaches 87.67% vs 85.95% for InternVL2.5-8B and 56.12% for NeSyCoCo.

Surprising findings

• NePTune degrades Qwen2VL-7B on CLEVR (93.55% to 82.55%), attributed to its weak analogical concept grounding (same color/shape/size).
• Standard fine-tuning of the VLM on Ref-GTA reaches only 4.19% while neuro-symbolic fine-tuning of the base framework yields much higher accuracy.

Limitations

• Performance depends on the quality of the LLM program generator and VLM grounder; logical errors in composition remain the main failure mode.
• Degrades performance on models weak at analogical relations (Qwen2VL on CLEVR).
• Relies on GPT-4o as judge for some CLEVR answer matching.

Experimental setup

Datasets: CLEVR CLEVR-Humans CLEVR-Ref CLEVR-Puzzles CLEVR-RPM RefCOCO-Adversarial Ref-GTA Visual Genome GQA

Benchmarks: CLEVR CLEVR-Humans CLEVR-Ref CLEVR-Puzzles CLEVR-RPM RefCOCO-Adversarial Ref-GTA

Metrics: accuracy IoU>=0.5 grounding accuracy micro F1 macro F1 execution success rate

Baselines: ViperGPT VisProg NeSyCoCo LEFT NAVER InternVL2.5-8B Qwen2VL-7B Ovis1.6-9B MDETR Florence2-L

Builds on: VisProg ViperGPT LEFT NeSyCoCo NAVER

Closest papers by meaning

• Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.91)
• NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization (AAAI 2025 · similarity 0.90)
• Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.90)
• Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models (AAAI 2026 · similarity 0.89)
• NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning (NeurIPS 2026 · similarity 0.89)
• GENOME: Generative Neuro-Symbolic Visual Reasoning by Growing and Reusing Modules (ICLR 2024 · similarity 0.88)