Research map / Vision-language and multimodal / Human and embodied modeling
Relational Programming with Foundational Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vieira, a declarative probabilistic relational framework (extending Scallop) that treats foundation models as stateless relational functions via a foreign interface with plugins, evaluated on 9 no-training tasks.
Problem
Foundation models are powerful but incomplete (hallucination, poor structured-data reasoning, awkward multi-modal composition), and existing augmentation mechanisms lack a unified programming framework.
Approach
Model foundation models as stateless functions with relational inputs and outputs, exposed via a foreign predicate/attribute interface in the Scallop probabilistic Datalog compiler, so LLM/vision models compose with logic programs and probabilistic reasoning without training.
Key results
- Vieira reaches 100% on tracking shuffled objects vs 84% for GPT-4 CoT and 92.41% on date reasoning vs 87.26% for GPT-4 CoT.
- On GQA, Recall@1 of 0.579 vs 0.419 for PNP-VQA (previous zero-shot SOTA) and on CLEVR Recall@1 0.463 vs 0.241 for ViLT-VQA.
- On HotpotQA Vieira gets 67.3% EM without fine-tuning, a few points below fine-tuned C2FM (72.07%) and FE2H (71.89%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Relational program over GPT-4 extracted relations instead of direct GPT-4 answering (few-shot shots contain only extracted facts) | Date reasoning · DR (BIG-bench date understanding) | accuracy (%): 92.41 vs 87.26 for GPT-4 CoT | Helped |
| Symbolic relational reasoning over GPT-4-extracted possessions and swaps | Tracking shuffled objects · TSO | accuracy (%): 100.00 vs 84.00 for GPT-4 CoT | Helped |
| Kinship graph extraction with GPT-4 plus external kinship composition rules | CLUTRR kinship reasoning · CLUTRR | accuracy (%): 72.50 vs 43.10 for GPT-4 3-shot | Helped |
| GPT-4 semantic parsing into DSL expressions evaluated symbolically, instead of direct LM answering | GSM8K math reasoning · GSM8K | accuracy (%): 90.60 vs 92.00 for GPT-4 CoT (5-shot); vs 87.10 0-shot | Hurt |
| Retrieval (embedding top-2 documents) followed by GPT QA, without fine-tuning | HotpotQA · HotpotQA distractor | EM (%): 67.3% vs 71.89-72.07% for fine-tuned methods | Hurt |
| Semantic search with positive/negative query decomposition using ada-002 embeddings | Amazon ESCI product search · Amazon ESCI | nDCG: 0.798 vs 0.797 for MIPS (ada-002), essentially tied; below fine-tuned 0.830/0.857 | Helped |
| VisProg-style DSL with OWL-ViT, ViLT, and CLIP modules, with GPT-4 semantic parsing | GQA visual QA · GQA | Recall@1: 0.579 vs 0.419 for PNP-VQA; 0.049 for ViLT-VQA | Helped |
| DSL with OWL-ViT and CLIP scene graph plus logical query evaluation | CLEVR visual QA · CLEVR | Recall@1: 0.463 vs 0.241 for ViLT-VQA | Helped |
Surprising findings
- On GSM8K, Vieira (90.60%) scores slightly below GPT-4 CoT (92.00%), which the authors attribute to heavier demands on GPT-4 for structured extraction.
Limitations
- Relies on GPT-4 for semantic parsing and relation extraction, so performance depends on LM extraction quality.
- Evaluation is no-training and in-context only; end-to-end fine-tuning is future work.
- Several image-task results rely on manual inspection of small samples (50 and 20 items).
Experimental setup
Datasets: BIG-bench date understanding BIG-bench tracking shuffled objects CLUTRR GSM8K HotpotQA Amazon ESCI GQA CLEVR VQAR OFCP IGP20
Benchmarks: BIG-bench date understanding BIG-bench tracking shuffled objects CLUTRR GSM8K HotpotQA Amazon ESCI GQA CLEVR VQAR OFCP IGP20
Metrics: exact match nDCG Recall@1 Recall@3 manual semantic correctness rate lines of code
Baselines: GPT-4 zero-shot GPT-4 chain-of-thought ViLT-VQA PNP-VQA C2FM FE2H MIPS with text-embedding-ada-002 InstructPix2Pix BERT CE-MPNet
Builds on: Scallop DeepProbLog VisProg FE2H Prompt-to-Prompt
Closest papers by meaning
- When No Paths Lead to Rome: Benchmarking Systematic Neural Relational Reasoning (NeurIPS 2025 · similarity 0.85)
- Relational Neurosymbolic Markov Models (AAAI 2025 · similarity 0.84)
- Right for the Right Reasons: Avoiding Reasoning Shortcuts via Prototypical Neurosymbolic AI (NeurIPS 2025 · similarity 0.84)
- Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models (ICML 2025 · similarity 0.84)
- REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Serving (NeurIPS 2025 · similarity 0.83)
- Logically Consistent Language Models via Neuro-Symbolic Integration (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.