Research map / Large language models / Agents, code and math
Inductive Learning of Logical Theories with LLMs: A Expressivity-graded Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an iterative LLM-plus-Prolog feedback loop to induce logic theories on RuDaS-generated datasets graded by rule expressivity, comparing GPT-4o, GPT-3.5, Llama3-8B, Mixtral and Gemma against the Popper ILP system.
Problem
Understanding how well LLMs can perform inductive logic theory induction across rule-expressivity levels and noise, relative to a state-of-the-art ILP system, since their inductive properties are not well quantified.
Approach
Combine stock LLMs (zero-shot prompting with iterative refinement) with a Prolog interpreter that scores induced theories on TP/FP/FN/TN and feeds misclassified examples back as critique, evaluated on synthetic datasets graded by rule dependency category (Chain, RDG, DRDG, Mixed, with recursion).
Key results
- GPT-4o reaches F1 0.75 on RDG at noise 0.1 but drops to 0.22 at noise 0.3, while Popper achieves F1 0.74 on RDG at noise 0.1 and 0.50 at noise 0.3.
- The paper reports the largest LLMs achieve F1 within about 0.25 of Popper on average, with no LLM beating Popper on CHAIN at all noise levels.
- Llama3-8B produced F1 of 0.00 on most CHAIN and RDG settings at noise 0.1, and Gemma-7B-it produced no valid theories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o instead of Popper | Theory induction · CHAIN, all noise levels | F1 | Hurt |
| Increase noise from 0.1 to 0.3 | GPT-4o · RDG | F1: 0.75 to 0.22 | Hurt |
| Increase noise from 0.1 to 0.3 | GPT-3.5-Turbo · RDG | F1: 0.56 to 0.0 | Hurt |
| Increase noise from 0.1 to 0.3 | Mixtral-8x7B · RDG | F1: 0.60 to 0.0 | Hurt |
| Increase iteration limit from 1 to 4 (ablation) | LLM iterative theory refinement · Logic theory induction | evaluation metrics | Mixed |
| Increase iteration limit beyond 3 (ablation) | LLM iterative theory refinement · Logic theory induction | overall performance | No clear effect |
| Increase noise from 0.1 to 0.2 | Mixtral-8x7B · All categories | average F1: over 0.8 | Hurt |
| Use GPT-4o instead of GPT-3.5-Turbo | LLM theory induction · Noise 0.1 | average F1: almost twice | Helped |
| Use GPT-4o instead of Llama3-8B | LLM theory induction · Noise 0.1 | F1: 21 times higher | Helped |
| Use iterative LLM theory induction instead of Popper | Theory induction · All classes and tested noise intervals | processing time: about 3 orders of magnitude lower | Helped |
| Use Gemma-7B-It for theory generation | LLM theory induction · All tested categories | valid theory generation | Hurt |
| Use GPT-4o instead of Mixtral-8x7B | LLM theory induction · Noise 0.3 | F1 | Mixed |
Surprising findings
- Long predicate relationship chains (CHAIN category) were harder for LLMs than higher dependency complexity such as disjunction or recursion.
- Model size did not reliably correlate with noise resilience: Llama3-8B occasionally beat GPT-3.5 and Mixtral, and scored higher at intermediate noise than at low noise.
Limitations
- Evaluation limited to the category typology of RuDaS and does not capture other ruleset properties such as workspace size or unification rate.
- Only five datasets per category at XS size; results are relative capability assessments, not SOTA performance.
- Recursive rules were removed from LLM-induced theories before evaluation.
Experimental setup
Datasets: RuDaS-generated synthetic logic datasets (105 datasets across 7 rule categories and 3 noise levels)
Benchmarks: RuDaS synthetic theory induction datasets (CHAIN, CHAIN REC., RDG, RDG REC., DRDG, DRDG REC., MIXED)
Metrics: F1 accuracy precision recall processing time
Baselines: Popper GPT-3.5-Turbo Llama3-8B-Instruct Mixtral-8x7B-Instruct-v0.1 Gemma-7B-it
Builds on: Popper Self-Refine RuDaS
Closest papers by meaning
- Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus (NeurIPS 2024 · similarity 0.90)
- Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · similarity 0.89)
- Logically Consistent Language Models via Neuro-Symbolic Integration (ICLR 2025 · similarity 0.88)
- Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement (ICLR 2024 · similarity 0.88)
- MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models (ICLR 2025 · similarity 0.88)
- RuAG: Learned-rule-augmented Generation for Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.