Research map / Large language models / Agents, code and math
RuAG: Learned-rule-augmented Generation for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RuAG, which uses LLM-formulated predicates and MCTS to distill offline data into first-order logic rules that are translated to natural language and injected into LLM prompts, evaluated on relation extraction, log anomaly detection, a cooperative game, and an industrial fraud task.
Problem
LLM reasoning needs domain knowledge from large external datasets, but ICL and RAG are limited by context window size, and fine-tuning and knowledge graphs are costly or hard to scale.
Approach
LLMs define target and body predicates from commonsense to shrink the search space, MCTS searches for first-order logic rules over data, and the resulting rules are translated into natural language and placed in prompts instead of raw demonstrations.
Key results
- On DWIE relation extraction with GPT-4, the method reaches 60.42% F1 versus 52.59% for HtT (GPT-4) and 49.84% for BERT.
- On HDFS log anomaly detection, the method reaches 92.59% F1, above LogRobust (87.31%) and LogGPT (72.56%).
- On the private UPA fraud task, the rule-augmented LLM reaches 96% F1 versus 22.64% for vanilla LLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace HtT with learned-rule-augmented generation | GPT-4 relation extraction on DWIE · DWIE test | F1: 52.59\% -> 60.42\% | Helped |
| use learned-rule-augmented generation instead of LogRobust | HDFS anomaly detection · HDFS test | F1: 87.31\% -> 92.59 | Helped |
| use RuAG instead of the Grounded policy | Alice&Bob · Alice&Bob | WR: 0.9 -> 0.7 | Hurt |
| add MCTS-extracted rule guidance | GPT-4 Unauthorized Party Abuse detection · Unauthorized Party Abuse | F1: 22.64\% -> 96\% | Helped |
| add CoT without rules (ablation) | GPT-3.5 relation extraction · DWIE | F1: 18.94\% -> 19.85\% | Helped |
| add rule guidance (ablation) | GPT-3.5 relation extraction · DWIE | F1: 18.94\% -> 26.63\% | Helped |
| add CoT without rules (ablation) | GPT-4 HDFS anomaly detection · HDFS | F1: 60.10\% -> 76.11\% | Helped |
| add rule guidance (ablation) | GPT-4 HDFS anomaly detection · HDFS | F1: 60.10\% -> 92.59\% | Helped |
| add CoT without rules (ablation) | GPT-4 Alice&Bob · Alice&Bob | WR: 0.0 -> 0.0 | No clear effect |
| add rule guidance (ablation) | GPT-4 Alice&Bob · Alice&Bob | WR: 0.0 -> 0.7 | Helped |
| increase MCTS search episodes from 50 to 1000 (ablation) | relation extraction rule search · Relation extraction rules | Precision: 100\% -> 95.65\% | Hurt |
| increase MCTS search episodes from 50 to 500 (ablation) | anomaly detection rule search · Anomaly detection rules | F1: 65.75\%\% -> 91.30\% | Helped |
Surprising findings
- Chain-of-thought alone sometimes lowers game performance (GPT-4 CoT win rate 0 vs vanilla 0), showing domain knowledge matters more than reasoning prompts.
- Increasing MCTS episodes beyond a point yielded lower precision in relation extraction (95.65% at 1000 episodes vs 100% at 500).
Limitations
- Rule quality and coverage depend on the LLM-suggested predicates and the Gini-based discretization of continuous features.
- Evaluated on a small set of tasks, including a private industrial dataset that is not released, with limited test sizes (e.g., 97 test documents).
Experimental setup
Datasets: DWIE HDFS Alice&Bob trajectories private UPA fraud dataset
Benchmarks: DWIE HDFS Alice&Bob private UPA abuse detection
Metrics: F1 precision recall accumulative reward win rate average episode length
Baselines: CNN BiLSTM BERT Context-Aware HtT LogCluster DeepLog LogRobust LogGPT Behavior Cloning Offline Tabular Q PLLB
Builds on: Monte Carlo Tree Search HtT ICL RAG
Closest papers by meaning
- Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus (NeurIPS 2024 · similarity 0.88)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.88)
- LLM-DR: A Novel LLM-Aided Diffusion Model for Rule Generation on Temporal Knowledge Graphs (AAAI 2025 · similarity 0.88)
- Boosting the Potential of Large Language Models with an Intelligent Information Assistant (NeurIPS 2024 · similarity 0.88)
- Automated Creation of Reusable and Diverse Toolsets for Enhancing LLM Reasoning (AAAI 2025 · similarity 0.87)
- Inductive Learning of Logical Theories with LLMs: A Expressivity-graded Analysis (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.