Research map / Large language models / Retrieval and knowledge
Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes LLaMA-2 on complex logical queries over incomplete KGs using binary-tree decomposition of queries and a three-stage curriculum ordered by query difficulty, reporting new SOTA MRR on FB15K, FB15K-237 and NELL995.
Problem
Answering complex first-order logical (EFO-1) queries over incomplete knowledge graphs; embedding-based methods lack world knowledge and generalize poorly, while prior LLM approaches only prompt closed models.
Approach
Convert each complex query into a binary computation tree, serialize it into a stepwise decomposition chain for instruction tuning of an LLM, and order training samples by number of decomposed sub-queries using a curriculum (80% easy first stage, then medium/hard).
Key results
- LACT achieves avg_p MRR of 82.6 on FB15K, 57.0 on FB15K-237 and 60.1 on NELL995, outperforming LARK and QTO baselines.
- Ablation on FB15k hard queries: adding binary tree decomposition raises Llama2 IT accuracy from 94.6/68.8 (1p/2p) and e.g. 2i from 84.5 to 89.7; combining BTD and CL reaches 94.8 on 1p and 88.1 on 3i.
- Base-model scaling on FB15K-237 shows Mistral-7B and Qwen1.5-7B give comparable MRR to Llama2-7B-based LACT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LACT instead of the strongest listed baseline | Complex KG reasoning · FB15K | MRR (avg_p): 74.0 -> 82.6 | Helped |
| Use LACT instead of the strongest listed baseline | Complex KG reasoning · FB15K-237 | MRR (avg_p): 50.7 -> 57.0 | Helped |
| Use LACT instead of the strongest listed baseline | Complex KG reasoning · NELL995 | MRR (avg_p): 52.9 -> 60.1 | Helped |
| Add binary tree decomposition to instruction tuning (ablation) | LLaMA-2 + IT · FB15K hard queries, including 3p and 3i | accuracy: 60.2 -> 65.6; 44.8 -> 75.2 | Helped |
| Add curriculum learning to binary-tree-decomposed training (ablation) | LLaMA-2 + IT + BTD · FB15K hard queries, including 3p and 3i | accuracy: 65.6 -> 69.2; 75.2 -> 88.1 | Helped |
| Add curriculum learning without binary tree decomposition (ablation) | LLaMA-2 + IT · FB15K hard queries, including 3p and 3i | accuracy: 60.2 -> 61.3; 44.8 -> 72.7 | Helped |
| Train with curriculum learning rather than random-shuffle sequence training (ablation) | LACT decomposed training corpus · FB15K high-difficulty queries | PPL | Helped |
| Use curriculum learning on undecomposed training samples (ablation) | LACT without logical-chain decomposition · FB15K | training performance | No clear effect |
Surprising findings
- Curriculum learning gave very little gain on training data not decomposed into logical chains, suggesting the two components are coupled.
Limitations
- Samples exceeding 4096 tokens were discarded and scored as zero at evaluation.
- Difficulty is defined simply by the number of decomposed sub-queries, which is a heuristic.
- Evaluation on a relatively narrow set of standard KG benchmarks with 14 query types.
Experimental setup
Datasets: FB15K FB15K-237 NELL995
Benchmarks: FB15K FB15K-237 NELL995
Metrics: MRR accuracy
Baselines: GQE Query2Box BetaE CQD ConE GNN-QE QTO LARK ChatGPT (GPT-3.5-turbo)
Builds on: BetaE QTO LARK Llama 2 chain-of-thought
Closest papers by meaning
- Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning (ICLR 2024 · similarity 0.88)
- Rethinking Complex Queries on Knowledge Graphs with Neural Link Predictors (ICLR 2024 · similarity 0.88)
- Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models (ICML 2025 · similarity 0.88)
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (ICML 2025 · similarity 0.87)
- Debate on Graph: A Flexible and Reliable Reasoning Framework for Large Language Models (AAAI 2025 · similarity 0.87)
- Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.