Research map / Large language models / Retrieval and knowledge
LightPROF: A Lightweight Reasoning Framework for Large Language Model on Knowledge Graph
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LightPROF, a retrieve-embed-reason framework that trains only a Transformer/BERT-based Knowledge Adapter to map KG reasoning-path embeddings into soft prompts for a frozen small LLM (LLaMA2-7B / LLaMA3-8B) for KGQA.
Problem
LLM reasoning over knowledge graphs usually verbalizes KG content as text, which loses structural information, inflates input tokens, and often relies on large or closed LLMs for KGQA.
Approach
Retrieve relation-link-based reasoning graphs with a BERT-predicted hop count and LLM-based ranking, encode each reasoning path into a single token fusing textual and structural (head-relation-tail) information via a small adapter, and project it into the frozen LLM's embedding space as a soft prompt alongside hard text prompts.
Key results
- LightPROF (LLaMA3-8B) reaches 83.8% Hits@1 on WebQSP versus 75.1% for UniKGQA and 59.3% on CWQ versus 57.6% for ToG (LLaMA2-70B).
- LightPROF (LLaMA2-7B) reaches 71.2% on WebQSP and 48.5% on CWQ, versus 61.4% and 31.5% for plain LLaMA2-7B-Chat.
- On WebQSP efficiency comparison with StructGPT, total input tokens drop from 24,750,610 to 365,380 (about 98% reduction) and time cost from 1:42:12 to 1:11:49.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LightPROF instead of UniKGQA | LLaMa3-8B · WebQSP | Hits@1: 75.1 -> 83.8 | Helped |
| use LightPROF instead of ToG | LLaMa3-8B · CWQ | Hits@1: 57.6 -> 59.3 | Helped |
| add LightPROF to LLaMa2-7B | LLaMa2-7B · WebQSP | Hits@1: 61.36 -> 71.19 | Helped |
| add LightPROF to LLaMa2-7B | LLaMa2-7B · CWQ | Hits@1: 31.49 -> 48.48 | Helped |
| add LightPROF to LLaMa3-8B | LLaMa3-8B · WebQSP | Hits@1: 66.83 -> 83.77 | Helped |
| add LightPROF to LLaMa3-8B | LLaMa3-8B · CWQ | Hits@1: 48.87 -> 59.26 | Helped |
| add structural information to knowledge embedding (ablation) | LightPROF · WebQSP | Hits@1: 82.36 -> 83.77 | Helped |
| add structural information to knowledge embedding (ablation) | LightPROF · CWQ | Hits@1: 58.05 -> 59.26 | Helped |
| train the Knowledge Encoder (ablation) | LightPROF · WebQSP | Hits@1: 80.37 -> 83.77 | Helped |
| train the Knowledge Encoder (ablation) | LightPROF · CWQ | Hits@1: 55.63 -> 59.26 | Helped |
| use accurate retrieval instead of random reasoning-path retrieval (ablation) | LightPROF · WebQSP | Hits@1: 53.44 -> 83.77 | Helped |
| use accurate retrieval instead of random reasoning-path retrieval (ablation) | LightPROF · CWQ | Hits@1: 46.84 -> 59.26 | Helped |
Surprising findings
- Random retrieval of reasoning paths drops WebQSP Hits@1 from 83.77 to 53.44, showing retrieval quality dominates performance.
Limitations
- The adapter is trained per KG and does not generalize to unseen KG data without retraining, as the authors note.
- Comparisons are mixed: several baselines are zero-shot or differ in LLM size, and the StructGPT efficiency comparison is on one dataset only.
- Hardware is named (NVIDIA A800) but the GPU count and training time are not reported.
Experimental setup
Datasets: WebQSP CWQ Freebase
Benchmarks: WebQSP CWQ
Metrics: Hits@1 time cost total input tokens tokens per request
Baselines: KV-Mem EmbedKGQA NSM KGT5 GraftNet PullNet TransferNet UniKGQA LLaMA2-7B-Chat LLaMA2-70B-Chat ToG StructGPT
Builds on: KAPING ToG StructGPT UniKGQA
Closest papers by meaning
- Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR 2024 · similarity 0.91)
- Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models (ICML 2025 · similarity 0.91)
- Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning (ICLR 2024 · similarity 0.91)
- Debate on Graph: A Flexible and Reliable Reasoning Framework for Large Language Models (AAAI 2025 · similarity 0.89)
- A Prompt-Based Knowledge Graph Foundation Model for Universal In-Context Reasoning (NeurIPS 2024 · similarity 0.89)
- Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.