Research map / Large language models / Retrieval and knowledge
Promoting Knowledge Base Question Answering by Directing LLMs to Generate Task-relevant Logical Forms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Llama-2-7B with multi-task learning to generate both logical forms and the index of answer-relevant reasoning paths (retrieved by a T5 yes/no retriever) for KBQA, using the path output as fallback when LF are non-executable.
Problem
Semantic-parsing KBQA with LLMs produces many non-executable or inaccurate logical forms, and fine-tuning LLMs to generate answer-relevant LF conflicts with their prior tendency to generate many LF when many reasoning paths are retrieved.
Approach
Add a simple auxiliary objective where the LLM outputs the index of reasoning paths leading to correct answers (deductive verification of propositions), trained jointly with LF generation, so the LF task dominates while the LLM attends to answer-relevant paths; non-executable LF fall back to the selected paths.
Key results
- FM-KBQA reaches 84.24 F1 and 87.34 Hits@1 on WebQSP, and 68.68 F1 and 79.50 Hits@1 on CWQ, outperforming FiDeLiS (gpt-4) as the best baseline.
- Ablation: LF+RP combination gives 84.24 F1 on WQSP vs 77.29 for LF-only and 79.96 for RP-only; on CWQ 68.68 vs 53.01 and 55.16.
- Adding the reasoning-path auxiliary task reduces non-executable LF on WQSP from 14.39% (LF-KBQA) to 8.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FM-KBQA instead of FiDeLiS | KBQA · WebQSP | F1: 78.32 -> 84.24 | Helped |
| use FM-KBQA instead of FiDeLiS | KBQA · WebQSP | Hits@1: 84.39 -> 87.34 | Helped |
| use FM-KBQA instead of FiDeLiS | KBQA · CWQ | F1: 64.32 -> 68.68 | Helped |
| use FM-KBQA instead of FiDeLiS | KBQA · CWQ | Hits@1: 71.47 -> 79.50 | Helped |
| add reasoning-path selection to logical-form generation (ablation) | FM-KBQA · WQSP | F1: 77.29 -> 84.24 | Helped |
| add reasoning-path selection to logical-form generation (ablation) | FM-KBQA · CWQ | F1: 53.01 -> 68.68 | Helped |
| add logical-form generation to reasoning-path selection (ablation) | FM-KBQA · WQSP | F1: 79.96 -> 84.24 | Helped |
| add logical-form generation to reasoning-path selection (ablation) | FM-KBQA · CWQ | F1: 55.16 -> 68.68 | Helped |
| add the auxiliary path-selection task during fine-tuning (ablation) | LF-KBQA · WQSP | non-executable LF%: 14.39 -> 8.7 | Helped |
Surprising findings
- Fine-tuning LLMs with retrieved context (queries plus retrieved paths) underperforms fine-tuning on queries alone for logical form generation.
- Prompting GPT-4 with the question alone performs poorly on KBQA (e.g., 59.71 F1 on WebQSP zero-shot), indicating LLMs lack KG knowledge.
Limitations
- CWQ non-executable LF rate remains high (33.08%), and reasoning-path fallback only partly compensates.
- Relies on pre-identified question entities and GPT-3.5-generated conclusions for training data construction.
- Evaluated only on two Freebase-based benchmarks with a single 7B LLM backbone.
Experimental setup
Datasets: WebQuestionsSP ComplexWebQuestions
Benchmarks: WebQuestionsSP ComplexWebQuestions
Metrics: F1 Hits@1
Baselines: Rigel TIARA UniK-QA UniKGQA HGNet GPT-4 zero-shot GPT-4 few-shot GPT-4 CoT ToG FiDeLiS NSM DeCAF
Builds on: ChatKBQA DECAF FiDeLiS RoG T5
Closest papers by meaning
- Code-Style In-Context Learning for Knowledge-Based Question Answering (AAAI 2024 · similarity 0.92)
- KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search (ICML 2025 · similarity 0.91)
- FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering (AAAI 2024 · similarity 0.91)
- Cost-efficient Knowledge-based Question Answering with Large Language Models (NeurIPS 2024 · similarity 0.89)
- Explore What LLM Does Not Know in Complex Question Answering (AAAI 2025 · similarity 0.88)
- KG-Guard: Graph-Based Hallucination Detection for Knowledge Base Question Answering (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.