Research map / Large language models / Retrieval and knowledge
FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FlexKBQA uses an LLM as a program translator to turn automatically sampled KB programs into synthetic question-program pairs for training a lightweight KBQA parser, then applies execution-guided self-training and LLM inherent reasoning for few-shot KBQA.
Problem
Few-shot knowledge base question answering with scarce annotated data, where in-context learning with LLMs is limited by context window, inference cost, and difficulty with domain-specific KB entities and relations.
Approach
Reverse the usual LLM use: sample executable programs (SPARQL/S-expressions) from the KB and have an LLM translate them into natural language questions to build a synthetic training set for a lightweight parser; then iteratively self-train on unlabeled user questions with execution-based filtering and LLM answer agreement.
Key results
- On GrailQA with 25 shots, FlexKBQA reaches 62.8 EM / 69.4 F1 on the test set, above Pangu's 100-shot 62.7 F1 and about 93% of the supervised RnG-KBQA result.
- On WebQSP with 100 shots, FlexKBQA achieves 60.6 F1 versus 54.5 for Pangu.
- On KQA Pro with 100 shots, FlexKBQA reaches 46.83 accuracy versus 31.75 for LLM-ICL, but below supervised BART+SPARQL at 89.68.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add execution-guided self-training (EGST) (ablation) | FlexKBQA · GrailQA, 25 shots | F1: 57.7 -> 69.4; 10.3 | Helped |
| add execution-guided self-training (EGST) (ablation) | FlexKBQA · WebQSP, 100 shots | F1: 51.1 -> 60.6; 7.1 | Helped |
| add execution-guided self-training (EGST) (ablation) | FlexKBQA · KQA Pro, 100 shots | accuracy: 23.10 -> 46.83; 10.2 | Helped |
| add Inherent Reasoning (IR) (ablation) | FlexKBQA · GrailQA, 25 shots | F1: 68.0 -> 69.4; 1.4 | Helped |
| add Inherent Reasoning (IR) (ablation) | FlexKBQA · WebQSP, 100 shots | F1: 58.2 -> 60.6; 2.4 | Helped |
| add Inherent Reasoning (IR) (ablation) | FlexKBQA · KQA Pro, 100 shots | accuracy: 33.32 -> 46.83; 13.5 | Helped |
| increase in-context examples from 10 to 60 (ablation) | FlexKBQA data generation · GrailQA test | F1: 66.61; 67.20 | No clear effect |
| train with synthetic data rather than real data | RnG-KBQA ranker · WebQSP test | F1: 68.2 -> 35.5 | Hurt |
| use real programs with synthetic questions instead of real data | RnG-KBQA ranker · WebQSP test | F1: 68.2 -> 56.3 | Hurt |
| use Progressive Execution-Guided Self-Training (PEGST) instead of EGST (ablation) | FlexKBQA · WebQSP | F1 | Helped |
Surprising findings
- Synthetic data alone trained on WebQSP gives only 35.5 F1 versus 68.2 for real data, showing a large distribution gap.
- Varying the number of in-context examples from 10 to 60 had little effect on final performance.
Limitations
- Performance gap on KQA Pro because there is no entity linking stage, producing unexecutable programs for unseen entities or relations.
- Synthetic data distribution differs substantially from real questions, requiring self-training to close the gap.
- Zero-shot results are notably below few-shot results on WebQSP and KQA Pro.
Experimental setup
Datasets: GrailQA WebQSP KQA Pro
Benchmarks: GrailQA WebQSP KQA Pro
Metrics: EM F1 accuracy
Baselines: Pangu KB-BINDER Fine-Tuning LLM-ICL RnG-KBQA ArcaneQA DecAF ReTraCk
Builds on: RnG-KBQA KB-BINDER Pangu BART-SPARQL teaching via data
Closest papers by meaning
- Promoting Knowledge Base Question Answering by Directing LLMs to Generate Task-relevant Logical Forms (AAAI 2025 · similarity 0.91)
- Code-Style In-Context Learning for Knowledge-Based Question Answering (AAAI 2024 · similarity 0.90)
- Cost-efficient Knowledge-based Question Answering with Large Language Models (NeurIPS 2024 · similarity 0.90)
- KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search (ICML 2025 · similarity 0.90)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.89)
- KG-Guard: Graph-Based Hallucination Detection for Knowledge Base Question Answering (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.