Research map / Large language models / Retrieval and knowledge
Code-Style In-Context Learning for Knowledge-Based Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KB-Coder, a training-free KBQA method that reformulates S-Expression logical form generation as Python meta-function call generation under code-style in-context learning with GPT-3.5-turbo, plus a retrieved related relation hint for zero-shot generalization.
Problem
LLMs rarely see logical forms in pre-training, so in-context semantic parsing for KBQA suffers high format error rates, and zero-shot generalization to unseen schema items is poor.
Approach
Define seven Python meta-functions implementing S-Expression operations and prompt the LLM to generate step-by-step function call sequences (code) from few-shot demos; additionally insert one retrieved KB relation most similar to the question as a hint, and link entities/relations via SimCSE+Faiss with candidate enumeration.
Key results
- KB-Coder(6)-R reaches 61.3 overall EM and 75.2 F1 on the GrailQA-style 40-shot setting (table reports overall EM 56.3 for KB-Coder(6)-R, 61.3 F1), versus 58.5 F1 for KB-BINDER(6)-R.
- On GraphQ 100-shot, KB-Coder(6) lowers format error rate to 0.6% from KB-BINDER(6)'s 2.8% and raises F1 to 35.8 from 34.5.
- On WebQSP 100-shot, KB-Coder(6)-R achieves 75.2 F1 versus 71.1 for KB-BINDER(6)-R, with FER 0.3% vs 0.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use code-style function-call generation instead of directly generating logical forms | KB-Coder vs KB-BINDER · WebQSP (100-shot) | F1: 52.6(1.1) -> 55.7(1.3) | Helped |
| Use code-style function-call generation instead of directly generating logical forms | KB-Coder vs KB-BINDER · GraphQ (100-shot) | FER: 15.4(1.6) -> 6.3(2.5) | Helped |
| Provide one question-related relation as a reference | KB-Coder · GrailQA zero-shot | F1 | Helped |
| Provide one question-related relation as a reference (ablation) | KB-Coder · GrailQA I.I.D and compositional | F1 | No clear effect |
| Add instruction definitions for the meta-functions (ablation) | KB-Coder · GrailQA | F1 | Helped |
| Add instruction definitions for the meta-functions (ablation) | KB-Coder · GrailQA | FER | Helped |
| Include demonstration examples (ablation) | KB-Coder · GrailQA | F1 | Helped |
| Increase the number of related relations beyond one | KB-Coder · GrailQA subset | F1 | Hurt |
| Increase the number of demo examples | KB-Coder · GrailQA subset | F1 | Helped |
| Increase the number of answers participating in majority vote | KB-Coder · GrailQA subset | FER | Helped |
| Select the highest-F1 candidate instead of the first candidate with a non-empty query | KB-Coder (6)-R · GraphQ | F1: 36.6 to 50.2 | Helped |
Surprising findings
- Entity and relation linking strategy strongly limits performance: choosing the best-F1 candidate instead of the first executable one raises KB-Coder(6)-R on GraphQ from 36.6 to 50.2 F1.
- Adding more than one retrieved related relation degrades performance, so a single relation works best.
Limitations
- Performance fluctuates more than KB-BINDER across runs, per the authors.
- Relies on API access to a deprecated/closed model (gpt-3.5-turbo), and entity/relation linking remains a bottleneck.
- Zero-shot gains depend on retrieving a relation hint; more relations hurt performance.
Experimental setup
Datasets: WebQSP GrailQA GraphQ
Benchmarks: WebQSP GrailQA GraphQ
Metrics: F1 EM format error rate
Baselines: KB-BINDER RnG-KBQA DecAF TIARA ArcaneQA SPARQA BERT + Ranking
Builds on: KB-BINDER S-Expression SimCSE Faiss
Closest papers by meaning
- Promoting Knowledge Base Question Answering by Directing LLMs to Generate Task-relevant Logical Forms (AAAI 2025 · similarity 0.92)
- FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering (AAAI 2024 · similarity 0.90)
- Cost-efficient Knowledge-based Question Answering with Large Language Models (NeurIPS 2024 · similarity 0.89)
- InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models (NeurIPS 2024 · similarity 0.89)
- KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search (ICML 2025 · similarity 0.88)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.