Research map / Large language models / Retrieval and knowledge

Cost-efficient Knowledge-based Question Answering with Large Language Models

NeurIPS 2024 · Large language models › Retrieval and knowledge · approach: Knowledge-augmented question answering · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Coke, a multi-armed bandit strategy that routes each KBQA question to either LLMs or small knowledge-graph models under a cost budget, combining cluster-level Thompson Sampling, contextual linear bandits, and a cost-regret penalty.

Problem

KBQA with LLMs is costly and LLMs lack domain-specific knowledge, while small KG-based models are cheap but weaker overall; no principled way exists to jointly trade off accuracy and API/call cost when choosing among models per question.

Approach

Model per-question model selection as a bandit: cluster-level Thompson Sampling chooses between the LLM cluster and the KGM cluster, a contextual linear bandit (RoBERTa question embeddings) picks the expert arm within the cluster, and a cost-regret term penalizes arms that waste budget on failed answers.

Key results

• Coke reaches 0.824 IHtest accuracy on CSQA and 0.908 test accuracy on OBQA versus GPT-4's 0.802 and 0.902, with up to 20.89% API fee savings vs GPT-4 on CSQA.
• On MedQA, Coke with HamQA, Llama2 and Llama3 arms achieves 0.692 test accuracy vs Llama3 8B's 0.691, with 0.66% fewer calls.
• Coke matches GPT-4 accuracy at roughly 60% of budget on CSQA and OBQA according to the budget study.

Measured effects reported in the paper

Surprising findings

• Removing the cost-regret term on CSQA yields only a small accuracy drop but lower cost savings, while removing the cluster expectation on CSQA and OBQA reduces cost savings by 30-48%.
• Llama2 (7b) and Llama3 (8b) underperform traditional KGMs on CSQA and OBQA, so LLM arms were only used on MedQA.

Limitations

• On MedQA, where KGMs are weak, the policy must call LLMs frequently, so cost savings are small (4.32% vs GPT-4).
• Accuracy-cost tradeoff depends on a hyperparameter lambda and budget B that must be tuned.
• Reported improvement figures are inconsistent in the text (e.g., 2.03% vs 2.56% on CSQA) and the 'Coke' rows report only per-dataset best results.

Experimental setup

Datasets: CommonsenseQA OpenBookQA MedQA-USMLE

Benchmarks: CommonsenseQA OpenBookQA MedQA-USMLE

Metrics: accuracy cost saving (API fees $) cost saving (calls) Pareto frontier error rate vs budget

Baselines: Bert-base Bert-large RoBERTa-large MHGRN QA-GNN HamQA JointLK GreaseLM GrapeQA GPT-3 GPT-3.5 GPT-4

Builds on: Thompson Sampling LinUCB-style contextual bandits QA-GNN HamQA

Closest papers by meaning

• FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering (AAAI 2024 · similarity 0.90)
• Code-Style In-Context Learning for Knowledge-Based Question Answering (AAAI 2024 · similarity 0.89)
• Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning (ICLR 2024 · similarity 0.89)
• Promoting Knowledge Base Question Answering by Directing LLMs to Generate Task-relevant Logical Forms (AAAI 2025 · similarity 0.89)
• KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search (ICML 2025 · similarity 0.89)
• Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.88)