Research map / Large language models / Retrieval and knowledge
SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SM3-Text-to-Query, a synthetic (Synthea/SNOMED-CT) medical text-to-query benchmark spanning PostgreSQL, MongoDB, Neo4j and GraphDB with 10K questions per query language, and evaluates in-context learning for four LLMs.
Problem
Text-to-Query systems have been evaluated only on single database models and query languages, so the effect of database model (relational, document, graph) and query language (SQL, MQL, Cypher, SPARQL) on LLM query generation for EHR data is unknown.
Approach
A template-based benchmark built from one synthetic SNOMED-CT-aligned patient dataset, loaded into four database systems with aligned content, so the same questions are paired with gold queries in SQL, MQL, Cypher and SPARQL, enabling cross-model comparison of ICL methods.
Key results
- Llama3-70b with schema and 5-shot achieves 57.50% EA on SQL, 57.07% on Cypher, 40.35% on MQL and 30.49% on SPARQL on the test set.
- BM25-based similarity-selected 5-shot examples with GPT-3.5 raise SQL EA to 88.55% and MQL to 78.70%, but SPARQL reaches only 75.75%.
- Schema information more than doubles 5-shot accuracy for SQL, Cypher and MQL, whereas it has little effect on SPARQL, suggesting LLMs know SNOMED CT from pretraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select 5 few-shot examples per question using BM25 similarity instead of fixed examples (ablation) | GPT-3.5 with schema information · SQL (PostgreSQL) | execution accuracy: +32.25 | Helped |
| Select 5 few-shot examples per question using BM25 similarity instead of fixed examples (ablation) | GPT-3.5 with schema information · SPARQL (GraphDB) | execution accuracy: +52.17 | Helped |
| Select 5 few-shot examples per question using BM25 similarity instead of fixed examples (ablation) | GPT-3.5 with schema information · Cypher (Neo4j) | execution accuracy: +37.18 | Helped |
| Select 5 few-shot examples per question using BM25 similarity instead of fixed examples (ablation) | GPT-3.5 with schema information · MQL (MongoDB) | execution accuracy: +43.64 | Helped |
| Add 5 few-shot examples to schema information (ablation) | Llama3-70b · SQL (PostgreSQL) test | execution accuracy: 47.05 -> 57.50 | Helped |
| Add 5 few-shot examples to schema information (ablation) | Llama3-70b · SPARQL (GraphDB) test | execution accuracy: 1.00 -> 30.49 | Helped |
| Add 5 few-shot examples to schema information (ablation) | Llama3-70b · Cypher (Neo4j) test | execution accuracy: 34.45 -> 57.07 | Helped |
| Add 5 few-shot examples to schema information (ablation) | Llama3-70b · MQL (MongoDB) test | execution accuracy: 21.55 -> 40.35 | Helped |
| Include database schema information in the prompt (ablation) | LLMs across SQL, Cypher, MQL, and SPARQL · SM3-Text-to-Query test | execution accuracy | Mixed |
Surprising findings
- Schema information has little effect on SPARQL accuracy, unlike the other query languages, which the authors attribute to LLM familiarity with SNOMED CT.
- Cypher outperforms MQL in some settings despite MQL being the more popular language on Stack Overflow by post count, which the authors attribute to Cypher queries being much shorter.
Limitations
- Questions are template-based and only guided by health professionals rather than written by them.
- Only English questions and database values are supported.
- SQL and Cypher queries are relatively simple (under 10 tokens on average), so complexity may be limited.
Experimental setup
Datasets: Synthea synthetic patient data (default config, 100 living and 10 deceased patients)
Benchmarks: SM3-Text-to-Query test set SM3-Text-to-Query dev set
Metrics: execution accuracy valid efficiency score
Baselines: Llama3-8b Llama3-70b Gemini 1.0 Pro GPT-3.5-turbo-0125
Builds on: EHRSQL MIMICSQL Synthea SNOMED CT BM25
Closest papers by meaning
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.87)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.87)
- EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries (NeurIPS 2024 · similarity 0.86)
- From Conversation to Query Execution: Benchmarking User and Tool Interactions for EHR Database Agents (ICLR 2026 · similarity 0.86)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.86)
- MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.