Research map / Large language models / Agents, code and math
Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Matchmaker, a zero-shot compositional LLM program for schema matching (retrieval plus reasoning candidate generation, refinement, MCQ confidence scoring) that self-improves via bootstrapped synthetic in-context demonstrations selected by an LLM evaluator.
Problem
Schema matching across heterogeneous databases without labeled data; prior ML methods need heavy supervision and LLM approaches have poor zero-shot accuracy and scale poorly with many LLM calls.
Approach
Formulate schema matching as information retrieval with a multi-stage LLM program (diverse candidate generation, LLM refinement, MCQ-style confidence scoring with an abstain option), and optimize it zero-shot by bootstrapping self-generated traces scored by an LLM evaluator into in-context demonstrations.
Key results
- On MIMIC-OMOP, Matchmaker reaches 62.20 acc@1 versus 42.50 for ReMatch and 29.59 for LLM-DP; on Synthea-OMOP it reaches 70.20 acc@1 versus 50.50 for ReMatch.
- Self-improvement via optimized synthetic demonstrations improves MIMIC acc@1 from 57.90 (No-IC) to 62.20, outperforming random demonstrations (55.36) and self-reflection (57.10).
- On WikiData Matchmaker scores 0.95 versus 0.84 for ReMatch; on Magellan both achieve 1.00.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Matchmaker instead of ReMatch | Schema matching · MIMIC-OMOP | accuracy@1: 42.50 -> 62.20 ± 2.40 | Helped |
| use Matchmaker instead of ReMatch | Schema matching · Synthea-OMOP | accuracy@1: 50.50 -> 70.20 ± 1.70 | Helped |
| systematically select synthetic in-context examples rather than selecting them randomly (ablation) | Matchmaker · MIMIC-OMOP | accuracy@1: 55.36 ± 2.15 -> 62.20 ± 2.40 | Helped |
| add systematically selected in-context examples (ablation) | Matchmaker · MIMIC-OMOP | accuracy@1: 57.90 ± 1.20 -> 62.20 ± 2.40 | Helped |
| use systematic example selection rather than self-reflection (ablation) | Matchmaker · MIMIC-OMOP | accuracy@1: 57.10 ± 0.60 -> 62.20 ± 2.40 | Helped |
| systematically select synthetic in-context examples rather than selecting them randomly (ablation) | Matchmaker · Synthea-OMOP | accuracy@1: 67.76 ± 1.38 -> 70.20 ± 1.70 | Helped |
| add systematically selected in-context examples (ablation) | Matchmaker · Synthea-OMOP | accuracy@1: 65.40 ± 0.90 -> 70.20 ± 1.70 | Helped |
| use systematic example selection rather than self-reflection (ablation) | Matchmaker · Synthea-OMOP | accuracy@1: 67.80 ± 1.40 -> 70.20 ± 1.70 | Helped |
Surprising findings
- Semantic similarity retrieval alone is insufficient for schema matching, with low accuracy@k in the authors' analysis.
- On the non-healthcare Magellan dataset, the gains are negligible because both methods saturate at 1.00.
Limitations
- Not a complete automation solution; requires human-in-the-loop for reliability.
- Only represents one component of the broader data harmonization process.
- ReMatch results are reported from the original paper because its code was unavailable.
Experimental setup
Datasets: MIMIC-OMOP Synthea-OMOP Magellan WikiData
Benchmarks: MIMIC-OMOP Synthea-OMOP Magellan WikiData
Metrics: accuracy@1 accuracy@3 accuracy@5 F1-score
Baselines: ReMatch LLM-DP Jellyfish-13b Jellyfish-7b SMAT Matchmaker (No-IC) Matchmaker (Random) Matchmaker (Self-reflection)
Builds on: DSPy bootstrapping ColBERT-v2 chain-of-thought prompting ReMatch
Closest papers by meaning
- Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale (ICML 2025 · similarity 0.84)
- It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.84)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.83)
- Induced Model Matching: Restricted Models Help Train Full-Featured Models (NeurIPS 2024 · similarity 0.83)
- MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models (ICML 2025 · similarity 0.83)
- UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.