Research map / Large language models / Agents, code and math
MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAGIC, a three-agent (manager, feedback, correction) method that automatically generates a self-correction guideline from LLM failures on the training set, improving DIN-SQL and other few-shot text-to-SQL methods on BIRD and Spider.
Problem
Self-correction for text-to-SQL relies on manually written guidelines by human experts, which are labor-intensive and limited in covering the LLM's error patterns.
Approach
A manager agent orchestrates feedback and correction agents over the training-set failures of an initial text-to-SQL method, stores feedback that led to successful revisions, and aggregates them batch-by-batch into a self-correction guideline that is then injected into the prompt at inference.
Key results
- On BIRD dev, MAGIC guideline raises DIN-SQL execution accuracy from 56.52 to 59.13, above the human expert guideline (58.52 vs 57.76 total scored in table; reported as 55.80 for all-SQL setting).
- On Spider dev, MAGIC guideline reaches 85.66 EX for correcting incorrect SQLs versus 78.62 without self-correction and 81.15 for the best human guideline.
- Applying the MAGIC guideline to MAC-SQL improves EX from 59.39 to 61.92 on BIRD dev.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MAGIC-generated guideline to DIN-SQL | DIN-SQL · BIRD dev, correcting incorrect SQLs | EX: 56.52 -> 59.13 | Helped |
| Add MAGIC-generated guideline to DIN-SQL | DIN-SQL · Spider dev, correcting incorrect SQLs | EX: 78.62 -> 85.66 | Helped |
| Apply self-correction to all initially predicted SQLs | DIN-SQL with human-written guideline · BIRD dev, all SQLs | EX: 56.52 -> 55.80 | Hurt |
| Aggregate feedback in 1 batch instead of 0 (ablation) | MAGIC guideline generation · BIRD dev | EX: 56.52 -> 57.4 | Helped |
| Aggregate feedback in 5 batches instead of 0 (ablation) | MAGIC guideline generation · BIRD dev | EX: 56.52 -> 58.8 | Helped |
| Aggregate feedback in 10 batches instead of 0 (ablation) | MAGIC guideline generation · BIRD dev | EX: 56.52 -> 59.13 | Helped |
| Increase feedback batches from 10 to 39 (ablation) | MAGIC guideline generation · BIRD dev | EX: 59.13 -> 59.13 | No clear effect |
| Add the manager agent (ablation) | MAGIC, maximum iterations 2 · BIRD train | EX: 73.01 -> 78.94 | Helped |
| Add the manager agent (ablation) | MAGIC, maximum iterations 2 · BIRD train | VES: 68.29 -> 80.22 | Helped |
| Add the manager agent (ablation) | MAGIC, maximum iterations 5 · BIRD train | EX: 78.60 -> 81.81 | Helped |
| Add the manager agent (ablation) | MAGIC, maximum iterations 5 · Spider train | EX: 83.28 -> 91.78 | Helped |
| Add MAGIC-generated guideline | Zero-shot GPT-4 · BIRD dev | EX: 40.18 -> 48.19 | Helped |
Surprising findings
- Applying self-correction to all predicted SQLs with the human-written DIN-SQL guideline reduces effectiveness, since correct predictions can be changed to incorrect ones.
- Excluding the manager agent (equivalent to critic-refine) yields fewer corrected SQLs within the same iteration budget.
Limitations
- Only one GPT-4 version used and results depend on it; baselines were reproduced, and reproduction differed from reported numbers.
- Some questions with controversial ground truth cannot be corrected by feedback.
- Evaluated mainly with DIN-SQL as the initial system and GPT-4 backbone.
Experimental setup
Datasets: BIRD train set Spider train set
Benchmarks: BIRD dev Spider dev
Metrics: Execution Accuracy Valid Efficiency Score average iterations
Baselines: DIN-SQL human expert guideline MAC-SQL human expert guideline Self-Debugging Self-Consistency Multiple-Prompt Zero-shot GPT-4 Few-shot CoT GPT-4
Builds on: DIN-SQL MAC-SQL Self-Debugging critic-refine
Closest papers by meaning
- Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024 · similarity 0.90)
- Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs (NeurIPS 2025 · similarity 0.89)
- Training Language Models to Self-Correct via Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Small Language Model Can Self-Correct (AAAI 2024 · similarity 0.87)
- A Theoretical Understanding of Self-Correction through In-context Alignment (NeurIPS 2024 · similarity 0.87)
- SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.