Research map / Large language models / Language, safety and interpretability

PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Multilingual LLM data and culture · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PARROT, a benchmark of 598 real-world translation pairs (plus Diverse and Simple variants) for evaluating LLMs on cross-system SQL-to-SQL translation across 22 database systems, finding low accuracy (around 17-60%).

Problem

LLMs are evaluated mainly on Text-to-SQL, while cross-system SQL translation (adapting a query from one DBMS dialect to another) lacks suitable benchmarks, since existing ones focus mostly on SQLite and omit system-specific dialect features.

Approach

A curated benchmark built from real-world SQLs (38 open-source benchmarks, translation-tool test cases, and a ByteDance business workload) with a pipeline of anonymization, clustering-based selection, ensemble rule-based annotation, parser and LLM-based equivalence checking, and execution-based accuracy metrics.

Key results

• LLMs achieve below 38.53% accuracy on average on PARROT translation pairs.
• DeepSeek-R1 32B reaches 58.82% on translation to MySQL, outperforming DeepSeek-R1 671B (44.12%) and GPT-4o at 50.00%.
• On the ByteHouse real-world workload, o3-mini achieves the highest Acc_EX of 58.60%, while GPT-4o reaches 23.91%.

Measured effects reported in the paper

Surprising findings

• Larger or more advanced models do not consistently outperform smaller ones; DeepSeek-R1 32B beats DeepSeek-R1 671B on several targets.
• Advanced reasoning model o3-mini performs poorly on several public-target dialect translations (e.g., 8.82% to MySQL).

Limitations

• Only 8 dialects are covered by the main 598-pair set; many dialects appear only in the variants.
• Equivalence checking relies on LLM-generated test data and execution results, which may miss semantic differences.
• Efficiency of translated queries is discussed but not evaluated.

Experimental setup

Datasets: BIRD Spider TPC-DS DSB TPC-H FROID SQLGlot test cases ByteHouse production workload

Benchmarks: PARROT PARROT-Diverse PARROT-Simple

Metrics: Acc_EX (dialect compatibility) Acc_RES (result consistency)

Baselines: DeepSeek-R1 7B DeepSeek-R1 32B DeepSeek-Coder-V2 Lite DeepSeek-V3 671B DeepSeek-R1 671B GPT-4o o3-mini Claude 3.7 Sonnet o1-preview Llama3.1-8B

Builds on: SQLGlot ANTLR

Closest papers by meaning

• SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications (NeurIPS 2025 · similarity 0.87)
• Structure-Guided Large Language Models for Text-to-SQL Generation (ICML 2025 · similarity 0.87)
• Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (ICLR 2025 · similarity 0.87)
• ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL (ICLR 2025 · similarity 0.86)
• CogSQL: A Cognitive Framework for Enhancing Large Language Models in Text-to-SQL Translation (AAAI 2025 · similarity 0.86)
• SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL (NeurIPS 2025 · similarity 0.86)