Research map / Large language models / Language, safety and interpretability
Grammar Prompting for Domain-Specific Language Generation with Large Language Models
Bailin Wang, Zi Wang, Xuezhi Wang, Yuan Cao, Rif A. Saurous, Yoon Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add predicted specialized grammar before the program | standard prompting · GeoQuery few-shot | execution accuracy (%): 81.5 -> 87.5 | Improved |
| add predicted specialized grammar before the program | standard prompting · Overnight-Blk few-shot | program accuracy (%): 29.3 -> 34.8 | Improved |
| use grammar prompting instead of linearized derivation tree prompting | derivation tree prompting · GeoQuery few-shot | execution accuracy (%): 77.5 -> 87.5 | Improved |
| add grammar constraint on predicted grammar (ablation) | grammar prompting · Overnight-Blk few-shot | execution accuracy (%): 57.4 -> 60.4 | Improved |
| add program constraint on top of grammar constraint (ablation) | grammar prompting · GeoQuery few-shot | program accuracy (%): 67.9 -> 69.6 | Improved |
| add constrained decoding to standard prompting (ablation) | standard prompting · Overnight-Blk few-shot | execution accuracy (%): 54.7 -> 54.7 | No clear effect |
| replace predicted grammar with oracle grammar (ablation) | grammar prompting · SMCalFlow few-shot | program accuracy (%): 50.8 -> 80.0 | Improved |
| grammar prompting in retrieval-based ICL | standard prompting · GeoQuery retrieval | execution accuracy (%): 96.8 -> 97.9 | Improved |
| grammar prompting on unseen functions | standard prompting · GeoQuery NewFunc | execution accuracy (%): 63.3 -> 90.8 | Improved |
| grammar prompting with different base LLMs | standard prompting · GeoQuery 100-example subset | execution accuracy (%): GPT-4 85 -> 98; PaLM 2-L 90 -> 87 | Mixed |
| grammar prompting for molecule generation | standard prompting · Chain Extenders | validity (%): 60.3 -> 96.3 | Improved |
| grammar prompting for molecule generation | standard prompting · Isocyanates | diversity: 0.82 -> 0.79 | Mixed |
Closest papers by meaning
- Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.87)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.86)
- Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo (ICLR 2025 · similarity 0.86)
- BLADE: Enhancing Black-Box Large Language Models with Small Domain-Specific Models (AAAI 2025 · similarity 0.86)
- Recursive Language Models (NeurIPS 2026 · similarity 0.86)
- Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
