Research map / Large language models / Agents, code and math
When Prompt Engineering Meets Software Engineering: CNL-P as Natural and Robust "APIs'' for Human-AI Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CNL-P, a controlled natural language for prompts that combines prompt-engineering best practices with software-engineering principles, plus an LLM-based NL-to-CNL-P converter and a parser/semantic linter.
Problem
Natural language prompts are ambiguous and hard to structure, and existing prompt approaches either wrap prompts in programming-language code (tight coupling, parsing overhead, inaccessible to non-technical staff) or use NL templates that lack expressiveness and state management.
Approach
A grammar-driven controlled natural language (CNL-P) with keyword-delimited modules (PERSONA, CONSTRAINTS, DATA_TYPE, VARIABLES, WORKER, CALL_API) that is converted from NL by an LLM agent and statically checked by a two-stage parser plus semantic linter, applying static analysis to natural language prompts for the first time.
Key results
- In RQ3, the linting tool reached 100% accuracy with 0% redundancy on 47 error instances, versus 76.60% accuracy and 19% redundancy for GPT-4o.
- In RQ2, LLMs (GPT-4o, Gemini-1.5-Pro-002, GPT-4o-Mini, Llama3-70B, Claude-3-Haiku) given CNL-P as system prompt achieved performance comparable to natural language prompts without explanations or few-shot examples.
- In RQ1, CNL-P was rated above RISEN and RODES by the LLM judge and by technical and non-technical human groups, with the clearest advantage on Modularity and Extensibility.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the CNL-P linting tool instead of GPT-4o to verify CNL-P prompts | CNL-P static checking · 47 task instances (RQ3) | accuracy: 76.60% -> 100% | Helped |
| Use the CNL-P linting tool instead of GPT-4o to verify CNL-P prompts | CNL-P static checking · 47 task instances (RQ3) | redundancy rate: 19% -> 0% | Helped |
| Convert natural-language prompts into CNL-P rather than use the original natural-language prompts | LLMs completing six classification tasks · Six classification tasks (RQ2) | accuracy | No clear effect |
| Use CNL-P rather than RISEN or RODES templates | Prompt conversion evaluation · 93 prompt instances (RQ1) | modularity | Helped |
| Use CNL-P rather than RISEN or RODES templates | Prompt conversion evaluation · 93 prompt instances (RQ1) | extensibility and maintainability | Helped |
| Use CNL-P rather than RISEN or RODES templates | Prompt conversion evaluation · 93 prompt instances (RQ1) | process rigor | Helped |
| Use CNL-P rather than RISEN or RODES templates | Prompt conversion evaluation by non-technical evaluators · 30 converted prompts (RQ1) | readability and structural clarity | Mixed |
| Use RISEN rather than CNL-P | Prompt conversion evaluation by technical evaluators · 93 prompt instances (RQ1) | adherence to original intent | Hurt |
Surprising findings
- On Readability, non-technical evaluators rated CNL-P lower despite finding its structure visually clearer, because they needed more time to understand types and variables.
- Ratings for RISEN and RODES on Adherence to Original Intent were lower from the technical group, who viewed RISEN's Expectation module as narrowing the agent's scope.
Limitations
- Evaluation sample is small and uses fixed-format classification tasks; RQ3 dataset is synthetically generated by GPT-4o.
- Linter's WORKER parsing relies on LLMs, and the paper states that the LLM and program are not yet fully integrated for semantic checks.
- Non-technical evaluators found the CNL-P syntax harder to read without training; the NL-to-CNL-P agent was not evaluated in the experiments.
Experimental setup
Datasets: 93 prompts from a GitHub prompt collection (113K stars) six classification tasks from a NLP task benchmark (tasks 1162, 1424, 1678, among others) 47 synthetic error task instances generated with GPT-4o
Benchmarks: six classification tasks (RQ2) 93 GitHub ChatGPT prompts converted to CNL-P/RISEN/RODES (RQ1) 47 synthetic CNL-P error instances (RQ3)
Metrics: five rubric dimension scores (0-100) accuracy redundancy rate relative accuracy vs best prompt
Baselines: RISEN RODES Natural language prompts
Builds on: RISEN RODES TypeChat Pydantic chain-of-thought prompting
Closest papers by meaning
- PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization (ICLR 2024 · similarity 0.86)
- How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptation (NeurIPS 2025 · similarity 0.86)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.85)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.85)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.85)
- StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.