Research map / Large language models / Agents, code and math
IaC-Eval: A Code Generation Benchmark for Cloud Infrastructure-as-Code Programs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IaC-Eval, a 458-scenario benchmark for generating AWS Terraform IaC programs, checked by compile-time Terraform plan and OPA Rego intent specifications, and finds that contemporary LLMs score poorly (GPT-4 pass@1 19.36%).
Problem
LLMs may help write infrastructure-as-code, but no benchmark exists to measure their ability to generate IaC programs (Terraform/HCL) that meet user infrastructure intent without deployment.
Approach
A benchmark that pairs natural-language IaC problems with human-written Rego intent specifications and checks generated Terraform via terraform plan plus OPA policy evaluation, avoiding costly cloud deployment while accounting for multiple valid configurations.
Key results
- GPT-4 achieves pass@1 of 19.36% on IaC-Eval versus 86.6% on EvalPlus.
- Intent specifications removed roughly 50% of false positives from compile-only checking across models.
- Retrieval-augmented generation was the most effective enhancement, improving GPT-4 from 19.36% to 36.70%, while few-shot and CoT reduced GPT-4 scores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add few-shot prompting (ablation) | GPT-4 · IaC-Eval | pass@1 accuracy: 19.36% -> 10.64% | Hurt |
| Add chain-of-thought prompting (ablation) | GPT-4 · IaC-Eval | pass@1 accuracy: 19.36% -> 9.31% | Hurt |
| Add two-turn interpreter feedback (ablation) | Top-4 models on IaC-Eval · IaC-Eval | pass@1 accuracy: +5.5% on average | Helped |
| Add retrieval-augmented generation (ablation) | Evaluated models on IaC-Eval · IaC-Eval | pass@1 accuracy: +6.14% average increase | Helped |
| Add intent specifications to Terraform compilation checks (ablation) | LLM-generated IaC programs · IaC-Eval | false positives removed: over ≈50% | Helped |
| Increase IaC problem difficulty (ablation) | Evaluated LLMs · IaC-Eval | average score | Hurt |
Surprising findings
- Few-shot and chain-of-thought prompting reduced GPT-4's score (to 10.64% and 9.31%) rather than improving it.
- BLEU and CodeBERTScore fail to separate correct from incorrect IaC programs, and the GPT-4 LLM-judge had low precision (10-30%).
Limitations
- Covers only Terraform and AWS services; no function generation and no other providers or IaC tools.
- Evaluation is compile-time only via terraform plan and Rego intent checks, not actual deployment.
- Intent specifications are manually crafted and leave some details underspecified.
Experimental setup
Datasets: IaC-Eval
Benchmarks: IaC-Eval EvalPlus
Metrics: pass@1 pass@k BLEU CodeBERTScore LLM-judge precision
Baselines: GPT-3.5-turbo WizardCoder-33B-V1.1 Magicoder-S-CL-7B Gemini 1.0 Pro CodeLlama Instruct (7B/13B/34B) BLEU CodeBERTScore LLM-as-a-judge
Builds on: HumanEval EvalPlus OPA Rego Terraform
Closest papers by meaning
- EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations (NeurIPS 2024 · similarity 0.85)
- BaxBench: Can LLMs Generate Correct and Secure Backends? (ICML 2025 · similarity 0.84)
- CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules (ICLR 2024 · similarity 0.84)
- DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation (AAAI 2025 · similarity 0.84)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.83)
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.