znyang/ambig-iac

Ambig-IaC: Ambiguous Infrastructure-as-Code Benchmark

A benchmark dataset of 300 tasks for testing AI agents that generate Infrastructure-as-Code (Terraform) configurations from ambiguous natural language intents.

Project page: https://zyang37.github.io/ambig-iac.github.io/

Dataset Description

This dataset is sourced from IaC-Eval. We performed manual fixes to the original Terraform configurations and validated that all 300 tasks pass terraform plan. Each task also includes the corresponding plan output in JSON format (plan_json field), which is easier to parse and compare programmatically than raw HCL.

Given an ambiguous infrastructure request (e.g., "I need a way to track which lookups are being made against our domain"), the agent is expected to generate a valid Terraform configuration. Each task includes the fully specified intent, a reference Terraform implementation, and OPA/Rego validation policies for evaluating the agent's output.

This dataset is useful for studying:

• —How well AI agents generate correct IaC configurations from underspecified requirements
• —Disambiguation strategies in IaC generation
• —Iterative refinement of infrastructure configurations

Fields

Usage

from datasets import load_dataset

ds = load_dataset("znyang/ambig-iac")

# Access a task
task = ds["train"][0]
print(task["prompt"])        # Ambiguous requirement
print(task["intent"])        # Detailed specification
print(task["main_tf"])       # Reference Terraform code

Dataset Statistics

• —Tasks: 300
• —Avg prompt length: ~60 words
• —Avg intent length: ~12 lines
• —Avg main.tf length: ~98 lines
• —Avg checks.rego length: ~69 lines
• —Domain: AWS infrastructure (Route 53, CloudWatch, VPC, Kinesis, IAM, etc.)

Citation

If you find this work useful, please cite:

@misc{yang2026ambigiacmultileveldisambiguationinteractive,
      title={Ambig-IaC: Multi-level Disambiguation for Interactive Cloud Infrastructure-as-Code Synthesis}, 
      author={Zhenning Yang and Kaden Gruizenga and Tongyuan Miao and Patrick Tser Jern Kon and Hui Guan and Ang Chen},
      year={2026},
      eprint={2604.02382},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2604.02382}, 
}

License

MIT