datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
iac-eval
IaC-Eval dataset (v1.1)
IaC-Eval dataset is the first human-curated and challenging Cloud Infrastructure-as-Code (IaC) dataset tailored to more rigorously benchmark large language models' IaC code generation capabilities.
This dataset contains 458 questions ranging from simple to difficult across various cloud services (targeting AWS for now).
| Github | 🏆 Leaderboard TBD | 📖 NeurIPS 2024 Paper |
2. Usage instructions
Option 1: Running the evaluation… See the full description on the dataset page: https://huggingface.co/datasets/autoiac-project/iac-eval.Multi-IaC-Eval
Multi-IaC-Eval
We present Multi-IaC-Eval is a novel benchmark dataset for evaluating LLM-based IaC generation and mutation across AWS CloudFormation, Terraform, and Cloud Development Kit (CDK) formats. The dataset consists of triplets containing initial IaC templates, natural language modification requests, and corresponding updated templates, created through a synthetic data generation pipeline with rigorous validation.
Cloudformation: 263
Terraform: 446
CDK (Python): 64
CDK… See the full description on the dataset page: https://huggingface.co/datasets/AmazonScience/Multi-IaC-Eval.iac-eval-v2
IaC-Eval v2
Modernised Terraform code-generation benchmark — 186 tasks, Terraform 1.15 + OPA 1.16 (Rego v1).
An updated and extended version of the IaC-Eval NeurIPS 2024 benchmark.
Scoring is deterministic: the generated HCL either passes terraform plan + opa eval, or it doesn't — no LLM-as-judge.
Dataset summary
Field
Value
Tasks
186 (AWS only)
Difficulty
1–6 (distribution: 1→35, 2→40, 3→51, 4→22, 5→9, 6→13)
AWS services
34 distinct
Terraform… See the full description on the dataset page: https://huggingface.co/datasets/iac-eval-v2/iac-eval-v2.Azure_IaC_testAzure_IaC_validationAzure_IaCAzure_IaC_reducedAzure_IaC_trainmirror-iac-eval
IaC-Eval dataset (v1.1)
IaC-Eval dataset is the first human-curated and challenging Cloud Infrastructure-as-Code (IaC) dataset tailored to more rigorously benchmark large language models' IaC code generation capabilities.
This dataset contains 458 questions ranging from simple to difficult across various cloud services (targeting AWS for now).
| Github | 🏆 Leaderboard TBD | 📖 NeurIPS 2024 Paper |
2. Usage instructions
Option 1: Running the… See the full description on the dataset page: https://huggingface.co/datasets/alucent/mirror-iac-eval.