SASVAAI/terraform-multicloud

terraform-multicloud

78,912 natural-language → Terraform (HCL) pairs, drawn from 25,264 public GitHub repositories across 15 provider families. Every target is human-written HCL that passes terraform validate. Every row carries its source repository, licence and file path.

Built by the SASVA AI Model Cognition Labs (MCL) Team and used to train `SASVAAI/qwen38-27b-terraform`.

What one row looks like

One complete row from data/train.jsonl, reproduced in full with nothing elided:

{
  "instruction": "Write a Terraform configuration for AWS that satisfies the requirement below. Emit only HCL — no explanation, no markdown fences.",
  "input": "Create a VPC with a CIDR block specified by the `vpc_cidr` variable, enable IPv6 assignment, and ensure DNS support and DNS hostnames are both enabled; tag the VPC with the project name provided via the `project_name` variable.",
  "output": "\n# Create a VPC to launch our instances into\nresource \"aws_vpc\" \"kubernetes\" {\n  assign_generated_ipv6_cidr_block = true\n  cidr_block                       = var.vpc_cidr\n\n  # ENPOINT fix\n  enable_dns_support   = true\n  enable_dns_hostnames = true\n\n  \n\n  tags = {\n    Name = \"${var.project_name}\"\n  }\n}",
  "repo": "dvogt/aws-kube-sandbox",
  "license": "Apache-2.0",
  "file_path": "aws-kube-tf-kubernetes/modules/network/vpc.tf",
  "provider_family": "aws",
  "resource_types": [
    "aws_vpc"
  ],
  "signature": "fa677c3c8d1c83372b1079db5d480539"
}

The output field above is a single JSON string; rendered, it is the file exactly as its author committed it:

# Create a VPC to launch our instances into
resource "aws_vpc" "kubernetes" {
  assign_generated_ipv6_cidr_block = true
  cidr_block                       = var.vpc_cidr

  # ENPOINT fix
  enable_dns_support   = true
  enable_dns_hostnames = true

  

  tags = {
    Name = "${var.project_name}"
  }
}

`output` is human-written. `instruction` and `input` are model-written — see Provenance.

Statistics

Provider families

provider_family is the build lane — the extraction pass a row came from — not a parse of the instruction text. The two can disagree for cross-cutting configs (an EKS module is built in the k8s lane but its instruction says AWS).

The corpus is AWS-heavy — roughly 60% of rows, about 3:2 against every other family combined.

How it was built

TerraDS archive  →  extract  →  verify  →  dedup  →  back-translate  →  seal  →  split

1. 1.Extract. Candidate module directories pulled from the TerraDS archive (Zenodo 10.5281/zenodo.20339474), filtered to a 15-licence permissive allowlist.
2. 2.Verify. terraform init -backend=false + terraform validate at the module-directory level, against a one-version offline provider mirror. Terraform resolves references across every .tf file in a directory, so the directory is the only thing that can be validated — but a training pair is one resource-bearing .tf file. Files without a resource block (variables.tf, outputs.tf, versions.tf) are kept as sibling context and never become pairs.
3. 3.Dedup. By signature, an order-insensitive hash over resource and data blocks. HCL is declarative: the same resources in a different order are the same config. Scoped to resource/data blocks only — including provider/terraform scaffolding made the contamination seal nearly inert.
4. 4.Back-translate. A teacher model reads the HCL and writes the plain-language description. Runs after dedup, so the teacher is never paid for rows about to be discarded.
5. 5.Seal. Rows whose signature collides with an IaC-Eval reference configuration are removed, so a model trained here can still be scored on that benchmark honestly.
6. 6.Split. Grouped by repository, never by row.

Split discipline

The split is repo-disjoint and verified: 0 of 25,264 repositories appear on both sides. This matters more than usual here. The corpus has a high near- duplicate rate between repositories that copied each other's modules, so a row-level split would put near-identical configs on both sides and turn any eval loss into a recall measurement.

Per-family splits were merged with a resolver rather than concatenated: 5.8% of repositories appear in more than one family lane, and independent per-family splits put 42 repositories train-side in one file and val-side in another. The resolver drops the train-side rows and keeps validation — never the reverse, because shrinking validation changes the eval set and losses are not comparable across eval-set sizes.

Provenance

The HCL is human-written and unaltered. Each output is the source file exactly as its author committed it. The verification stage rewrites dead syntax (stale version pins, 0.11-era type constraints) only inside a throwaway work directory so that terraform init can run offline; those rewrites are discarded and the candidate's original text is what ships.

The instructions are model-written. instruction and input were back-translated from the HCL by a teacher model — the model read the answer and wrote a description of it.

Teacher model. Qwen/Qwen3-Coder-Next.

Licensing

This dataset is released under CC-BY-4.0, inherited from TerraDS, its upstream source.

The code inside remains under its original licences. Every row carries repo, license and file_path, and `ATTRIBUTION.csv` lists all 25,264 source repositories with their licence and row count, so attribution is possible per-row and in bulk.

Source repositories were filtered to a permissive allowlist before extraction, so unlike a header-based label this is a repository-level licence:

76,450 rows (96.9%) are under licences that require attribution and retention of the licence text. If you redistribute this dataset or a derivative of it, carry ATTRIBUTION.csv with it. If you redistribute generated HCL, note that MIT and Apache-2.0 obligations attach to the training material, not automatically to model output — but the two are close enough here that the conservative reading is the safe one.

Apache-2.0 §4(b) requires retention of NOTICE files. Those files are not carried in this dataset: the unit is a single .tf file, and repository-root NOTICE files were outside the extraction. ATTRIBUTION.csv plus repo is the route back to them.

Removal requests

If your repository appears in ATTRIBUTION.csv and you want it removed, open a discussion on this dataset. Removals are applied by repo across both splits and the file is regenerated.

Intended use

Supervised fine-tuning for natural-language → Terraform generation. The training rendering used for the reference model is:

<|im_start|>system
{SYSTEM}<|im_end|>
<|im_start|>user
{instruction}

{input}

<|im_start|>assistant
{output}<|im_end|>

The user turn's payload is fenced. The reference model additionally trained with an empty <think></think> block before the answer.

from datasets import load_dataset
ds = load_dataset("SASVAAI/terraform-multicloud")           # train: 74,909 · validation: 4,003
aws = ds["train"].filter(lambda r: r["provider_family"] == "aws")

Out of scope. This is not a benchmark. The validation split is a held-out sample of the same distribution, scored by loss; it has no execution harness, no Rego policies and no pass/fail oracle. For scored evaluation use IaC-Eval, which this corpus is deliberately disjoint from.

Citation

@misc{terraform_multicloud_2026,
  title  = {terraform-multicloud: Natural-language to Terraform pairs from permissively licensed repositories},
  author = {{SASVA AI Model Cognition Labs (MCL) Team}},
  year   = {2026},
  url    = {https://huggingface.co/datasets/SASVAAI/terraform-multicloud}
}

@dataset{terrads,
  title     = {TerraDS: A Dataset for Terraform HCL Programs},
  year      = {2025},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.20339474}
}

@inproceedings{iaceval2024,
  title     = {IaC-Eval: A Code Generation Benchmark for Infrastructure-as-Code Programs},
  booktitle = {NeurIPS Datasets and Benchmarks},
  year      = {2024}
}