OpenxAILabs/nix-reviewer-training

nix-reviewer-training

Training data for `OpenxAILabs/nix-reviewer-1.5b` — 445 (broken Nix config, structured review) pairs, fully synthetic, Apache-2.0 clean.

How it was built

Every row was generated by:

1. 1.Picking a known failure pattern from the pattern catalog (3 patterns in v0.1).
2. 2.Synthesizing an original Nix configuration that exhibits it (our code, our package choices, no content copied from anywhere).
3. 3.Running the synthesized config through a nixos/nix Docker container — nix eval --json produces the ground-truth error message, line number, and column.
4. 4.Composing the ideal review comment from the pattern's template + the oracle's real line/message.
5. 5.Dropping any pair where Nix didn't error, the error didn't match the expected class, or the prompt hash collided with a prior row.

The Nix compiler is the teacher. No LLM labelled these examples, no forum post was scraped, no verbatim third-party code appears in any row.

Distribution

v0.2 will add: option_renamed_across_channels, module_namespace_mismatch, missing_module_import, and negative (non-Nix) examples for refusal training.

Schema

One JSON object per line:

{
  "id": "8e76a3f1c2d4b5e6",
  "pattern_id": "package_attr_path_drift",
  "strategy": "expr",
  "prompt": "{ pkgs, ... }:\n{\n  environment.systemPackages = with pkgs; [\n    vim\n    vvim\n  ];\n}\n",
  "completion": "[{\"line\": 5, \"severity\": \"error\", \"message\": \"`vvim` is not a nixpkgs attribute — did you mean `vim`?\"}]",
  "oracle_line": 5,
  "oracle_message": "undefined variable 'vvim'",
  "label": {"old_path": "vvim", "new_path": "vim",
            "hint": "`vvim` is not a nixpkgs attribute — did you mean `vim`?"}
}

Fields:

• —prompt — the broken Nix source the model is shown at train time
• —completion — the ideal JSON review; the training target
• —oracle_line, oracle_message — the real Nix eval output, kept for audit
• —pattern_id, label — metadata for dataset analysis; not fed to the model
• —id — sha256-16 of prompt, for deduplication

Intended use

Train specialist Nix-review models. The dataset is deliberately narrow — 3 patterns, 445 pairs — to prove out the synthesis pipeline. For production-scale training, expect v0.2 (~5,000 pairs, ≥6 patterns).

Provenance and ethics

• —No forum content. Pattern selection was informed by qualitative analysis of publicly available NixOS community discussions, but no direct text, code, or attribution from those discussions entered any row.
• —Apache-2.0. All prompt / completion / label content is our original synthesis. Upstream attribution for the Nix evaluator (LGPL-2.1-or-later, invoked at generation time but not redistributed) and for nixpkgs source paths referenced during pattern design (MIT) lives in the companion NOTICE file on the project repo.

Reproduce it

The full generation pipeline is in johnforfar/nix-assistant → scrape/generate_pairs.py. Every row in this dataset can be regenerated deterministically from a seed + the pattern catalog.

License

Apache-2.0.