curtburk/nemotron3-nano-kd-corpus

Nemotron 3 Nano KD Corpus

4,302 verified reasoning traces for coding problems, generated by DeepSeek-V4-Flash (284B) and filtered by executing the generated code against real test suites.

Built entirely on-premises on a single HP ZGX Fury (NVIDIA GB300 Grace Blackwell Ultra). No cloud APIs were used at any stage.

Used to distill NVIDIA-Nemotron-3-Nano-30B-A3B-BF16, raising held-out pass@1 from 23.0% to 63.4%. The corpus is model-agnostic and should be usable for any student.

Contents

Source distribution in train:

Difficulty: 1,186 easy, 2,212 medium, 474 hard. Average quality score 4.57 / 5. Roughly 7.1M tokens.

Format

OpenAI chat format, one JSON object per line:

{
  "conversations": [
    {"role": "system",    "content": "..."},
    {"role": "user",      "content": "problem statement"},
    {"role": "assistant", "content": "## Analysis ... ## Solution ```python ... ```"}
  ],
  "metadata": {
    "problem_id": "APPS_4975",
    "source": "apps",
    "category": "benchmark",
    "difficulty": "easy",
    "quality_score": 5.0,
    "trace_id": "APPS_4975_t0"
  }
}

Every assistant turn follows the same structure — Analysis, Approach, Implementation Plan, Solution, Verification — and every one ends with a fenced Python code block. There are no examples without code (verified: 0 of 3,872).

How it was built

1. 1.Seed problems pulled from HumanEval+ (164), MBPP full split (500), and APPS test split (5,000), plus 327 synthetic enterprise-flavoured problems generated by the teacher.
2. 2.Trace generation — three candidate traces per problem from DeepSeek-V4-Flash-NVFP4 at temperature=1.0, top_p=1.0 (the sampling the model card recommends), served with vLLM at 12-way concurrency. 17,973 traces total.
3. 3.Execution filtering — Python extracted from each trace and run against the real test suite in a network-isolated, memory-capped, read-only container. Assertion tests for HumanEval+/MBPP, stdin/stdout pairs for APPS. 92.7% of teacher-generated code passed.
4. 4.Quality scoring — surviving traces scored 1–5 by the teacher on reasoning clarity and correctness. Threshold 3.5.
5. 5.Assembly — highest-scoring trace kept per unique problem, stratified into splits.

Full pipeline runtime: about 15 hours on one GB300.

Caveats

The splits are held out from a student, not from the teacher. The teacher generated traces for every problem including those in validation and test. If you evaluate the teacher on these splits, treat the result as a contaminated ceiling.

Enterprise synthetic examples are underrepresented. 327 seeds produced only 40 surviving examples (~12%), because the generation validator required at least two assert statements and most synthetic problems did not produce them cleanly. That validator is probably too strict.

Traces are teacher output, not human-verified reasoning. Code correctness is verified by execution; the reasoning narrative is not. Quality scores are the teacher grading itself.

License

MIT, following DeepSeek-V4-Flash, from which the traces were generated.

Underlying problem statements come from HumanEval+, MBPP, and APPS and carry their own licenses.

Citation

@misc{deepseekai2026deepseekv4,
      title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
      author={DeepSeek-AI},
      year={2026},
}

Code

Pipeline and evaluation harness: https://github.com/curtburk/distillation-nemotron3