jordangong/jupyter-scripts-smollm3

The Stack v2 Jupyter Notebooks as Scripts

This dataset contains script representations of the Jupyter notebooks in The Stack v2. It was created from the materialized Jupyter_Notebook split in `jordangong/the-stack-v2-smollm3`.

The output schema follows the Jupyter-script schema used by `bigcode/starcoderdata`, but this release is not deduplicated, PII-filtered, or otherwise equivalent to StarCoderData's filtered split.

Relationship to the SmolLM3 training mix

This release is an independent reproduction of the jupyter-scripts dataset component used in the SmolLM3 training mix. The pinned SmolLM3 stage-1 8T configuration lists jupyter-scripts among both its source dataset paths and tokenized dataset folders.

The reproduction also follows the guidance in BigCode's StarCoder2Data extras discussion. That discussion explains that StarCoder2's Jupyter-script extra was not published because the Software Heritage-backed data requires a separate agreement, and points users to the Stack v2 build code. This dataset reconstructs that missing component from an independently materialized Stack v2 snapshot.

Because the original SmolLM3/StarCoder2 training artifact is not publicly available for direct comparison, this release is not claimed to be byte-for-byte identical to it. The exact source commits, conversion procedure, validation results, and known differences are documented below.

Dataset summary

The dataset has one train split. The Parquet files are stored directly under data/ as train-00000-of-00420.parquet through train-00419-of-00420.parquet.

Usage

Streaming is recommended because the Parquet payload is approximately 48 GB:

from datasets import load_dataset

dataset = load_dataset(
    "jordangong/jupyter-scripts-smollm3",
    split="train",
    streaming=True,
)

Remove streaming=True to download and prepare the complete dataset locally.

Schema

Conversion

The converter adapts BigCode's Stack v1 notebook conversion to stream Stack v2 Parquet shards and produce Parquet output directly.

1. 1.Notebook JSON is parsed with nbformat.
2. 2.The script format is selected from notebook language metadata. Python kernels are recognized as a fallback.
3. 3.If metadata is missing or cannot be rendered, code cells are classified by GuessLang. Predictions below 0.5 are rejected.
4. 4.The notebook is rendered as a script with Jupytext.
5. 5.Unpaired Unicode surrogates are replaced with U+FFFD before Arrow serialization; valid Unicode is preserved.

Rows that cannot be parsed, classified, or rendered are omitted from train and recorded as JSON Lines under _audit/. The complete worker manifests and checkpoints are retained under _state/ for provenance.

The conversion used:

• —materialized source repository commit: ffd83e8e2dd7c14fcbb7eca12e28b2f8afc71f0e
• —upstream bigcode/the-stack-v2 commit recorded by that source repository: e565caa3a78c2423bd374333a472b049eb090e47
• —13,425 source Parquet files
• —32 source Parquet files per output Parquet
• —converter semantics: bigcode-jupytext-script-v2-merged

Validation

Before publication, all 420 Parquet footers were read from the Hub. Every file had the same 11-column schema, every file was non-empty, and the footer row total exactly matched the two completed worker checkpoints: 9,478,641 rows. All 210 audit files were also parsed; their 177,459 valid JSON records exactly matched the checkpoint failure totals.

Limitations and responsible use

• —Conversion can change notebook formatting and does not preserve rich output, widget state, or every notebook metadata field.
• —Language detection can be incorrect, especially for short or polyglot notebooks.
• —This release does not perform deduplication, decontamination, malware scanning, secret removal, or PII filtering.
• —Repository and license metadata is provided for provenance but is not legal advice. Code remains subject to its original license.
• —The data can contain insecure, malicious, offensive, or sensitive material present in public source repositories.

Users must follow the original licenses and the The Stack v2 terms and responsible-use guidance, including Software Heritage's principles for language-model training.

The Stack v2 is updated to honor validated removal requests. Users should check the upstream dataset for the latest usable version and applicable removals before redistributing or training on this derivative.

Acknowledgements

The source corpus was produced by the BigCode and Software Heritage teams. The notebook-to-script strategy and output schema build on BigCode's Stack v1 and StarCoderData preprocessing work.