Avelina/python-edu-cleaned

SmolLM-Corpus: Python-Edu (Cleaned)

This dataset contains the python-edu subset of SmolLM-Corpus with the contents of the files stored in a new text field. All files were downloaded from the S3 bucket on January the 8th 2025, using the blob IDs from the original dataset with revision 3ba9d605774198c5868892d7a8deda78031a781f. Only 1 file was marked as not found and the corresponding row removed from the dataset (content/39c3e5b85cc678d1d54b4d93a55271c51d54126c which I suspect is caused by a mallformed blob ID), and 980 files were removed (see the next section for more details).

Please refer to the original README for all other information.

Cleaning

980 files were removed from the dataset due to being low quality. The primary culprits were hardcoded python calculators, massive hardcoded unit tests, files containing lists words, and TSV/CSV data stored as python files. There was also one file that contained an entire Harry Potter book.

Dataset Features

• —blob_id (string): Software Heritage (SWH) ID of the file on AWS S3.
• —repo_name (string): Repository name on GitHub.
• —path (string): The file path within the repository.
• —length_bytes (int64): Length of the file content in UTF-8 bytes.
• —score (float32): The output of the educational scoring model.
• —int_score (uint8): The rounded educational score.
• —text (string): The downloaded python file text.