nyuuzyou/notabug-code

NotaBug Code Dataset

Dataset Description

This dataset was compiled from code repositories hosted on NotaBug.org, a free code hosting platform that emphasizes software freedom and privacy. NotaBug is built on a fully free software stack and is popular among free software advocates and privacy-conscious developers.

Dataset Summary

Key Features

• —Free software focused corpus: Contains code from repositories on a platform dedicated to software freedom
• —Diverse language coverage: Spans thousands of file types identified by file extension
• —Rich metadata: Includes repository name, file path, detected language, license information, and file size

Languages

The dataset includes files from many programming languages and file types. Languages are detected by file extension. The top 30 languages by file count:

Licenses

The dataset includes files from repositories with various licenses:

Dataset Structure

Data Fields

Data Format

• —Format: Apache Parquet with Zstd compression
• —File Structure: 12 files (notabug_0000.parquet to notabug_0011.parquet)

Data Splits

All examples are in the train split. There is no validation or test split.

Example Data Point

{
    'code': '#!/usr/bin/env python2\n# -*- coding: utf-8 -*-\n# Copyright (C) 2014...',
    'repo_name': 'intermsofthewhole/libreboot',
    'path': 'resources/utilities/i945gpu/intel-regs.py',
    'language': 'Python',
    'license': 'mit',
    'size': 3733
}

Dataset Creation

Source Data

All data originates from public repositories hosted on NotaBug.org.

Language Detection

Programming languages are detected by file extension inference.

License Detection

Licenses are detected by scanning for license files in repositories and matching against known license patterns. Repositories without a detectable license are marked as "unknown".

File Filtering

• —Long Lines: Files with any line exceeding 1,000 characters were excluded
• —Deduplication: No deduplication was performed on the dataset

Considerations for Using the Data

Personal and Sensitive Information

The dataset may contain:

• —Email addresses in code comments or configuration files
• —API keys or credentials that were accidentally committed
• —Personal information in comments or documentation

Users should exercise caution and implement appropriate filtering when using this data.

Licensing Information

This dataset is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in this dataset must abide by the terms of the original licenses, including attribution clauses when relevant. The license field in each data point indicates the license of the source repository.