Ujjwal-Tyagi/moshub

Mos.Hub Code Dataset

A comprehensive code dataset compiled from Mos.Hub, Moscow's official code hosting platform operated by the Moscow Government. This dataset is designed to support training code models with authentic Russian development practices and documentation.

Overview

The Mos.Hub Code Dataset represents a significant code corpus from Russia's governmental and municipal code hosting platform, capturing diverse projects across 297 programming languages. It serves as a valuable resource for developing multilingual code understanding models reflecting Russian coding practices and standards.

Key Statistics

Dataset Characteristics

Scope and Coverage

This dataset captures code from over 16,000 repositories hosted on Mos.Hub, including:

• —Russian government and public sector code: Repositories from Moscow Government and related organizations, featuring Russian comments and documentation
• —Diverse language ecosystem: Support for 297 distinct programming languages
• —Wide project range: From municipal services to development projects and tools
• —Quality-assured: Binary files and low-quality content have been systematically removed

Programming Languages

The dataset encompasses 297 languages. The 30 most represented languages by file count are:

Dataset Structure

Data Fields

Each record contains three fields providing content and metadata:

Sample Record

{
    "file_text": "package main\n\nimport \"fmt\"\n\nfunc main() {\n    fmt.Println(\"Hello\")\n}\n",
    "language": "Go",
    "file_name": "main.go"
}

File Format

• —Format: Apache Parquet
• —Structure: Single consolidated file (data.parquet)
• —Encoding: UTF-8
• —Split: All examples are included in a single training split (no validation or test splits)

Data Creation Process

Language Detection Methodology

Programming languages are identified using github-linguist, GitHub's robust library for language detection. This ensures consistent and reliable classification across all files in the dataset.

Source Data

All data originates from public repositories hosted on Mos.Hub, Moscow's official code hosting platform managed by the Moscow Government.

Quality Filtering

The dataset has undergone systematic filtering to ensure quality and usability:

Deduplication

• —Files have been deduplicated to ensure each code file appears only once in the dataset

Binary File Removal

• —Binary files have been systematically excluded from the dataset
• —Only text-based source code files are retained

Text Encoding Validation

• —Files must be valid UTF-8 encoded text to be included
• —Non-text and improperly encoded files are excluded

Usage Considerations

Data Privacy and Security

The dataset may contain sensitive information that requires careful handling:

• —Email Addresses: Present in code comments, documentation, or configuration files
• —Credentials: Accidentally committed API keys or authentication tokens
• —Personal Information: Names, phone numbers, and other identifiable data in comments or documentation

Users should implement appropriate filtering and anonymization when preparing data for model training.

Licensing and Attribution

This dataset has been compiled with careful consideration of the licenses used in source repositories. Any use of code or data derived from this dataset should be done responsibly and ethically.

Users are responsible for:

• —Respecting the intellectual property rights of original authors
• —Using the data responsibly and ethically
• —Understanding any applicable license restrictions
• —Providing appropriate attribution when required

Responsible Use

The dataset should be used with awareness of:

• —The public sector origin of many repositories
• —The importance of code security and data protection
• —Compliance with Russian data protection regulations
• —Ethical considerations in code model training and deployment

Technical Details

Source: Public repositories hosted on Mos.Hub

Annotations: Machine-generated (language detection)

Multilingual Support: Includes multilingual code and documentation with emphasis on Russian content

Task Categories: Text generation, code modeling, language understanding

Tags: Code, Russian language, multilingual, public sector development