nyuuzyou/ms-codeplex-archive

Microsoft CodePlex Archive Dataset

Dataset Description

Source code from the Microsoft CodePlex Archive on the Internet Archive. CodePlex was Microsoft's open-source project hosting service from 2006 to 2017, popular for .NET and Windows projects.

Dataset Summary

Features

• —38K repositories from the 2006-2017 period
• —91 programming languages, dominated by C# and .NET
• —Metadata: repository name, file path, language, license, file size
• —Filtered to remove vendor code, build artifacts, and generated files

Languages

Top 30 of 91 languages by file count:

Licenses

Dataset Structure

Data Fields

Data Format

• —Format: Apache Parquet with Zstd compression
• —File Structure: 10 files (codeplex_0000.parquet to codeplex_0009.parquet)

Data Splits

Train split only.

Example Data Point

{
    "code": "using System;\nusing System.Collections.Generic;\nusing System.ComponentModel;\nusing System.Data;\nusing System.Drawing;\nusing System.Text;\nusing System.Windows.Forms;\n\nnamespace KeygenApp\n{\n    public partial class KeygenForm : Form\n    {\n        public KeygenForm()\n        {\n            InitializeComponent();\n        }\n    }\n}",
    "repo_name": "2atgroup",
    "path": "TiffBrowserTestCSharp/Keygen/KeygenForm.cs",
    "language": "C#",
    "license": "Microsoft Reciprocal License (Ms-RL)",
    "size": 1259
}

Dataset Creation

Pipeline

1. 1.Load project metadata from zips.csv (108,508 repositories)
2. 2.Extract 122 tar archives from the Internet Archive tars/ directory
3. 3.Extract source code from sourceCode/sourceCode.zip within each project
4. 4.Parse license/license.json for license metadata
5. 5.Filter non-code files
6. 6.Write to Parquet with Zstd compression

Language Detection

By file extension.

License Detection

Parsed from license/license.json in each project archive.

File Filtering

Size Limits

Excluded Directories

• —Config: .git/, .github/, .gitlab/, .vscode/, .idea/, .vs/, .settings/, .eclipse/, .project/, .metadata/
• —Vendor: node_modules/, bower_components/, jspm_packages/, vendor/, third_party/, 3rdparty/, external/, packages/, deps/, lib/vendor/, target/dependency/, Pods/
• —Build: build/, dist/, out/, bin/, target/, release/, debug/, .next/, .nuxt/, _site/, _build/, __pycache__/, .pytest_cache/, cmake-build-*, .gradle/, .maven/, obj/
• —CodePlex metadata: discussions/, issues/, releases/, wiki/, wikiRender/
• —Tests: test/, tests/, spec/, specs/, __tests__/

Excluded Files

• —Lock files: package-lock.json, yarn.lock, pnpm-lock.yaml, Gemfile.lock, Cargo.lock, poetry.lock, Pipfile.lock, composer.lock, go.sum, mix.lock, packages.lock.json
• —Minified: files containing .min.
• —Hidden: files starting with . (except .htaccess)
• —Binary: .exe, .dll, .so, .dylib, .a, .lib, .o, .obj, .jar, .war, .ear, .class, .pyc, .pyo, .wasm, .bin, .dat, .pdf, .doc, .docx, .xls, .xlsx, .ppt, .pptx, .zip, .tar, .gz, .bz2, .7z, .rar, .jpg, .jpeg, .png, .gif, .bmp, .ico, .svg, .mp3, .mp4, .avi, .mov, .wav, .flac, .ttf, .otf, .woff, .woff2, .eot, .pdb, .nupkg, .snk
• —System: .DS_Store, thumbs.db

Content Filtering

• —UTF-8 Validation: must be valid UTF-8
• —Binary Detection: no null bytes, <30% non-printable characters in first 8KB
• —Generated Files: excluded if first 500 bytes contain generated by, do not edit, auto-generated, @generated, etc.
• —Empty Files: excluded
• —Long Lines: excluded if any line >1,000 characters in first 10 lines

Source Data

Microsoft CodePlex Archive on Internet Archive, tars/ directory (122 tar archives). 108,508 repositories accessible out of 108,516 listed.

Archive Structure

Each CodePlex project archive contains:

• —sourceCode/sourceCode.zip - source code (extracted)
• —license/license.json - license metadata (parsed)
• —discussions/, issues/, releases/, wiki/ - excluded

Considerations

Sensitive Information

May contain email addresses, accidentally committed credentials, or personal information in comments. Filter accordingly.

Licensing

Use must comply with the original repository licenses. The license field indicates each file's source license.