mdonigian/starcoder-curated

StarCoderData Curated

A curated subset of StarCoderData optimised for training a 500M parameter model focused on structured data output (JSON generation, function calling, schema compliance).

Dataset Summary

• —Total code files: 5,203,508
• —Total tokens: 3.9B (target: 3.5B)
• —Classifier-scored files: 1,553,596 (1.7B tokens)
• —Non-classified files: 3,649,912 (2.2B tokens) — filtered by heuristics, not the classifier
• —Source: bigcode/starcoderdata
• —Classifier: mdonigian/code-curator-v1 (UniXcoder-base, multi-task)
• —Curation: Per-language-slice filtering + compression ratio pre-filter + MinHash deduplication

Filtering Strategy

Different language groups need different curation approaches. Not every slice goes through the GPU classifier — schema languages and GitHub issues are filtered with cheaper heuristics because the classifier was trained on general-purpose code and isn't the right tool for inherently structured formats.

All slices share these pre-filters:

• —zlib compression ratio < 0.10 (catches extreme repetition)
• —MinHash LSH deduplication (128 perms, 5-line shingles, 0.7 Jaccard threshold)

Classifier-Scored Slices (relevance_filter)

These languages were scored by the multi-task classifier. Files were ranked by structured data relevance and filtered to keep only those with relevance ≥ 2.0 and quality ≥ 1.5, sampled down to the per-slice token budget:

• —TypeScript: ~600M tokens — strong type system, filter by SD relevance ≥ 2
• —Python: ~600M tokens — filter by SD relevance ≥ 2
• —Rust/Go/Java: ~600M tokens — strongly typed, filter by SD relevance ≥ 2

Non-Classified Slices

These languages were not run through the classifier. Their quality, structured_data, and content_type columns contain default placeholder values (0.0 / "unclassified") and should be ignored:

• —Schema languages (JSON/YAML/SQL/protobuf/thrift/XSLT): ~800M tokens — inherently structured data formats; quality floor + random sample to budget
• —GitHub Issues (technical): ~500M tokens — keyword filter matching structured-data topics (JSON, schema, API, protobuf, gRPC, etc.)
• —General code (78 other languages): ~1B tokens — random sample for language diversity; quality floor only

Language Slice Distribution

Classifier-Scored Slices — Detail

The quality and structured data scores below apply only to the 1,553,596 files (1.7B tokens) that went through the classifier. Non-classified slices are excluded from these statistics.

Content Group Distribution (classifier-scored files only)

Structured Data Relevance (classifier-scored files only)

The strongest classifier signal (Spearman 0.81 on held-out data). SD2+ files contain significant structured data patterns (API endpoints, JSON parsing, schema definitions, etc.).

Quality mean: 3.79, Median: 3.88.

Quality Distribution (classifier-scored files only)

Non-Classified Slices — Detail

These slices were filtered using heuristics. The classifier columns (quality, structured_data, content_type) are set to defaults and do not reflect actual code quality — the filtering was done by other means:

Programming Languages

Token Count Distribution

Schema

Each row contains:

Tip: To work with only classifier-scored data, filter on language_slice in {"typescript", "python", "rust_go_java"}.

Methodology

1. 1.Download: All language folders from bigcode/starcoderdata.
2. 2.Classification: Multi-task UniXcoder-base model (3 heads: quality, SD relevance, content type) runs on TypeScript, Python, Rust, Go, and Java Schema languages, GitHub issues, and general code skip this step.
3. 3.Pre-filtering: zlib compression ratio filter removes repetitive boilerplate before GPU inference.
4. 4.Filtering: Per-slice strategy — relevance-based ranking for classified languages, keyword matching for GitHub issues, random sampling for schema/general code. All slices enforce a quality floor.
5. 5.Deduplication: MinHash LSH (128 perms, 5-line shingles, 0.7 Jaccard threshold). Highest-relevance file kept from each cluster.