Pinkstack/luau-pretrain-corpus-filtered

Luau / Roblox Corpus

An openly-licensed dataset of Luau and Roblox-flavored Lua source code, built from The Stack v3 (revision ef074e272dd3e5e75c4d78e62d6483ec9281a77d).

What's in it

This dataset combines two sources of Luau/Roblox code:

1. 1.Pure Luau: files The Stack v3's own language classifier already tagged as Luau with high confidence. Included unconditionally, with no score-based filtering.
2. 2.Extracted Roblox/Luau candidates: files tagged generic Lua by the classifier, but heuristically identified as Roblox/Luau code anyway (mixed .lua/.luau repos and older tooling frequently cause classifiers to miss real Luau code this way, see file_type below).

Why two sources instead of one classifier pass

Language classifiers built for general-purpose code are not Luau-aware in a robust way, and Roblox development in particular tends to produce repos with inconsistent file extensions and mixed tooling. Relying solely on the upstream language tag would silently drop a meaningful amount of genuine Roblox/Luau code that happens to be tagged Lua. This dataset closes that gap with an explicit, score-based heuristic pass, while still trusting the classifier's confident Luau tags outright.

file_type column — how to filter without losing pure Luau

This is the column to filter on, not the score columns.

Why not just filter on score?

It's tempting to assume "score 0 means pure Luau" or "higher score = more confident," but score does not reliably indicate origin in either direction:

• —A pure Luau file can legitimately score 0 on roblox_api_score — a small math/utility module never calls game:GetService(...), but it's still 100% genuine Luau.
• —An extracted Lua-tagged file can score just as high, or higher, than some pure Luau files, depending on how much Roblox-specific API surface it happens to touch.

Score reflects content, file_type reflects origin. If you filter by score alone (e.g. "keep only rows with luau_syntax_score >= 3"), you will silently drop pure Luau files that had no reason to score highly on that particular pattern set — even though they are unambiguously real Luau code.

Correct pattern, if you want to apply a stricter score threshold to the heuristically-extracted rows while always keeping every pure Luau file:

SELECT *
FROM luau_full_corpus_open_license.parquet
WHERE file_type = 'luau'
   OR luau_syntax_score >= 3
   OR roblox_api_score  >= 3

Incorrect pattern (silently drops legitimate pure Luau files):

-- DON'T do this
SELECT * FROM luau_full_corpus_open_license.parquet WHERE luau_syntax_score >= 3

Columns

License

This dataset compilation, the selection, filtering, deduplication logic, and added columns (file_type, admission_reason, computed scores) — is released under the Open Data Commons Attribution License (ODC-By) v1.0: https://opendatacommons.org/licenses/by/1-0/

This matches the license Stack v3 itself uses for the same reason: ODC-By covers rights over the compilation as a database, the act of collecting, selecting, and arranging this data, it does not and cannot grant rights over the underlying copyrighted content of each individual file.

Every individual file in this dataset keeps its own original author's license. Only files with a confirmed permissive license are included: every row has license_type = 'permissive', but that still spans at least 27 distinct license identifiers (Apache-2.0, Unlicense, CC0-1.0, WTFPL, MIT, and others, check detected_licenses per row). Aggregating code into a dataset does not relicense it: an Apache-2.0 file is still Apache-2.0 after being included here, and using that file is still subject to whatever its original license requires (in most cases, attribution).

repo_path, commit_id, and detected_licenses are kept on every row specifically so this provenance can always be traced back and honored — check detected_licenses for the specific file before reusing its code.

Known limitations

• —No deduplication. Forks, near-identical copies, and reuploaded repos are not collapsed. If you need a deduplicated set, dedupe on content hash or (repo_path, file_path) before training.
• —No quality filtering. Trivial snippets, generated code, and test fixtures are not excluded.
• —`readme` coverage is partial for some rows. Some rows may have readme = null even where a README exists upstream, depending on when the underlying file was originally fetched.
• —Heuristic thresholds are approximate. The lua-origin rows are admitted via pattern-matching, not a verified classifier: expect some false positives (generic Lua that happens to mention a matching pattern) and some false negatives (genuine Roblox code that didn't hit any pattern).
• —A small number of `detected_licenses` values are non-standard identifiers (LicenseRef-scancode-*: ScanCode's label for licenses it detected but couldn't map to a standard SPDX identifier). These are a very small fraction of the dataset; check detected_licenses if this matters for your use case.

Provenance

Built from The Stack v3, pinned to a single frozen revision for reproducibility, then filtered to files with a confirmed permissive license only.