Pinkstack/luau-pretrain-corpus-unfiltered

Luau / Roblox Corpus (Unfiltered)

**opt out info**

A dataset of Luau and Roblox-flavored Lua source code, built from The Stack v3 (revision ef074e272dd3e5e75c4d78e62d6483ec9281a77d).

File: luau_full_corpus.parquet, 845,351 files This is the unfiltered companion release to `luau-pretrain-corpus-filtered`, which contains only files with a confirmed permissive license. This unfiltered variant additionally includes files where no license was detected at all (license_type = 'no_license'), see "License" below for what that means and why it matters before you use this data.

!!@WARNING: This dataset will have "bad" code in it (aka exploit related code). You may want to filter this dataset.

What's in it

This dataset combines two sources of Luau/Roblox code:

1. 1.Pure Luau, files The Stack v3's own language classifier already tagged as Luau with high confidence. Included unconditionally, with no score-based filtering.
2. 2.Extracted Roblox/Luau candidates, files tagged generic Lua by the classifier, but heuristically identified as Roblox/Luau code anyway (mixed .lua/.luau repos and older tooling frequently cause classifiers to miss real Luau code this way, see file_type below).

Why two sources instead of one classifier pass

Language classifiers built for general-purpose code are not Luau-aware in a robust way, and Roblox development in particular tends to produce repos with inconsistent file extensions and mixed tooling. Relying solely on the upstream language tag would silently drop a meaningful amount of genuine Roblox/Luau code that happens to be tagged Lua. This dataset closes that gap with an explicit, score-based heuristic pass, while still trusting the classifier's confident Luau tags outright.

file_type column, how to filter without losing pure Luau

This is the column to filter on, not the score columns.

Why not just filter on score?

It's tempting to assume "score 0 means pure Luau" or "higher score = more confident," but score does not reliably indicate origin in either direction:

• —A pure Luau file can legitimately score 0 on roblox_api_score, a small math/utility module never calls game:GetService(...), but it's still 100% genuine Luau.
• —An extracted Lua-tagged file can score just as high, or higher, than some pure Luau files, depending on how much Roblox-specific API surface it happens to touch.

Score reflects content, file_type reflects origin. If you filter by score alone (e.g. "keep only rows with luau_syntax_score >= 3"), you will silently drop pure Luau files that had no reason to score highly on that particular pattern set, even though they are unambiguously real Luau code.

Correct pattern, if you want to apply a stricter score threshold to the heuristically-extracted rows while always keeping every pure Luau file:

SELECT *
FROM luau_full_corpus.parquet
WHERE file_type = 'luau'
   OR luau_syntax_score >= 3
   OR roblox_api_score  >= 3

Incorrect pattern (silently drops legitimate pure Luau files):

-- DON'T do this
SELECT * FROM luau_full_corpus.parquet WHERE luau_syntax_score >= 3

Columns

License

This dataset compilation, the selection, filtering, deduplication logic, and added columns (file_type, admission_reason, computed scores), is released under the Open Data Commons Attribution License (ODC-By) v1.0: https://opendatacommons.org/licenses/by/1-0/

This matches the license Stack v3 itself uses for the same reason: ODC-By covers rights over the compilation as a database, the act of collecting, selecting, and arranging this data, it does not and cannot grant rights over the underlying copyrighted content of each individual file.

`license_type` in this dataset is one of:

*Note: for filtering keep in mind that some readme's have licensing information that was not detected. Deep filtering is needed to get the most use out of the dataset.

Every individual file, regardless of license_type, keeps its own original author's copyright. Aggregating code into a dataset does not change that. repo_path and commit_id are kept on every row so provenance can always be traced back to the source.

OPT OUT

If you are the author of code in this dataset

If you'd like your repository (or all of your repositories) removed from this dataset, please reach out:

• —By opening a discussion here on HF
• —Email: pinkstack433@gmail.com
• —X (Twitter): @PinkStack_

Please include the repository URL (or your GitHub username, if requesting removal of everything) so it can be identified and removed. Requests are tracked and applied to future releases of this dataset.

Known limitations

• —No deduplication. Forks, near-identical copies, and re-uploaded repos are not collapsed. If you need a deduplicated set, dedupe on content hash or (repo_path, file_path) before training.
• —No quality filtering. Trivial snippets, generated code, and test fixtures are not excluded.
• —`readme` coverage is partial for some rows. Some rows may have readme = null even where a README exists upstream, depending on when the underlying file was originally fetched.
• —Heuristic thresholds are approximate. The lua-origin rows are admitted via pattern-matching, not a verified classifier, expect some false positives (generic Lua that happens to mention a matching pattern) and some false negatives (genuine Roblox code that didn't hit any pattern).
• —`no_license` rows carry some legal ambiguity. See "License" above. If you need a dataset with less ambiguity, use the filtered release instead.

Provenance

Built from The Stack v3, pinned to a single frozen revision for reproducibility.