changliu8541/assemblage-rust

Assemblage-Rust

Consider using your coding agent to test, download and process the data, repository size is 1.5T and it is highly likely you only want a portion of it, but do remember to check the agent outputs.

Produced by Assemblage, a distributed binary-corpus generator, a cite would be greatly appreciated!

Teh dataset contains127,165 compiled Rust binaries from 77,004 builds of 10,766 permissively licensed GitHub repositories, each paired with DWARF-derived function and line metadata, the source tree it was compiled from, and for a subset of it comes with compiler IRs.

Unprocessed. Raw build outputs, filtered only by license, not cleaned, deduplicated or processed into normal Assemblage database+binary format.

Layout

One uncompressed tar per repository (~150 MB average). Everything inside is already compressed — zstd for binaries and metadata, gzip for IR — so the tar is pure container and you can stream one repository without fetching the rest.

repos/{owner}__{project}.tar        builds: binaries, metadata, IR
repos2/{owner}__{project}.tar       ... overflow, see note below
sources/{owner}__{project}.tar.gz   the source tree each was built from
sources2/{owner}__{project}.tar.gz  ... overflow
sources/manifest.json               source file -> repo URL, commit, license, tar
index.jsonl        one line per build: dir, tar, repo_url, commit, license, sizes
LICENSES.csv       build directory -> repo URL, commit, license

Why `repos2/` The Hub caps a directory at 10,000 files and this corpus has 10,766 repositories, so the remainder lives in repos2/ / sources2/. Treat each pair as one logical namespace. The tar field in index.jsonl and the file/tar fields in sources/manifest.json are always the authoritative paths — resolve through those rather than assuming repos/.

repos*/X.tar pairs with sources*/X.tar.gz — same slug, so a repository's two archives share a filename even when they sit in different directories. Each tar expands to one directory per build:

{owner}_{project}_{sha12}-{flag}-{backend}-{mode}/
├── binaries/<name>.zst                ELF binaries, zstd -12
├── metadata/assemblage_meta.json.zst  DWARF functions, RVAs, line mappings
├── ir/<stage>.tar.gz                  LLVM-IR / MIR / HIR / THIR / ASM (subset)
└── export.json                        provenance, license, byte counts

Build mode is in the directory name because Debug, RelWithDebInfo and Release of the same commit are distinct binaries.

sources/ ships the builder's own git clone --recursive snapshot, captured before the build ran — no target/, and .git included, so history and the exact tree hash are recoverable. Source correspondence is a property of the release, not of an upstream that may be deleted or force-pushed later.

Quickstart

Always pass a token (huggingface-cli login, or HF_TOKEN): unauthenticated requests get a lower rate limit, and it is the most common cause of throttling.

# resolve a repository's paths from index.jsonl (they may be in repos/ or repos2/)
jq -r 'select(.repo_url=="https://github.com/tokio-rs/tokio") | .tar' index.jsonl | head -1
# -> repos/tokio-rs__tokio.tar

# one repository's builds + its source tree
huggingface-cli download changliu8541/assemblage-rust \
    repos/tokio-rs__tokio.tar sources/tokio-rs__tokio.tar.gz \
    --repo-type dataset --local-dir .
tar xf repos/tokio-rs__tokio.tar
mkdir -p src && tar xzf sources/tokio-rs__tokio.tar.gz -C src

zstd -d */binaries/mytool.zst -o mytool
zstd -dc */metadata/assemblage_meta.json.zst | jq .

# find builds without downloading anything
jq -r 'select(.license=="MIT License" and .has_ir) | .tar' index.jsonl | sort -u

Build matrix

Toolchain: nightly-2026-06-15, symbol mangling v0, x86_64-unknown-linux-gnu. 17,174 builds carry IR dumps (llvm-ir, mir, hir, thir, asm), emitted only for the LLVM RelWithDebInfo tiers and scoped to repository crates.

assemblage_meta.json records, per binary, every function recovered from DWARF: function_name (mangled), demangled_name, source_file, function_info (RVA ranges), lines (line -> RVA), and origin ∈ in_repo | dependency | stdlib | other.

Known limitations

Please read these — they are properties of the data, not bugs to work around.

1. 1.~26% of builds with binaries have empty or partial function metadata. DWARF extraction runs under a wall-clock budget; over budget, the binary still ships but its Binary_info_list entry is dropped. Check its length first.
2. 2.Lib-only crates produce metadata but no binaries — 17,891 builds (23%) are metadata-only.
3. 3.`Release` binaries carry little repository DWARF by design. Repo symbols survive in .symtab; most DWARF present belongs to the precompiled stdlib. Filter on origin.
4. 4.The `gcc` (cg_gcc) backend is a name/address corpus. Mangling and symbol tables are sound, but repo-level source_file and line info is largely absent and stdlib paths are recorded relative.
5. 5.IR builds were compiled with --emit
6. 6.Binaries over 1 GiB uncompressed are excluded

Licensing and attribution

Every repository carries an OSI-recognized permissive license, verified by database join against the scraper's license records. Copyleft (GPL/AGPL/LGPL/MPL/EPL/EUPL) and unidentified-license repositories are excluded.

LICENSES.csv maps every build directory to its upstream URL, commit and license; the same fields appear in each export.json and in sources/manifest.json. MIT, BSD and Apache-2.0 require that copyright and license notices accompany redistribution — because sources/ ships each tree verbatim, those notices travel with the data rather than only being pointed at. Shipped trees include .git, so they also carry commit history and its author names and emails, as published by those repositories on GitHub.

If you are an author of an included repository and want it removed, open a discussion on this dataset and it will be taken out.

Changlog

• —Aug 25th, 2027 Full dataset upload, 127k binaries.
• —Aug 27th, 2027 If you are using the dataset version between Aug 25, 2026 to Aug 26th, 2026, the citation bib entry is wrong, hallucinated by my agent uploading the new dataset (Claude Opus 5), we sincerely apologize for this mistake.