llvm-ml/ComPile

Dataset Card for ComPile: A Large IR Dataset from Production Sources

Table of Contents

• —Table of Contents
• —Dataset Description
• —Changelog
• —Dataset Summary
• —Languages
• —Dataset Usage
• —Dataset Structure
• —Data Fields
• —Dataset Size
• —Licensing
• —How to Cite

Dataset Description

• —Homepage: https://llvm-ml.github.io/ComPile/
• —Paper: https://arxiv.org/abs/2309.15432
• —Leaderboard: N/A

Changelog

Dataset Summary

ComPile contains over 2.7TB of permissively-licensed source code compiled to (textual) LLVM intermediate representation (IR) covering C/C++, Rust, Swift, and Julia. The dataset was created by hooking into LLVM code generation either through the language's package manager or the compiler directly to extract the dataset of intermediate representations from production grade programs using our dataset collection utility for the LLVM compilation infrastructure.

Dataset Size

The public release of ComPile contains over 2.7TB of textual LLVM-IR, which tokenizes into 1.3+T tokens using the Llama tokenizer.

ComPile is distributed as bitcode, which is a compressed format that can be easily converted to and from the textual representation of LLVM-IR. To collect token counts, we disassembled the bitcode to convert it into textual IR and then ran a tokenizer over it. We used the standard Llama tokenizer and then ran fastBPE using a custom vocabulary trained on a multi-GB sample of textual IR representativie of all languages in ComPile at two different two different vocab sizes, particularly 10k and 50k. LLVM-IR is quite formulaic, so using custom vocabulary significantly reduces the number of tokens generated.

Languages

The dataset contains 5 programming languages as of v1.0.

"c++", "c", "rust", "swift", "julia"

Dataset Usage

To use ComPile we recommend HuggingFace's datasets library. To e.g. load the dataset:

from datasets import load_dataset

ds = load_dataset('llvm-ml/ComPile', split='train')

By default this will download the entirety of the 550GB+ dataset, and cache it locally at the directory specified by the environment variable HF_DATASETS_CACHE, which defaults to ~/.cache/huggingface. To load the dataset in a streaming format, where the data is not saved locally:

ds = load_dataset('llvm-ml/ComPile', split='train', streaming=True)

For further arguments of load_dataset, please take a look at the loading a dataset documentation, and the streaming documentation. Bear in mind that this is significantly slower than loading the dataset from a local storage. For experimentation that requires more performance but might not require the whole dataset, you can also specify a portion of the dataset to download. For example, the following code will only download the first 10% of the dataset:

ds = load_dataset('llvm-ml/ComPile', split='train[:10%]')

Once the dataset has been loaded, the individual module files can be accessed by iterating through the dataset or accessing specific indices:

# We can iterate through the dataset
next(iter(ds))
# We can also access modules at specific indices
ds[0]

If you're interested in getting textual IR instead of bitcode, you can simply run llvm-dis over the bitcode which will return the IR in textual form. Using Python's subprocess module to do this looks something like this:

bitcode_module = next(iter(ds))['content']
dis_command_vector = ['llvm-dis', '-']
with subprocess.Popen(
    dis_command_vector,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    stdin=subprocess.PIPE) as dis_process:
  output = dis_process.communicate(
      input=bitcode_module)[0].decode('utf-8')
  # the variable output contains the textual IR that can be used downstream.

Filtering and map operations can be performed with the primitives available within the HuggingFace datasets library.

Dataset Structure

Data Fields

Each row in the dataset consists of an individual LLVM-IR Module along with some metadata. There are six columns associated with each row:

• —content (string): This column contains the raw bitcode that composes the module. This can be written to a .bc file and manipulated using the standard llvm utilities or passed in directly through stdin if using something like Python's subprocess.
• —license_expression (string): This column contains the SPDX expression describing the license of the project that the module came from.
• —license_source (string): This column describes the way the license_expression was determined. This might indicate an individual package ecosystem (eg spack), license detection (eg go_license_detector), or might also indicate manual curation (manual).
• —license_files: This column contains an array of license files. These file names map to licenses included in /licenses/licenses-0.parquet.
• —package_source (string): This column contains information on the package that the module was sourced from. This is typically a link to a tar archive or git repository from which the project was built, but might also contain a mapping to a specific package ecosystem that provides the source, such as Spack.
• —language (string): This column indicates the source language that the module was compiled from.

License Constraints and Deduplication

The raw size is the size obtained directly from building all the projects. The license constraints column shows the size per language after license information is taken into account. The last column shows the size when both license constraints and deduplication are taken into account, which is what is included in the dataset.

Note that the sizes displayed here are of the compressed bitcode representation rather than textual IR. We see an expansion ratio of 2-5x, averaging around 4x when converting from compressed bitcode to textual IR. Specific per-language numbers are available in the section above on dataset size.

Dataset Construction

Exact details on how the dataset is constructed are available in our paper describing the dataset. The packages for v1.0 of the dataset were downloaded and built on 1/12/24-1/13/24.

Licensing

The individual modules within the dataset are subject to the licenses of the projects that they come from. License information is available in each row, including the SPDX license expression, the license files, and also a link to the package source where license information can be further validated.

The curation of these modules is licensed under a CC-BY-4.0 license.

Contact Info

1. 1.Aiden Grossman (amgrossman@ucdavis.edu)
2. 2.Ludger Paehler (paehlerludger@gmail.com)
3. 3.Johannes Doerfert (doerfert1@llnl.gov)

How to Cite

Please cite the dataset in the following format:

@article{grossman2023compile,
  title={ComPile: A Large IR Dataset from Production Sources},
  author={Grossman, Aiden and Paehler, Ludger and Parasyris, Konstantinos and Ben-Nun, Tal and Hegna, Jacob and Moses, William and Diaz, Jose M Monsalve and Trofin, Mircea and Doerfert, Johannes},
  journal={arXiv preprint arXiv:2309.15432},
  year={2023}
}