datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
pi-extensions-sessions
Coding agent session traces for thomasmustier/pi-extensions-sessions
This dataset contains redacted coding agent session traces collected while working on tmustier/pi-extensions. The traces were exported with pi-share-hf from a local pi workspace and filtered to keep only sessions that passed deterministic redaction and LLM review.
Data description
Each *.jsonl file is a redacted pi session. Sessions are stored as JSON Lines files where each line is a structured… See the full description on the dataset page: https://huggingface.co/datasets/thomasmustier/pi-extensions-sessions.finewebedu-20BThis is a subset of the HuggingFaceFW/fineweb-edu/100BT dataset.
I extracted (in order) the initial 20,200,000 rows where, ideally, 20M are meant for training and 200k for validation.
Tokenised configs:
bpe32000minipile: 21.6B tokens
License
For the license, refer to the original dataset (HuggingFaceFW/fineweb-edu).
pile-deduped-pythia-preshuffledThis dataset contains the fully prepared data, which has been tokenized and pre-shuffled, used to train the Pythia (deduplicated) models.
You can find these models under the EleutherAI organisation, and they are also listed in my Memorisation-Profiles collection.
This data is the same as the one found in EleutherAI/pile-deduped-pythia-preshuffled,
but it is presented in a more manageable format. Instead of using the Megatron format used by the GPT-NeoX library, I have stored the data in a… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/pile-deduped-pythia-preshuffled.pierogue
Pierogue
Pierogue is a small open-licensed machine-generated dataset that contains fifteen short texts in English covering five topics, provided with the relevance judgements (qrels), designed for educational purposes.
Topics: cosmos, nature, music, technology, fashion
Splits: train (10 documents, 375 qrels) and test (5 documents, 150 qrels)
Texts were generated by ChatGPT 3.5. Queries, qrels, and analogies were generated by GPT-4. Words were provided with Word2Vec embeddings… See the full description on the dataset page: https://huggingface.co/datasets/dustalov/pierogue.pie-gem5-pairs
PIE gem5-timed code optimization (src,tgt pairs)
C++ program-optimization data derived from the PIE dataset
("Learning Performance-Improving Code Edits"),
re-timed end-to-end with gem5 (x86 Skylake, syscall-emulation mode) at
per-test-case granularity. One row per official (source, target) program pair.
This dataset is reward-agnostic: it ships the full per-test-case reference timings and
case manifests so a downstream RL / eval pipeline decides at runtime how many cases to use… See the full description on the dataset page: https://huggingface.co/datasets/stablegradients/pie-gem5-pairs.pie-gem5-bysrc
PIE gem5-timed code optimization (per source program)
C++ program-optimization data derived from the PIE dataset
("Learning Performance-Improving Code Edits"),
re-timed end-to-end with gem5 (x86 Skylake, syscall-emulation mode) at
per-test-case granularity. One row per unique (problem, source) program; the best surviving target is kept as an oracle ceiling.
This dataset is reward-agnostic: it ships the full per-test-case reference timings and
case manifests so a downstream RL /… See the full description on the dataset page: https://huggingface.co/datasets/stablegradients/pie-gem5-bysrc.pile-deduped
Repo Structure
Each file contains 1M documents (apart from the last file, which contains the remaining documents).
Each file is around 2GB in size (slight differences are due to certain documents being longer or shorter than the "average" across files).
Each document has a unique id assigned (a simply sequential int).
/data: The raw documents. This config is the same as EleutherAI/the_pile_deduplicated.
One minor point is that, instead of copying those data, I detokenised the data… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/pile-deduped.