datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
project_gutenberg_preprocessed
Gutenberg
Our version of the project gutenberg corpus, so as used to pretrain Apertus (v1 being used before 9T, v2 between 9T and 12T).
More details about data provenance, preparation, and statistics can be found in our tech report.
Sampling, filtering and data-preparation scripts can be found in our dedicated GitHub repository.
Feel free to reach out for any questions or suggestions 😊
lfqa-preprocessed-ittexthumanizer-preprocessed-dataPreprocessed_Solidity_Dataset_V1This dataset consists of 4,134 unique Solidity files. The files were gathered from three sources: Etherscan, Github and DISL dataset. Six preprocessing steps were applied:
Step 1 "Cleaning": Unnecessary parts such as comments or blank lines were removed from each file.
Step 2 "Formatting": Each file was converted with Prettier (and the corresponding Solidity-plugin) so that the final model only generates code in a correct format.
Step 3 "Slither Analysis": Each file has been checked for… See the full description on the dataset page: https://huggingface.co/datasets/fbnhnsl/Preprocessed_Solidity_Dataset_V1.