togethercomputer/RedPajama-Data-1T

1---2task_categories:3- text-generation4language:5- en6pretty_name: Red Pajama 1T7---8### Getting Started9 10The dataset consists of 2084 jsonl files.11You can download the dataset using HuggingFace:12```python13from datasets import load_dataset14ds = load_dataset("togethercomputer/RedPajama-Data-1T")15```16 17Or you can directly download the files using the following command:18 19```20wget 'https://data.together.xyz/redpajama-data-1T/v1.0.0/urls.txt'21while read line; do22    dload_loc=${line#https://data.together.xyz/redpajama-data-1T/v1.0.0/}23    mkdir -p $(dirname $dload_loc)24    wget "$line" -O "$dload_loc"25done < urls.txt26```27 28After downloading the files, you can load the dataset from disk by setting the `RED_PAJAMA_DATA_DIR` environment variable to the directory containing the files:29 30```python31import os32from datasets import load_dataset33os.environ["RED_PAJAMA_DATA_DIR"] = "/path/to/download"34ds = load_dataset("togethercomputer/RedPajama-Data-1T")35```36 37A smaller 1B-token sample of the dataset can be found [here](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T-Sample).38 39A full set of scripts to recreate the dataset from scratch can be found [here](https://github.com/togethercomputer/RedPajama-Data).40 41### Dataset Summary42 43RedPajama is a clean-room, fully open-source implementation of the LLaMa dataset.44 45| Dataset       | Token Count |46|---------------|-------------|47| Commoncrawl   | 878 Billion        |48| C4            | 175 Billion        |49| GitHub        | 59 Billion         |50| ArXiv         | 28 Billion         |51| Wikipedia     | 24 Billion         |52| StackExchange | 20 Billion         |53| Total         | 1.2 Trillion      |54 55### Languages56 57Primarily English, though the Wikipedia slice contains multiple languages.58 59## Dataset Structure60 61The dataset structure is as follows:62 63```json64{65    "text": ...,66    "meta": {"url": "...", "timestamp": "...", "source": "...", "language": "...", ...},67    "red_pajama_subset": "common_crawl" | "c4" | "github" | "arxiv" | "wikipedia" | "stackexchange"68}69```70 71## Dataset Creation72 73This dataset was created to follow the LLaMa paper as closely as possible to try to reproduce its recipe.74 75### Source Data76 77#### Commoncrawl78 79We download five dumps from Commoncrawl, and run the dumps through the official `cc_net` pipeline.80We then deduplicate on the paragraph level, and filter out low quality text using a linear classifier trained to 81classify paragraphs as Wikipedia references or random Commoncrawl samples.82 83#### C484 85C4 is downloaded from Huggingface. The only preprocessing step is to bring the data into our own format.86 87#### GitHub88 89The raw GitHub data is downloaded from Google BigQuery. We deduplicate on the file level and filter out low quality 90files and only keep projects that are distributed under the MIT, BSD, or Apache license.91 92#### Wikipedia93We use the Wikipedia dataset available on Huggingface, which is based on the Wikipedia dump from 2023-03-20 and contains94text in 20 different languages. The dataset comes in preprocessed format, so that hyperlinks, comments and other 95formatting boilerplate has been removed.96 97#### Gutenberg and Books398 99 100<div class="course-tip course-tip-orange bg-gradient-to-br dark:bg-gradient-to-r before:border-orange-500 dark:before:border-orange-800 from-orange-50 dark:from-gray-900 to-white dark:to-gray-950 border border-orange-50 text-orange-700 dark:text-gray-400">101  <p><b>Defunct:</b> The 'book' config is defunct and no longer accessible due to reported copyright infringement for the Book3 dataset contained in this config.</p>102</div>103 104 105#### ArXiv106ArXiv data is downloaded from Amazon S3 in the `arxiv` requester pays bucket. We only keep latex source files and 107remove preambles, comments, macros and bibliographies.108 109#### Stackexchange110The Stack Exchange split of the dataset is download from the 111[Internet Archive](https://archive.org/download/stackexchange). Here we only keep the posts from the 28 largest sites,112remove html tags, group the posts into question-answer pairs, and order answers by their score.113 114### SHA256 Checksums115 116SHA256 checksums for the dataset files for each data source are available here:117 118```119https://data.together.xyz/redpajama-data-1T/v1.0.0/sha256/arxiv_SHA256SUMS.txt120https://data.together.xyz/redpajama-data-1T/v1.0.0/sha256/c4_SHA256SUMS.txt121https://data.together.xyz/redpajama-data-1T/v1.0.0/sha256/common_crawl_SHA256SUMS.txt122https://data.together.xyz/redpajama-data-1T/v1.0.0/sha256/github_SHA256SUMS.txt123https://data.together.xyz/redpajama-data-1T/v1.0.0/sha256/stackexchange_SHA256SUMS.txt124https://data.together.xyz/redpajama-data-1T/v1.0.0/sha256/wikipedia_SHA256SUMS.txt125```126 127To cite RedPajama, please use:128 129```130@software{together2023redpajama,131  author = {Together Computer},132  title = {RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset},133  month = April,134  year = 2023,135  url = {https://github.com/togethercomputer/RedPajama-Data}136}137```138 139### License140Please refer to the licenses of the data subsets you use.141 142* [Common Crawl Foundation Terms of Use](https://commoncrawl.org/terms-of-use/full/)143* [C4 license](https://huggingface.co/datasets/allenai/c4#license)144* GitHub was limited to MIT, BSD, or Apache licenses only145* [ArXiv Terms of Use](https://info.arxiv.org/help/api/tou.html)146* [Wikipedia License](https://huggingface.co/datasets/wikipedia#licensing-information)147* [StackExchange license on the Internet Archive](https://archive.org/details/stackexchange)148 149<!--150### Annotations151#### Annotation process152[More Information Needed]153#### Who are the annotators?154[More Information Needed]155### Personal and Sensitive Information156[More Information Needed]157## Considerations for Using the Data158### Social Impact of Dataset159[More Information Needed]160### Discussion of Biases161[More Information Needed]162### Other Known Limitations163[More Information Needed]164## Additional Information165### Dataset Curators166[More Information Needed]167### Licensing Information168[More Information Needed]169### Citation Information170[More Information Needed]171### Contributions172[More Information Needed]173-->