redpajama
RedPajama-Data-V2RedPajama V2: an Open Dataset for Training Large Language Modelsraw_redpajamas
Getting Started
The dataset is built from the redpajamas dataset after filtering by marketing keywords list that can be found here
The full scripts to recreate the raw dataset before sharding can be found here.
The dataset includes:
~4.8B tokens from raw contents.
Downloading the dataset
To start exploring and get to know the dataset you can run the script:
import datasets
ds = datasets.load_dataset("marketeam/raw_redpajamas", split="train")
for sample in ds:… See the full description on the dataset page: https://huggingface.co/datasets/marketeam/raw_redpajamas.RedPajama-Data-1TRedPajama is a clean-room, fully open-source implementation of the LLaMa dataset.RedPajama-Data-1T-Sample-Backupredpajama-data-v2_urls
Dataset Card for redpajama-data-v2_urls
This dataset provides the URLs and top-level domains associated with training records in togethercomputer/RedPajama-Data-V2. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/redpajama-data-v2_urls.RedPajama-Data-1T-Sample