latam-gpt/red_pajama_es_hq
RedPajama's High Quality Spanish subset What is this? The following is a high-quality dataset distilled from the Spanish subsection of RedPajama-Data-v2, created using the methodology proposed in FineWEB-Edu. Usage from datasets import load_dataset ds = load_dataset("latam-gpt/red_pajama_es_hq") Filtering by quality score Documents in this corpus are scored on academic quality from 2.5 to 5, with higher scores indicating better… See the full description on the dataset page: https://huggingface.co/datasets/latam-gpt/red_pajama_es_hq.
RedPajama's High Quality Spanish subset
What is this?
The following is a high-quality dataset distilled from the Spanish subsection of RedPajama-Data-v2, created using the methodology proposed in FineWEB-Edu.
Usage
from datasets import load_dataset
ds = load_dataset("latam-gpt/red_pajama_es_hq") Filtering by quality score
Documents in this corpus are scored on academic quality from 2.5 to 5, with higher scores indicating better quality. The dataset can be filtered by score using standard filtering methods.
from datasets import load_dataset
ds = load_dataset("latam-gpt/red_pajama_es_hq")
# filter the dataset for scores > 3
filtered_ds = ds.filter(lambda x: x['score'] > 3)Dataset creation
In a nutshell, we use Llama-3.1-70B to grade the educational quality of 550k samples from the original dataset. Then, we used these samples to train a encoder-based classifier, so that it learns to assign a score from 0 to 5. Since this model is cheaper to use than a GPT, we can run it at scale over the entire dataset, thus allowing us to filter a high-quality section from it.
Here is an overview of the architecture:
<div align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/61b15c3f20037ec5d7c91aa6/H5xPOHy_4RhMEDtGvsnTE.png" width="400"> </div>
For more detailed information on how this dataset was created, refer to our open implementation.
What is Latam-GPT?
Latam-GPT is a Latin American initiative to develop a large language model built entirely in the region. The project encompasses all development stages — from data collection and pre-training to final model refinement — making it the first foundation model created completely within Latin America.
License
The text documents of the source database (RedPajama-Data-v2) were collected using 84 CommonCrawl snapshots, processed using the CCNet pipeline, and also provided under an Apache 2.0 license by the Together Computer team under the jurisdiction of the United States of America. There may be differences between the jurisdiction of the USA and Latin American countries. In order to comply with the terms of use of the Common Crawl Foundation and in the search for the greatest possible transparency, we provide the following contact to ask any questions, comments or complaints: eugenio.herrera@cenia.cl.