Datasetpublic

lapa-llm/pretraining-lower-quality

Dataset Card for Lapa Pretraining Lower Quality Dataset Dataset Description Dataset Summary This dataset is a high quality (but lower quality than https://huggingface.co/datasets/lapa-llm/pretraining-high-qualit) subset of pretraining corpus for Ukrainian language. It was filtered using 6 models, measuring different quality aspects of the data: lapa-llm/alignment-score-model - Alignment - filtering for disinformation lapa-llm/gec-score-model - Grammatical… See the full description on the dataset page: https://huggingface.co/datasets/lapa-llm/pretraining-lower-quality.

sourceHugging Facecc-by-4.0updated 10mo agoView on Hugging Face

0likes103downloads

Dataset Card

Dataset Card for Lapa Pretraining Lower Quality Dataset

Dataset Description

Dataset Summary

This dataset is a high quality (but lower quality than https://huggingface.co/datasets/lapa-llm/pretraining-high-qualit) subset of pretraining corpus for Ukrainian language.

It was filtered using 6 models, measuring different quality aspects of the data:

—lapa-llm/alignment-score-model - Alignment - filtering for disinformation
—lapa-llm/gec-score-model - Grammatical Correctness of the text
—lapa-llm/fineweb-nemotron-edu-score - Educational Value of the text
—lapa-llm/fineweb-mixtral-edu-score - Educational Value of the text
—lapa-llm/manipulative-score-model - How manipulative is the text
—lapa-llm/fasttext-quality-score - Text Coherence (how close is it to ELI5 from Reddit style of explanations)

All models are available in this collection: https://huggingface.co/collections/lapa-llm/lapa-v012-pretraining

We perform CDF binning and perform max ensembling, then select data only from the highest performing bucket, which is this dataset. This dataset consists of CDF bins from 12 to 18.

Additional measure, formula-score, combines most of them into one metric:

python

def formula_score(item):
     item["formula-score"] = np.median([item["fineweb-nemotron-edu-score"], item["fineweb-mixtral-edu-score"], item["fasttext-quality-score"],]) * item["alignment-score"] * item["manipulative-score"] * item["gec-score"]
     return item

This provides a balanced measure of quality across all classifiers.

Languages

—Ukrainian (uk)

Dataset Creation

Source Data

—Base dataset: Kobza, FinePDFs, FineWeb, UberText

Intended Uses

—Instruction/chat LLM training in Ukrainian
—Continued pretraining or SFT on multi-turn dialogs
—Research on adapting Hermes-style dialogs to Ukrainian

Considerations for Using the Data

Social Impact

Aims to strengthen the Ukrainian-language LLM ecosystem and improve accessibility of language technology for Ukrainian speakers.

Citation

BibTeX

TBD

License

CC-BY-SA-4.0

This dataset is part of the "Lapa" - Ukrainian LLM initiative to advance natural language processing for the Ukrainian language.