datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SlimPajama-Meta-rater-Readability-30B
Top 30B token SlimPajama Subset selected by the Readability rater
This repository contains the dataset described in the paper Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models.
Code: https://github.com/opendatalab/Meta-rater
Dataset Description
This dataset contains the top 30B tokens from the SlimPajama-627B corpus, selected using the Readability dimension of the PRRC (Professionalism, Readability, Reasoning, Cleanliness) framework.… See the full description on the dataset page: https://huggingface.co/datasets/opendatalab/SlimPajama-Meta-rater-Readability-30B.low-readability-text
Low Readability Text Dataset
This dataset consists of high-complexity English web text with an estimated readability at or above the U.S. Grade 12 level. The content typically features advanced, highly technical prose or verbose syntactical structures, making it well-suited for researching complex language understanding and automation.
Primary Use Cases
Text Simplification: Training and evaluating models to translate complex text into plain English.
Information… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/low-readability-text.