Datasetpublic

MonumentalSystems/chinchilla-master-corpus-v1

Chinchilla Master Corpus v1 This dataset contains parquet shards for a curated text corpus with train and validation splits. Files data/train-*.parquet data/validation-*.parquet _dedup.sqlite Notable columns text, domain, doremi_domain, doremi_weight, source_id, source_ref, source_type, split_source, content_type, subject, reading_level, complexity, curriculum_stage, difficulty, flesch, mtld, quality_pass, quality_reason, word_count, char_count… See the full description on the dataset page: https://huggingface.co/datasets/MonumentalSystems/chinchilla-master-corpus-v1.

sourceHugging Faceotherupdated 5mo agoView on Hugging Face

0likes410downloads

Dataset Card

Chinchilla Master Corpus v1

This dataset contains parquet shards for a curated text corpus with train and validation splits.

Files

—data/train-*.parquet
—data/validation-*.parquet
—_dedup.sqlite

Notable columns

text, domain, doremi_domain, doremi_weight, source_id, source_ref, source_type, split_source, content_type, subject, reading_level, complexity, curriculum_stage, difficulty, flesch, mtld, quality_pass, quality_reason, word_count, char_count, token_count_est, checksum, row_id