ezipe/lichess_2023_janoct_shards

Installation

pip install datasets numpy zstandard

Usage

i've given up on trying to get this to work natively with huggingface datasets. this is nice because it allows streaming (https://huggingface.co/docs/datasets/en/aboutmapstylevs_iterable) and has functions like 'map' which easily parallelize operations over the dataset. maybe i'll try to get this working in the future, but for now it gets stuck as the download and extracted zstd files are not decompressible for some reason, and it has to rewrite the entire dataset into arrow first.

<!-- ``` from datasets import load_dataset

dataset = loaddataset("ezipe/lichess2023janoctshards")

or more slowly,

kconda install git-lfs git lfs install git clone https://huggingface.co/datasets/ezipe/lichess2023janoct_shards/