pico-lm/pretokenized-paloma-tinsy

The Tinsy Pretokenized Paloma Dataset A small version of the pretokenized-paloma benchmark dataset. This dataset is a sub-sampled version of pretokenized-paloma, and can be used in place of pretokenized-paloma. We release the exact scripts we use to create this dataset in our pico-lm/pico-dataset GitHub repo.