Datasetpublic

sudy-super/piece-of-refined-oscar

Descrption This dataset is part of the OSCAR-2301 cleaned. There are about 0.5b tokens counted by calm2 tokenizer. NOTE This dataset has not passed sentence end boundary determination or Perplexity Filtering, so there is room for improvement in quality.

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face

7likes33downloads

Dataset Card

Descrption

This dataset is part of the OSCAR-2301 cleaned.

There are about 0.5b tokens counted by calm2 tokenizer.

NOTE

This dataset has not passed sentence end boundary determination or Perplexity Filtering, so there is room for improvement in quality.