fromziro/jetoncount_corpus

JetonCount's Corpus

This is the corpus used to train JetonCount.

JSONL Format

{
  "source_file": "token_stats\\HuggingFaceFW_fineweb-edu00000.jsonl",
  "dataset_dir": "HuggingFaceFW/fineweb-edu",
  "index": 4020,
  "chars": 2178,
  "words": 335,
  "avg_chars_per_word": 5.504478,
  "longest_word_chars": 33,
  "punctuation_ratio": 0.037649,
  "symbol_ratio": 0.00551,
  "tokens": 664,
  "vocab_size": 2560,
  "tokenizer_dir": "fromziro/Er-Tiny-1.3M"
}

Field Descriptions

1. 1.source_file: The original file used to generate the statistics on our local system.
2. 2.dataset_dir: The Hugging Face dataset identifier from which the sample originated.
3. 3.index: The index of the sample within the dataset when iterating through it using the Hugging Face datasets library.
4. 4.chars: The total number of characters in the text sample.
5. 5.words: The total number of words in the text sample.
6. 6.avg_chars_per_word: The average number of characters per word in the text sample.
7. 7.longest_word_chars: The length, in characters, of the longest word in the text sample.
8. 8.punctuation_ratio: The fraction of characters in the text sample that are punctuation characters.
9. 9.symbol_ratio: The fraction of characters in the text sample that are symbols (such as @, #, $, %, &, etc.).
10. 10.tokens: The number of tokens produced by the tokenizer specified in tokenizer_dir.
11. 11.vocab_size: The vocabulary size of the tokenizer used to generate the token count. (Ranges from 250 to 256,000)
12. 12.tokenizer_dir: The Hugging Face repository containing the tokenizer used to generate the token count.

Use Cases

1. 1.Training ML models: Train token count predictors, tabular models, neural networks, gradient boosting models, or anything else you want.
2. 2.Tokenizer research: Study how text characteristics such as length, vocabulary size, and symbol density affect tokenization behavior.
3. 3.Benchmarking tokenizers: Compare tokenization efficiency across different tokenizers and vocabulary sizes using a large and diverse collection of text samples.
4. 4.Feature engineering research: Explore which textual features are most predictive of token counts and develop new feature sets for token estimation tasks.
5. 5.Dataset analysis: Analyze the statistical properties of large text corpora.
6. 6.Educational purposes: Learn about tokenization, text statistics, feature engineering, and regression.
7. 7.Anything else: The dataset is released to the community to be used in any way.