seonglae/wikipedia-256

This is Wikidedia passages dataset for ODQA retriever. Each passages have 256~ tokens splitteed by gpt-4 tokenizer using tiktoken. Token count {'~128': 1415068, '128~256': 1290011, '256~512': 18756476, '512~1024': 667, '1024~2048': 12, '2048~4096': 0, '4096~8192': 0, '8192~16384': 0, '16384~32768': 0, '32768~65536': 0, '65536~128000': 0, '128000~': 0} Text count {'~512': 1556876,'512~1024': 6074975, '1024~2048': 13830329, '2048~4096': 49, '4096~8192': 2, '8192~16384': 3, '16384~32768': 0… See the full description on the dataset page: https://huggingface.co/datasets/seonglae/wikipedia-256.