alibayram/cosmos-corpus-encoded

Cosmos Corpus Encoded for Embedding Distillation This dataset is a pre-tokenized version of alibayram/cosmos-corpus-0-05-with-embeddings, designed for efficient embedding distillation training of MFT and TabiBERT models. Pre-processing & Filtering The dataset was processed using two different tokenizers to support multiple student architectures: MFT Tokenizer: A custom morphologically informed tokenizer. TabiBERT Tokenizer: A BERT-based tokenizer with 32k… See the full description on the dataset page: https://huggingface.co/datasets/alibayram/cosmos-corpus-encoded.