CoolFace
Datasetpublic

alibayram/cosmos-corpus-encoded

Cosmos Corpus Encoded for Embedding Distillation This dataset is a pre-tokenized version of alibayram/cosmos-corpus-0-05-with-embeddings, designed for efficient embedding distillation training of MFT and TabiBERT models. Pre-processing & Filtering The dataset was processed using two different tokenizers to support multiple student architectures: MFT Tokenizer: A custom morphologically informed tokenizer. TabiBERT Tokenizer: A BERT-based tokenizer with 32k… See the full description on the dataset page: https://huggingface.co/datasets/alibayram/cosmos-corpus-encoded.

sourceHugging Facemitupdated 8mo agoView on Hugging Face
0likes88downloads
Dataset Card

Cosmos Corpus Encoded for Embedding Distillation

This dataset is a pre-tokenized version of alibayram/cosmos-corpus-0-05-with-embeddings, designed for efficient embedding distillation training of MFT and TabiBERT models.

Dataset Description

  • —Source: alibayram/cosmos-corpus-0-05-with-embeddings
  • —Language: Turkish
  • —Task: Embedding Distillation (Teacher-Student Training)
  • —Total Examples: 224,807 (Filtered from 300,000)
  • —Max Sequence Length: 2048 tokens

Pre-processing & Filtering

The dataset was processed using two different tokenizers to support multiple student architectures:

  1. 1.MFT Tokenizer: A custom morphologically informed tokenizer.
  2. 2.TabiBERT Tokenizer: A BERT-based tokenizer with 32k vocabulary.

Filtering:

  • —Original size: 300,000 examples.
  • —Filtered size: 224,807 examples (~75%).
  • —Criterion: Both mft_input_ids and tabi_input_ids must be <= 2048 tokens.
  • —Sequences longer than 2048 tokens were dropped to ensure efficient training within context limits.

Dataset Structure

The dataset contains the following columns:

ColumnTypeDescription
textstringThe original raw text content.
mft_input_idslist[int]Token IDs encoded using the MFT tokenizer.
tabi_input_idslist[int]Token IDs encoded using the TabiBERT tokenizer.
teacher_embedding_finallist[float]Final layer embeddings from the teacher model (Gemma-2-9b-it).

Data Instances

python
{
  'text': 'Makine öğrenmesi, verilerden öğrenen algoritmaların çalışılmasıdır.',
  'mft_input_ids': [124, 5921, ...],
  'tabi_input_ids': [101, 2341, ...],
  'teacher_embedding_final': [0.021, -0.054, ...]  # 3584-dimensional vectors
}

Usage

This dataset is optimized for the EmbeddingDistillationTrainer. You can load it directly without needing to re-tokenize during training.

python
from datasets import load_dataset

dataset = load_dataset("alibayram/cosmos-corpus-encoded")

Training Example

To train a model using the mft_input_ids column:

python
from embedding_trainer import EmbeddingDistillationTrainer, EmbeddingTrainerConfig

config = EmbeddingTrainerConfig(
    student_model="alibayram/mft-downstream-task-embeddinggemma",
    input_ids_column="mft_input_ids",  # or "tabi_input_ids"
    embedding_column="teacher_embedding_final",
    loss_type="cosine",
    batch_size=256
)

trainer = EmbeddingDistillationTrainer(config)
trainer.train("alibayram/cosmos-corpus-encoded")

Creation Details

  • —Created by: Ali Bayram
  • —Date: 2026-01-25
  • —Teacher Model: google/gemma-2-9b-it (Embeddings extracted via sartify-llm/Gemma-2-9b-it-v2-embedding)
  • —Processing Script: prepare_dataset.py

License

MIT