fineweb-retrieval/fineweb-edu-emb
This file contains the embeddings for the full fineweb-edu dataset. The dataset has been deduplicated (using only exact deduplication). The emb folder contains for each parquet file a new_{parquet_name}.npy and old_{parquet}.npy file. The old refers to text that has been seen in the smaller 10B/100B/350B data samples. Cohere embed-multilingual-v3.0 model has been used. The index of the dataset can be found here: https://huggingface.co/datasets/Cohere/fineweb-edu-index The corpus can be found… See the full description on the dataset page: https://huggingface.co/datasets/fineweb-retrieval/fineweb-edu-emb.
Conversations for this repository live on Hugging Face.
CoolFace shows imported repositories read-only. Posting into someone else’s repository from here would need an authorised integration and the account holder’s consent, so the link goes to the source instead.
Open discussions on Hugging Face