fineweb-retrieval/fineweb-edu-index

This dataset contains the embeddings for the full fineweb-edu, embedded with the Cohere Embed V3 model. You can search on this dataset with just 500MB of memory using DiskVectorIndex. Installation & Usage Get your free Cohere API key from cohere.com. You must set this API key as an environment variable: export COHERE_API_KEY=your_api_key Install the package: pip install DiskVectorIndex You can then search via: from DiskVectorIndex import DiskVectorIndex index =… See the full description on the dataset page: https://huggingface.co/datasets/fineweb-retrieval/fineweb-edu-index.