Datasetpublic

fineweb-retrieval/fineweb-edu-index

This dataset contains the embeddings for the full fineweb-edu, embedded with the Cohere Embed V3 model. You can search on this dataset with just 500MB of memory using DiskVectorIndex. Installation & Usage Get your free Cohere API key from cohere.com. You must set this API key as an environment variable: export COHERE_API_KEY=your_api_key Install the package: pip install DiskVectorIndex You can then search via: from DiskVectorIndex import DiskVectorIndex index =… See the full description on the dataset page: https://huggingface.co/datasets/fineweb-retrieval/fineweb-edu-index.

sourceHugging Facecc-by-nc-sa-4.0updated 1y agoView on Hugging Face

0likes25kdownloads

Dataset Card

This dataset contains the embeddings for the full fineweb-edu, embedded with the Cohere Embed V3 model.

You can search on this dataset with just 500MB of memory using DiskVectorIndex.

Installation & Usage

Get your free Cohere API key from cohere.com. You must set this API key as an environment variable:

export COHERE_API_KEY=your_api_key

Install the package:

pip install DiskVectorIndex

You can then search via:

python

from DiskVectorIndex import DiskVectorIndex

index = DiskVectorIndex("Cohere/fineweb-edu-index")

while True:
    query = input("\n\nEnter a question: ")
    docs = index.search(query, top_k=3)
    for doc in docs:
        print(doc)
        print("=========")

License

Please observe the License for the Fineweb-edu. The license displayed here is just for the embeddings.