Datasetpublic

rulins/MassiveDS-140B

We release the raw passages, embeddings, and index of MassiveDS. Website: https://retrievalscaling.github.io We release two versions of MassiveDS: MassiveDS-1.4T, which contains 1.4T tokens in the datastore. MassiveDS-140B, which is a subsampled version containing 140B tokens in the datastore. File structure: raw_data: plain data in JSONL files. passages: chunked raw passages with passage IDs. Each passage is chunked to have no more than 256 words. embeddings: embeddings of the passages… See the full description on the dataset page: https://huggingface.co/datasets/rulins/MassiveDS-140B.

sourceHugging Facemitupdated 2y agoView on Hugging Face

7likes3.7kdownloads

Dataset Card

We release the raw passages, embeddings, and index of MassiveDS.

Website: https://retrievalscaling.github.io

We release two versions of MassiveDS:

1.MassiveDS-1.4T, which contains 1.4T tokens in the datastore.
2.MassiveDS-140B, which is a subsampled version containing 140B tokens in the datastore.

File structure:

—raw_data: plain data in JSONL files.
—passages: chunked raw passages with passage IDs. Each passage is chunked to have no more than 256 words.
—embeddings: embeddings of the passages encoded with Contriever-MSMACRO.
—index: flat index built with embeddings.

Note: due to large data volume, we are still uploading the data for MassiveDS-1.4T. MassiveDS-140B is ready to go. Code support to run with MassiveDS is in https://github.com/RulinShao/retrieval-scaling.