CoolFace
Datasetpublic

KyrieX/cole-arxiv-cc-e5-retriever

CoLe arXiv CC E5 Retriever A non-Wikipedia retrieval corpus for the CoLe/R2–R4a experiments. It contains 280,738 arXiv documents selected from the Common Pile filtered arXiv collection, plus an E5-base-v2 dense index. Data provenance and license handling Source: common-pile/arxiv_papers_filtered, revision 033cf7f. The source records are converted arXiv papers with per-document license metadata. This release keeps records whose metadata is CC BY, CC0, or Public… See the full description on the dataset page: https://huggingface.co/datasets/KyrieX/cole-arxiv-cc-e5-retriever.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face
0likes139downloads
Dataset Card

CoLe arXiv CC E5 Retriever

A non-Wikipedia retrieval corpus for the CoLe/R2–R4a experiments. It contains 280,738 arXiv documents selected from the Common Pile filtered arXiv collection, plus an E5-base-v2 dense index.

Data provenance and license handling

  • —Source: `common-pile/arxiv_papers_filtered`, revision 033cf7f.
  • —The source records are converted arXiv papers with per-document license metadata.
  • —This release keeps records whose metadata is CC BY, CC0, or Public Domain. CC BY-SA records are excluded by default because the share-alike condition needs separate downstream handling.
  • —The per-document license, url, source shard, and content hash are retained in arxiv_cc_manifest.jsonl.gz.
  • —The source dataset card notes that license metadata can be incomplete or inaccurate. This release does not provide a blanket legal warranty: users are responsible for checking the record-level license and complying with the applicable terms before redistribution or use.

The resulting corpus is a mixed-license collection, so the repository metadata intentionally uses license: other rather than claiming a single corpus-wide license.

Files

  • —hf_shards/arxiv_cc_corpus.jsonl.gz.part-*: ordered 16 MiB shards of arxiv_cc_corpus.jsonl.gz; concatenate them to reconstruct the gzip corpus.
  • —hf_shards/arxiv_cc_manifest.jsonl.gz.part-*: ordered shards of the provenance/license manifest.
  • —hf_shards/emb_e5.memmap.part-*: ordered shards of the float32 passage embeddings, shape 280,738 x 768.
  • —hf_shards/e5_Flat.index.part-*: ordered shards of the FAISS IndexFlatIP file.
  • —index/index_config.json: embedding and index parameters.
  • —arxiv_cc_corpus.jsonl.gz, arxiv_cc_manifest.jsonl.gz, index/emb_e5.memmap, and index/e5_Flat.index are retained in the local build directory but are represented as shards in this Hub release so each network transfer stays small.
  • —stats.json: source and filtering counts.
  • —build_arxiv_cc_e5_retriever.py: reproducible build/verification script.
  • —checksums.sha256: SHA256 checksums for the local complete files.

To reconstruct the complete files after downloading the repository, run:

bash
cat hf_shards/arxiv_cc_corpus.jsonl.gz.part-* > arxiv_cc_corpus.jsonl.gz
cat hf_shards/arxiv_cc_manifest.jsonl.gz.part-* > arxiv_cc_manifest.jsonl.gz
cat hf_shards/emb_e5.memmap.part-* > index/emb_e5.memmap
cat hf_shards/e5_Flat.index.part-* > index/e5_Flat.index

Embedding recipe

The index uses intfloat/e5-base-v2 with the Search-R1-compatible recipe: passage: prefix, mean pooling over the attention mask, L2 normalization, 256-token truncation, float32 storage, and inner-product FAISS search. Queries should use the E5 query: prefix when encoded with the same model.

Minimal use

python
import faiss
from datasets import load_dataset

corpus = load_dataset("json", data_files="arxiv_cc_corpus.jsonl.gz", split="train")
index = faiss.read_index("index/e5_Flat.index")
# corpus[i]["contents"] corresponds to vector i.

To reproduce locally, run the build script in this repository with download-source, build-corpus, build-index, and verify as documented by --help.

References