CoolFace
Datasetpublic

Sicheng-Chroma/wikipedia-en-splade-bge

Wikipedia English with SPLADE and BGE-M3 Pre-computed SPLADE sparse and BGE-M3 dense embeddings for 6.4M English Wikipedia articles. Direct Usage HuggingFace automatically discovers parquet files. You can load this dataset directly: from datasets import load_dataset # Stream the entire dataset (recommended for large dataset) dataset = load_dataset("Sicheng-Chroma/wikipedia-en-splade-bge", streaming=True) # Load specific splits train_dataset =… See the full description on the dataset page: https://huggingface.co/datasets/Sicheng-Chroma/wikipedia-en-splade-bge.

sourceHugging Facecc-by-sa-4.0updated 1y agoView on Hugging Face
1likes89downloads
Dataset Card

Wikipedia English with SPLADE and BGE-M3

Pre-computed SPLADE sparse and BGE-M3 dense embeddings for 6.4M English Wikipedia articles.

Dataset Description

  • —Source: HuggingFaceFW/clean-wikipedia (English)
  • —Size:
  • —Train: 6,406,711 documents
  • —Test: 256 queries
  • —Format:
  • —Train: 7 Parquet files (~1M records each) in train/ directory
  • —Test: 1 Parquet file in test/ directory
  • —Embeddings:
  • —Sparse: SPLADE PP - ~265 non-zero dims
  • —Dense: BGE-M3 - 1024 dimensions

Direct Usage

HuggingFace automatically discovers parquet files. You can load this dataset directly:

python
from datasets import load_dataset

# Stream the entire dataset (recommended for large dataset)
dataset = load_dataset("Sicheng-Chroma/wikipedia-en-splade-bge", streaming=True)

# Load specific splits
train_dataset = load_dataset("Sicheng-Chroma/wikipedia-en-splade-bge", split="train")
test_dataset = load_dataset("Sicheng-Chroma/wikipedia-en-splade-bge", split="test")

# Load specific chunk
dataset = load_dataset("Sicheng-Chroma/wikipedia-en-splade-bge", 
                       data_files="train/train-00000-of-00007.parquet")

# Load multiple chunks
dataset = load_dataset("Sicheng-Chroma/wikipedia-en-splade-bge", 
                       data_files=["train/train-00000-of-00007.parquet", "train/train-00001-of-00007.parquet"])

Dataset Structure

Each row contains:

FieldTypeDescription
textstringFull Wikipedia article text
titlestringArticle title
urlstringWikipedia URL
sparse_embedding_indiceslist[int32]SPLADE indices (non-zero positions)
sparse_embedding_valueslist[float32]SPLADE values (weights)
dense_embeddinglist[float32]BGE-M3 1024-dim dense vector

Example Usage

python
from datasets import load_dataset

# Load the dataset (streaming recommended for large dataset)
ds = load_dataset("Sicheng-Chroma/wikipedia-en-splade-bge", streaming=True)

# Iterate through articles
for article in ds['train']:
    print(f"Title: {article['title']}")
    print(f"URL: {article['url']}")
    print(f"Text preview: {article['text'][:200]}...")
    print(f"Sparse embedding: {len(article['sparse_embedding_indices'])} non-zero dims")
    print(f"Dense embedding: {len(article['dense_embedding'])} dims")
    break

# Load into memory (for smaller chunks)
dataset = load_dataset("Sicheng-Chroma/wikipedia-en-splade-bge", 
                       data_files="train/train-00000-of-00007.parquet")
print(f"Loaded {len(dataset['train'])} articles")

File Organization

.
├── train/
│   ├── train-00000-of-00007.parquet - First 1M articles
│   ├── train-00001-of-00007.parquet - Next 1M articles
│   ├── train-00002-of-00007.parquet
│   ├── train-00003-of-00007.parquet
│   ├── train-00004-of-00007.parquet
│   ├── train-00005-of-00007.parquet
│   └── train-00006-of-00007.parquet - Final ~400K articles
└── test/
    └── test-00000-of-00001.parquet - 256 test queries

License

CC BY-SA 4.0 (inherited from Wikipedia)