splade
Datasets
All datasets matching “splade”japanese-splade-v1-hard-negatives日本語 SPLADE v2 の学習に用いたデータセットです。
SPLADE モデルである hotchpotch/japanese-splade-base-v1-mmarco-only, japanese-splade-base-v1_5 を用いてハードネガティブマイニングを行なっています。また BAAI/bge-reranker-v2-m3 を用いたリランカースコアを付与しています。
mqa, mmarco はhpprc/emb のデータを用いています。
mqa の query 作成には MinHash を用い約40万件になるようフィルタしました。
msmarco-ja は hpprc/msmarco-jaのデータを用いています。
ライセンスは、各データセットのライセンスを継承します。
dbpedia-entities-efficient-splade-100K
DBPedia SPLADE + OpenAI: 100,000 SPLADE Sparse Vectors + OpenAI Embedding
This dataset has both OpenAI and SPLADE vectors for 100,000 DBPedia entries. This adds SPLADE Vectors to KShivendu/dbpedia-entities-openai-1M/
Model id used to make these vectors:
model_id = "naver/efficient-splade-VI-BT-large-doc"
For processing the query, use this:
model_id = "naver/efficient-splade-VI-BT-large-query"
If you'd like to extract the indices and weights/values from the vectors, you can do so… See the full description on the dataset page: https://huggingface.co/datasets/nirantk/dbpedia-entities-efficient-splade-100K.wikipedia-en-splade-bge
Wikipedia English with SPLADE and BGE-M3
Pre-computed SPLADE sparse and BGE-M3 dense embeddings for 6.4M English Wikipedia articles.
Direct Usage
HuggingFace automatically discovers parquet files. You can load this dataset directly:
from datasets import load_dataset
# Stream the entire dataset (recommended for large dataset)
dataset = load_dataset("Sicheng-Chroma/wikipedia-en-splade-bge", streaming=True)
# Load specific splits
train_dataset =… See the full description on the dataset page: https://huggingface.co/datasets/Sicheng-Chroma/wikipedia-en-splade-bge.msmarco_v2.1_segmented-spladev3-anserinidbpedia-entity.splade-v3.cache
dbpedia-entity.splade-v3.cache
Description
TODO: What is the artifact?
Usage
# Load the artifact
import pyterrier_alpha as pta
artifact = pta.Artifact.from_hf('pyterrier/dbpedia-entity.splade-v3.cache')
# TODO: Show how you use the artifact
Benchmarks
TODO: Provide benchmarks for the artifact.
Reproduction
# TODO: Show how you constructed the artifact.
Metadata
{
"type": "indexer_cache",
"format": "lz4pickle"… See the full description on the dataset page: https://huggingface.co/datasets/pyterrier/dbpedia-entity.splade-v3.cache.hotpotqa.splade-v3.cache
hotpotqa.splade-v3.cache
Description
TODO: What is the artifact?
Usage
# Load the artifact
import pyterrier_alpha as pta
artifact = pta.Artifact.from_hf('pyterrier/hotpotqa.splade-v3.cache')
# TODO: Show how you use the artifact
Benchmarks
TODO: Provide benchmarks for the artifact.
Reproduction
# TODO: Show how you constructed the artifact.
Metadata
{
"type": "indexer_cache",
"format": "lz4pickle",
"package_hint":… See the full description on the dataset page: https://huggingface.co/datasets/pyterrier/hotpotqa.splade-v3.cache.