Datasetpublic

KShivendu/miriad-mlateon-colbert-smoke

MIRIAD 200, encoded with mLateOn-medical Multi-vector (ColBERT-style) embeddings for tomaarsen/miriad-benchmark-200k, produced with multi-vector-encoder/mLateOn-medical. passages 200 token vectors 176,014 mean vectors / passage 880.07 dim 128 stored dtype float16 embeddings size 0.05 GB raw text encoded 1 MB The embeddings are 49x larger than the text they came from, which is why late-interaction retrieval needs quantization or pooling.… See the full description on the dataset page: https://huggingface.co/datasets/KShivendu/miriad-mlateon-colbert-smoke.

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes68downloads

Dataset Card

MIRIAD 200, encoded with mLateOn-medical

Multi-vector (ColBERT-style) embeddings for `tomaarsen/miriad-benchmark-200k`, produced with `multi-vector-encoder/mLateOn-medical`.


passages	200
token vectors	176,014
mean vectors / passage	880.07
dim	128
stored dtype	float16
embeddings size	0.05 GB
raw text encoded	1 MB

The embeddings are 49x larger than the text they came from, which is why late-interaction retrieval needs quantization or pooling.

Usage

embedding is flattened; reshape it to recover one vector per token.

python

import numpy as np
from datasets import load_dataset

ds = load_dataset("KShivendu/miriad-mlateon-colbert-smoke", split="train")
row = ds[0]
vecs = np.asarray(row["embedding"], dtype=np.float16).reshape(row["n_tokens"], 128)

MaxSim, the late-interaction score:

python

score = (query_vecs @ doc_vecs.T).max(axis=1).sum()

Queries, qrels and the raw layout live alongside in queries.npy, queries_offs.npy and qrels.json.