CoolFace
Datasetpublic

moorcheh/mair-floating-point-vs-quantized-benchmarks

Dataset Coverage The benchmarks evaluate performance across 14+ specialized datasets covering: Legal & Regulatory: ACORDAR, AILA2019-Case, AILA2019-Statutes, LeCaRDv2, LegalQuAD, REGIR-EU2UK, REGIR-UK2EU. Financial: ConvFinQA, FinanceBench, FinQA, FiQA, HC3Finance. Medical & Clinical: NFCorpus. General/API: Apple Documentation. Metrics NDCG@10: Normalized Discounted Cumulative Gain at rank 10, measuring retrieval quality. Latency (ms): Mean search latency… See the full description on the dataset page: https://huggingface.co/datasets/moorcheh/mair-floating-point-vs-quantized-benchmarks.

sourceHugging Facemitupdated 8mo agoView on Hugging Face
0likes110downloads
Dataset Card

Dataset Coverage

The benchmarks evaluate performance across 14+ specialized datasets covering:

  • —Legal & Regulatory: ACORDAR, AILA2019-Case, AILA2019-Statutes, LeCaRDv2, LegalQuAD, REGIR-EU2UK, REGIR-UK2EU.
  • —Financial: ConvFinQA, FinanceBench, FinQA, FiQA, HC3Finance.
  • —Medical & Clinical: NFCorpus.
  • —General/API: Apple Documentation.

Metrics

  1. 1.NDCG@10: Normalized Discounted Cumulative Gain at rank 10, measuring retrieval quality.
  2. 2.Latency (ms): Mean search latency measured on the server-side and end-to-end.

Benchmark Configurations

The data is organized into four primary splits, which you can switch between using the "Viewer" tab above:

  1. 1.NDCG@10 - Floating Point (`ndcg_vector_fp`): Evaluation of retrieval accuracy using standard 32-bit floating-point vectors.

NDCG@10 Performance of Floating-Point Vector Embeddings on MAIR Datasets

  1. 1.NDCG@10 – Floating Point (Categorized) (`ndcg_vector_fp_categorized`): Evaluation of retrieval accuracy using standard 32-bit floating-point vectors, reported per dataset category.

NDCG@10 Performance of Floating-Point Vector Embeddings - Categorized

  1. 1.Latency - Floating Point (`latency_vector_fp`): Search performance (ms) for floating-point vector retrieval.
  2. 2.NDCG@10 - Quantized (`ndcg_vector_quantized`): Evaluation of retrieval accuracy using optimized quantized/binary embedding paths.

NDCG@10 Performance of Quantized Vector Embeddings on MAIR Datasets

  1. 1.NDCG@10 – Quantized (Categorized) (`ndcg_vector_quantized_categorized`): Evaluation of retrieval accuracy using optimized quantized or binary vector representations, reported per dataset category.

NDCG@10 Performance of Quantized Embeddings - Categorized

  1. 1.Latency - Quantized (`latency_vector_quantized`): Search performance (ms) for optimized quantized vector retrieval.

Tested Providers

Results include comparisons across:

  • —Moorcheh
  • —Elasticsearch
  • —Pinecone (with Cohere Rerank)
  • —PGVector (PostgreSQL)
  • —Qdrant

How to use in Python

You can load these results directly into a Pandas DataFrame using the Hugging Face datasets library:

python
from datasets import load_dataset

# Load the Latency results for Floating Point vectors
dataset = load_dataset("moorcheh/Benchmarks", split="latency_vector_fp")
df = dataset.to_pandas()

print(df.head())

Citation

If you use this dataset, please cite: