CoolFace
Datasetpublic

moorcheh/mair-ndcg10-results-all-providers

MAIR Benchmark: NDCG@10 Performance Across Vector Providers This dataset contains comprehensive NDCG@10 (Normalized Discounted Cumulative Gain) accuracy results across various vector database providers and retrieval configurations. The benchmarks compare performance using both Quantized and Floating-Point vectors across multiple specialized domains (Legal, Financial, Medical, and API Documentation). 📊 Overview of Comparisons The results compare the retrieval… See the full description on the dataset page: https://huggingface.co/datasets/moorcheh/mair-ndcg10-results-all-providers.

sourceHugging Facemitupdated 8mo agoView on Hugging Face
0likes21downloads
Dataset Card

MAIR Benchmark: NDCG@10 Performance Across Vector Providers

This dataset contains comprehensive NDCG@10 (Normalized Discounted Cumulative Gain) accuracy results across various vector database providers and retrieval configurations. The benchmarks compare performance using both Quantized and Floating-Point vectors across multiple specialized domains (Legal, Financial, Medical, and API Documentation).

📊 Overview of Comparisons

The results compare the retrieval accuracy of Moorcheh against industry standards:

  • —Providers: Elasticsearch, Pinecone (with Cohere), PGVector, Qdrant, and Moorcheh.
  • —Data Types: Quantized Vectors vs. Floating-Point Vectors.
  • —Datasets: 14 specialized datasets ranging from 197 to 57,638 corpus records.

📂 Dataset Structure

The main file mair-ndcg10-results-all-providers.csv includes the following columns:

ColumnDescription
DatasetName of the benchmark dataset (e.g., AILA2019, FiQA, FinanceBench).
CategoryDomain of the data (Legal & Regulatory, Financial, Medical, etc.).
Dataset SizeTotal number of documents in the corpus.
[Provider] - Vector (Quantized)NDCG@10 score using 8-bit or similar quantization.
[Provider] - Vector (Floating-Point)NDCG@10 score using standard FP32/FP16 precision.

NDCG@10 Comparison of Floating-Point and Quantized Vector Embeddings on MAIR Datasets

📈 Performance Summary

Based on the benchmark data:

  • —Moorcheh maintains highly competitive accuracy, often outperforming or matching standard PGVector and Qdrant implementations in legal and financial domains.
  • —Quantization Impact: The data tracks the minimal "accuracy drop" when moving from Floating-Point to Quantized vectors, demonstrating the efficiency of modern embedding compression.

🛠 How to Use

Loading with Python

python
from datasets import load_dataset

# Load the NDCG@10 results
dataset = load_dataset("moorcheh/mair-ndcg10-results-all-providers", split="all_providers_ndcg10")

# Convert to Pandas for analysis
df = dataset.to_pandas()

# Filter for Financial datasets
financial_results = df[df['Category'] == 'Financial']
print(financial_results)

Citation

If you use this dataset, please cite: