Datasetpublic

Convence/ParseEmbed

ParseEmbed Hard, parse-sensitive retrieval evaluation for embedding models. ParseEmbed is a compact benchmark for embedding models. It tests whether a model can retrieve the exact correct document when hard negatives share nearly all surface tokens with the answer. Tasks Task ID Split What it measures mean mean Semantic scope, negation, numeric values, temporal conditions, and exception handling text_formatting text_formatting Meaning carried by… See the full description on the dataset page: https://huggingface.co/datasets/Convence/ParseEmbed.

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

5likes75downloads

Dataset Card

ParseEmbed

Hard, parse-sensitive retrieval evaluation for embedding models.

ParseEmbed is a compact benchmark for embedding models. It tests whether a model can retrieve the exact correct document when hard negatives share nearly all surface tokens with the answer.

Tasks

Task ID	Split	What it measures
`mean`	`mean`	Semantic scope, negation, numeric values, temporal conditions, and exception handling
`text_formatting`	`text_formatting`	Meaning carried by Markdown-like formatting such as code spans, headings, quotes, and strike-through
`table`	`table`	Row/column grounding in compact Markdown tables

Files

—eval.yaml: Hugging Face Benchmark definition.
—mean.jsonl, text_formatting.jsonl, table.jsonl: benchmark splits.
—corpus.jsonl: retrieval corpus.
—queries.jsonl: retrieval queries.
—qrels.jsonl: relevance judgments.
—evaluate_parseembed.py: reference evaluator for embedding models.
—requirements.txt: evaluator dependencies.

Quick Start

Install the evaluator dependencies:

bash

pip install -r requirements.txt

Run a local evaluation:

bash

python evaluate_parseembed.py   --model sentence-transformers/all-MiniLM-L6-v2   --task all   --k 10   --output results.json

The evaluator reports ndcg_at_10, recall_at_10, and mrr_at_10 for each task and an average across tasks.

Evaluation Protocol

ParseEmbed is evaluated as dense retrieval:

1.Embed every document in corpus.jsonl.
2.Embed the queries from one benchmark split.
3.Rank corpus documents by cosine similarity.
4.Score the rank of each query's positive_doc_id.

The official leaderboard metric should be ndcg_at_10. recall_at_10 and mrr_at_10 are included as secondary diagnostics.

Data Format

Each benchmark split contains one JSON object per query with these fields: id, query, positive_doc_id, positive_text, hard_negative_doc_ids, hard_negative_texts, answer, style, and difficulty.

The corpus is stored separately in corpus.jsonl, and qrels.jsonl provides standard retrieval relevance judgments with query-id, corpus-id, and score.

Eval Results

Model repositories can report results with .eval_results/parseembed.yaml:

yaml

- dataset:
    id: Convence/ParseEmbed
    task_id: mean
    revision: <dataset_commit_hash>
  value: <ndcg_at_10_score>
  notes: "metric=ndcg_at_10"

Use task IDs mean, text_formatting, and table.

Benchmark Registration

This repository is prepared for Hugging Face's beta Benchmark system:

—It contains a root eval.yaml.
—evaluation_framework is set to parseembed.
—The benchmark has three task leaderboards: mean, text_formatting, and table.
—A working reference evaluator is included in evaluate_parseembed.py.

Because Hugging Face's Benchmark feature is allow-listed, parseembed must be added by the Hugging Face team before the official Benchmark tag appears.

Usage

python

from datasets import load_dataset

mean = load_dataset("Convence/ParseEmbed", "parse-embed", split="mean")
corpus = load_dataset("Convence/ParseEmbed", "corpus", split="test")
queries = load_dataset("Convence/ParseEmbed", "queries", split="test")
qrels = load_dataset("Convence/ParseEmbed", split="test")

Dataset Construction

The benchmark is generated deterministically from generate_parseembed.py with seed 91247. Each query has one positive document and three hard negatives. The negatives are constructed by changing a decisive detail while preserving most of the vocabulary.

No private or external source text is used. The benchmark is synthetic and is intended as a diagnostic stress test alongside natural retrieval benchmarks.

Reproducibility

ParseEmbed is fully deterministic. Regenerating from generate_parseembed.py with the same seed produces the same queries, corpus IDs, hard negatives, and relevance judgments.

Size

—Queries: 720
—Corpus documents: 2,880
—Queries per task: 240

License

Apache 2.0