CoolFace
Datasetpublic

Nitin1211/dbpedia-hindi-benchie-ground-truth

DBpedia Hindi — BenchIE Ground Truth The first DBpedia property ground truth for the Hindi BenchIE benchmark — 139 canonical triples across 112 sentences, built for the DBpedia Hindi Chapter (Google Summer of Code 2026). Why This Was Needed BenchIE contains human-verified gold subject/relation/object spans, but was designed for open information extraction evaluation, not DBpedia alignment — it had no mapping to DBpedia properties before this work.… See the full description on the dataset page: https://huggingface.co/datasets/Nitin1211/dbpedia-hindi-benchie-ground-truth.

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face
0likes19downloads
Dataset Card

DBpedia Hindi — BenchIE Ground Truth

The first DBpedia property ground truth for the Hindi BenchIE benchmark — 139 canonical triples across 112 sentences, built for the DBpedia Hindi Chapter (Google Summer of Code 2026).

Why This Was Needed

BenchIE contains human-verified gold subject/relation/object spans, but was designed for open information extraction evaluation, not DBpedia alignment — it had no mapping to DBpedia properties before this work.

Format

json
{
  "sent_id": "<BenchIE sentence ID>",
  "sentence": "<full Hindi sentence>",
  "subject": "...", "relation": "...", "object": "...",
  "candidates": ["<top-10 dbo: candidates>"],
  "llm_reasoning": "<full GPT-OSS-120B reasoning>",
  "ground_truth_dbo": "dbo:propertyName or null"
}

How This Was Built

  1. 1.Parsed 139 canonical Cluster-1 triples from BenchIE's gold annotations
  2. 2.Retrieved top-10 candidate DBpedia properties per relation using e5-large-instruct against the full 2,710-property catalog
  3. 3.GPT-OSS-120B selected the correct property from the candidates, given full sentence context, or returned NONE if none genuinely fit
  4. 4.All 139 entries manually verified; corrections applied to borderline and corrupted cases

Results

CategoryCount%
Labeled with a real DBpedia property7352.5%
Correctly returned NONE (no DBpedia property fits)6647.5%

Note: BenchIE was built for general information extraction, not DBpedia alignment — many sentences describe abstract or procedural content with no corresponding DBpedia property. The ~47.5% NONE rate reflects that honestly, not a gap in this mapping effort.

Used For

Full-pipeline evaluation (extraction + predicate linking): F1 = 0.173. Extraction-only evaluation: F1 = 0.056 (the gap is explained by BenchIE's gold spans retaining Hindi postpositions that the model consistently trims — see project README).

Part of a Larger Pipeline

Full code and documentation: https://github.com/singhhnitin/neural-extraction-framework/tree/gsoc26h-development/GSoC26_H