PatSnap/novelty-search-bench
PatSnap Novelty Search Bench A Bench for evaluating patent novelty search (prior art search) systems. Each sample contains a query patent publication number (PN) along with ground truth X-type (novelty-destroying) prior art references identified by patent examiners. The Bench is deliberately designed as a 50% public release of an internal full evaluation set that combines two complementary sample types — cross-jurisdiction family-expanded samples and single-jurisdiction… See the full description on the dataset page: https://huggingface.co/datasets/PatSnap/novelty-search-bench.
PatSnap Novelty Search Bench
A Bench for evaluating patent novelty search (prior art search) systems. Each sample contains a query patent publication number (PN) along with ground truth X-type (novelty-destroying) prior art references identified by patent examiners.
The Bench is deliberately designed as a 50% public release of an internal full evaluation set that combines two complementary sample types — cross-jurisdiction family-expanded samples and single-jurisdiction non-expanded samples — so that retrieval systems can be probed on the most informative prior-art scenarios alongside a standard non-expanded baseline.
📄 Companion paper: Research on Evaluation Methods for Patent Novelty Search Systems and Empirical Analysis — Shu Zhang, LiSha Zhang, Kai Duan, XinKai Sun · arXiv **2508.17782** (Aug 2025). ⚠️ Note on the paper. The dataset has been updated since the paper (arXiv, Aug 2025) was published; a revised paper is in preparation.
🐙 GitHub mirror: This dataset is also published as part of the `patsnap/patent-bench` monorepo, where you can find the reference metric scripts (novelty_metrics.py), evaluation examples, and additional sub-Benches.Dataset Overview
Quick Start
from datasets import load_dataset
ds = load_dataset("PatSnap/novelty-search-bench", split="test")
print(f"Total samples: {len(ds)}")
print(ds[0])
# Build ground truth and evaluate your retrieval system
def build_gt(sample, collapsed=False):
if collapsed:
return set(sample["pn_x_family"]) | set(sample["pn_family_x_family"])
return set(sample["pn_x"]) | set(sample["pn_family_x"])
for sample in ds:
gt = build_gt(sample, collapsed=False)
query_text = sample["description"] # ← bundled, no external lookup
# ranked = your_search_system(query_text)
# hits = sum(1 for r in ranked[:100] if r in gt)
# ...How to Use the Query
✅ Query input is bundled in the `description` field of every sample. Feed it straight into your retrieval system — no external lookup needed. Thedescriptionis the full specification text of the query patent, in its originallang. If the system under test has an input character limit, extracting only the Summary of Invention (发明内容) section is usually a workable fallback. The PNs used here are in PatSnap standardized format and are resolvable by all major public patent databases if you want to cross-reference. Production system: [PatSnap Eureka](https://eureka.patsnap.com/?from=benchmark_huggingface).
Data Fields
Ground Truth Construction
The same formula pn_x ∪ pn_family_x applies uniformly to both sample types — for type=public, the union with an empty set degenerates to pn_x.
Evaluation Metrics
The reference metric scripts (with strict / leaderboard mode by default, PN canonicalization, and ranked-list de-duplication) are available in the `patsnap/patent-bench` GitHub repo.
Distribution
By Sample Type
By Jurisdiction
By Language
By IPC Section
Limitations
- Retrieval-only Bench: Evaluates the search/retrieval step only.
- GT based on examiner citations: Not exhaustive prior art coverage.
- Temporal snapshot: Newer prior art after examination not included.
- Language bias: Only Chinese and English patents.
- Public release is a 50% sample of the internal full evaluation set.
Citation
@article{zhang2025novelty,
title = {Research on Evaluation Methods for Patent Novelty Search Systems and Empirical Analysis},
author = {Zhang, Shu and Zhang, LiSha and Duan, Kai and Sun, XinKai},
journal = {arXiv preprint arXiv:2508.17782},
year = {2025},
url = {https://arxiv.org/abs/2508.17782}
}
@dataset{patsnap_novelty_search_bench_2026,
title = {PatSnap Novelty Search Bench},
author = {PatSnap},
year = {2026},
url = {https://huggingface.co/datasets/PatSnap/novelty-search-bench},
note = {Open dataset accompanying Zhang et al. 2025 (arXiv:2508.17782)}
}License
Released under CC BY-NC 4.0 — research and non-commercial evaluation purposes only.
Try the Production System
Experience the PatSnap Novelty Search AI Agent — the commercial system referenced in this Bench.