Datasetpublic

PatSnap/design-fto-bench

PatSnap Design FTO Bench A Bench for evaluating design patent Freedom-To-Operate (FTO) retrieval systems on cross-modal image search. Each sample provides a query product image (or design patent figure) plus the ground truth set of target design patents that constitute infringement risk, as confirmed by patent invalidation proceedings. 🐙 GitHub mirror: This dataset is also published as part of the patsnap/patent-bench monorepo, where you can find the reference metric scripts… See the full description on the dataset page: https://huggingface.co/datasets/PatSnap/design-fto-bench.

sourceHugging Facecc-by-nc-4.0updated 4mo agoView on Hugging Face

4likes50downloads

Dataset Card

PatSnap Design FTO Bench

A Bench for evaluating design patent Freedom-To-Operate (FTO) retrieval systems on cross-modal image search. Each sample provides a query product image (or design patent figure) plus the ground truth set of target design patents that constitute infringement risk, as confirmed by patent invalidation proceedings.

🐙 GitHub mirror: This dataset is also published as part of the `patsnap/patent-bench` monorepo, where you can find the reference metric scripts (search_metrics.py) and additional sub-Benches.

Dataset Overview

Property	Value
Total samples	91
Source	Real patent invalidation proceedings
Jurisdictions	CN (100% for the released subset)
Modality	PNG images (product photo ↔ patent line drawing / photo)
Image directory	`data/image/<jurisdiction>/<class>/<sub>/<pn>/<file>.png` (91 PNG files, ~6.8 MB)
Ground truth	Patent pairs confirmed as infringement-equivalent through patent invalidation proceedings
Locarno (LOC) coverage	All 26 first-level LOC classes
License	CC BY-NC 4.0

Quick Start

python

from datasets import load_dataset

ds = load_dataset("PatSnap/design-fto-bench", split="test")
print(f"Total samples: {len(ds)}")

# Inspect one sample
sample = ds[0]
print(sample["query_pn"], sample["pair_name"])

# query_img_path is a PIL Image (bytes embedded in the Parquet, no external lookup needed)
img = sample["query_img_path"]
print(f"Query image: {img.size}, mode={img.mode}")

# Targets are the set of design patents whose images constitute infringement risk
print(sample["target_pns"], sample["target_img_ids"])

Data Fields

Field	Type	Description
`id`	int64	Sample identifier
`query_img_id`	string	Identifier of the query image
`query_pn`	string	Publication number of the query patent (PatSnap standardized PN)
`query_img_path`	string	Relative path to the query image under `data/image/`
`target_pns`	list[string]	Ground truth target design-patent PNs that constitute infringement risk
`target_img_ids`	list[string]	Image identifiers of the target patents
`pair_name`	string	Pair identifier from the invalidation proceeding
`picture_type`	string	Source of the GT pair (e.g. 无效 = invalidation proceeding)
`one_level_loc`	string	First-level Locarno classification code
`two_level_loc`	string	Second-level Locarno classification code (e.g. 14-03)
`country`	list[string]	Country/jurisdiction codes of the sample
`version`	string	Dataset version (e.g. 1.1)

How to Use the Query

The query input is the query product image at data/image/<query_img_path>. Each sample's target_pns lists the design patents that an FTO retrieval system should return.

Evaluation Metrics

Metric	Description
Hit Rate @ K	% of samples with ≥1 GT patent in top K (K = 10, 50, 100, 200)
PRES @ N	Patent Retrieval Evaluation Score (Magdy & Jones 2010, with miss-penalty correction): single score in `[0, 1]` jointly capturing how many GT patents are retrieved within top-N and how highly they are ranked. PRES = 1.0 means every GT patent appears at the top; PRES = 0 means none are found within N. Default N = 200.

The reference metric scripts (with strict / leaderboard mode by default and ranked-list schema validation) are available in the `patsnap/patent-bench` GitHub repo.

Scoring Grades (Hit Rate @ Top@100)

Grade	Hit Rate	Description
A	≥ 90%	Excellent — suitable for direct professional use
B	≥ 75%	Good — effective as a high-efficiency screening tool
C	≥ 60%	Acceptable — requires human review of key results
D	< 60%	Below standard — model improvement needed

Distribution

By Jurisdiction

Jurisdiction	Count	Percentage
CN	91	100%

The v1.1 public release contains only invalidation-proceeding samples (CN). Future releases (v2) will incorporate cross-jurisdiction TRO data (US/EP/JP).

By Locarno Classification

Coverage spans all 26 first-level LOC classes.

Limitations

—Retrieval-only Bench: Evaluates the search/retrieval step only; does not cover infringement adjudication or court-ruling outcomes.
—GT based on invalidation proceedings: This subset (v1.1) is restricted to CN invalidation-proceeding pairs. E-commerce infringement-complaint samples are retained internally for client confidentiality.
—Single-jurisdiction: CN only in this release.
—Visual similarity ≠ legal infringement: A retrieval system returning a top-1 hit does not constitute a legal infringement determination; results are inputs to professional FTO review.

Citation

bibtex

@dataset{patsnap_design_fto_bench_2026,
  title  = {PatSnap Design FTO Bench},
  author = {PatSnap},
  year   = {2026},
  url    = {https://huggingface.co/datasets/PatSnap/design-fto-bench},
  note   = {A Bench for evaluating design-patent freedom-to-operate image-retrieval systems}
}

License

Released under CC BY-NC 4.0 — research and non-commercial evaluation purposes only.

Try the Production System

Experience the PatSnap Design FTO AI Agent — the commercial system referenced in this Bench.

🔗 Try it on PatSnap Eureka