justicedao/patent-legal-ir-graphrag

Patent Legal IR GraphRAG

Single-dataset retrieval release at `justicedao/patent-legal-ir-graphrag`.

Layout matches Publicus IR GraphRAG releases (Publicus/cvefixes-security-ir-graphrag, Publicus/skillcenter-ir):

Sparse BM25 formula (Publicus)

• —Tokenizer: cvefixes-ascii-code-nfkc-casefold/v1 (Publicus HF sparse formula)
• —Ranking params: k1=1.2, b=0.75, titleweight=5.0, bodyweight=1.0
• —Terms are sorted lexicographically; postings shards expose first_term/last_term
• —IDF: FTS5 log((N - df + 0.5) / (df + 0.5)) with floor 1e-6
• —Shard bounds: ≤ 4096 rows / ≤ 4096 terms per postings shard

Counts

• —corpus documents: 2,174
• —BM25 terms: 30,514
• —BM25 postings: 854,046
• —vectors: 2,174
• —graph nodes: 37,764 (incl. BM25 terms: 30,514)
• —graph edges: 883,628 (contains_term: 854,046)

Efficient retrieval

1. 1.Load meta indexes under indexes/ (keyword shards, vector chunks, adjacency).
2. 2.Probe BM25 by term range → download only matching data/bm25/postings/part-*.parquet.
3. 3.Probe vectors by centroid → download only relevant vector shards.
4. 4.Expand via graph adjacency pages; BM25 term nodes share the lexical vocabulary.

BM25 layout fragment:

{
  "average_document_length": 1771.7778288868444,
  "b": 0.75,
  "body_weight": 1.0,
  "k1": 1.2,
  "max_query_terms": 64,
  "posting_rows_per_record": 4096,
  "terms_per_shard": 4096,
  "title_weight": 5.0,
  "tokenizer": "cvefixes-ascii-code-nfkc-casefold/v1"
}

Primary identity

• —Primary key: content ID (entry_cid / CIDv1 bafk…)
• —Secondary pointer: dense document_index (0..N-1) used only inside sparse postings and shard ranges
• —Graph endpoints use node_cid / edge_cid / subject_cid / object_cid

Coverage

Query client (local / e2e)

From the ipfs_datasets_py repository:

# Rule retrieval evaluation suite (BM25 + hybrid, citation-aware ranking)
python3 scripts/ops/legal_data/query_patent_legal_ir_hf.py \
  --release-dir /path/to/local/checkout-or-download \
  evaluate --strict

# Interactive BM25 (citation + topic→section boost ON by default)
python3 scripts/ops/legal_data/query_patent_legal_ir_hf.py \
  --release-dir /path/to/local/checkout-or-download \
  bm25 "duty of candor and good faith" --top-k 10

# Explicit section ranking
python3 scripts/ops/legal_data/query_patent_legal_ir_hf.py \
  --release-dir /path/to/local/checkout-or-download \
  bm25 "information disclosure statement 37 CFR 1.97"

Citation-aware re-ranking:

• —explicit cues: 37 CFR 1.97, 35 USC 103, MPEP § 2001, bare 1.56
• —topic lexicon: “duty of candor” → 1.56 / MPEP 2001; “IDS timing” → 1.97/1.98/609
• —family-safe matching (avoids false friends like CFR 2.103 for USC 103)

Verified e2e (local): 9/9 rule cases PASS; pytest integration suite 13/13 PASS.

Snapshot roots

• —corpus_root_cid: bafkreiak2bzrnblycry6t34kyusupep3nusxpptwlaffaz7glur3cpeiem
• —release_root_cid: bafkreidjgszp2welhobw42nhsvxebcjt7uugqsbufgzph46ghvj2r6vose
• —layout: publicus-ir-graphrag/v1
• —schema: patent.public_legal_ir_hf_release/v1