CoolFace
Datasetpublic

stefdoerr/cidoc-crm-corpus

CIDOC CRM SIG corpus — derived artifacts The built corpus for cidoc-crm-mcp: 26 years of CIDOC CRM Special Interest Group mailing list, its issue register and meeting minutes, cleaned, threaded, and indexed for BM25 and vector search. These are derived artifacts, not the source. They exist because the code repository cannot carry them: ~876MB, git-ignored, rebuilt from a 143MB mbox that is distributed separately. Fetching uv run python build.py fetch… See the full description on the dataset page: https://huggingface.co/datasets/stefdoerr/cidoc-crm-corpus.

sourceHugging Faceotherupdated 11d agoView on Hugging Face
0likes79downloads
Dataset Card

CIDOC CRM SIG corpus — derived artifacts

The built corpus for cidoc-crm-mcp: 26 years of CIDOC CRM Special Interest Group mailing list, its issue register and meeting minutes, cleaned, threaded, and indexed for BM25 and vector search.

These are derived artifacts, not the source. They exist because the code repository cannot carry them: ~876MB, git-ignored, rebuilt from a 143MB mbox that is distributed separately.

Fetching

uv run python build.py fetch # everything uv run python build.py fetch --no-vectors # 91MB, skips the vectors

Without the vectors you still get full-text search over the whole archive and you do not need torch or an embedding model.

Contents

pathwhat
data/clean.jsonlevery message, cleaned, with extracted CRM identifiers
data/threads.jsonmessages grouped into threads
data/documents.jsonlthe specification, issue pages and minutes, chunked ready to search
data/episodes.jsonlsummarised decision episodes
data/issues.jsonthe SIG issue register
stores/*/fts.sqlite3SQLite FTS5 indexes (BM25)
stores/*/chroma.sqlite3 and friendsvector indexes
stores/*/meta.jsonwhich embedding model built each store, and the sha256 of its source

Provenance

Each vector store records the embedding model it was built with and a source_sha256 of the input it was built from. build.py fetch checks these, because a store embedded with one model and queried with another returns confident nonsense.

Licence

The mailing list is published by the CIDOC CRM SIG; posts remain their authors'. This is a derived index redistributed for research use. Contact the SIG for reuse beyond that.