NHLOCAL/judaic-texts-corpus

Judaic Texts Corpus

Dataset Summary

Judaic Texts Corpus is a machine-readable Hebrew and Aramaic corpus of Judaic texts derived from the Otzaria library release archives. It is intended for language-model training, retrieval, search, digital humanities research, and other NLP workflows that need structured access to rabbinic and traditional Jewish texts.

The current dataset build is produced from the official Otzaria/otzaria-library release assets, which package the Otzaria text library and its conversion scripts.

• —otzaria_latest.zip
• —otzaria_dicta_latest.zip

Each row represents one source text file. The original source path, collection, book title, author metadata, release tag, and licensing note are preserved in the metadata struct.

Supported Tasks

This dataset can be used for:

• —Hebrew and Aramaic language modeling
• —Retrieval-augmented generation over Judaic texts
• —Text classification and clustering
• —Question answering over source texts
• —Corpus analysis and digital humanities research

Dataset Structure

Data Files

The dataset uses a single configuration named default.

data/*.parquet

The dataset has one split:

train

Dataset sizes and row counts are intentionally not hard-coded in this card. They are computed by the Hugging Face Dataset Viewer from the uploaded Parquet files.

Data Fields

Each row has the following top-level fields:

• —text (string): The full text extracted from a source .txt file.
• —source (string): The source label for this corpus. Current builds use Otzaria Library.
• —metadata (struct): Structured metadata about the source text.

The metadata struct contains:

• —title (string): Display title from the Otzaria metadata when available, otherwise the filename-derived book name.
• —author (string): Author name when available.
• —book_name (string): Book name derived from the source filename.
• —category (string): Category path derived from the source path.
• —source_collection (string): Top-level source collection inside the release archive.
• —source_path (string): Relative path of the original text file inside the release archive.
• —file_hash (string): Hash from the release manifest when available.
• —github_release (string): Otzaria release tag used for the build, for example library-143.
• —pub_date (string): Publication date metadata when available.
• —pub_place (string): Publication place metadata when available.
• —comp_date (string): Composition date metadata when available.
• —comp_place (string): Composition place metadata when available.
• —description (string): Hebrew description from the source metadata when available.
• —license_note (string): Per-row licensing note derived from the source collection.

Data Instance

{
  "text": "מאימתי קורין את שמע בערבית...",
  "source": "Otzaria Library",
  "metadata": {
    "title": "ברכות",
    "author": "",
    "book_name": "ברכות",
    "category": "תלמוד/בבלי",
    "source_collection": "sefariaToOtzaria",
    "source_path": "sefariaToOtzaria/sefaria_export/ספרים/אוצריא/תלמוד/ברכות.txt",
    "file_hash": "abc123",
    "github_release": "library-143",
    "pub_date": "",
    "pub_place": "",
    "comp_date": "",
    "comp_place": "",
    "description": "",
    "license_note": "Source collection: sefariaToOtzaria. Verify licensing against the original source terms."
  }
}

How to Use

from datasets import load_dataset

dataset = load_dataset("NHLOCAL/judaic-texts-corpus")
print(dataset["train"][0])

The metadata column is a nested struct. For example:

row = dataset["train"][0]
print(row["text"])
print(row["metadata"]["title"])
print(row["metadata"]["source_path"])

Dataset Creation

Source Data

The source material comes from the official Otzaria release archives published by Otzaria/otzaria-library. The dataset builder streams .txt files directly from the release archives and writes sharded Parquet files.

Otzaria aggregates and converts books from multiple projects and collections. The source collection is preserved in metadata.source_collection, and the original path is preserved in metadata.source_path so users can trace each row back to its upstream collection.

Processing

The build pipeline:

1. 1.Downloads the selected Otzaria release assets.
2. 2.Extracts files_manifest.json and metadata.json from otzaria_latest.zip.
3. 3.Streams .txt files from otzaria_latest.zip and otzaria_dicta_latest.zip.
4. 4.Trims empty text, skips non-text files, and removes duplicate texts by SHA-256 hash of the normalized text content.
5. 5.Writes Parquet shards with the schema declared in this dataset card.

Versioning

The metadata.github_release field records the upstream Otzaria release tag used to build each row. This keeps the dataset aligned with Otzaria's original release numbering, such as library-143.

Considerations for Using the Data

Licensing

The code in this repository is released under the Apache License 2.0. The text content is not covered by a single dataset-wide license. It comes from multiple upstream collections, and each row should be evaluated according to its metadata.source_collection, metadata.source_path, and original source terms.

The dataset card declares cc-by-4.0 as the general license for this dataset package and metadata. This declaration does not override the license or usage terms of the underlying source texts.

Original Otzaria project content is generally distributed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0), unless another license is specified for a source collection. Imported collections keep their own license terms.

The main source-collection terms are:

Do not assume that all rows are covered by a single license. Check metadata.source_collection, metadata.source_path, and metadata.license_note, and verify the terms of the original source collection before using the content. This summary is informational and is not legal advice.

Limitations

This is a large aggregated corpus built from many source collections. Some texts may contain formatting artifacts, transcription errors, OCR mistakes, duplicated material not caught by exact text hashing, or inconsistent source metadata.

The dataset is provided as a research and engineering resource. Users are responsible for validating fitness, accuracy, and legal suitability for their own use case.

Citation

If you use this dataset, cite the dataset and the upstream Otzaria project:

@dataset{judaic_texts_corpus,
  author = {NHLOCAL},
  title = {Judaic Texts Corpus},
  publisher = {Hugging Face},
  year = {2026},
  url = {https://huggingface.co/datasets/NHLOCAL/judaic-texts-corpus}
}

Credits and Acknowledgements

This dataset builds on the Otzaria project and the Otzaria library maintainers, who collect, convert, organize, and publish the source library.

Acknowledgements also go to the upstream text projects and collections included in the Otzaria release archives:

• —Sefaria
• —Dicta Library
• —Pninim
• —Torat Emet
• —Hebrew Wikisource
• —The collaborative Jewish books archive
• —Project Ben-Yehuda
• —Orayta
• —OnYourWay / ובלכתך בדרך
• —Tashma
• —Or Breslev
• —Otzaria contributors and editors who created, corrected, or organized additional books through the Otzaria project and editing site

This dataset is a machine-readable packaging of those sources. Preserve attribution to Otzaria and to the relevant original source collections when using or redistributing content.