Nerthus-Project/Generated_OE_Gregory_Dialogues_Text_and_Evaluation

Generated Old English Gregory's Dialogues (variatio)

A complete, machine-generated Old English variatio of the Old English Dialogues of Gregory the Great (Waerferth's translation), produced on 19 July 2026, together with the full generation and evaluation apparatus: prompt, constraint lexicon scripts, validator, dependency parses, word embeddings, and all quantitative evaluation results.

The project is described in:

Martin Arista, J., & Nunez, M. Evaluating Generated Old English Text: A Dependency-Based Method Built on the Old English Dependency Treebank and Pre-Trained Word Embeddings. (Under review.)

Authors: Javier Martin Arista (Universidad de La Rioja) and Matias Nunez (INIBIOMA-CONICET).

What this is

The generated text is a sentence-by-sentence rewriting of the whole work in the same language and register: one generated Old English sentence for each of the 4,217 source sentences (114,451 running tokens), each with its own literal Modern English translation. The exercise is the classical variatio: same content, same style, different wording.

The generator was Claude Fable 5 (Anthropic), reached through the Anthropic Messages API. The source was divided into 71 batches of about sixty sentences. Four constraints were enforced during generation, not checked afterwards: every token had to belong to a 227,899-form lexicon of attested Old English surface forms; no diacritics were allowed; each line had to stay between 0.8 and 1.25 times the length of its source; and no line could exceed a token-set Jaccard similarity of 0.7 with its source. A validator script rejected offending lines and generation continued until all 4,217 lines passed.

Repository structure

data/gregory_dialogues_generated_oe.parquet   the generated text (4,217 rows, 3 columns)
data/gregory_dialogues_generated_oe.tsv       the same data as tab-separated text
generation/PROMPT.md                      the generation instructions
generation/build_lexicon.py               compiles the allowed-forms lexicon
generation/validate_batch.py              the four-check validator
generation/assemble_final.py              assembles validated batches
evaluation/RESULTS_INDEX.md               which file each published figure comes from
evaluation/scripts/parse_new_text.py      parses the generated text (Stanza, ang)
evaluation/scripts/bootstrap_cis.py       bootstrap confidence intervals (1,000 replicates)
evaluation/scripts/evaluate_native_pt_embeddings.py   embedding-space comparison
evaluation/scripts/rare_hapax_corpus_test.py          rare/hapax depletion test (p < 1e-6)
evaluation/scripts/ngram_cutoff_sensitivity.py        character-track coverage and cut-off checks
evaluation/results/*.json                 all quantitative results
evaluation/results/*.tsv                  aggregate counts behind the two runnable tests
parses/regenerated.conllu                 dependency parse of the generated text (UD)
embeddings/ang_embeddings.pt              Old English word embeddings (57,186 x 100)

evaluation/RESULTS_INDEX.md is the place to start if you are checking a particular number in the paper against this repository.

Data fields

The dataset is loaded from data/gregory_dialogues_generated_oe.parquet (this is the file the Hub's dataset viewer reads); data/gregory_dialogues_generated_oe.tsv carries the same rows as UTF-8 tab-separated text with a header line. The fields are:

Example row:

[GD 1 (C) 000100 (1.11.3)]	Geo waes in thaere maegthe Samni sumes welegan mannes ham...	Long ago there was in the province of Samni a wealthy man's homestead...

(The actual file uses the Old English characters thorn, eth and ash; they are shown transliterated in this example only.)

The sentence codes preserve the alignment with the source: each row answers exactly one sentence of the original Dialogues, so the generated text can be re-aligned with any copy of the original obtained from the sources listed below.

Fixing the corpus

The generation ran through a commercial access point, so the run itself cannot be made to repeat exactly. What can be fixed is the object of study. The article cites the digest below, so that any later generation can be compared against this one line by line even where it cannot be made to reproduce it:

sha256  a11c1fb3c9f7717e0c55dea37b664b343a77c15a5fe3ce87f9438f8fdf3fe1ee
file    data/gregory_dialogues_generated_oe.tsv

Verify with shasum -a 256 data/gregory_dialogues_generated_oe.tsv (or sha256sum on GNU systems). The digest covers the TSV, not the Parquet file, since Parquet encoding is not byte-stable across writer versions.

Evaluation summary

The text was evaluated against two references: the original Old English Dialogues (fidelity to the source) and the Dictionary of Old English Corpus (compatibility with the language at large). Full details are in the paper; the component scores are:

The parse in parses/regenerated.conllu was produced with the same Stanza ang pipeline used for the reference parses, trained on the Old English Dependency Treebank (OEDT).

Every published figure is traced to its file in evaluation/RESULTS_INDEX.md.

The two tests you can run here

Most of the evaluation compares the generated text against the DOEC and against the original Dialogues, neither of which can be redistributed, so those results are released as output files rather than as reproducible runs. Two checks are exceptions, because they can be driven from aggregate counts:

cd evaluation/scripts
python3 rare_hapax_corpus_test.py        # rare/hapax depletion, p < 1e-6
python3 ngram_cutoff_sensitivity.py      # character-track coverage and cut-off

Both need numpy and nothing else, both are seeded, and together they take under half a minute.

The first reproduces the contrast quoted in the Discussion: the rare and hapax bins of the DOEC frequency index hold 3.4591% of generated token mass against 7.2891% of the original, a gap of 3.83 percentage points with a 95% confidence interval of [3.69, 3.97]. The reported p is the bound from a paired permutation over the 4,217 aligned line pairs, the line pair being the unit of resampling because tokens are clustered within lines: in 2,000,000 replicates none reached the observed difference, so p ≤ 5e-7, reported as p < 1e-6.

The second reproduces the character track and the checks on its vocabulary cut: the 12,000 most frequent DOEC 3/4/5-grams cover 70.0% of the corpus's 50,535,036 running character n-grams, the last n-gram admitted still occurs 660 times, and recomputing at 6,000 and 24,000 moves the generated–original cosine by +0.00125 and −0.00082 while leaving the ordering of the three corpus pairs unchanged on both cosine and JSD.

They read evaluation/results/rare_hapax_per_line_counts.tsv and evaluation/results/char_ngram_top24000_counts.tsv. Both hold derived numbers only: token counts per line, and three-to-five character sequences with corpus frequencies. Neither can reconstruct running text, so no corpus material is redistributed.

Embeddings

embeddings/ang_embeddings.pt contains the 100-dimensional Old English word embeddings used in the evaluation (57,186 vocabulary items). Row i corresponds to token id2unit[i+5] of the Stanza ang pretrain nerthus.pt, with which the matrix is identical from row 5 onward; the mapping is verified in evaluation/scripts/evaluate_native_pt_embeddings.py.

Previously published embeddings and models of the Nerthus Project are available at:

• —https://huggingface.co/Nerthus-Project
• —https://stanfordnlp.github.io/stanza/ner_models.html (Stanza models for Old English, ang)

What is not included, and where to find it

The original Old English text. The original Dialogues is not distributed here. The edition used in the project is:

Hecht, H. (Ed.). (1900-1907). Bischof Waerferths von Worcester Uebersetzung der Dialoge Gregors des Grossen (Bibliothek der angelsaechsischen Prosa 5). Georg H. Wigand.

The electronic text is part of the Dictionary of Old English Corpus (DOEC), available through the Oxford Text Archive at http://hdl.handle.net/20.500.12024/2488. For the corpus and its licensing, see the Dictionary of Old English Project: https://doe.artsci.utoronto.ca/

The Dictionary of Old English Corpus (DOEC). Used as the broad historical baseline of the evaluation; obtain it from the Oxford Text Archive (link above) or from the Dictionary of Old English Project.

Because neither the DOEC nor the original Dialogues can be redistributed, the aggregate counts needed to re-run two of the statistical checks are released in their place, in evaluation/results/: per-line rare/hapax and token counts for both corpora, and the 24,000 most frequent DOEC character 3/4/5-grams with their counts in each corpus. See evaluation/RESULTS_INDEX.md.

Old English dictionaries. The dictionaries used to compile the lexical resources of the project cannot be redistributed. They can be obtained from:

• —Bosworth-Toller, An Anglo-Saxon Dictionary - digital edition: https://bosworthtoller.com/
• —Bosworth-Toller - scanned copies for download: https://archive.org/details/anglosaxondictio00boswuoft and the Germanic Lexicon Project: http://www.ling.upenn.edu/~kurisuto/germanic/oebosworthtollerabout.html
• —Clark Hall, A Concise Anglo-Saxon Dictionary: https://archive.org/details/conciseanglosaxo00hall
• —Sweet, The Student's Dictionary of Anglo-Saxon: https://archive.org/details/studentsdictiona00sweeuoft
• —Dictionary of Old English (A to Le), University of Toronto: https://doe.artsci.utoronto.ca/

Project-internal lexical resources. The lemma-and-inflection dictionary (37,522 lemmas, 149,286 attested inflectional forms), the orthographic variant dictionary (162,253 attested spelling variants), and the compiled allowed-forms lexicon derived from them are based on third-party sources and are not released here. They are available under conditions: requests should be addressed to javier.martin@unirioja.es.

Citation

If you use this dataset, please cite:

@misc{gregory-generated-oe-2026,
  title     = {Generated Old English Gregory's Dialogues (variatio)},
  author    = {Mart{\'i}n Arista, Javier and N{\'u}{\~n}ez, Mat{\'i}as},
  year      = {2026},
  publisher = {Hugging Face},
  note      = {Universidad de La Rioja / INIBIOMA-CONICET}
}

Contact

Javier Martin Arista, Universidad de La Rioja: javier.martin@unirioja.es