CGICAI/cherokee-english-translation

Cherokee–English Parallel Corpus (Archivist Project)

A curated Cherokee (ᏣᎳᎩ / Tsalagi) ↔ English parallel corpus for machine translation, assembled from public sources, deduplicated, benchmark-decontaminated, and conflict-cleaned. Built to train and evaluate English→Cherokee translation models for one of the most endangered languages in North America.

Files

Format: JSONL with input / output fields (plus source, dialect, and instruction metadata where applicable). Cherokee text uses the Cherokee syllabary (U+13A0–U+13FF) throughout.

A note on dialects

Cherokee has two principal living dialects: Kituwah (ᎩᏚᏩ), the Middle dialect of the Eastern Band of Cherokee Indians in North Carolina, and Otali (ᎣᏔᎵ), the Overhill/Western dialect of the Cherokee Nation and the United Keetoowah Band in Oklahoma. They differ in phonology, vocabulary, and aspects of usage, and we do not treat those differences as negligible.

For this release, however, training and evaluation were conducted without dialect markers, merging material from both dialect communities. This was a pragmatic decision driven by resource constraints, not a linguistic judgment: there is simply not enough Eastern Band (Kituwah) text available to train a dedicated dialect model, and the corpus as a whole skews toward Western / Cherokee Nation (Otali) sources, which constitute the majority of the parallel material we could obtain. A properly dialect-conditioned corpus — one that could serve Kituwah and Otali speakers on their own terms — remains a goal for future work, and we would warmly welcome contributions of verified Eastern Band text.

Provenance

The corpus aggregates parallel material from public sources, including datasets already hosted on Hugging Face, repositories on GitHub, and artifacts shared by other Cherokee language projects. Principal components: the ChrEn benchmark corpus, cherokeedictionary.net news translations, biblical text, DAILP (Digital Archive of Indigenous Language Persistence) materials, terminology glossaries, and Cherokee Wikipedia (chrwiki) alignments. Roughly 20–30K pairs are full sentences; the remainder are terms and short phrases, which we include deliberately for vocabulary coverage. Users should be aware that the sentence-level material skews toward older and religious registers.

Third-party source data remains under its original terms; this release contains the derived, cleaned compilation and our original evaluation assets.

Methodology and project history

This corpus is the product of roughly nine months of iterative work:

1. 1.Data archaeology (late 2025): collection and provenance-tracking of ~750K raw records, split into a hash-verified gold archive and a staging area; extraction from PDFs, translation memories (TMX/TBX), Wikipedia dumps, and web sources.
2. 2.Cleaning and audits (early 2026): deduplication, encoding fixes, removal of identity pairs and garbled rows; construction of fixed eval holdouts spanning every source.
3. 3.Decontamination (April 2026): removal of 5,401 rows overlapping the ChrEn benchmark dev/test splits, rows with conflicting targets, and casing-heuristic junk — yielding the v2 training set released here.
4. 4.Model training: after several unsuccessful attempts with a custom Cherokee tokenizer on Qwen3-8B, the project pivoted to google/translategemma-4b-it with QLoRA (r=512), which produced a working en→chr translator (BLEU 39.9 / chrF 45.0 / 98.9% script purity on the 192-sample holdout released here).

Current status

Development paused in April 2026 following a hardware failure that took one of our two training GPUs offline. Work is expected to resume in the near future; a retrained adapter on the v2 corpus has completed training but is unpublished pending evaluation.

Why this project exists

Cherokee has on the order of two thousand fluent first-language speakers remaining. We believe language models can be instruments of preservation: that future models should encompass all languages, carrying historical culture forward within the very weights themselves — and that this must happen through open-source, community-driven work, not behind closed doors.

There is a well-documented tendency for language models to flatten the distinctiveness of the cultures they absorb — to smooth every voice into the same homogenized, median-internet register. For dominant languages this is an annoyance. For endangered ones it is an extinction event wearing the costume of inclusion: the language appears to be supported while its actual texture — its dialects, its registers, its way of carving up the world — is averaged away. Preservation has to mean preserving the differences.

We offer this corpus and the accompanying model as a test bed: for Cherokee, and for the many other Native American and indigenous languages at risk of being lost entirely. If you build on it, we ask that you do so in the open, and in partnership with the communities whose language this is.

ᏣᎳᎩ ᎦᏬᏂᎯᏍᏗ ᏱᎩ — it is the Cherokee language.

Citation

@misc{archivist_cherokee_2026,
  title  = {Cherokee--English Parallel Corpus (Archivist Project)},
  author = {CGICAI},
  year   = {2026},
  url    = {https://huggingface.co/datasets/CGICAI/cherokee-english-translation}
}