Datasetpublic

ZurichNLP/document-level-word-alignment

Document-Level Word Alignment Document-level word alignment data for six language pairs — English–French (en-fr), English–Romanian (en-ro), English–Japanese (en-ja), English–Chinese (en-zh), Latin–Ancient Greek (la-gr), and English–Czech (en-cz) — reconstructed from existing sentence-level, human-annotated word alignment gold standards. Document-level examples are built by grouping sentence-level annotations by document membership and sentence order; the dataset does not contain… See the full description on the dataset page: https://huggingface.co/datasets/ZurichNLP/document-level-word-alignment.

sourceHugging Faceotherupdated 29d agoView on Hugging Face

0likes199downloads

Dataset Card

Document-Level Word Alignment

Document-level word alignment data for six language pairs — English–French (en-fr), English–Romanian (en-ro), English–Japanese (en-ja), English–Chinese (en-zh), Latin–Ancient Greek (la-gr), and English–Czech (en-cz) — reconstructed from existing sentence-level, human-annotated word alignment gold standards.

Document-level examples are built by grouping sentence-level annotations by document membership and sentence order; the dataset does not contain new human annotation beyond what already exists in the source corpora (see Sources & Licensing). This dataset accompanies the paper *"Scaling Unsupervised Word Alignment to Documents with Structural Constraints"*.

Dataset Structure

Each configuration corresponds to a language pair, e.g. en-ja. Each row is one document pair. Splits are dev and test, except en-cz, which is test-only.

Data Fields

Field	Type	Description
`sent_id`	string	Unique identifier for the document
`doc_id`	string	Identifier of the parent document
`lang`	string	Language pair code, e.g. `"en-ja"`
`text_a`	string	Source-language text (whitespace-tokenized)
`text_b`	string	Target-language text (whitespace-tokenized)
`sure_align`	string	Space-separated `i-j` sure alignment links (0-indexed word positions, `i` in `text_a`, `j` in `text_b`)
`possible_align`	string	Space-separated `i-j` possible alignment links

Example

json

{"sent_id": "trial", "doc_id": "trial", "lang": "en-fr",
 "text_a": "source document text ...", "text_b": "target document text ...",
 "sure_align": "0-0 3-5", "possible_align": "1-1"}

Dataset Statistics (dev / test)

Lang pair	Docs (dev/test)	Mean doc length (chars)
en-fr	1 / 1	22,408
en-ro	1 / 10	2,901
en-ja	8 / 7	5,516
en-zh	47 / 52	646
la-gr	1 / 1	5,837
en-cz	— / 33	6,318

Lengths are target-side (text_b) character counts.

Sources & Licensing

This is a derived/reconstructed dataset. Each language pair originates from a separate, independently licensed source.

Lang pair	Source	License
en-ja	Neubig (2011)	CC BY-SA 3.0
la-gr	Yousef et al. (2022)	CC BY-SA 4.0
en-fr, en-ro	Mihalcea & Pedersen (2003)	Not specified by the source
en-zh	Liu and Sun (2015)	Not specified by the source
en-cz	Mareček (2011)	CC BY-SA 4.0

Dataset Creation

Document-level examples were constructed by concatenating sentence-level source/target texts within a document in their original order and re-indexing sure_align/possible_align accordingly.

Citation

If you use this dataset, please cite both this work and the original source(s) listed in Sources & Licensing for the language pair(s) you use.

bibtex

@misc{wastl2026scalingunsupervisedwordalignment,
      title={Scaling Unsupervised Word Alignment to Documents via Structural Constraints}, 
      author={Michelle Wastl and Jannis Vamvas and Rico Sennrich},
      year={2026},
      eprint={2608.21023},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.21023}, 
}