Datasetpublic

TigreGotico/desacordo_ortografico

Portuguese Orthographies — Parallel Corpus A parallel corpus for detecting and converting between Portuguese orthographies. Each record is one Portuguese sentence written in five orthographic norms, so the same content can be aligned across the spelling reforms of the language. Norms (one column each) column norm etymological pre-1911 pseudo-etymological spelling pt_1973 pre-AO1990 European (Convenção 1945 + 1973 mini-reform) ao1990_pt Acordo… See the full description on the dataset page: https://huggingface.co/datasets/TigreGotico/desacordo_ortografico.

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes28downloads

Dataset Card

Portuguese Orthographies — Parallel Corpus

A parallel corpus for detecting and converting between Portuguese orthographies. Each record is one Portuguese sentence written in five orthographic norms, so the same content can be aligned across the spelling reforms of the language.

Norms (one column each)

column	norm
`etymological`	pre-1911 pseudo-etymological spelling
`pt_1973`	pre-AO1990 European (Convenção 1945 + 1973 mini-reform)
`ao1990_pt`	Acordo Ortográfico de 1990, European variant
`br_1971`	pre-AO1990 Brazilian (Formulário 1943 + 1971 mini-reform)
`ao1990_br`	Acordo Ortográfico de 1990, Brazilian variant

The European line (etymological, pt_1973, ao1990_pt) and the Brazilian line (br_1971, ao1990_br) may differ in lexis as well as spelling (e.g. ficheiro / arquivo); within each line the cells differ only by the documented era transforms.

Configs

—parallel (default) — 24196 sentences × 5 norms, each with a features list naming the orthographic phenomena it exercises (digraphs, silent consonants, nasal vowels, trema, differential accents, hyphenation, …) and a source field (see Construction).
—sisters — short samples of Mirandese, Galician and Barranquenho (recognised varieties that are not Portuguese orthographies) plus Portuguese controls, for a language-guard task.

Construction

The source field marks how each row was built:

—authored — written from linguistic knowledge against an explicit per-norm rule card (a hand-written core plus a fan-out of language models over distinct topic domains), independently of any converter, then audited sentence-by-sentence by a language-model panel and tightened to reference quality.
—derived — a natural modern-European sentence whose other four norms are produced by a verified derivation: each word is transformed only via a per-word form confirmed against a curated lexicon (the silent-consonant, trema, nasal-vowel, dual and differential vocabularies were classified word-by-word by a language-model panel); any sentence containing a word whose variation is not verified is dropped, so a derived row is correct by construction. Spelling correctness was confirmed by independent sampling.

Within each line (European: etymological/pt_1973/ao1990_pt; Brazilian: br_1971/ao1990_br) the cells differ only by the documented era transforms; the two national lines may also differ lexically in the authored rows.

A third tag, coverage, marks short carrier sentences (one quoted term) added so that every AO1990-changed word in the reference lexicon appears at least once.

Word coverage

Every AO1990-changed word in the reference lexicon appears at least once — silent c/p (acção, óptimo, director), trema (freqüência, lingüiça), open-diphthong accent (idéia, heróico), and dual/divergence (facto/fato, húmido/úmido) are each at 100% of their inventory.

Uses

—Benchmark or train a Portuguese orthography detector (classify which norm a text is in).
—Benchmark an orthography converter (transform between norms) — note that the two national lines can diverge lexically, which an orthographic converter does not translate.

Source

Produced for the `desacordo_ortografico` library. Apache-2.0.