worthing-zhang/wmt26-sorbian-synthetic-data

WMT26 Sorbian synthetic and derived data

This repository contains the redistributable portions of the synthetic and derived data used by the HITSZ WMT 2026 Sorbian system, together with prompts, processing scripts, filtering procedures, provenance manifests, and license metadata.

For examples whose upstream licenses or provenance do not permit unambiguous redistribution, only reproducibility metadata and processing artifacts are provided.

The associated system paper is “HITSZ at WMT 2026: Mixed Continued Pre-training and Supervised Fine-tuning for Low-Resource Sorbian.” The work was submitted in the WMT 2026 Sorbian shared-task context. This repository is a data-release companion, not a complete dump of every training example.

Release scope

Data availability

License and provenance warning

The repository has no single blanket data license. Licensing is attached to each source family and, where possible, to each record. See `DATA_LICENSES.md` before reuse.

QA construction

The accepted QA data contain 10,000 Upper Sorbian (hsb) and 10,000 Lower Sorbian (dsb) items derived from native-language Wikipedia contexts. The data comprise deterministic statement-selection and boolean-verification items. Each released row preserves the context, evidence, options, answer, question type, formal validation metadata, source identifier, source URL, and upstream-license label.

The original DataFactory source and template files were deleted during project cleanup and could not be recovered byte-for-byte. The implementation in `qa/reconstructed/` is a reconstruction from preserved audit records and final accepted examples. It is not claimed to be the exact code used during system development, and the original context pool needed for an exact regeneration test is no longer available.

GC construction

The GC component covers 2,400 Qwen3.7-Max adjective-agreement minimal pairs, 1,200 per Sorbian language. All pairs passed the preserved one-token and acceptance checks. Publication follows the recorded source license:

• —317 Apache-2.0 examples: full text;
• —209 examples labelled CC-BY-NC-SA: full text in a separate file;
• —1,444 WMT25 research-use examples: non-text metadata only;
• —430 WMT22 research-use examples: non-text metadata only.

The surviving prompt construction, generation, filtering, and export code is included under `gc/scripts/`. Machine-local input defaults were replaced by relative paths or environment-variable overrides for this public copy; prompt and validation logic were left unchanged. The recorded CC-BY-NC-SA label does not identify a license version, so this release does not invent one.

MR translation

The MR release contains only Sorbian translated problem statements and source mapping for GSM8K, SVAMP, and ASDiv. It does not redistribute the English problem statement, answer, calculator expression, or reasoning trace.

Preserved per-record metadata identifies GPT-5.5 as the translation model. Preserved production audit records establish temperature 0.0 and batch size

1. 1.Only the problem statement was supplied to the translation model; answers and reasoning traces were excluded. Deterministic checks covered numbers, symbols, foreign-function words, length ratio, format, and contamination status.

The original 07_translate_mr.py and prompt file were deleted and could not be recovered exactly. Files under `mr/reconstructed/` are clearly marked reconstructions. They document and implement the preserved contract but are not claimed to reproduce the accepted files byte-for-byte.

CPT instruction restriction

The surviving run selected 955 source records and produced 955 HSB plus 955 DSB translations. The exact upstream dataset identifier and revision could not be unambiguously reconstructed from the surviving local provenance. The local label source: magpie is not sufficient to prove a specific public dataset revision.

Therefore, translated CPT instruction records are not redistributed because the exact upstream dataset revision could not be unambiguously reconstructed. This repository includes only the surviving prompt and processing code, selection-level non-text hashes and local row metadata, aggregate processing metadata, and file-level hashes of the withheld artifacts. The SQLite state database is also withheld.

Reconstructed-code disclaimer

The QA and MR reconstruction files begin with the following notice:

This is a reconstructed implementation based on preserved audit records, generation parameters, and final accepted examples. It is not the exact original implementation used during system development.

No claim of byte-identical regeneration is made. The GC and CPT processing implementations survived; public copies only remove machine-local path defaults where documented.

Reproduction and integrity

• —Data are JSON Lines with one JSON object per line.
• —`RELEASE_MANIFEST.json` records counts, release boundaries, source-snapshot hashes, and file metadata.
• —`SHA256SUMS` contains checksums for the release files.
• —`tools/audit_release.py` checks counts, uniqueness, provenance coverage, restricted-text schemas, CPT withholding, unexpected files, local paths, and credential-shaped strings.
• —`AUDIT_REPORT.json` records the completed local audit.
• —`RECONSTRUCTION_TESTS.md` records offline smoke tests and their deliberately limited claims.

Dataset statistics

Model

The associated released model is available at: <https://huggingface.co/worthing-zhang/sorbian-sft-v3-ck3250>

Citation

Citation will be updated after publication.