yhn112/WMT_Limited_Resources_Sorbian

WMT 2026 Limited-Resources Sorbian Multitask Model

This repository contains the single merged BF16 model used for our primary Sorbian-track submission to the WMT 2026 Shared Task Multitask LLMs with Limited Resources.

The model handles all five task families with one checkpoint:

• —machine translation (MT);
• —multiple-choice question answering (QA);
• —spelling correction (SC);
• —grammar correction (GC);
• —mathematical reasoning (MR).

It is a text-only Qwen3_5ForCausalLM artifact derived from `Qwen/Qwen3.5-2B`. The trained LoRA adapter was merged into the base model, so PEFT is not required for loading this repository.

Training data

Fine-tuning used 278,488 unique records:

Data separation and MT source overlap

Official development/test records were not copied into the final CLEAN-J2 training mixture. Hidden MT test references were unavailable and were not used. Sorbian certificate QA material, translated Sorbian MR data, MR development/test examples, and PolyMath-derived material were not used as training examples.

However, 763 retained official MT-training records share normalized source strings with the released MT test set: 762 in German-to-Lower-Sorbian and one in Upper-to-Lower-Sorbian. Matching used Unicode NFC normalization followed by whitespace collapsing, within each translation direction. These are source-only matches. The official training records include translations of these sources, but agreement with hidden test references could not be checked.

The audit of the original official MT training files found 778 source matches: 777 against the 5,352 German-to-Lower-Sorbian test rows (14.52%) and one against the 4,000 Upper-to-Lower-Sorbian test rows. The other four directions had none. We recorded the matching IDs and source hashes and continued ordinary use of the organizer-provided MT training data. These overlaps were neither selected nor upweighted using test information, and the overlap list was not used as a translation lookup. No dedicated test-source-overlap filter was applied to the MT backbone. Fifteen of the original matches were removed incidentally when synthetic holdout groups were excluded, leaving the 763 records above.

Data separation followed different policies for the training components:

• —MT and development data: four bilingual German–Lower-Sorbian training pairs matching complete normalized development pairs were removed from the initial SFT pool. Because each pair was used in both directions, this removed eight SFT records. Single-side development matches were retained.
• —Synthetic SC, GC, and QA: candidate texts were screened against released development/test texts after normalization; matches were used only to remove candidates. Source groups were split before generation, and all synthetic holdout groups were removed from MT replay. This excluded 8,170 MT records from 4,085 source groups; four malformed MT records were removed separately.
• —MR: the English GSM8K training source was screened against all released WMT26 MR development/test files. Exact and normalized checks found no matches. The final mixture used English questions with answer-only targets.

The internal label CLEAN-J2 and the manifest field official_dev_or_test_training_rows: 0 describe the exclusion of records taken directly from evaluation files. They do not establish disjoint MT training and test sources.

Method

The base model was adapted with all-linear LoRA (r=16, alpha=32, dropout 0.05) using weighted, globally token-normalized supervised cross-entropy. The MT/SC/GC/MR/QA record weights were 1/1/10/6/12. Training used a maximum sequence length of 512 and an effective global batch size of 32.

The selected checkpoint is the final step-8703 checkpoint of a conservative additional training dose on the same frozen mixture. Checkpoint selection used an unchanged official Sorbian development harness and a pre-registered multitask retention rule; hidden-test scores were not used for checkpoint selection. The same merged model, official prompts, and official output converter were used for every submitted task.

Research workflow and compute constraint

This project also served as a test of an autoresearch-style workflow. Within bounded experiment plans, automated agents prepared reproducible trials, monitored runs, audited outputs and artifacts, and compared candidates against fixed quality and safety gates. Final model choices and external submissions remained human-controlled.

A deliberately strict engineering constraint was that the main optimization and production training had to fit on two consumer NVIDIA RTX 3090 GPUs with 24 GB of memory each. We profiled batch geometry, compilation and attention paths, optimizer variants, distributed-training settings, and power limits. The validated short-context production benchmark reached 80.7 samples/s with an effective global batch size of 32. The final, longer-context multitask J2 run sustained 52.21 samples/s and completed two epochs over 278,488 records in 10,670 seconds (5.93 GPU-hours). These measurements are specific to this hardware and workload; “optimal” here means the best validated throughput/quality trade-off among the configurations we tested, not a universal optimum.

Loading

The model was validated with transformers==5.13.0 and BF16 inference.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "yhn112/WMT_Limited_Resources_Sorbian"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
)

messages = [
    {
        "role": "user",
        "content": "Translate from German to Upper Sorbian: Guten Morgen!",
    }
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

For reproducing the shared-task submission, use the official task prompts and output converter rather than the illustrative prompt above.

Artifact integrity

• —Format: merged BF16 Transformers model
• —Parameters: 1,881,825,088
• —Weight file: model.safetensors
• —Weight SHA-256: 59e8d935db2e74e8c6e29ec183db0c054ef4a8bd0f44eec010fe3e388ea7ffc4
• —Source checkpoint: clean-J2 additional-dose step 8703

merge_manifest.json and source_train_manifest.json record additional artifact and training provenance.

Limitations

The MT source overlap described above limits interpretation of test scores as performance on unseen sources. It presents a risk of transductive contamination; it does not establish leakage of hidden test labels. We did not train an otherwise matched model with all test-source overlaps removed, and we have no hidden-reference evaluation broken down by overlapping versus non-overlapping sources. The effect of these overlaps on the reported MT scores is therefore unknown.

This is a shared-task research model, not a general-purpose Sorbian language model. Raw generative responses can continue after the first complete task answer; the official converter extracts the task-formatted answer. Grammar correction and mathematical reasoning remain weak, and the generated Sorbian text has not received comprehensive native-speaker validation. Users should validate outputs for their own domain and should not rely on the model for high-stakes decisions.

License

This derivative model follows the Apache License 2.0 of the base `Qwen/Qwen3.5-2B` model.