Modelpublic

Luki-G/augmented-residuals-culturax24-downstream

sourceHugging Facemitupdated 27d agoView on Hugging Face

0likes

Model Card

CulturaX-24 XNLI and PAWS-X full-finetuned checkpoints

Six sequence-classification models obtained by full fine-tuning the 115,200-step CulturaX-24 (40 GB, SentencePiece 120k) plain residual, four-stream mHC-lite, and full Attention Residuals language-model checkpoints.

Training used English examples for five epochs (effective batch size 32, learning rate 2e-5, seed 42). Evaluation is multilingual over all task test languages. Values below are accuracies in percent; test macro is the unweighted mean over languages.

Architecture	XNLI English val	XNLI test macro	PAWS-X English val	PAWS-X test macro
plain	79.40	63.22	90.20	74.84
mhc-4stream	79.40	63.80	91.40	74.78
attnres-full	79.04	65.10	91.40	76.34

Files

Each <architecture>/<task>/ directory contains:

—model.pt: a PyTorch dictionary with model_state_dict, summary, and label_names. The state dict covers the full custom backbone plus the sequence-classification head.
—summary.json: training configuration, validation history, and per-language evaluation metrics.

The architecture directory names are plain, mhc-4stream, and attnres-full; task names are xnli and pawsx.

These are project-native PyTorch checkpoints, not Transformers save_pretrained directories. Load the corresponding 115,200-step backbone with the multi-mhc code, construct eval.models.SequenceClassifier, and restore model_state_dict. The shared SentencePiece tokenizer is the one distributed with the CulturaX-24 pretraining setup.

The complete evaluation and training metadata needed to distinguish these runs from linear probes is included in every summary.json.