TungCan/vietnamese-correction-lora-v2

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

vietnamese-correction-lora-v2

This model is a fine-tuned version of vinai/bartpho-syllable on an unknown dataset. It achieves the following results on the evaluation set:

• —Loss: 0.1776123046875
• —Sacrebleu: 25.07128550273525
• —Precision: 0.9230769230769231
• —Recall: 0.6
• —F1 Score: 0.7272727272727274
• —evalsamplesper_second: 6.776

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

DatasetDict({
    train: Dataset({
        features: ['input', 'output'],
        num_rows: 800000
    })
    val: Dataset({
        features: ['input', 'output'],
        num_rows: 200000
    })
    test: Dataset({
        features: ['input', 'output'],
        num_rows: 40000
    })
})

Training procedure

The following bitsandbytes quantization config was used during training:

• —loadin8bit: False
• —loadin4bit: True
• —llmint8threshold: 6.0
• —llmint8skip_modules: None
• —llmint8enablefp32cpu_offload: False
• —llmint8hasfp16weight: False
• —bnb4bitquant_type: nf4
• —bnb4bitusedoublequant: True
• —bnb4bitcompute_dtype: float16

Training hyperparameters

The following hyperparameters were used during training:

• —trainable params: 25,165,824 || all params: 326,801,408 || trainable%: 7.700647360735974
• —Num examples = 800,000
• —Num Epochs = 2
• —Instantaneous batch size per device = 12
• —Total train batch size (w. parallel, distributed & accumulation) = 72
• —Gradient Accumulation steps = 6
• —Total optimization steps = 22,222
• —Number of trainable parameters = 25,165,824

Framework versions

• —PEFT 0.4.0
• —PEFT 0.14.0
• —Transformers 4.47.0
• —Pytorch 2.5.1+cu121
• —Datasets 3.3.1
• —Tokenizers 0.21.0

library_name: peft ---

Training procedure

The following bitsandbytes quantization config was used during training:

• —loadin8bit: False
• —loadin4bit: True
• —llmint8threshold: 6.0
• —llmint8skip_modules: None
• —llmint8enablefp32cpu_offload: False
• —llmint8hasfp16weight: False
• —bnb4bitquant_type: nf4
• —bnb4bitusedoublequant: True
• —bnb4bitcompute_dtype: float16

Framework versions

• —PEFT 0.4.0