Weni/ZeroShot-2.2.1-Llama2-13b-Multilanguage-3.0.3

Training Hyperparameters

• —evaluation_strategy: epoch
• —predictionlossonly: False
• —perdevicetrainbatchsize: 2
• —perdeviceevalbatchsize: 8
• —pergputrainbatchsize: None
• —pergpuevalbatchsize: None
• —gradientaccumulationsteps: 2
• —evalaccumulationsteps: 1
• —eval_delay: 0
• —learning_rate: 0.0004
• —weight_decay: 0.01
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —maxgradnorm: 0.3
• —numtrainepochs: 10
• —max_steps: -1
• —lrschedulertype: cosine
• —warmup_ratio: 0.1
• —warmup_steps: 0
• —log_level: passive
• —loglevelreplica: warning
• —logoneach_node: True
• —logging_strategy: steps
• —loggingfirststep: False
• —logging_steps: 500
• —loggingnaninf_filter: True
• —save_strategy: epoch
• —save_steps: 500
• —savetotallimit: 5
• —save_safetensors: True
• —saveoneach_node: False
• —no_cuda: False
• —usempsdevice: False
• —seed: 42
• —data_seed: None
• —jitmodeeval: False
• —use_ipex: False
• —bf16: False
• —fp16: True
• —fp16optlevel: O1
• —halfprecisionbackend: auto
• —bf16fulleval: False
• —fp16fulleval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpunumcores: None
• —tpumetricsdebug: False
• —debug: []
• —dataloaderdroplast: False
• —eval_steps: None
• —dataloadernumworkers: 0
• —past_index: -1
• —removeunusedcolumns: True
• —label_names: None
• —loadbestmodelatend: True
• —metricforbestmodel: evalloss
• —greaterisbetter: False
• —ignoredataskip: False
• —sharded_ddp: []
• —fsdp: []
• —fsdpminnum_params: 0
• —fsdpconfig: {'fsdpminnumparams': 0, 'xla': False, 'xlafsdpgrad_ckpt': False}
• —fsdptransformerlayerclsto_wrap: None
• —deepspeed: None
• —labelsmoothingfactor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —groupbylength: False
• —ddpfindunused_parameters: None
• —ddpbucketcap_mb: None
• —ddpbroadcastbuffers: None
• —dataloaderpinmemory: True
• —skipmemorymetrics: True
• —uselegacyprediction_loop: False
• —pushtohub: True
• —resumefromcheckpoint: None
• —hubstrategy: allcheckpoints
• —gradient_checkpointing: True

Training procedure

The following bitsandbytes quantization config was used during training:

• —loadin8bit: False
• —loadin4bit: True
• —llmint8threshold: 6.0
• —llmint8skip_modules: None
• —llmint8enablefp32cpu_offload: False
• —llmint8hasfp16weight: False
• —bnb4bitquant_type: nf4
• —bnb4bitusedoublequant: True
• —bnb4bitcompute_dtype: bfloat16

• —

Framework versions

• —PEFT 0.4.0