Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-1.0
<a href="https://ibb.co/NtQ3QfF"><img src="https://i.ibb.co/RYZSZtg/model.webp" alt="model" border="0" alt="Model-card-peft-lora-1.0" align="center">></a> alt="Model-card-peft-lora-1.0" align="center">
Model Card for Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-1.0
Model Details
Model Description
- Finetuned from model:[Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT]
Training Details
Parameters used for Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-1.0
model = FastLanguageModel.getpeftmodel(
model, r = 16, targetmodules = ["qproj", "kproj", "vproj", "oproj", "gateproj", "upproj", "downproj",],
loraalpha = 16, loradropout = 0.05, bias = "none",
usegradientcheckpointing = "unsloth", randomstate = 3407, userslora = False, loftq_config = None, )
trainer = SFTTrainer(
model = model, tokenizer = tokenizer, traindataset = dataset, datasettext_field = "completion",
maxseqlength = maxseqlength, datasetnumproc = 2, packing = False,
args = TrainingArguments( perdevicetrainbatchsize = 6, gradientaccumulationsteps = 4, warmupsteps = 5, maxsteps=5000, learningrate = 2e-4, fp16 = not isbfloat16_supported(),
bf16 = isbfloat16supported(), loggingsteps = 1, optim = "adamw8bit", weightdecay = 0.01, lrschedulertype = "linear", seed = 3407, outputdir = "outputs2", pushtohub=True, hubalways_push=True, ), )
Dataset: Berkeley-nest/Nectar
Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[berkeley-nest/Nectar]
Evaluation
MMLU Pro 0-shot: 0.2927
Evaluation Data
<!-- This should link to a Dataset Card if possible. -->
[TIGER-AI-Lab/MMLU-Pro]
Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).