Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-2.0

<a href="https://ibb.co/4jPnqYk"><img src="https://i.ibb.co/qRk3SW6/DALL-E-2024-08-08-05-52-48-Craft-an-epic-and-historic-image-for-a-model-card-blending-elements-of-an.webp" alt="DALL-E-2024-08-08-05-52-48-Craft-an-epic-and-historic-image-for-a-model-card-blending-elements-of-an" border="0"></a>

Model Card for Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-2.0

Model Details

Model Description

• —Finetuned from model:[Na0s/Llama-3.1-8b-Pruned-4-Layers-1.0]

<!-- Provide a quick summary of what the model is/does. -->

Training Details

model = FastLanguageModel.getpeftmodel( model, r = 4, targetmodules = ["qproj", "kproj", "vproj", "oproj", "gateproj", "upproj", "downproj",], loraalpha = 4, loradropout = 0.05, bias = "none",

usegradientcheckpointing = "unsloth", randomstate = 3407, userslora = False, loftq_config = None, )

from trl import SFTTrainer from transformers import TrainingArguments from unsloth import isbfloat16supported

trainer = SFTTrainer( model = model, tokenizer = tokenizer, traindataset = dataset, datasettextfield = "completion", maxseqlength = maxseqlength, datasetnumproc = 2, packing = False, args = TrainingArguments( perdevicetrainbatchsize = 10, gradientaccumulationsteps = 4, warmupsteps = 5, maxsteps=5000, learningrate = 2e-4, fp16 = not isbfloat16supported(), bf16 = isbfloat16supported(), loggingsteps = 1, optim = "adamw8bit", weightdecay = 0.01, lrschedulertype = "cosine", seed = 3407, outputdir = "outputs4", pushtohub=True, hubalways_push=True, ), )

Training Data

[meta-math/MetaMathQA]

Evaluation

MMLU Pro 0-shot: 0.2872

Evaluation Data

<!-- This should link to a Dataset Card if possible. -->

[TIGER-AI-Lab/MMLU-Pro]

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).