Modelpublic

ShahzebKhoso/t5-small-opencode-lora

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face

0likes

Model Card

license: apache-2.0 tags:

—code-generation
—t5
—lora
—peft
—transformers libraryname: peft basemodel: t5-small datasets: nvidia/OpenCodeReasoning model-index:
—name: T5-Small with LoRA on OpenCodeReasoning results:
—task: type: text2text-generation name: Code Generation dataset: name: OpenCodeReasoning type: nvidia/OpenCodeReasoning metrics:
—name: Loss type: loss value: 4.69 ---

T5-Small with LoRA on OpenCodeReasoning

This is a LoRA fine-tuned version of T5-small on a subset of NVIDIA's OpenCodeReasoning dataset using PEFT. Improved version to be uploaded soon.

Loss Curve

Step	Train Loss	Val Loss
50	8.63	8.17
100	6.04	5.35
150	5.31	4.90
200	5.19	4.71
250	4.94	4.59
300	4.95	4.51
350	4.79	4.46
400	4.89	4.42
450	4.69	4.40

Final Train Loss: 4.69 Final Eval Loss: 4.40

Notes

Trained on subset of OpenCodeReasoning due to Colab memory limits

Use PeftModel with t5-small base

Metrics used: Loss (BLEU skipped due to output structure)

License

Apache 2.0

Example Usage

python

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel, PeftConfig

config = PeftConfig.from_pretrained("ShahzebKhoso/t5-small-opencode-lora")
base_model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path)
model = PeftModel.from_pretrained(base_model, "ShahzebKhoso/t5-small-opencode-lora")
tokenizer = AutoTokenizer.from_pretrained("ShahzebKhoso/t5-small-opencode-lora")

inputs = tokenizer("generate code: write a function to reverse a string", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
'''