wandgibaut/qwen-1.7b-gpt-oss-20b-pt-BR-distilled

Qwen3-1.7B-GPT-OSS-20B-Distilled

This model is a fine-tuned version of `Qwen/Qwen3-1.7B`, distilled from `openai/gpt-oss-20b` using LoRA and SFTTrainer.

Model Description

This is a student model created through knowledge distillation from a larger, more capable teacher model. The goal is to transfer the knowledge and capabilities of the openai/gpt-oss-20b (teacher) to a smaller, more efficient Qwen/Qwen3-1.7B (student) model.

Training Details

• —Teacher Model: openai/gpt-oss-20b
• —Dataset: Synthetic dataset of 1000 samples in Brazilian Portuguese generated by the teacher using random samples from dominguesm/alpaca-data-pt-br.
• —LoRA Config: r=16, alpha=32, dropout=0.05
• —Training Hyperparams: 3 epochs, learning rate=0.0002, batch size=2

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
model = PeftModel.from_pretrained(model, "wandgibaut/qwen-1.7b-gpt-oss-20b-distilled")