CoolFace
Modelpublic

UgurI/Qwen3.8-27B-Turkish-CPT-LoRA-Beta

sourceHugging Facecc-by-nc-sa-4.0updated 1mo agoView on Hugging Face
1likes28downloads
Model Card

Qwen3.8-27B-Turkish-CPT-LoRA-1M

A continued-pretrained Low-Rank Adaptation (LoRA) adapter for Qwen3.8-27B, trained on high-quality filtered Turkish web text from the Kumru corpus.

Model Summary

  • —Base Model: Qwen/Qwen3.8-27B (Text-only causal LM backbone)
  • —Adapter Type: PEFT LoRA (Rank 16, Alpha 16)
  • —Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
  • —Training Tokens: 1,000,000 tokens (512 token packed context)
  • —Training Precision: 4-bit NF4 Base + BF16 Compute / Gradients
  • —Parameters: 79,691,776 trainable parameters (304 MiB safetensors)

Evaluation & Benchmarks

MetricBase Model (Qwen3.8-27B NF4)Turkish-CPT-1M AdapterImprovement
Turkish Validation Perplexity7.58386.9348-8.56%
Turkish Test Perplexity8.05347.3303-8.98%
TurkishMMLU (90-Q Stratified)60.00%75.56%+15.56%
Belebele Reading Comprehension (TR)83.33%90.00%+6.67%
Belebele Reading Comprehension (EN)96.67%96.67%100% Retained

Usage with Transformers & PEFT

python
import torch
from transformers import AutoTokenizer, BitsAndBytesConfig, Qwen3_5ForCausalLM
from peft import PeftModel

base_model_id = "Qwen/Qwen3.8-27B"
adapter_id = "UgurInanc12/Qwen3.8-27B-Turkish-CPT-LoRA-1M"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = Qwen3_5ForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=quantization_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

prompt = "Türkiye'de yapay zeka araştırmalarının geleceği hakkında bir paragraf yaz."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation & Attribution

  • —Training Corpus: VNGRS-AI (vngrs-ai/vngrs-web-corpus)
  • —Base Model: Qwen Team (Qwen/Qwen3.8-27B)
  • —Author: Ugur Inanc