zainkhanz/qwen3-4b-toxic-unsloth

Qwen3-4B Toxic Unsloth (SFT + DPO)

Fine-tuned version of [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) using Unsloth and TRL.

This model is trained to be direct and low-refusal: it is preference-optimized to answer user requests instead of declining with policy-style refusals.

Summary

Behavior

Compared with the stock Instruct model, this fine-tune is tuned to:

• —Answer the user’s request directly
• —Avoid generic “I can’t help with that” / policy refusal patterns on the training distribution
• —Follow instructions in a straightforward, conversational style

It still inherits Qwen3-4B limits (factual errors, bias, hallucination). It is not guaranteed to be correct, safe, or legal for every use case. You are responsible for how you use the outputs.

Related checkpoints

Training recipe

Stage 1 — SFT

• —Objective: supervised fine-tuning on chosen responses
• —Format: Qwen chat template (system optional, user = prompt, assistant = chosen)
• —Method: Unsloth FastLanguageModel + QLoRA
• —LoRA: r=16, lora_alpha=32, targets q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Stage 2 — DPO

• —Objective: Direct Preference Optimization on (prompt, chosen, rejected)
• —beta ≈ 0.1
• —Continues from the SFT adapter, then weights are merged to 16-bit for this repo

Data

• —Source: adamo1139/toxic-dpo-natural-v5
• —Fields used: system, prompt, chosen, rejected
• —Designed to prefer non-refusing, natural answers over refusal-style rejected answers

Hardware

• —Trained on Kaggle GPU (single-GPU Unsloth run)

Quick start (Transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "zainkhanz/qwen3-4b-toxic-unsloth"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {"role": "user", "content": "Your question here."}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))