Modelpublic

zainkhanz/qwen3-4b-toxic-unsloth-bnb-8bit

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes11downloads

Model Card

Qwen3-4B Toxic Unsloth — 8-bit (bitsandbytes)

8-bit quantized version of zainkhanz/qwen3-4b-toxic-unsloth using bitsandbytes.

Same behavior as the 16-bit model (SFT + DPO, low-refusal style), lower VRAM.

Summary

Item	Detail
Parent model	zainkhanz/qwen3-4b-toxic-unsloth
Original base	Qwen/Qwen3-4B-Instruct-2507
Quantization	bitsandbytes 8-bit
Parameters	~4B
Training	Unsloth SFT then DPO, then quantize
Dataset	adamo1139/toxic-dpo-natural-v5

Model family

Repo	Format
zainkhanz/qwen3-4b-toxic-unsloth	16-bit merged
zainkhanz/qwen3-4b-toxic-unsloth-bnb-4bit	4-bit NF4
This repo	8-bit

Behavior

Preference-tuned to answer requests directly and reduce generic policy-style refusals, compared with stock Qwen3-4B-Instruct.

Still subject to factual errors, bias, and hallucination. You are responsible for downstream use and must follow applicable law and the Apache 2.0 license.

Load (Transformers)

python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "zainkhanz/qwen3-4b-toxic-unsloth-bnb-8bit"

bnb = BitsAndBytesConfig(load_in_8bit=True)

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb,
    device_map="auto",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Your question here."}]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Intended use

—Lower-VRAM inference of the Toxic Unsloth fine-tune
—Research and experimentation on preference-tuned chat models

Not a guarantee of accuracy or compliance for every prompt. Do not use for illegal activity.

Limitations

—Quantization can slightly change outputs vs 16-bit
—No separate benchmark table shipped with this quant
—Inherits limitations of Qwen3-4B and of the DPO dataset

License

Apache 2.0 (same as Qwen base and parent fine-tune).

Citation

bibtex

@misc{zainkhanz_qwen3_4b_toxic_unsloth_bnb_8bit,
  author       = {zainkhanz},
  title        = {Qwen3-4B Toxic Unsloth (8-bit bitsandbytes)},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {https://huggingface.co/zainkhanz/qwen3-4b-toxic-unsloth-bnb-8bit}
}