Modelpublic

CoNDeNse-AI/GLM-5.1-Qwen3-1.7B-CoNDeNse

sourceHugging Faceupdated 4mo agoView on Hugging Face

5likes31downloads

Model Card

CoNDeNse-AI/GLM-5.1-Qwen3-1.7B-CoNDeNse

Part of the CoNDeNse project — compressing the reasoning capability of large models into small, deployable ones.

Model Details

—Base model: Qwen/Qwen3-1.7B
—Method: LoRA (r=32, α=64)
—Target modules: qproj, kproj, vproj, oproj, gateproj, upproj, down_proj
—Dtype: float16

Training

—Dataset: Jackrong/GLM-5.1-Reasoning-1M-Cleaned (75,000 examples)
—Optimizer: AdamW 8-bit
—Learning rate: 2e-4 with cosine scheduler
—Batch size: 1 × 16 gradient accumulation (effective batch = 16)
—Max sequence length: 4096
—Packing: enabled

Notes

May HALLUCINATE

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-1.7B",
    torch_dtype=torch.float16,
    device_map="cuda",
)

tokenizer = AutoTokenizer.from_pretrained("CoNDeNse-AI/GLM-5.1-Qwen3-1.7B-CoNDeNse")
model = PeftModel.from_pretrained(base_model, "CoNDeNse-AI/GLM-5.1-Qwen3-1.7B-CoNDeNse")

prompt = "<|im_start|>user\nYour question here<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))