CoolFace
Modelpublic

tayaee/Llama-3.1-Tulu-3-8B-AWQ-4bit

sourceHugging Facellama3.1updated 8d agoView on Hugging Face
0likes22downloads
Model Card

tayaee/Llama-3.1-Tulu-3-8B-AWQ-4bit

allenai/Llama-3.1-Tulu-3-8B를 AWQ 4bit(W4A16_ASYM, group 128, lm_head 제외)로 양자화한 모델이다.

  • —Tool: llmcompressor (AWQModifier(duo_scaling="both") + QuantizationModifier(scheme="W4A16_ASYM"))
  • —Calibration: wikitext (wikitext-2-raw-v1, 8 samples, seq 512, 학습용 축소 설정)
  • —실전 권장: 256~512 samples, seq 2048~4096

사용법 (transformers)

python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("tayaee/Llama-3.1-Tulu-3-8B-AWQ-4bit")
m = AutoModelForCausalLM.from_pretrained("tayaee/Llama-3.1-Tulu-3-8B-AWQ-4bit", torch_dtype="auto")

사용법 (vLLM)

python
from vllm import LLM, SamplingParams
llm = LLM(model="tayaee/Llama-3.1-Tulu-3-8B-AWQ-4bit", quantization="compressed-tensors")
print(llm.generate(["Hello, my name is"], SamplingParams(max_tokens=60))[0].outputs[0].text)