Modelpublic

dolfsai/Jan-nano-128k-W8A8

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face

1likes9downloads

Model Card

dolfsai/Jan-nano-128k-W8A8

This is a compressed version of Menlo/Jan-nano-128k using llm-compressor with the following scheme: W8A8 Usage instructions and details here

Model Details

—Original Model: Menlo/Jan-nano-128k
—Quantization Method: GPTQ
—Compression Libraries: llm-compressor
—Calibration Dataset: ultrachat_200k (1024 samples)
—Optimized For: Inference with vLLM
—License: same as original model