KnutJaegersberg/Galactica-120B-GPTQ-2-bit-64g

Experimental quantization.

Working inference code (regular inference with autogptq does not work without returntokentype_ids=False, didn't get it to work with textgen-webui):

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

from transformers import AutoTokenizer, TextGenerationPipeline

tokenizer = AutoTokenizer.frompretrained(quantizedmodeldir, usefast=True)

model = AutoGPTQForCausalLM.fromquantized(quantizedmodeldir, device="cuda:0", usetriton=False)

inputids = tokenizer("Question: What is the purpose of life?\n\nAnswer:", returntensors="pt").input_ids.to("cuda:0")

out = model.generate(inputids=inputids,max_length=300)

print(tokenizer.decode(out[0]))

or

print(tokenizer.decode(model.generate(**tokenizer("test is", returntensors="pt", returntokentypeids=False).to("cuda:0"))[0]))