SeanScripts/pixtral-12b-nf4

1---2license: apache-2.03base_model:4- mistral-community/pixtral-12b5pipeline_tag: image-text-to-text6library_name: transformers7---8 9Converted from [mistral-community/pixtral-12b](https://huggingface.co/mistral-community/pixtral-12b) using BitsAndBytes with NF4 (4-bit) quantization. Not using double quantization.10Requires `bitsandbytes` to load.11 12Example usage for image captioning:13```python14from transformers import LlavaForConditionalGeneration, AutoProcessor, BitsAndBytesConfig15from PIL import Image16import time17 18# Load model19model_id = "SeanScripts/pixtral-12b-nf4"20model = LlavaForConditionalGeneration.from_pretrained(21    model_id,22    use_safetensors=True,23    device_map="cuda:0"24)25# Load tokenizer26processor = AutoProcessor.from_pretrained(model_id)27 28# Caption a local image29IMG_URLS = [Image.open("test.png").convert("RGB")]30PROMPT = "<s>[INST]Caption this image:\n[IMG][/INST]"31 32inputs = processor(images=IMG_URLS, text=PROMPT, return_tensors="pt").to("cuda")33prompt_tokens = len(inputs['input_ids'][0])34print(f"Prompt tokens: {prompt_tokens}")35 36t0 = time.time()37generate_ids = model.generate(**inputs, max_new_tokens=512)38t1 = time.time()39total_time = t1 - t040generated_tokens = len(generate_ids[0]) - prompt_tokens41time_per_token = generated_tokens/total_time42print(f"Generated {generated_tokens} tokens in {total_time:.3f} s ({time_per_token:.3f} tok/s)")43 44output = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]45print(output)46```47 48On a 4090, this is getting about 10 - 12 tok/s (without flash attention) and the captions seem pretty good, though I haven't tested very many. It uses about 10 GB VRAM.49 50You can get a set of ComfyUI custom nodes for running this model here:51[https://github.com/SeanScripts/ComfyUI-PixtralLlamaVision](https://github.com/SeanScripts/ComfyUI-PixtralLlamaVision)52