RichardErkhov/TilQazyna_-_llama-kaz-instruct-8B-1-8bits

Quantization made by Richard Erkhov.

Github

Discord

Request more models

llama-kaz-instruct-8B-1 - bnb 8bits

• —Model creator: https://huggingface.co/TilQazyna/
• —Original model: https://huggingface.co/TilQazyna/llama-kaz-instruct-8B-1/

Original model description: --- base_model:

• —meta-llama/Meta-Llama-3-8B language:
• —kk license: apache-2.0 tags:
• —text-generation-inference
• —transformers
• —llama
• —trl ---

Uploaded model

• —Developed by: Til-Qazyna
• —License: apache-2.0
• —Finetuned from model : Meta-Llama-3-8B

This model underwent Continuous Pretraining (CPT) on an extensive Kazakh text corpus to optimize LLAMA3 for the Kazakh language. It was subsequently fine-tuned with Kazakh-language instructional data. The model demonstrates strong performance in processing Kazakh text, answering text-based questions, correcting punctuation and grammar, and summarizing text. However, there is still room for improvement in handling open-ended questions.

Requirements

To install the necessary dependencies, use the following commands:

!pip install --no-deps "xformers<0.0.27" "trl<0.9.0"
!pip install peft accelerate bitsandbytes triton

Loading in 8bit with transformers

  from transformers import AutoTokenizer, AutoModelForCausalLM
  
  model_name = "TilQazyna/llama-kaz-instruct-8B-1"
  hf_token = "<ENTER YOUR TOKEN>"

  # enable load_in_4bit=True for faster results but slighlty lower accuracy
  model = AutoModelForCausalLM.from_pretrained(model_name, load_in_8bit=True, use_auth_token=hf_token)
  tokenizer = AutoTokenizer.from_pretrained(model_name, use_auth_token=hf_token)

Running simple inference

  from transformers import TextStreamer
  inputs = tokenizer("Тапсырма: Келесі мәтіндегі пунктуацияларды және грамматикалық қателерді дұрыста. \n\nМәтін: Жаналыктар леби осиндай \n\nЖауабы:", return_tensors="pt")
  text_streamer = TextStreamer(tokenizer)
  _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128)