RichardErkhov/autopilot-ai_-_Indic-sentence-completion-4bits

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Indic-sentence-completion - bnb 4bits

• —Model creator: https://huggingface.co/autopilot-ai/
• —Original model: https://huggingface.co/autopilot-ai/Indic-sentence-completion/

Original model description: --- language:

• —hi
• —gu
• —pa
• —as
• —ta
• —mr
• —bn
• —te
• —ml
• —kn --- Indic-Sentence-Completion --- license: other ---

Details

The model cannot be commercially used. It's a fine-tuned Bloom-3B in several Indian languages:

• —Gujarati
• —Marathi
• —Bangali
• —Punjabi
• —Kannada
• —Malayalam
• —Telugu
• —Tamil
• —Hindi

Architecture

Same as Bloom-3B, the model is decoder only.

Motivation behind the model fine-tuning

• —The model can be fine-tuned for any downstream task that requires the use of the aforementioned Indian languages
• —PEFT LoRA is advised.
• —Can be stacked with an Encoder if needed for any Sequence to Sequence task that requires aforementioned Indian languages

Example of getting inference from the model

from transformers import AutoModel, AutoConfig, AutoModelForCausalLM, AutoTokenizer

# Path to the directory containing the model files modeldirectory = "autopilot-ai/Indic-sentence-completion" tokenizer = AutoTokenizer.frompretrained(modeldirectory) model = AutoModelForCausalLM.frompretrained( modeldirectory, loadin8bit=True, devicemap="auto", )

# Load the model configuration config = AutoConfig.frompretrained(modeldirectory)

# Load the model model = AutoModel.frompretrained(modeldirectory, config=config) batch = tokenizer("હેલો કેમ છો?", return_tensors='pt')

with torch.cuda.amp.autocast(): outputtokens = model.generate(**batch, maxnew_tokens=10)

print('\n\n', tokenizer.decode(outputtokens[0], skipspecial_tokens=True))

## To run the above code snippet (in 8 bits), make sure to install the following pip install accelerate bitsandbytes