NovelAI/genji-python-6B-split

1---2language:3- en4tags:5- pytorch6- causal-lm7license: apache-2.08datasets:9- the Pile10---11 12# Genji-python 6B13 14For example usage or to easily use the model you can check our colab notebook:15[Notebook](https://colab.research.google.com/drive/1PnWpx02IEUkY8jhLKd_NewUGEXahAska?usp=sharing)16 17## Model Description18 19Genji is a transformer model finetuned on EleutherAI's GPT-J 6B model. This particular model is trained on python only code approaching 4GB in size.20Split model has the checkpoints splitted, which makes it use less system RAM while loading and makes it faster to load.21This model needs more effort to set up as you need to install git-lfs and pull the repo.22 23| Hyperparameter    | Value  | 24|-------------------|--------|25| n_parameters      | 6,053,381,344 |26| n_layers          | 28*    |27| d_model           | 4,096  |28| d_ff              | 16,384 |29| n_heads           | 16     |30| d_head            | 256    |31| n_ctx             | 2,048  |32| n_vocab           | 50,400 (same tokenizer as GPT-2/3)  |33| position encoding | [Rotary position encodings (RoPE)](https://arxiv.org/abs/2104.09864) |34| RoPE dimensions   | [64](https://github.com/kingoflolz/mesh-transformer-jax/blob/f2aa66e0925de6593dcbb70e72399b97b4130482/mesh_transformer/layers.py#L223) |35 36`*` each layer consists of one feedforward block and one self attention block37 38The model consists of 28 layers with a model dimension of 4096, and a feedforward dimension of 16384. The model39dimension is split into 16 heads, each with a dimension of 256. Rotary position encodings (RoPE) was applied to 6440dimensions of each head. The model is trained with a tokenization vocabulary of 50257, using the same set of BPEs as41GPT-2/GPT-3.42 43## Training data44 45GPT-J 6B was pretrained on the [Pile](pile.eleuther.ai), a large scale curated dataset created by EleutherAI for the purpose of training this model. After the pre-training, it's finetuned on the python code that was taken from the Pile.46 47## Training procedure48 49Genji-python-6B is trained for 20k steps on around 655 million tokens with learning rate of 2e-0650 51## Intended Use52 53This model is trained for assistence on writing python code and having fun trying weird stuff with it. 54 55### How to use56 57This model is only usable with our fork because GPT-J is not merged to the main transformers repo yet. When it's merged, we will make this model easily loadable.58For now, you need to use this fork:59[Fork](https://github.com/finetuneanon/transformers)60 61to install with pip:62```bash63pip install git+https://github.com/finetuneanon/transformers@gpt-neo-localattention3-rp-b64```65 66**git-lfs** also needs to be installed, on ubuntu:67```bash68apt install git-lfs69```70 71after it's installed, initialize git-lfs:72```bash73git lfs install74```75 76then clone this repo:77```bash78git clone https://huggingface.co/NovelAI/genji-python-6B-split79```80 81Now we can load the model.82 83We recommend the usage of the model as FP16. That way, it fits in 16GB VRAM cards.84 85How to use:86```python87from transformers import (88    AutoTokenizer,89    AutoModelForCausalLM,90    GPTNeoForCausalLM,91)92 93model = AutoModelForCausalLM.from_pretrained("genji-python-6B-split/model").half().eval().cuda()94tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B")95 96text = '''def print_customer_name'''97 98tokens = tokenizer(text, return_tensors="pt").input_ids99generated_tokens = model.generate(tokens.long().cuda(), use_cache=True, do_sample=True, top_k=50, temperature=0.3, top_p=0.9, repetition_penalty=1.125, min_length=1, max_length=len(tokens[0]) + 400, pad_token_id=tokenizer.eos_token_id)100last_tokens = generated_tokens[0][len(tokens[0]):]101generated_text = tokenizer.decode(last_tokens)102print("Generation:\n" + generated_text)103```104When ran, this code generates:105```python106Prompt:107def print_customer_name108Generation:109(self, customer):110        """Print the name of a customer."""111        if not self.is_valid():112            return113 114        print("Customer: {}".format(customer))115```116 117For example usage, you can see our colab notebook as well:118[Notebook](https://colab.research.google.com/drive/1PnWpx02IEUkY8jhLKd_NewUGEXahAska?usp=sharing)119 120## Eval results121 122TBD123 124## Acknowledgements125 126This project was possible because of the compute provided by the127[TPU Research Cloud](https://sites.research.google/trc/) and [EleutherAI](https://eleuther.ai/) for pretraining of the GPT-J 6B.128 129Thanks to everyone who contributed to this project:130- [Aero](https://github.com/AeroScripts)131- [Finetune](https://github.com/finetuneanon)132- [Kurumuz](https://github.com/kurumuz)