kernelmachine/silo-pd-1.3b

1---2license: apache-2.03language:4- en5pipeline_tag: text-generation6tags:7- text-generation8- openlm9- silo10---11 12# Silo Language Models: Isolating Legal Risk in a Datastore13 14This is Silo-PD, first introduced in [Silo Language Models]() by researchers at University of Washington, UC Berkeley, and the Allen Institute for AI.15 16### NOTE: Dependencies17 18To use the model, you need to install a specific transformers fork:19 20```21pip install git+https://github.com/kernelmachine/transformers@openlm#egg=transformers22```23 24The model also depends on `xformers`, install via 25 26```27pip install xformers28```29 30### Model Description31 32 33Silo-PD is a 1.3B parameter, decoder-only language model trained on data in the public domain from [the Open License Corpus (OLC)](https://huggingface.co/datasets/kernelmachine/open-license-corpus). 34 35The model is based on the LLaMA architecture as implemented in (OpenLM)[].36 37The model is trained with 128 A100 GPUs across 16 nodes.38 39 40### Model and Training Hyperparameters41 42We follow the model architecture of LLaMa, and we use the GPT-NeoX-20B tokenizer, with 50432 BPE types. 43 44During training, we use 2,048 token sequences that are packed across document boundaries, and we pre-pend a beginning-of-text token to every document. 45 46We use weight decay of 0.1, the Adam optimizer with beta_2 of 0.95, 2,000 steps of warmup, with a cosine learning rate scheduler. 47 48 49| Model  | #L  | #H  | d_model | LR     | Batch  |50|--------|-----|-----|-------------|--------|--------|51| 1.3B   | 24  | 16  | 2048        | 1e-3   | 2.6M   |52 53 54 55### Training data56Specifically, it was trained on the following domain proportions (please see the OLC repository for more details on the data sources for each domain):57 58 59| Domain          | Tokens (B) | %     |60|-----------------|------------|-------|61| Legal           | 27.1       | 86.2  | 62| Books           | 2.9        | 9.3   |63| Science         | 1.2        | 3.8   |64| News            | 0.2        | 0.7   |65| Total           | 31.4       | 100.0 | 66 67We train with early stopping for 60B tokens in total, for a total of 2 epochs of training over this subset 68 69Since the distribution of OLC is highly skewed, we perform a simple upweighting scheme where we upsample all data that accounts for less than 5% of the corpus by a factor of 3x, which we found to work well after a sweep of different settings.70 71### Intended Uses and Limitations72 73This model can be used for prompting for evaluation of downstream tasks as well as text generation. 74 75### How to use76 77 78You can use this model directly with a pipeline for text generation.79 80 81```python82from transformers import pipeline83generator = pipeline('text-generation', model="kernelmachine/silo-pd-1.3b", device='cuda')84generator("Hello")85[{'generated_text': 'Hello, my dear," said the old man, "I have been waiting for you\na long'}]86```87 88By default, generation is deterministic. In order to use the top-k sampling, please set do_sample to True.89 90 91```python92from transformers import pipeline, set_seed93set_seed(42)94generator = pipeline('text-generation', model="kernelmachine/silo-pd-1.3b", device='cuda', do_sample=True)95generator("Hello")96[{'generated_text': 'Hello, Mother," he called.\n\n"Hello, Son.  Have you got a car'}]97```98 99### Limitations and Bias100 101Silo-PD inherits the biases and limitations of public domain data, which carry risks of toxic or otherwise unfair output, due to the prevalence of older copyright-expired text.102 103Silo-PD may also output personally identifiable information, because we did not filter that out of training data.104