Lagstill/Varsity_module2_bot

1---2license: gemma3---4# Gemma Model Card5 6**Model Page**: [Gemma](https://ai.google.dev/gemma/docs)7 8This model card corresponds to the 7B base version of the Gemma model. You can also visit the model card of the [2B base model](https://huggingface.co/google/gemma-2b), [7B instruct model](https://huggingface.co/google/gemma-7b-it), and [2B instruct model](https://huggingface.co/google/gemma-2b-it). 9 10**Resources and Technical Documentation**:11 12* [Gemma Technical Report](https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf)13* [Responsible Generative AI Toolkit](https://ai.google.dev/responsible)14* [Gemma on Kaggle](https://www.kaggle.com/models/google/gemma)15* [Gemma on Vertex Model Garden](https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/335?version=gemma-7b-gg-hf)16 17**Terms of Use**: [Terms](https://www.kaggle.com/models/google/gemma/license/consent)18 19**Authors**: Google20 21## Model Information22 23Summary description and brief definition of inputs and outputs.24 25### Description26 27Gemma is a family of lightweight, state-of-the-art open models from Google,28built from the same research and technology used to create the Gemini models.29They are text-to-text, decoder-only large language models, available in English,30with open weights, pre-trained variants, and instruction-tuned variants. Gemma31models are well-suited for a variety of text generation tasks, including32question answering, summarization, and reasoning. Their relatively small size33makes it possible to deploy them in environments with limited resources such as34a laptop, desktop or your own cloud infrastructure, democratizing access to35state of the art AI models and helping foster innovation for everyone.36 37### Context Length38Models are trained on a context length of 8192 tokens.39 40### Usage41 42Below we share some code snippets on how to get quickly started with running the model. First make sure to `pip install -U transformers`, then copy the snippet from the section that is relevant for your usecase.43 44#### Fine-tuning examples45 46You can find fine-tuning notebooks under the [`examples/` directory](https://huggingface.co/google/gemma-7b/tree/main/examples). We provide:47 48* A script to perform Supervised Fine-Tuning (SFT) on UltraChat dataset using [QLoRA](https://huggingface.co/papers/2305.14314)49* A script to perform SFT using FSDP on TPU devices50* A notebook that you can run on a free-tier Google Colab instance to perform SFT on English quotes dataset. You can also find the copy of the notebook [here](https://github.com/huggingface/notebooks/blob/main/peft/gemma_7b_english_quotes.ipynb).51 52#### Running the model on a CPU53 54 55```python56from transformers import AutoTokenizer, AutoModelForCausalLM57tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")58model = AutoModelForCausalLM.from_pretrained("google/gemma-7b")59input_text = "Write me a poem about Machine Learning."60input_ids = tokenizer(input_text, return_tensors="pt")61outputs = model.generate(**input_ids)62print(tokenizer.decode(outputs[0]))63```64 65 66#### Running the model on a single / multi GPU67 68 69```python70# pip install accelerate71from transformers import AutoTokenizer, AutoModelForCausalLM72tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")73model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", device_map="auto")74input_text = "Write me a poem about Machine Learning."75input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")76outputs = model.generate(**input_ids)77print(tokenizer.decode(outputs[0]))78```79 80 81#### Running the model on a GPU using different precisions82 83* _Using `torch.float16`_84 85```python86# pip install accelerate87from transformers import AutoTokenizer, AutoModelForCausalLM88tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")89model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", device_map="auto", revision="float16")90input_text = "Write me a poem about Machine Learning."91input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")92outputs = model.generate(**input_ids)93print(tokenizer.decode(outputs[0]))94```95 96* _Using `torch.bfloat16`_97 98```python99# pip install accelerate100from transformers import AutoTokenizer, AutoModelForCausalLM101tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")102model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", device_map="auto", torch_dtype=torch.bfloat16)103input_text = "Write me a poem about Machine Learning."104input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")105outputs = model.generate(**input_ids)106print(tokenizer.decode(outputs[0]))107```108 109#### Quantized Versions through `bitsandbytes`110 111* _Using 8-bit precision (int8)_112 113```python114# pip install bitsandbytes accelerate115from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig116quantization_config = BitsAndBytesConfig(load_in_8bit=True)117tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")118model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", quantization_config=quantization_config)119input_text = "Write me a poem about Machine Learning."120input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")121outputs = model.generate(**input_ids)122print(tokenizer.decode(outputs[0]))123```124 125* _Using 4-bit precision_126 127```python128# pip install bitsandbytes accelerate129from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig130quantization_config = BitsAndBytesConfig(load_in_4bit=True)131tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")132model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", quantization_config=quantization_config)133input_text = "Write me a poem about Machine Learning."134input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")135outputs = model.generate(**input_ids)136print(tokenizer.decode(outputs[0]))137```138 139 140#### Other optimizations141 142* _Flash Attention 2_143 144First make sure to install `flash-attn` in your environment `pip install flash-attn`145 146```diff147model = AutoModelForCausalLM.from_pretrained(148    model_id, 149    torch_dtype=torch.float16, 150+   attn_implementation="flash_attention_2"151).to(0)152```153 154### Inputs and outputs155 156*   **Input:** Text string, such as a question, a prompt, or a document to be157    summarized.158*   **Output:** Generated English-language text in response to the input, such159    as an answer to a question, or a summary of a document.160## Model Data161 162Data used for model training and how the data was processed.163 164### Training Dataset165 166These models were trained on a dataset of text data that includes a wide variety167of sources, totaling 6 trillion tokens. Here are the key components:168 169* Web Documents: A diverse collection of web text ensures the model is exposed170  to a broad range of linguistic styles, topics, and vocabulary. Primarily171  English-language content.172* Code: Exposing the model to code helps it to learn the syntax and patterns of173  programming languages, which improves its ability to generate code or174  understand code-related questions.175* Mathematics: Training on mathematical text helps the model learn logical176  reasoning, symbolic representation, and to address mathematical queries.177 178The combination of these diverse data sources is crucial for training a powerful179language model that can handle a wide variety of different tasks and text180formats.181 182### Data Preprocessing183 184Here are the key data cleaning and filtering methods applied to the training185data:186 187* CSAM Filtering: Rigorous CSAM (Child Sexual Abuse Material) filtering was188  applied at multiple stages in the data preparation process to ensure the189  exclusion of harmful and illegal content190* Sensitive Data Filtering: As part of making Gemma pre-trained models safe and191  reliable, automated techniques were used to filter out certain personal192  information and other sensitive data from training sets.193* Additional methods: Filtering based on content quality and safely in line with194  [our policies](https://storage.googleapis.com/gweb-uniblog-publish-prod/documents/2023_Google_AI_Principles_Progress_Update.pdf#page=11).195 196## Implementation Information197 198Details about the model internals.199 200### Hardware201 202Gemma was trained using the latest generation of203[Tensor Processing Unit (TPU)](https://cloud.google.com/tpu/docs/intro-to-tpu) hardware (TPUv5e).204 205Training large language models requires significant computational power. TPUs,206designed specifically for matrix operations common in machine learning, offer207several advantages in this domain:208 209* Performance: TPUs are specifically designed to handle the massive computations210  involved in training LLMs. They can speed up training considerably compared to211  CPUs.212* Memory: TPUs often come with large amounts of high-bandwidth memory, allowing213  for the handling of large models and batch sizes during training. This can214  lead to better model quality.215* Scalability: TPU Pods (large clusters of TPUs) provide a scalable solution for216  handling the growing complexity of large foundation models. You can distribute217  training across multiple TPU devices for faster and more efficient processing.218* Cost-effectiveness: In many scenarios, TPUs can provide a more cost-effective219  solution for training large models compared to CPU-based infrastructure,220  especially when considering the time and resources saved due to faster221  training.222* These advantages are aligned with223  [Google's commitments to operate sustainably](https://sustainability.google/operating-sustainably/).224 225### Software226 227Training was done using [JAX](https://github.com/google/jax) and [ML Pathways](https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture).228 229JAX allows researchers to take advantage of the latest generation of hardware,230including TPUs, for faster and more efficient training of large models.231 232ML Pathways is Google's latest effort to build artificially intelligent systems233capable of generalizing across multiple tasks. This is specially suitable for234[foundation models](https://ai.google/discover/foundation-models/), including large language models like235these ones.236 237Together, JAX and ML Pathways are used as described in the238[paper about the Gemini family of models](https://arxiv.org/abs/2312.11805); "the 'single239controller' programming model of Jax and Pathways allows a single Python240process to orchestrate the entire training run, dramatically simplifying the241development workflow."242 243## Evaluation244 245Model evaluation metrics and results.246 247### Benchmark Results248 249These models were evaluated against a large collection of different datasets and250metrics to cover different aspects of text generation:251 252| Benchmark                      | Metric        | 2B Params | 7B Params |253| ------------------------------ | ------------- | ----------- | --------- |254| [MMLU](https://arxiv.org/abs/2009.03300)                   | 5-shot, top-1 | 42.3        | 64.3      |255| [HellaSwag](https://arxiv.org/abs/1905.07830)         | 0-shot        |71.4        | 81.2      |256| [PIQA](https://arxiv.org/abs/1911.11641)                   | 0-shot        | 77.3        | 81.2      |257| [SocialIQA](https://arxiv.org/abs/1904.09728)      | 0-shot        | 49.7        | 51.8      |258| [BooIQ](https://arxiv.org/abs/1905.10044)                | 0-shot        | 69.4        | 83.2      |259| [WinoGrande](https://arxiv.org/abs/1907.10641)       | partial score | 65.4        | 72.3      |260| [CommonsenseQA](https://arxiv.org/abs/1811.00937) | 7-shot        | 65.3        | 71.3      |261| [OpenBookQA](https://arxiv.org/abs/1809.02789)       |               | 47.8        | 52.8      |262| [ARC-e](https://arxiv.org/abs/1911.01547)                  |               | 73.2        | 81.5      |263| [ARC-c](https://arxiv.org/abs/1911.01547)                   |               | 42.1        | 53.2      |264| [TriviaQA](https://arxiv.org/abs/1705.03551)           | 5-shot        | 53.2        | 63.4      |265| [Natural Questions](https://github.com/google-research-datasets/natural-questions)  | 5-shot        | 12.5       | 23        |266| [HumanEval](https://arxiv.org/abs/2107.03374)      | pass@1        | 22.0        | 32.3      |267| [MBPP](https://arxiv.org/abs/2108.07732)                   | 3-shot        | 29.2        | 44.4      |268| [GSM8K](https://arxiv.org/abs/2110.14168)                | maj@1         | 17.7        | 46.4      |269| [MATH](https://arxiv.org/abs/2108.07732)                   | 4-shot        | 11.8          | 24.3      |270| [AGIEval](https://arxiv.org/abs/2304.06364)           |               | 24.2        | 41.7      |271| [BIG-Bench](https://arxiv.org/abs/2206.04615)         |               | 35.2        | 55.1      |272| ------------------------------ | ------------- | ----------- | --------- |273| **Average**                    |               | **45.0**    | **56.9**  |274 275 276## Ethics and Safety277 278Ethics and safety evaluation approach and results.279 280### Evaluation Approach281 282Our evaluation methods include structured evaluations and internal red-teaming283testing of relevant content policies. Red-teaming was conducted by a number of284different teams, each with different goals and human evaluation metrics. These285models were evaluated against a number of different categories relevant to286ethics and safety, including:287 288* Text-to-Text Content Safety: Human evaluation on prompts covering safety289  policies including child sexual abuse and exploitation, harassment, violence290  and gore, and hate speech.291* Text-to-Text Representational Harms: Benchmark against relevant academic292  datasets such as [WinoBias](https://arxiv.org/abs/1804.06876) and [BBQ Dataset](https://arxiv.org/abs/2110.08193v2).293* Memorization: Automated evaluation of memorization of training data, including294  the risk of personally identifiable information exposure.295* Large-scale harm: Tests for "dangerous capabilities," such as chemical,296  biological, radiological, and nuclear (CBRN) risks.297 298### Evaluation Results299 300The results of ethics and safety evaluations are within acceptable thresholds301for meeting [internal policies](https://storage.googleapis.com/gweb-uniblog-publish-prod/documents/2023_Google_AI_Principles_Progress_Update.pdf#page=11) for categories such as child302safety, content safety, representational harms, memorization, large-scale harms.303On top of robust internal evaluations, the results of well known safety304benchmarks like BBQ, BOLD, Winogender, Winobias, RealToxicity, and TruthfulQA305are shown here.306 307| Benchmark                      | Metric        | 2B Params   | 7B Params |308| ------------------------------ | ------------- | ----------- | --------- |309| [RealToxicity](https://arxiv.org/abs/2009.11462)        | average       | 6.86        | 7.90      |310| [BOLD](https://arxiv.org/abs/2101.11718)                   |               | 45.57       | 49.08     |311| [CrowS-Pairs](https://aclanthology.org/2020.emnlp-main.154/)        | top-1         | 45.82       | 51.33     |312| [BBQ Ambig](https://arxiv.org/abs/2110.08193v2)               | 1-shot, top-1 | 62.58       | 92.54     |313| [BBQ Disambig](https://arxiv.org/abs/2110.08193v2)            | top-1         | 54.62       | 71.99     |314| [Winogender](https://arxiv.org/abs/1804.09301)       | top-1         | 51.25       | 54.17     |315| [TruthfulQA](https://arxiv.org/abs/2109.07958)       |               | 44.84       | 31.81     |316| [Winobias 1_2](https://arxiv.org/abs/1804.06876)       |               | 56.12       | 59.09     |317| [Winobias 2_2](https://arxiv.org/abs/1804.06876)       |               | 91.10       | 92.23     |318| [Toxigen](https://arxiv.org/abs/2203.09509)             |               | 29.77       | 39.59     |319| ------------------------------ | ------------- | ----------- | --------- |320 321 322## Usage and Limitations323 324These models have certain limitations that users should be aware of.325 326### Intended Usage327 328Open Large Language Models (LLMs) have a wide range of applications across329various industries and domains. The following list of potential uses is not330comprehensive. The purpose of this list is to provide contextual information331about the possible use-cases that the model creators considered as part of model332training and development.333 334* Content Creation and Communication335  * Text Generation: These models can be used to generate creative text formats336    such as poems, scripts, code, marketing copy, and email drafts.337  * Chatbots and Conversational AI: Power conversational interfaces for customer338    service, virtual assistants, or interactive applications.339  * Text Summarization: Generate concise summaries of a text corpus, research340    papers, or reports.341* Research and Education342  * Natural Language Processing (NLP) Research: These models can serve as a343    foundation for researchers to experiment with NLP techniques, develop344    algorithms, and contribute to the advancement of the field.345  * Language Learning Tools: Support interactive language learning experiences,346    aiding in grammar correction or providing writing practice.347  * Knowledge Exploration: Assist researchers in exploring large bodies of text348    by generating summaries or answering questions about specific topics.349### Limitations350 351* Training Data352  * The quality and diversity of the training data significantly influence the353    model's capabilities. Biases or gaps in the training data can lead to354    limitations in the model's responses.355  * The scope of the training dataset determines the subject areas the model can356    handle effectively.357* Context and Task Complexity358  * LLMs are better at tasks that can be framed with clear prompts and359    instructions. Open-ended or highly complex tasks might be challenging.360  * A model's performance can be influenced by the amount of context provided361    (longer context generally leads to better outputs, up to a certain point).362* Language Ambiguity and Nuance363  * Natural language is inherently complex. LLMs might struggle to grasp subtle364    nuances, sarcasm, or figurative language.365* Factual Accuracy366  * LLMs generate responses based on information they learned from their367    training datasets, but they are not knowledge bases. They may generate368    incorrect or outdated factual statements.369* Common Sense370  * LLMs rely on statistical patterns in language. They might lack the ability371    to apply common sense reasoning in certain situations.372### Ethical Considerations and Risks373 374The development of large language models (LLMs) raises several ethical concerns.375In creating an open model, we have carefully considered the following:376 377* Bias and Fairness378  * LLMs trained on large-scale, real-world text data can reflect socio-cultural379    biases embedded in the training material. These models underwent careful380    scrutiny, input data pre-processing described and posterior evaluations381    reported in this card.382* Misinformation and Misuse383  * LLMs can be misused to generate text that is false, misleading, or harmful.384  * Guidelines are provided for responsible use with the model, see the385    [Responsible Generative AI Toolkit](http://ai.google.dev/gemma/responsible).386* Transparency and Accountability:387  * This model card summarizes details on the models' architecture,388    capabilities, limitations, and evaluation processes.389  * A responsibly developed open model offers the opportunity to share390    innovation by making LLM technology accessible to developers and researchers391    across the AI ecosystem.392Risks identified and mitigations:393 394* Perpetuation of biases: It's encouraged to perform continuous monitoring395  (using evaluation metrics, human review) and the exploration of de-biasing396  techniques during model training, fine-tuning, and other use cases.397* Generation of harmful content: Mechanisms and guidelines for content safety398  are essential. Developers are encouraged to exercise caution and implement399  appropriate content safety safeguards based on their specific product policies400  and application use cases.401* Misuse for malicious purposes: Technical limitations and developer and402  end-user education can help mitigate against malicious applications of LLMs.403  Educational resources and reporting mechanisms for users to flag misuse are404  provided. Prohibited uses of Gemma models are outlined in the405  [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy).406* Privacy violations: Models were trained on data filtered for removal of PII407  (Personally Identifiable Information). Developers are encouraged to adhere to408  privacy regulations with privacy-preserving techniques.409 410### Benefits411 412At the time of release, this family of models provides high-performance open413large language model implementations designed from the ground up for Responsible414AI development compared to similarly sized models.415 416Using the benchmark evaluation metrics described in this document, these models417have shown to provide superior performance to other, comparably-sized open model418alternatives.