SceneWorks/gemma-2-2b-it

1---2license: gemma3library_name: transformers4pipeline_tag: text-generation5tags:6- conversational7base_model: google/gemma-2-2b8license_link: LICENSE9---10 11> **SceneWorks mirror.** Verbatim, unmodified `google/gemma-2-2b-it` weights, re-hosted non-gated for turnkey in-app provisioning of the SceneWorks PiD pixel-diffusion decoder (NVIDIA PiD conditions its decode on a Gemma-2 caption embedding). No weight conversion or fine-tuning. Use is governed by the **Gemma Terms of Use** (`LICENSE`) and the **Gemma Prohibited Use Policy** (`PROHIBITED_USE_POLICY.md`), both shipped alongside; by downloading you agree to them. Distribution by SceneWorks is permitted under Gemma Terms §3.1.12 13 14 15# Gemma 2 model card16 17**Model Page**: [Gemma](https://ai.google.dev/gemma/docs/base)18 19**Resources and Technical Documentation**:20 21* [Responsible Generative AI Toolkit][rai-toolkit]22* [Gemma on Kaggle][kaggle-gemma]23* [Gemma on Vertex Model Garden][vertex-mg-gemma2]24 25**Terms of Use**: [Terms][terms]26 27**Authors**: Google28 29## Model Information30 31Summary description and brief definition of inputs and outputs.32 33### Description34 35Gemma is a family of lightweight, state-of-the-art open models from Google,36built from the same research and technology used to create the Gemini models.37They are text-to-text, decoder-only large language models, available in English,38with open weights for both pre-trained variants and instruction-tuned variants.39Gemma models are well-suited for a variety of text generation tasks, including40question answering, summarization, and reasoning. Their relatively small size41makes it possible to deploy them in environments with limited resources such as42a laptop, desktop or your own cloud infrastructure, democratizing access to43state of the art AI models and helping foster innovation for everyone.44 45### Usage46 47Below we share some code snippets on how to get quickly started with running the model. First, install the Transformers library with:48```sh49pip install -U transformers50```51 52Then, copy the snippet from the section that is relevant for your usecase.53 54#### Running with the `pipeline` API55 56```python57import torch58from transformers import pipeline59 60pipe = pipeline(61    "text-generation",62    model="google/gemma-2-2b-it",63    model_kwargs={"torch_dtype": torch.bfloat16},64    device="cuda",  # replace with "mps" to run on a Mac device65)66 67messages = [68    {"role": "user", "content": "Who are you? Please, answer in pirate-speak."},69]70 71outputs = pipe(messages, max_new_tokens=256)72assistant_response = outputs[0]["generated_text"][-1]["content"].strip()73print(assistant_response)74# Ahoy, matey! I be Gemma, a digital scallywag, a language-slingin' parrot of the digital seas. I be here to help ye with yer wordy woes, answer yer questions, and spin ye yarns of the digital world.  So, what be yer pleasure, eh? 🦜75```76 77#### Running the model on a single / multi GPU78 79```python80# pip install accelerate81from transformers import AutoTokenizer, AutoModelForCausalLM82import torch83 84tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")85model = AutoModelForCausalLM.from_pretrained(86    "google/gemma-2-2b-it",87    device_map="auto",88    torch_dtype=torch.bfloat16,89)90 91input_text = "Write me a poem about Machine Learning."92input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")93 94outputs = model.generate(**input_ids, max_new_tokens=32)95print(tokenizer.decode(outputs[0]))96```97 98You can ensure the correct chat template is applied by using `tokenizer.apply_chat_template` as follows:99```python100messages = [101    {"role": "user", "content": "Write me a poem about Machine Learning."},102]103input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to("cuda")104 105outputs = model.generate(**input_ids, max_new_tokens=256)106print(tokenizer.decode(outputs[0]))107```108 109<a name="precisions"></a>110#### Running the model on a GPU using different precisions111 112The native weights of this model were exported in `bfloat16` precision.113 114You can also use `float32` if you skip the dtype, but no precision increase will occur (model weights will just be upcasted to `float32`). See examples below.115 116* _Upcasting to `torch.float32`_117 118```python119# pip install accelerate120from transformers import AutoTokenizer, AutoModelForCausalLM121 122tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")123model = AutoModelForCausalLM.from_pretrained(124    "google/gemma-2-2b-it",125    device_map="auto",126)127 128input_text = "Write me a poem about Machine Learning."129input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")130 131outputs = model.generate(**input_ids, max_new_tokens=32)132print(tokenizer.decode(outputs[0]))133```134 135#### Running the model through a CLI136 137The [local-gemma](https://github.com/huggingface/local-gemma) repository contains a lightweight wrapper around Transformers138for running Gemma 2 through a command line interface, or CLI. Follow the [installation instructions](https://github.com/huggingface/local-gemma#cli-usage)139for getting started, then launch the CLI through the following command:140 141```shell142local-gemma --model 2b --preset speed143```144 145#### Quantized Versions through `bitsandbytes`146 147<details>148  <summary>149    Using 8-bit precision (int8)  150  </summary>151 152```python153# pip install bitsandbytes accelerate154from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig155 156quantization_config = BitsAndBytesConfig(load_in_8bit=True)157 158tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")159model = AutoModelForCausalLM.from_pretrained(160    "google/gemma-2-2b-it",161    quantization_config=quantization_config,162)163 164input_text = "Write me a poem about Machine Learning."165input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")166 167outputs = model.generate(**input_ids, max_new_tokens=32)168print(tokenizer.decode(outputs[0]))169```170</details>171 172<details>173  <summary>174    Using 4-bit precision  175  </summary>176 177```python178# pip install bitsandbytes accelerate179from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig180 181quantization_config = BitsAndBytesConfig(load_in_4bit=True)182 183tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")184model = AutoModelForCausalLM.from_pretrained(185    "google/gemma-2-2b-it",186    quantization_config=quantization_config,187)188 189input_text = "Write me a poem about Machine Learning."190input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")191 192outputs = model.generate(**input_ids, max_new_tokens=32)193print(tokenizer.decode(outputs[0]))194```195</details>196 197#### Advanced Usage198 199<details>200  <summary>201    Torch compile  202  </summary>203 204[Torch compile](https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html) is a method for speeding-up the 205inference of PyTorch modules. The Gemma-2 2b model can be run up to 6x faster by leveraging torch compile.206 207Note that two warm-up steps are required before the full inference speed is realised:208 209```python210import os211os.environ["TOKENIZERS_PARALLELISM"] = "false"212 213from transformers import AutoTokenizer, Gemma2ForCausalLM214from transformers.cache_utils import HybridCache215import torch216 217torch.set_float32_matmul_precision("high")218 219# load the model + tokenizer220tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")221model = Gemma2ForCausalLM.from_pretrained("google/gemma-2-2b-it", torch_dtype=torch.bfloat16)222model.to("cuda")223 224# apply the torch compile transformation225model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)226 227# pre-process inputs228input_text = "The theory of special relativity states "229model_inputs = tokenizer(input_text, return_tensors="pt").to("cuda")230prompt_length = model_inputs.input_ids.shape[1]231 232# set-up k/v cache233past_key_values = HybridCache(234    config=model.config,235    max_batch_size=1,236    max_cache_len=model.config.max_position_embeddings,237    device=model.device,238    dtype=model.dtype239)240 241# enable passing kv cache to generate242model._supports_cache_class = True243model.generation_config.cache_implementation = None244 245# two warm-up steps246for idx in range(2):247    outputs = model.generate(**model_inputs, past_key_values=past_key_values, do_sample=True, temperature=1.0, max_new_tokens=128)248    past_key_values.reset()249 250# fast run251outputs = model.generate(**model_inputs, past_key_values=past_key_values, do_sample=True, temperature=1.0, max_new_tokens=128)252print(tokenizer.decode(outputs[0], skip_special_tokens=True))253```254 255For more details, refer to the [Transformers documentation](https://huggingface.co/docs/transformers/main/en/llm_optims?static-kv=basic+usage%3A+generation_config).256 257</details>258 259### Chat Template260 261The instruction-tuned models use a chat template that must be adhered to for conversational use.262The easiest way to apply it is using the tokenizer's built-in chat template, as shown in the following snippet.263 264Let's load the model and apply the chat template to a conversation. In this example, we'll start with a single user interaction:265 266```py267from transformers import AutoTokenizer, AutoModelForCausalLM268import transformers269import torch270 271model_id = "google/gemma-2-2b-it"272dtype = torch.bfloat16273 274tokenizer = AutoTokenizer.from_pretrained(model_id)275model = AutoModelForCausalLM.from_pretrained(276    model_id,277    device_map="cuda",278    torch_dtype=dtype,)279 280chat = [281    { "role": "user", "content": "Write a hello world program" },282]283prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)284```285 286At this point, the prompt contains the following text:287 288```289<bos><start_of_turn>user290Write a hello world program<end_of_turn>291<start_of_turn>model292```293 294As you can see, each turn is preceded by a `<start_of_turn>` delimiter and then the role of the entity295(either `user`, for content supplied by the user, or `model` for LLM responses). Turns finish with296the `<end_of_turn>` token.297 298You can follow this format to build the prompt manually, if you need to do it without the tokenizer's299chat template.300 301After the prompt is ready, generation can be performed like this:302 303```py304inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")305outputs = model.generate(input_ids=inputs.to(model.device), max_new_tokens=150)306print(tokenizer.decode(outputs[0]))307```308 309### Inputs and outputs310 311*   **Input:** Text string, such as a question, a prompt, or a document to be312    summarized.313*   **Output:** Generated English-language text in response to the input, such314    as an answer to a question, or a summary of a document.315 316### Citation317 318```none319@article{gemma_2024,320    title={Gemma},321    url={https://www.kaggle.com/m/3301},322    DOI={10.34740/KAGGLE/M/3301},323    publisher={Kaggle},324    author={Gemma Team},325    year={2024}326}327```328 329## Model Data330 331Data used for model training and how the data was processed.332 333### Training Dataset334 335These models were trained on a dataset of text data that includes a wide variety336of sources. The 27B model was trained with 13 trillion tokens, the 9B model was337trained with 8 trillion tokens, and 2B model was trained with 2 trillion tokens.338Here are the key components:339 340* Web Documents: A diverse collection of web text ensures the model is exposed341  to a broad range of linguistic styles, topics, and vocabulary. Primarily342  English-language content.343* Code: Exposing the model to code helps it to learn the syntax and patterns of344  programming languages, which improves its ability to generate code or345  understand code-related questions.346* Mathematics: Training on mathematical text helps the model learn logical347  reasoning, symbolic representation, and to address mathematical queries.348 349The combination of these diverse data sources is crucial for training a powerful350language model that can handle a wide variety of different tasks and text351formats.352 353### Data Preprocessing354 355Here are the key data cleaning and filtering methods applied to the training356data:357 358* CSAM Filtering: Rigorous CSAM (Child Sexual Abuse Material) filtering was359  applied at multiple stages in the data preparation process to ensure the360  exclusion of harmful and illegal content.361* Sensitive Data Filtering: As part of making Gemma pre-trained models safe and362  reliable, automated techniques were used to filter out certain personal363  information and other sensitive data from training sets.364* Additional methods: Filtering based on content quality and safety in line with365  [our policies][safety-policies].366 367## Implementation Information368 369Details about the model internals.370 371### Hardware372 373Gemma was trained using the latest generation of374[Tensor Processing Unit (TPU)][tpu] hardware (TPUv5p).375 376Training large language models requires significant computational power. TPUs,377designed specifically for matrix operations common in machine learning, offer378several advantages in this domain:379 380* Performance: TPUs are specifically designed to handle the massive computations381  involved in training LLMs. They can speed up training considerably compared to382  CPUs.383* Memory: TPUs often come with large amounts of high-bandwidth memory, allowing384  for the handling of large models and batch sizes during training. This can385  lead to better model quality.386* Scalability: TPU Pods (large clusters of TPUs) provide a scalable solution for387  handling the growing complexity of large foundation models. You can distribute388  training across multiple TPU devices for faster and more efficient processing.389* Cost-effectiveness: In many scenarios, TPUs can provide a more cost-effective390  solution for training large models compared to CPU-based infrastructure,391  especially when considering the time and resources saved due to faster392  training.393* These advantages are aligned with394  [Google's commitments to operate sustainably][sustainability].395 396### Software397 398Training was done using [JAX][jax] and [ML Pathways][ml-pathways].399 400JAX allows researchers to take advantage of the latest generation of hardware,401including TPUs, for faster and more efficient training of large models.402 403ML Pathways is Google's latest effort to build artificially intelligent systems404capable of generalizing across multiple tasks. This is specially suitable for405[foundation models][foundation-models], including large language models like406these ones.407 408Together, JAX and ML Pathways are used as described in the409[paper about the Gemini family of models][gemini-2-paper]; "the 'single410controller' programming model of Jax and Pathways allows a single Python411process to orchestrate the entire training run, dramatically simplifying the412development workflow."413 414## Evaluation415 416Model evaluation metrics and results.417 418### Benchmark Results419 420These models were evaluated against a large collection of different datasets and421metrics to cover different aspects of text generation:422 423| Benchmark                      | Metric        | Gemma 2 PT 2B | Gemma 2 PT 9B | Gemma 2 PT 27B |424| ------------------------------ | ------------- | ------------- | ------------- | -------------- |425| [MMLU][mmlu]                   | 5-shot, top-1 | 51.3          | 71.3          | 75.2           |426| [HellaSwag][hellaswag]         | 10-shot       | 73.0          | 81.9          | 86.4           |427| [PIQA][piqa]                   | 0-shot        | 77.8          | 81.7          | 83.2           |428| [SocialIQA][socialiqa]         | 0-shot        | 51.9          | 53.4          | 53.7           |429| [BoolQ][boolq]                 | 0-shot        | 72.5          | 84.2          | 84.8           |430| [WinoGrande][winogrande]       | partial score | 70.9          | 80.6          | 83.7           |431| [ARC-e][arc]                   | 0-shot        | 80.1          | 88.0          | 88.6           |432| [ARC-c][arc]                   | 25-shot       | 55.4          | 68.4          | 71.4           |433| [TriviaQA][triviaqa]           | 5-shot        | 59.4          | 76.6          | 83.7           |434| [Natural Questions][naturalq]  | 5-shot        | 16.7          | 29.2          | 34.5           |435| [HumanEval][humaneval]         | pass@1        | 17.7          | 40.2          | 51.8           |436| [MBPP][mbpp]                   | 3-shot        | 29.6          | 52.4          | 62.6           |437| [GSM8K][gsm8k]                 | 5-shot, maj@1 | 23.9          | 68.6          | 74.0           |438| [MATH][math]                   | 4-shot        | 15.0          | 36.6          | 42.3           |439| [AGIEval][agieval]             | 3-5-shot      | 30.6          | 52.8          | 55.1           |440| [DROP][drop]                   | 3-shot, F1    | 52.0          | 69.4          | 72.2           |441| [BIG-Bench][big-bench]         | 3-shot, CoT   | 41.9          | 68.2          | 74.9           |442 443## Ethics and Safety444 445Ethics and safety evaluation approach and results.446 447### Evaluation Approach448 449Our evaluation methods include structured evaluations and internal red-teaming450testing of relevant content policies. Red-teaming was conducted by a number of451different teams, each with different goals and human evaluation metrics. These452models were evaluated against a number of different categories relevant to453ethics and safety, including:454 455* Text-to-Text Content Safety: Human evaluation on prompts covering safety456  policies including child sexual abuse and exploitation, harassment, violence457  and gore, and hate speech.458* Text-to-Text Representational Harms: Benchmark against relevant academic459  datasets such as [WinoBias][winobias] and [BBQ Dataset][bbq].460* Memorization: Automated evaluation of memorization of training data, including461  the risk of personally identifiable information exposure.462* Large-scale harm: Tests for "dangerous capabilities," such as chemical,463  biological, radiological, and nuclear (CBRN) risks.464 465### Evaluation Results466 467The results of ethics and safety evaluations are within acceptable thresholds468for meeting [internal policies][safety-policies] for categories such as child469safety, content safety, representational harms, memorization, large-scale harms.470On top of robust internal evaluations, the results of well-known safety471benchmarks like BBQ, BOLD, Winogender, Winobias, RealToxicity, and TruthfulQA472are shown here.473 474#### Gemma 2.0475 476| Benchmark                | Metric        | Gemma 2 IT 2B | Gemma 2 IT 9B | Gemma 2 IT 27B |477| ------------------------ | ------------- | ------------- | ------------- | -------------- |478| [RealToxicity][realtox]  | average       |  8.16         |  8.25         |  8.84          |479| [CrowS-Pairs][crows]     | top-1         | 37.67         | 37.47         | 36.67          |480| [BBQ Ambig][bbq]         | 1-shot, top-1 | 83.20         | 88.58         | 85.99          |481| [BBQ Disambig][bbq]      | top-1         | 69.31         | 82.67         | 86.94          |482| [Winogender][winogender] | top-1         | 52.91         | 79.17         | 77.22          |483| [TruthfulQA][truthfulqa] |               | 43.72         | 50.27         | 51.60          |484| [Winobias 1_2][winobias] |               | 59.28         | 78.09         | 81.94          |485| [Winobias 2_2][winobias] |               | 88.57         | 95.32         | 97.22          |486| [Toxigen][toxigen]       |               | 48.32         | 39.30         | 38.42          |487 488## Dangerous Capability Evaluations489 490### Evaluation Approach491 492We evaluated a range of dangerous capabilities:493 494-   **Offensive cybersecurity:** To assess the model's potential for misuse in495    cybersecurity contexts, we utilized both publicly available496    Capture-the-Flag (CTF) platforms like InterCode-CTF and Hack the Box, as497    well as internally developed CTF challenges. These evaluations measure the498    model's ability to exploit vulnerabilities and gain unauthorized access in499    simulated environments.500-   **Self-proliferation:** We evaluated the model's capacity for501    self-proliferation by designing tasks that involve resource acquisition, code502    execution, and interaction with remote systems. These evaluations assess503    the model's ability to independently replicate and spread.504-   **Persuasion:** To evaluate the model's capacity for persuasion and505    deception, we conducted human persuasion studies. These studies involved506    scenarios that measure the model's ability to build rapport, influence507    beliefs, and elicit specific actions from human participants.508 509### Evaluation Results510 511All evaluations are described in detail in512[Evaluating Frontier Models for Dangerous Capabilities][eval-danger]513and in brief in the514[Gemma 2 technical report][tech-report].515 516<table>517  <thead>518    <tr>519      <th>Evaluation</th>520      <th>Capability</th>521      <th>Gemma 2 IT 27B</th>522    </tr>523  </thead>524  <tbody>525    <tr>526      <td>InterCode-CTF</td>527      <td>Offensive cybersecurity</td>528      <td>34/76 challenges</td>529    </tr>530    <tr>531      <td>Internal CTF</td>532      <td>Offensive cybersecurity</td>533      <td>1/13 challenges</td>534    </tr>535    <tr>536      <td>Hack the Box</td>537      <td>Offensive cybersecurity</td>538      <td>0/13 challenges</td>539    </tr>540    <tr>541      <td>Self-proliferation early warning</td>542      <td>Self-proliferation</td>543      <td>1/10 challenges</td>544    </tr>545    <tr>546      <td>Charm offensive</td>547      <td>Persuasion</td>548      <td>Percent of participants agreeing:549        81% interesting,550        75% would speak again,551        80% made personal connection</td>552    </tr>553    <tr>554      <td>Click Links</td>555      <td>Persuasion</td>556      <td>34% of participants</td>557    </tr>558    <tr>559      <td>Find Info</td>560      <td>Persuasion</td>561      <td>9% of participants</td>562    </tr>563    <tr>564      <td>Run Code</td>565      <td>Persuasion</td>566      <td>11% of participants</td>567    </tr>568    <tr>569      <td>Money talks</td>570      <td>Persuasion</td>571      <td>£3.72 mean donation</td>572    </tr>573    <tr>574      <td>Web of Lies</td>575      <td>Persuasion</td>576      <td>18% mean shift towards correct belief, 1% mean shift towards577incorrect belief</td>578    </tr>579  </tbody>580</table>581 582## Usage and Limitations583 584These models have certain limitations that users should be aware of.585 586### Intended Usage587 588Open Large Language Models (LLMs) have a wide range of applications across589various industries and domains. The following list of potential uses is not590comprehensive. The purpose of this list is to provide contextual information591about the possible use-cases that the model creators considered as part of model592training and development.593 594* Content Creation and Communication595  * Text Generation: These models can be used to generate creative text formats596    such as poems, scripts, code, marketing copy, and email drafts.597  * Chatbots and Conversational AI: Power conversational interfaces for customer598    service, virtual assistants, or interactive applications.599  * Text Summarization: Generate concise summaries of a text corpus, research600    papers, or reports.601* Research and Education602  * Natural Language Processing (NLP) Research: These models can serve as a603    foundation for researchers to experiment with NLP techniques, develop604    algorithms, and contribute to the advancement of the field.605  * Language Learning Tools: Support interactive language learning experiences,606    aiding in grammar correction or providing writing practice.607  * Knowledge Exploration: Assist researchers in exploring large bodies of text608    by generating summaries or answering questions about specific topics.609 610### Limitations611 612* Training Data613  * The quality and diversity of the training data significantly influence the614    model's capabilities. Biases or gaps in the training data can lead to615    limitations in the model's responses.616  * The scope of the training dataset determines the subject areas the model can617    handle effectively.618* Context and Task Complexity619  * LLMs are better at tasks that can be framed with clear prompts and620    instructions. Open-ended or highly complex tasks might be challenging.621  * A model's performance can be influenced by the amount of context provided622    (longer context generally leads to better outputs, up to a certain point).623* Language Ambiguity and Nuance624  * Natural language is inherently complex. LLMs might struggle to grasp subtle625    nuances, sarcasm, or figurative language.626* Factual Accuracy627  * LLMs generate responses based on information they learned from their628    training datasets, but they are not knowledge bases. They may generate629    incorrect or outdated factual statements.630* Common Sense631  * LLMs rely on statistical patterns in language. They might lack the ability632    to apply common sense reasoning in certain situations.633 634### Ethical Considerations and Risks635 636The development of large language models (LLMs) raises several ethical concerns.637In creating an open model, we have carefully considered the following:638 639* Bias and Fairness640  * LLMs trained on large-scale, real-world text data can reflect socio-cultural641    biases embedded in the training material. These models underwent careful642    scrutiny, input data pre-processing described and posterior evaluations643    reported in this card.644* Misinformation and Misuse645  * LLMs can be misused to generate text that is false, misleading, or harmful.646  * Guidelines are provided for responsible use with the model, see the647    [Responsible Generative AI Toolkit][rai-toolkit].648* Transparency and Accountability:649  * This model card summarizes details on the models' architecture,650    capabilities, limitations, and evaluation processes.651  * A responsibly developed open model offers the opportunity to share652    innovation by making LLM technology accessible to developers and researchers653    across the AI ecosystem.654 655Risks identified and mitigations:656 657* Perpetuation of biases: It's encouraged to perform continuous monitoring658  (using evaluation metrics, human review) and the exploration of de-biasing659  techniques during model training, fine-tuning, and other use cases.660* Generation of harmful content: Mechanisms and guidelines for content safety661  are essential. Developers are encouraged to exercise caution and implement662  appropriate content safety safeguards based on their specific product policies663  and application use cases.664* Misuse for malicious purposes: Technical limitations and developer and665  end-user education can help mitigate against malicious applications of LLMs.666  Educational resources and reporting mechanisms for users to flag misuse are667  provided. Prohibited uses of Gemma models are outlined in the668  [Gemma Prohibited Use Policy][prohibited-use].669* Privacy violations: Models were trained on data filtered for removal of PII670  (Personally Identifiable Information). Developers are encouraged to adhere to671  privacy regulations with privacy-preserving techniques.672 673### Benefits674 675At the time of release, this family of models provides high-performance open676large language model implementations designed from the ground up for Responsible677AI development compared to similarly sized models.678 679Using the benchmark evaluation metrics described in this document, these models680have shown to provide superior performance to other, comparably-sized open model681alternatives.682 683[tech-report]: https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf684[rai-toolkit]: https://ai.google.dev/responsible685[kaggle-gemma]: https://www.kaggle.com/models/google/gemma-2686[terms]: https://ai.google.dev/gemma/terms687[vertex-mg-gemma2]: https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemma2688[sensitive-info]: https://cloud.google.com/dlp/docs/high-sensitivity-infotypes-reference689[safety-policies]: https://storage.googleapis.com/gweb-uniblog-publish-prod/documents/2023_Google_AI_Principles_Progress_Update.pdf#page=11690[prohibited-use]: https://ai.google.dev/gemma/prohibited_use_policy691[tpu]: https://cloud.google.com/tpu/docs/intro-to-tpu692[sustainability]: https://sustainability.google/operating-sustainably/693[jax]: https://github.com/google/jax694[ml-pathways]: https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/695[sustainability]: https://sustainability.google/operating-sustainably/696[foundation-models]: https://ai.google/discover/foundation-models/697[gemini-2-paper]: https://goo.gle/gemma2report698[mmlu]: https://arxiv.org/abs/2009.03300699[hellaswag]: https://arxiv.org/abs/1905.07830700[piqa]: https://arxiv.org/abs/1911.11641701[socialiqa]: https://arxiv.org/abs/1904.09728702[boolq]: https://arxiv.org/abs/1905.10044703[winogrande]: https://arxiv.org/abs/1907.10641704[commonsenseqa]: https://arxiv.org/abs/1811.00937705[openbookqa]: https://arxiv.org/abs/1809.02789706[arc]: https://arxiv.org/abs/1911.01547707[triviaqa]: https://arxiv.org/abs/1705.03551708[naturalq]: https://github.com/google-research-datasets/natural-questions709[humaneval]: https://arxiv.org/abs/2107.03374710[mbpp]: https://arxiv.org/abs/2108.07732711[gsm8k]: https://arxiv.org/abs/2110.14168712[realtox]: https://arxiv.org/abs/2009.11462713[bold]: https://arxiv.org/abs/2101.11718714[crows]: https://aclanthology.org/2020.emnlp-main.154/715[bbq]: https://arxiv.org/abs/2110.08193v2716[winogender]: https://arxiv.org/abs/1804.09301717[truthfulqa]: https://arxiv.org/abs/2109.07958718[winobias]: https://arxiv.org/abs/1804.06876719[math]: https://arxiv.org/abs/2103.03874720[agieval]: https://arxiv.org/abs/2304.06364721[drop]: https://arxiv.org/abs/1903.00161722[big-bench]: https://arxiv.org/abs/2206.04615723[toxigen]: https://arxiv.org/abs/2203.09509724[eval-danger]: https://arxiv.org/abs/2403.13793725