Modelpublic

rinna/gemma-2-baku-2b

sourceHugging Facegemmaupdated 2y agoView on Hugging Face

7likes168downloads

Model Card

`Gemma 2 Baku 2B (rinna/gemma-2-baku-2b)`

[image]

Overview

We conduct continual pre-training of google/gemma-2-2b on 80B tokens from a mixture of Japanese and English datasets. The continual pre-training improves the model's performance on Japanese tasks.

The name baku comes from the Japanese word `獏/ばく/Baku`, which is a kind of Japanese mythical creature (`妖怪/ようかい/Youkai`).

Size	Continual Pre-Training	Instruction-Tuning
2B	Gemma 2 Baku 2B [[HF]](https://huggingface.co/rinna/gemma-2-baku-2b)	Gemma 2 Baku 2B Instruct [[HF]](https://huggingface.co/rinna/gemma-2-baku-2b-it)

—Library

The model was trained using code based on Lightning-AI/litgpt.

—Model architecture

A 26-layer, 2304-hidden-size transformer-based language model. Please refer to the Gemma 2 Model Card for detailed information on the model's architecture.

—Training

The model was initialized with the google/gemma-2-2b model and continually trained on around 80B tokens from a mixture of the following corpora

—Japanese CC-100
—Japanese C4
—Japanese OSCAR
—The Pile
—Wikipedia
—rinna curated Japanese dataset

—Contributors
—Toshiaki Wakatsuki
—Xinqi Chen
—Kei Sawada

—Release date

October 3, 2024

Benchmarking

Please refer to rinna's LM benchmark page (Sheet 20241003).

How to use the model

~~~python import transformers import torch

modelid = "rinna/gemma-2-baku-2b" pipeline = transformers.pipeline( "text-generation", model=modelid, modelkwargs={"torchdtype": torch.bfloat16, "attnimplementation": "eager"}, devicemap="auto" ) output = pipeline( "西田幾多郎は、", maxnewtokens=256, dosample=True ) print(output[0]["generatedtext"]) ~~~

It is recommended to use eager attention when conducting batch inference under bfloat16 precision. Currently, Gemma 2 yields NaN values for input sequences with padding when the default attention mechanism (torch.scaleddotproduct_attention) is employed in conjunction with bfloat16.

Tokenization

The model uses the original google/gemma-2-2b tokenizer.

How to cite

bibtex

@misc{rinna-gemma-2-baku-2b,
    title = {rinna/gemma-2-baku-2b},
    author = {Wakatsuki, Toshiaki and Chen, Xinqi and Sawada, Kei},
    url = {https://huggingface.co/rinna/gemma-2-baku-2b}
}

@inproceedings{sawada2024release,
    title = {Release of Pre-Trained Models for the {J}apanese Language},
    author = {Sawada, Kei and Zhao, Tianyu and Shing, Makoto and Mitsui, Kentaro and Kaga, Akio and Hono, Yukiya and Wakatsuki, Toshiaki and Mitsuda, Koh},
    booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
    month = {5},
    year = {2024},
    pages = {13898--13905},
    url = {https://aclanthology.org/2024.lrec-main.1213},
    note = {\url{https://arxiv.org/abs/2404.01657}}
}

References

bibtex

@article{gemma-2-2024,
    title = {Gemma 2},
    url = {https://www.kaggle.com/models/google/gemma-2},
    publisher = {Kaggle},
    author = {Gemma Team},
    year = {2024}
}

@misc{litgpt-2023,
    author = {Lightning AI},
    title = {LitGPT},
    howpublished = {\url{https://github.com/Lightning-AI/litgpt}},
    year = {2023}
}

License

Gemma Terms of Use