SemanticAlignment/Mistral-v0.1-Italian-LAPT-instruct

1---2library_name: transformers3tags: []4---5 6# Mistral-7B-v0.1-Italian-LAPT-instruct7<div align="center">8 9<img src="https://github.com/Andrew-Wyn/images/blob/master/sava/italian_adapt-img.jpg?raw=true" width="400" height="400" style="border-radius:10%" />10 11</div>12 13The **Mistral-7B-v0.1-Adapted** collection of large language models (LLMs), is a collection of adapted generative models in 7B (text in/text out), adapted models from **Mistral-7B-Base-v0.1**.14 15*Mistral-v0.1-Italian-LAPT-instruct* is a continually trained and instruction tuned Mistral model.16 17**Model developer:** SapienzaNLP, ISTI-CNR, ILC-CNR18 19**Model Architecture:** Mistral-7B-v0.1-Adapted is an auto-regressive language model that uses an optimized transformer architecture.20 21## Data used for the adaptation22 23The **Mistral-7B-v0.1-Adapted** models are trained on a collection of Italian and English data extracted from [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX).24The data are extracted to be skewed toward Italian language with a ration of one over four. Extracting the first 9B tokens from Italian part of CulturaX and the first 3B tokens from English part of CulturaX.25 26## Data used for the instruction tuning (SFT)27 28The data used in the instruction following training procedure:29 30| Dataset | Language | Instances |31|------|-----|------|32| [TÜLU-v3](https://huggingface.co/datasets/allenai/tulu-3-sft-mixture) | EN  | 940,000 |33| [LIMA](https://huggingface.co/datasets/GAIR/lima) | IT/EN  | 2,000 |34| [WildChat-IT](https://huggingface.co/datasets/allenai/WildChat-1M) | IT  | 5,000 |35| [TowerBlocks-v0.2](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2) | IT/EN  | 7,276 |36| [GPT-4o-ITA-Instruct](https://huggingface.co/datasets/DeepMount00/GPT-4o-ITA-INSTRUCT) | IT | 15,000 |37| [Aya](https://huggingface.co/datasets/CohereLabs/aya_dataset) | IT | 700 |38 39The model is trained for two epoches in the aforementioned data.40 41## Evaluation42 43Adapted models are evaluated on [ITA-Bench](https://github.com/SapienzaNLP/ita-bench).44 45| Model | MMLU (5-shots) | ARC-C (5-shots) | Hellaswag (0-shots) | IFEval (inst_level) |46|------|-----|------|------|------|47| Llama-3.1-SAVA | 56.9 | 42.3 | 58.1 | 62.3 |48| Llama-3.1-LAPT | 58.5 | 47.9 | 62.4 | 67.3 |49| Mistral-0.1-SAVA | 51.5 | 41.6 | 57.5 | 61.7 |50| **Mistral-0.1-LAPT** | 52.9 | 39.9 | 58.4 | 60.0 |51| Llama-3.1-Original | 47.4 | 43.1 | 57.9 | 66.8 |52| Mistral-0.1-Original | 41.6 | 38.9 | 50.0 | 42.2 |53 54## Use with Transformers55 56You can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.57 58Make sure to update your transformers installation via `pip install --upgrade transformers`.59 60```python61import transformers62import torch63 64model_id = "SemanticAlignment/Mistral-v0.1-Italian-LAPT-instruct"65 66tokenizer = AutoTokenizer.from_pretrained(model_name)67 68generator = pipeline(69    "text-generation",70    model=model_name,71    device_map="auto",72    dtype=torch.bfloat1673)74 75conversations.append([76    {"role": "system", "content": "Sei un assistente utile, rispondi in modo conciso e coerente."},77    {"role": "user", "content": "Cosa si può fare in una bella giornata di sole?"},78])79 80chat_samples = tokenizer.apply_chat_template(conversations, tokenize=False)81 82# get number of prompt tokens83prompt_tokens_number = len(tokenizer(chat_samples)["input_ids"])84 85outputs = generator(86    conversations,87    max_new_tokens=2048,88    eos_token_id=[89        tokenizer.eos_token_id,90        tokenizer.convert_tokens_to_ids("<|eot_id|>"),91    ],92)93 94```95 96Code: https://github.com/SapienzaNLP/sava97 98## Aknowledgement99Thanks to Leonardo Colosi (colosi@diag.uniroma1.it) for helping in instruction tuning phase.100 101We acknowledge ISCRA for awarding this project access to the LEONARDO supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CINECA (Italy).102 103## Citation104 105If you use any part of this work, please consider citing the paper as follows:106 107```bibtex108@misc{moroni2025optimizingllmsitalianreducing,109      title={Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation}, 110      author={Luca Moroni and Giovanni Puccetti and Pere-Lluis Huguet Cabot and Andrei Stefan Bejgu and Edoardo Barba and Alessio Miaschi and Felice Dell'Orletta and Andrea Esuli and Roberto Navigli},111      year={2025},112      eprint={2504.17025},113      archivePrefix={arXiv},114      primaryClass={cs.CL},115      url={https://arxiv.org/abs/2504.17025}, 116}117```