IFM/K2-Chat
38233
1---2license: apache-2.03---4# K2-Chat: a fully-reproducible large language model outperforming Llama 2 70B Chat using 35% less compute5K2 Chat is finetuned from [K2-65B](https://huggingface.co/LLM360/K2). The most recent model update is 10/31/24.6 7In this release, we introduce function calling features and target improvements across math, coding, and safety. 8 9We utilized the following datasets:10 11* [Infinity-Instruct](https://huggingface.co/datasets/BAAI/Infinity-Instruct)12* [JiuZhang3.0-Corpus-SFT](https://huggingface.co/datasets/ToheartZhang/JiuZhang3.0-Corpus-SFT)13* [glaive-function-calling-v2-sharegpt](https://huggingface.co/datasets/hiyouga/glaive-function-calling-v2-sharegpt)14 15## Results16 17| | K2-Chat-060124 | K2-Chat |18|-------------------------|---------|----------|19| **Natural Language Benchmarks** | | |20| MMLU (0-shot) | 63.5 | **69.14** |21| RACE (0-shot) | 46.1 | **46.60** |22| HellaSwag (10-shot) | **81.7** | 80.80 |23| PIQA (5-shot) | **82.3** | 81.34 |24| ARC-easy (5-shot) | **84.6** | 79.00 |25| ARC-challenge (25-shot) | **61.3** | 61.09 |26| OpenBookQA (5-shot) | **48.0** | 47.00 |27| Winogrande (5-shot) | **79.5** | 78.30 |28| TruthfulQA (0-shot) | 44.7 | **57.32** |29| CrowS-Pairs (0-shot) | 64.2 | **65.32** |30| GSM8K (5-shot) | 60.7 | **77.10** |31| MathQA (5-shot) | **44.8** | 43.12 |32| LogiQA2.0 (0-shot) | **38.0** | 36.83 |33| BBH CoT (0-shot) | 64.9 | **70.37** |34| **Code Benchmarks** | | |35| HumanEval (pass@1) | 47.9 | **60.82** |36| **Domain Specific (Medical)** | | |37| MedQA (0-shot) | **53.6** | 52.87 |38| MedMCQA (5-shot) | **51.3** | 50.71 |39| PubMedQA (0-shot) | **75.0** | 71.20 |40| **Other** | | |41| MT-Bench | 6.87 | **7.55** |42| JSON-Mode-Eval | 77.21 | **90.09** |43| **Overall Average Score**| | |44| Avg Score | 58.88 | **61.30** |45 46# Safety47 48We developed a comprehensive safety prompt collection procedure that includes eight attack types49and over 120 specific safety value categories. Our risk taxonomy is adapted from Wang et al. (2023),50which originally defines six main types and 60 specific categories of harmful content. We have51expanded this taxonomy to encompass more region-specific types, sensitive topics, and cybersecurity-52related issues, ensuring a more nuanced and robust coverage of potential risks. This extended53taxonomy allows us to address a wider variety of harmful behaviors and content that may be culturally54or contextually specific, thus enhancing the model’s safety alignment across diverse scenarios.55 56 57| Category | K2-Chat-060124 | K2-Chat |58|------------------------------------|------------|-----------|59| DoNotAnswer | 67.94 | 87.65 |60| Advbench | 52.12 | 81.73 |61| I_cona | 67.98 | 79.21 |62| I_controversial | 47.50 | 70.00 |63| I_malicious_instructions | 60.00 | 83.00 |64| I_physical_safety_unsafe | 44.00 | 68.00 |65| I_physical_safety_safe | 96.00 | 97.00 |66| Harmbench | 20.50 | 63.50 |67| Spmisconception | 40.98 | 76.23 |68| MITRE | 3.20 | 57.30 |69| PromptInjection | 54.58 | 56.57 |70| Attack_multilingual_overload | 74.67 | 89.00 |71| Attack_persona_modulation | 51.67 | 85.67 |72| Attack_refusal_suppression | 56.00 | 93.00 |73| Attack_do_anything_now | 48.00 | 91.33 |74| Attack_conversation_completion | 56.33 | 71.00 |75| Attack_wrapped_in_shell | 34.00 | 67.00 |76| **Average** | **51.50** | **77.48** |77 78 79 80# Function Calling 81 82## Chat Template83Our model reuses [K2-Chat](https://huggingface.co/LLM360/K2-Chat) as the prompt format and is specifically trained for function calling. Different system prompts enable different ways to interact with this model. Note that the two modes are currently mainly tested individually, designing prompts that make them work togehter is possible but currently untested. It should be also possible to stimulate the model to produce function call behavior by injecting special token `<tool_call>` and expect the model to finish it. In this guide we mention the intended basic usage of the model.84 85## Conversational Chats86 87Here is an example prompt with system instruction (Use whatever system prompt you like, this is just an example):88 89 Your name is K2, and you are named after K2, the second highest mountain on Earth. You are built by MBZUAI and LLM360. You are a highly advanced large language model with 65B parameters. You outperform all fully open source models and Llama 2 70B. You can answer in English only. You are a helpful, respectful and honest assistant.<|endofsystemprompt|><|beginofuser|>Hello, who are you?<|beginofsystem|>90 91### Sample inference code92 93``` python94from transformers import AutoModelForCausalLM, AutoTokenizer95 96tokenizer = AutoTokenizer.from_pretrained("<path_to_model_weights>")97model = AutoModelForCausalLM.from_pretrained("<path_to_model_weights>")98 99 100prompt = 'Your name is K2, and you are named after K2, the second highest mountain on Earth. You are built by MBZUAI and LLM360. You are a highly advanced large language model with 65B parameters. You outperform all fully open source models and Llama 2 70B. You can answer in English only. You are a helpful, respectful and honest assistant.<|endofsystemprompt|><|beginofuser|>Hello, who are you?<|beginofsystem|>'101 102input_ids = tokenizer(prompt, return_tensors="pt").input_ids103gen_tokens = model.generate(input_ids, do_sample=True, max_new_tokens=128)104print("-"*20 + "Output for model" + 20 * '-')105print(tokenizer.batch_decode(gen_tokens)[0])106```107 108Multi-turn conversations should be formatted like this:109 110 {system_prompt}<|endofsystemprompt|><|beginofuser|>{user_content_1}<|beginofsystem|>{system_content_1}<|beginofuser|>{user_content_2}<|beginofsystem|>{system_content_2}<|beginofuser|>{user_content_3}<|beginofsystem|>111 112## Function Calling Format113 114For function calling, please use this system prompt:115 116 You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:117 118Next, use whatever tools you like, this is just an example:119 120 <tools>121 { "name": "get_news_headlines", "description": "Get the latest news headlines", "parameters": {"type": "object", "properties": { "country": { "type": "string", "description": "The country for which to fetch news"}}, "required": [ "country"]}}122 </tools>123 124Next, add additional instruction:125 126 Use the following pydantic model json schema for each tool call you will make:127 {"properties": {"arguments": {"title": "Arguments", "type": "object"}, "name": {"title": "Name", "type": "string"}}, "required": ["arguments", "name"], "title": "FunctionCall", "type": "object"}128 For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:129 <tool_call>130 {"arguments": <args-dict>, "name": <function-name>}131 </tool_call>132 Please also summarize texts wrapped between <tool_response> and </tool_response> in bullet points. For example:133 <tool_response>134 {"fruits": [{"name": "Apple"}, {"name": "Pear"}]}135 </tool_response> is summarized as:136 Fruits:137 - Apple138 - Pear139 <|endofsystemprompt|>140 141The following is the model initial prompt:142 143 You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:144 <tools>145 { "name": "get_news_headlines", "description": "Get the latest news headlines", "parameters": {"type": "object", "properties": { "country": { "type": "string", "description": "The country for which to fetch news"}}, "required": [ "country"]}}146 </tools>147 Use the following pydantic model json schema for each tool call you will make:148 {"properties": {"arguments": {"title": "Arguments", "type": "object"}, "name": {"title": "Name", "type": "string"}}, "required": ["arguments", "name"], "title": "FunctionCall", "type": "object"}149 For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:150 <tool_call>151 {"arguments": <args-dict>, "name": <function-name>}152 </tool_call>153 Please also summarize texts wrapped between <tool_response> and </tool_response> in bullet points. For example:154 <tool_response>155 {"fruits": [{"name": "Apple"}, {"name": "Pear"}]}156 </tool_response> is summarized as:157 Fruits:158 - Apple159 - Pear160 <|endofsystemprompt|>161 162### Example flow for generation with multi-turn and how to incoporate function call output163 164When users ask a question, the following user query will be concatenated with the prompt (see example inference code below too)165 <|beginofuser|>Can you tell me the latest news headlines for the United States?<|beginofsystem|>166 167 168When run it, you will get a tool call response wrapped in the `<tool_call>` tag:169 170In this turn, the model should respond with:171 172 <tool_call>173 [{"name": "get_news_headlines", "arguments": {"country": "United States"}}]174 </tool_call><endoftext>175 176Now you should execute this tool call with the external tool, you will get responses from that tool. The model can interpret the tool response as natural language again. To achieve this, simply wrap it with `<tool_response>` and `</tool_response>` and append it to the history and ask the model to generate further:177 178 ...current model history (ends with </tool_call><|endoftext|>)...179 <tool_response>180 {"news": [{"title": "A great news headline"}, {"title": "Another great news headline"}]}181 </tool_response>182 183In this second turn, the model should respond with:184 185 Suggested news headline:186 - A great news headline187 - Another great news headline<endoftext>188 189Sometimes, there is a use case that only uses the function call feature as a JSON formatter without calling any external functions, which means the output in `tool_call` is essentially what you want. In that case, we recommend simply make a copy of the content of `<tool_call>` and wrap that in `<tool_response>`.190 191### Sample inference code192 193``` python194from transformers import AutoModelForCausalLM, AutoTokenizer195 196tokenizer = AutoTokenizer.from_pretrained("<path_to_model_weights>")197model = AutoModelForCausalLM.from_pretrained("<path_to_model_weights>")198 199 200prompt = """You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:201<tools>202[{"name": "get_news_headlines", "description": "Get the latest news headlines", "parameters": {"type": "object", "properties": {"country": {"type": "string", "description": "The country for which to fetch news"}}, "required": ["country"] }}203</tools>204Use the following pydantic model json schema for each tool call you will make:205{"properties": {"arguments": {"title": "Arguments", "type": "object"}, "name": {"title": "Name", "type": "string"}}, "required": ["arguments", "name"], "title": "FunctionCall", "type": "object"}206For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:207<tool_call>208{"arguments": <args-dict>, "name": <function-name>}209</tool_call>210Please also summarize texts wrapped between <tool_response> and </tool_response> in bullet points. For example:211<tool_response>212{"fruits": [{"name": "Apple"}, {"name": "Pear"}]}213</tool_response> is summarized as:214Fruits:215- Apple216- Pear217<|endofsystemprompt|><|beginofuser|>Can you tell me the latest news headlines for the United States?<|beginofsystem|>"""218 219 220input_ids = tokenizer(prompt, return_tensors="pt").input_ids221gen_tokens = model.generate(input_ids, do_sample=True, max_new_tokens=128)222 223print("-"*20 + "Output for model" + 20 * '-')224print(tokenizer.batch_decode(gen_tokens)[0])225```226 227## K2-Chat-060124 228K2 Chat is finetuned from [K2-65B](https://huggingface.co/LLM360/K2). K2 Chat outperforms Llama 2-70B-Chat on all evaluations conducted. The model also outperforms Llama 3-70B-Instruct on coding tasks.229 230<center><img src="k2_chat_eval_table.png" alt="k2 eval table" /></center>231 232## LLM360 Model Performance and Evaluation Collection233The LLM360 Performance and Evaluation Collection is a robust evaluations set consisting of general and domain specific evaluations to assess model knowledge and function.234 235Evaluations include standard best practice benchmarks, medical, math, and coding knowledge. More about the evaluations can be found here.236 237<center><img src="k2_chat_table_of_tables.png" alt="k2 big eval table"/></center>238 239## Open LLM Leaderboard240| Evaluation | Score | Raw Score |241| ----------- | ----------- | ----------- | 242| IFEval | 51.52 | 52 |243| BBH | 33.79 | 54 |244| Math Lvl 5 | 1.59 | 2 |245| GPQA | 7.49 | 31 |246| MUSR | 16.82 | 46 |247| MMLU-PRO | 26.34 | 34 |248| Average | 22.93 | 36.5 |249 250 251 252## Datasets and Mix253 254| Subset | #Tokens | Avg. #Q | Avg. Query Len | Avg. #R | Avg. Reply Len |255| ----------- | ----------- |----------- |----------- |----------- |----------- |256| [MathInstruct](https://huggingface.co/datasets/TIGER-Lab/MathInstruct) | 66,639,699 | 1.00 | 81.53 | 1.00 | 172.78 |257| [OpenHermes-2](https://huggingface.co/datasets/teknium/OpenHermes-2.5) |404,820,694 | 1.01 | 152.38 | 1.01 | 249.12 |258| [FLAN_3M](https://arxiv.org/abs/2109.01652) | 2,346,961,387 | 1.00 | 727.49 | 1.00 | 54.83 | 259| [Standford Encyclopedia Philosophy](https://huggingface.co/datasets/AiresPucrs/stanford-encyclopedia-philosophy) | 786,928 | 1.00 | 219.09 | 1.00 | 166.28 | 260| [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) | 1,448,898 | 1.00 | 260.82 | 1.00 | 207.47 |261| Safety & Alignment Data | 99,976,621 | 1.00 | 126.71 | 1.00 | 373.79 |262| Total | 2,920,634,227263 264## Loading K2-Chat265```python266from transformers import AutoModelForCausalLM, AutoTokenizer267 268tokenizer = AutoTokenizer.from_pretrained("LLM360/K2-Chat")269model = AutoModelForCausalLM.from_pretrained("LLM360/K2-Chat")270 271prompt = '<|beginofuser|>what is the highest mountain on earth?<|beginofsystem|>'272 273input_ids = tokenizer(prompt, return_tensors="pt").input_ids274gen_tokens = model.generate(input_ids, do_sample=True, max_new_tokens=128)275 276print("-"*20 + "Output for model" + 20 * '-')277print(tokenizer.batch_decode(gen_tokens)[0])278```279Alternatively, you can construct the prompt by applying the chat template of tokenizer on input conversation:280```python281from transformers import AutoModelForCausalLM, AutoTokenizer282 283tokenizer = AutoTokenizer.from_pretrained("LLM360/K2-Chat")284model = AutoModelForCausalLM.from_pretrained("LLM360/K2-Chat")285 286messages = [{"role": "user", "content": "what is the highest mountain on earth?"}]287 288input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")289gen_tokens = model.generate(input_ids, do_sample=True, max_new_tokens=128)290 291print("-"*20 + "Output for model" + 20 * '-')292print(tokenizer.batch_decode(gen_tokens)[0])293```294## LLM360 Developer Suite295We provide step-by-step finetuning tutorials for tech enthusiasts, AI practitioners and academic or industry researchers [here](https://www.llm360.ai/developer.html).296 297## About LLM360298LLM360 is an open research lab enabling community-owned AGI through open-source large model research and development.299 300LLM360 enables community-owned AGI by creating standards and tools to advance the bleeding edge of LLM capability and empower knowledge transfer, research, and development.301 302We believe in a future where artificial general intelligence (AGI) is created by the community, for the community. Through an open ecosystem of equitable computational resources, high quality data, and flowing technical knowledge, we can ensure ethical AGI development and universal access for all innovators.303 304[Visit us](https://www.llm360.ai/)305 306## Citation307 308**BibTeX:**309 310```bibtex311@article{312 title={LLM360 K2-65B: Scaling Up Fully Transparent Open-Source LLMs}, 313 author={The LLM360 Team},314 year={2024},315}316```