jdopensource/JoyAI-LLM-Flash-Base

1---2language:3- zh4- en5pipeline_tag: text-generation6---7<div align="center">8  <picture>9      <img src="figures/joyai-logo.png" width="30%" alt="JoyAI-LLM Flash-Base">10  </picture>11</div>12<hr>13 14 15 16<div align="center" style="line-height: 1;">17  <a href="https://huggingface.co/jdopensource" target="_blank"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-JD-ffc107?color=ffc107&logoColor=white"/></a>18  <a href="https://huggingface.co/jdopensource/JoyAI-LLM-Flash-Base/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Modified_MIT-f5de53?&color=f5de53"/></a>19</div>20 21 22 23 24## 1. Model Introduction25 26JoyAI-LLM Flash-Base is a state-of-the-art mixture-of-experts (MoE) language model with 3 billion activated parameters and 48 billion total parameters. Trained with the Muon optimizer, JoyAI Flash-base achieves exceptional performance across frontier knowledge, reasoning, and coding tasks while being meticulously optimized for agentic capabilities. JoyAI-LLM Flash series aim to accelarate high-throughput, latency-sensitive applications where cost per query must remain minimal.27 28### Key Features29 30- Training-Inference Collaboration: apply Muon optimizer with dense MTP, develop novel optimization techniques to resolve instabilities while scaling up, delivering 1.3× to 1.7× the throughput of the non-MTP version.31- Agentic Intelligence: Specifically designed for tool use, reasoning, and autonomous problem-solving.32 33## 2. Model Summary34 35|                                             |                          |36| :-----------------------------------------: | :----------------------: |37|              **Architecture**               | Mixture-of-Experts (MoE) |38|            **Total Parameters**             |           48B            |39|          **Activated Parameters**           |            3B            |40| **Number of Layers** (Dense layer included) |            40            |41|         **Number of Dense Layers**          |            1             |42|       **Attention Hidden Dimension**        |           2048           |43|    **MoE Hidden Dimension** (per Expert)    |           768            |44|        **Number of Attention Heads**        |            32            |45|            **Number of Experts**            |           256            |46|       **Selected Experts per Token**        |            8             |47|        **Number of Shared Experts**         |            1             |48|             **Vocabulary Size**             |           129K           |49|             **Context Length**              |           128K           |50|           **Attention Mechanism**           |           MLA            |51|           **Activation Function**           |          SwiGLU          |52|                   </div>                    |                          |53 54## 3. Evaluation Results55 56 57<table>58<thead>59<tr>60<th align="center">Benchmark</th>61<th align="center"><sup>JoyAI-LLM Flash-base</sup></th>62<th align="center"><sup>Qwen3-30B-A3B-base</sup></th>63</tr>64</thead>65<tbody>66 67 68<tr>69<td align="center" style="vertical-align: middle">MMLU</td>70<td align="center" style="vertical-align: middle"><strong>84.70</strong></td>71<td align="center" style="vertical-align: middle">82.12</td>72</tr>73<tr>74<td align="center" style="vertical-align: middle">MMLU-Pro</td>75<td align="center" style="vertical-align: middle"><strong>73.14</strong></td>76<td align="center" style="vertical-align: middle">61.76</td>77</tr>78<tr>79<td align="center" style="vertical-align: middle">CMMLU</td>80<td align="center" style="vertical-align: middle">83.09</td>81<td align="center" style="vertical-align: middle"><strong>83.60</strong></td>82</tr>83<tr>84</tr>85 86 87<tr>88<td align="center" style="vertical-align: middle">HumanEval</td>89<td align="center" style="vertical-align: middle">85.37</td>90<td align="center" style="vertical-align: middle"><strong>87.80</strong></td>91</tr>92<tr>93<td align="center" style="vertical-align: middle">LiveCodeBench</td>94<td align="center" style="vertical-align: middle"><strong>39.91</strong></td>95<td align="center" style="vertical-align: middle">37.34</td>96</tr>97<tr></tr>98 99<tr>100<td align="center" style="vertical-align: middle">GSM8K</td>101<td align="center" style="vertical-align: middle">88.78</td>102<td align="center" style="vertical-align: middle"><strong>90.37</strong></td>103</tr>104<tr>105</tr>106<tr>107<td align="center" style="vertical-align: middle">MATH</td>108<td align="center" style="vertical-align: middle"><strong>78.16</strong></td>109<td align="center" style="vertical-align: middle">59.60</td>110</tr>111<tr>112<td align="center" style="vertical-align: middle">MATH 500</td>113<td align="center" style="vertical-align: middle"><strong>77.00</strong></td>114<td align="center" style="vertical-align: middle">58.00</td>115</tr>116 117</tbody>118</table>119 120 121 122## 4. License123 124Both the code repository and the model weights are released under the [Modified MIT License](LICENSE).