pathcosmos/EVAFRILL-Mo-3B

1---2language:3  - ko4  - en5license: mit6library_name: pytorch7pipeline_tag: text-generation8tags:9  - mamba210  - hybrid11  - transformer12  - korean13  - from-scratch14  - dpo15  - slerp16  - orpo17  - nemotron-h18datasets:19  - heegyu/orca-math-korean-preference-cleaned20  - nayohan/preference-collection-ko-full21  - kuotient/orca-math-word-problems-193k-korean22  - FreedomIntelligence/alpaca-gpt4-korean23  - heegyu/orca_ko24  - HAERAE-HUB/KOFFQA-GuardInstruct-v125model-index:26  - name: EVAFRILL-Mo-3B27    results:28      - task:29          type: text-generation30          name: Text Generation31        dataset:32          type: hellaswag33          name: HellaSwag (0-shot, limit=500)34        metrics:35          - name: Accuracy36            type: accuracy37            value: 34.638      - task:39          type: text-generation40        dataset:41          type: arc_easy42          name: ARC-Easy (0-shot, limit=500)43        metrics:44          - name: Accuracy45            type: accuracy46            value: 32.047      - task:48          type: text-generation49        dataset:50          type: belebele51          name: Belebele Korean (0-shot, limit=500)52        metrics:53          - name: Accuracy54            type: accuracy55            value: 23.656      - task:57          type: text-generation58        dataset:59          type: mmlu60          name: Global MMLU Korean (0-shot, limit=500)61        metrics:62          - name: Accuracy63            type: accuracy64            value: 23.765---66 67> [한국어](#한국어) | [English](#english)68 69---70 71# 한국어72 73## EVAFRILL-Mo 3B — 하이브리드 Mamba-2 + Transformer74 75### 프로젝트 소개76 77EVAFRILL-Mo 3B는 NVIDIA [Nemotron-H](https://arxiv.org/abs/2504.03624) 아키텍처에서 영감을 받아 **밑바닥부터 직접 구현한** 30억 파라미터 하이브리드 언어 모델입니다.78 79- 7× NVIDIA B200 GPU로 55B 토큰 사전학습 (약 60시간)80- 한국어·영어·코드·수학 혼합 데이터셋 사용81- SFT → DPO → SLERP 전체 파이프라인을 단일 프로젝트에서 직접 구현82- 외부 프레임워크(Transformers Trainer, TRL) 없이 PyTorch 네이티브로 구현83 84### 아키텍처85 86```87Type:           Hybrid Mamba-2 + Transformer88Parameters:     2.94B (2,975,397,632)89Layers:         26 (24× Mamba-2 SSM + 2× Attention GQA)90d_model:        3,07291Vocabulary:     64,000 (custom SentencePiece)92Max seq length: 4,09693```94 95Mamba-2 SSM 블록이 장거리 의존성을 효율적으로 처리하고, 2개의 GQA Attention 블록이 전역 컨텍스트를 보완합니다.96표준 Transformer 대비 추론 시 KV 캐시 메모리를 크게 절감합니다.97 98### 개발 배경 및 히스토리99 100EVAFRILL-Mo는 6단계의 반복적 설계 과정을 거쳐 탄생했습니다:101 1021. **[FRANKENSTALLM](https://github.com/pathcosmos/FRANKENSTALLM)** — 순수 Transformer decoder-only LLM으로 시작한 전신 프로젝트. 한국어+영어+코드+수학 데이터로 커스텀 SentencePiece 토크나이저(64K 어휘)를 학습하고, DDP 학습 파이프라인을 구축했습니다.1032. **Nemotron-H 영감** — NVIDIA의 하이브리드 Mamba-2 + Transformer 설계를 핵심 원칙만 추출하여(fragmentation) 제한된 하드웨어에 맞게 축소·적용.1043. **체계적 규모 탐색** — 5개 규모(1B~3B) 모델을 7×B200에서 벤치마크하여 Chinchilla-optimal 최대 규모(3B, 93% 달성) 결정.1054. **1B → 3B 전환** — tok/s가 per-GPU 값임을 발견하여, 1B 과잉학습(681%)을 3B 적정학습(93%)으로 전환.1065. **3B 사전학습** — 319,772 steps, 55B tokens, 7×B200 FP8로 60시간 완료.1076. **Post-training** — H100 MIG 환경에서 SFT → DPO → SLERP → ORPO 실험까지 완수.108 109### 핵심 기술 하이라이트110 111| 기술 | 효과 |112|------|------|113| **Chunked Cross-Entropy** | 64K 어휘에서 logits 메모리 사용량을 1/8로 절감 |114| **Mamba Memory Cliff 발견** | batch 6→7에서 47GB→183GB+ 폭증 — selective scan의 구조적 제약 규명 |115| **FP8 네이티브 학습** | TransformerEngine MXFP8BlockScaling으로 B200에서 BF16 대비 ~2배 처리량 |116| **LoRA B-zeroing** | DPO reference model을 모델 복제 없이 LoRA B를 임시 0으로 만들어 계산 — VRAM 50% 절약 |117| **SLERP 체크포인트 병합** | SFT 지식 보존 + DPO 정렬을 구면 보간으로 균형 — alignment tax 완화 |118| **Native DPO/ORPO** | TRL 미사용, 커스텀 Mamba-2 하이브리드를 위해 처음부터 PyTorch로 구현 |119 120> 📖 **전체 개발 과정, 아키텍처 설계 근거, 하드웨어 최적화 상세는 [GitHub README](https://github.com/pathcosmos/EVAFRILL-Mo)를 참조하세요.**121 122### 모델 버전123 124이 저장소에는 학습 파이프라인 각 단계의 체크포인트 **7종**이 포함됩니다.125 126| 버전 | 디렉토리 | 크기 | 설명 | 권장 |127|------|----------|------|------|:----:|128| **SLERP** | `slerp/` | 6.3 GB | SFT + DPO R2 구면 선형 보간 (α=0.5) | ⭐ |129| Pretrain | `pretrain/` | 12.6 GB | 기반 모델 (319K 스텝, 55B 토큰) | |130| SFT v2 | `sft-v2/` | 6.3 GB | 명령어 파인튜닝 (65K 스텝) | |131| DPO R1 | `dpo-r1/` | 6.3 GB | 선호도 정렬 1라운드 (3K 스텝) | |132| DPO R2 | `dpo-r2/` | 6.3 GB | 보수적 파인튜닝 2라운드 (2K 스텝) | |133| ORPO | `orpo/` | 6.3 GB | SFT+정렬 동시 학습 실험 (10K 스텝) | |134| DPO R3 | `dpo-r3/` | 6.3 GB | 반복 억제 특화 실험 (1K 스텝) | |135 136### 학습 파이프라인137 138```139Pretrain (55B tokens, 7×B200, 60h)140  └─► SFT v2 (65K steps, H100 MIG, 5일)141        ├─► DPO R1 (3K steps) ─► DPO R2 (2K steps)142        │     └─► SLERP Merge (α=0.5) ⭐ 최종 권장143        └─► ORPO (10K steps, 실험)144              └─► DPO R3 (1K steps, 반복 특화 실험)145```146 147각 화살표는 독립된 체크포인트로 저장되어, 임의의 단계부터 재현·비교가 가능합니다.148 149### 벤치마크 결과150 151**평가 대상: SLERP 모델** (0-shot, limit=500)152 153| 벤치마크 | 정확도 |154|----------|:------:|155| HellaSwag | 34.6% |156| ARC-Easy | 32.0% |157| Belebele 한국어 | 23.6% |158| Global MMLU 한국어 | 23.7% |159 160**반복 생성 억제** (greedy decoding 기준)161 162| 설정 | 3-gram 반복률 |163|------|:-------------:|164| rep_penalty 없음 | 74.5% |165| rep_penalty=1.2 | **5.5%** |166 167권장 추론 파라미터: `temperature=0.7, repetition_penalty=1.2`168 169### DPO vs ORPO 비교170 171| 지표 | SLERP (SFT→DPO) | ORPO | 우세 |172|------|:---------------:|:----:|:----:|173| Greedy 반복률 | 74.5% | 87.1% | SLERP |174| 대화 품질 | 자연스러움 | 부자연스러움 | SLERP |175| HellaSwag | **39.0%** | 35.0% | SLERP |176| 학습 시간 | 5일+8시간 | **12.8시간** | ORPO |177 178ORPO의 약점: SFT 65K 스텝 대비 10K 스텝만 학습되어 기반 명령어 이해가 부족합니다.179 180### 사용법181 182> **GGUF/Ollama 미지원**: 커스텀 Mamba-2 하이브리드 아키텍처로 llama.cpp/GGUF/Ollama와 호환되지 않습니다. PyTorch 직접 추론만 가능합니다.183 184**사전 준비:**185 186```bash187# 1. 소스 코드 클론 (커스텀 아키텍처 모듈 필요)188git clone https://github.com/pathcosmos/EVAFRILL-Mo189cd EVAFRILL-Mo190 191# 2. 의존성 설치192pip install torch safetensors tokenizers PyYAML193```194 195**방법 1: safetensors 직접 로딩 (권장)**196 197```python198import json199import torch200from model.config import LMConfig201from model.transformer import LLM202from tokenizers import Tokenizer203from safetensors.torch import load_file as load_safetensors204 205CKPT = "path/to/EVAFRILL-Mo-3B/slerp"  # 이 저장소의 slerp/ 디렉토리206 207# Config & 모델 로드208with open(f"{CKPT}/config.json") as f:209    data = json.load(f)210for k in ("model_type", "architectures", "_variant", "_description"):211    data.pop(k, None)212cfg = LMConfig(**data)213cfg.use_flash_attn = False214 215model = LLM(cfg)216state = load_safetensors(f"{CKPT}/model.safetensors", device="cpu")217model.load_state_dict(state, strict=False)218model = model.to(device="cuda:0", dtype=torch.bfloat16)219model.eval()220 221tok = Tokenizer.from_file(f"{CKPT}/tokenizer.json")222 223# 생성 (권장: temp=0.7, rep_penalty=1.2)224prompt = "<|user|>\n인공지능이란 무엇인가요?\n<|assistant|>\n"225ids = torch.tensor([tok.encode(prompt).ids], device="cuda:0")226 227with torch.no_grad():228    for _ in range(256):229        logits, _ = model(ids)230        logits = logits[:, -1, :].float()231        for prev_id in set(ids[0].tolist()):232            if logits[0, prev_id] > 0: logits[0, prev_id] /= 1.2233            else: logits[0, prev_id] *= 1.2234        probs = torch.softmax(logits / 0.7, dim=-1)235        next_id = torch.multinomial(probs, 1)236        ids = torch.cat([ids, next_id], dim=1)237        if next_id.item() == tok.token_to_id("</s>"): break238 239print(tok.decode(ids[0].tolist()))240```241 242**방법 2: 평가 프레임워크 러너 사용**243 244[frankenstallm_test](https://github.com/pathcosmos/frankenstallm_test)의 `evafrill_runner.py`가 위 과정을 래핑합니다:245 246```python247from eval_framework.evafrill_runner import generate, unload_model248 249result = generate("한국어로 인사해주세요.")250print(result["response"])251print(f"속도: {result['tokens_per_sec']:.1f} TPS")252unload_model()253```254 255> 설정 방법: [frankenstallm_test README](https://github.com/pathcosmos/frankenstallm_test#evafrill-mo-모델-설정-pytorch-직접-추론) 참조256 257**시스템 요구사항**: GPU VRAM 8GB+ (BF16), CPU 추론 가능하지만 극히 느림 (~0.5 TPS)258 259### 재현 자료260 261| 경로 | 내용 |262|------|------|263| `data/combined_preference.jsonl` | 선호도 학습 데이터 (684K 쌍, 2.6 GB) |264| `data/repetition_preference.jsonl` | 반복 억제 선호도 데이터 (105 쌍, 자동 생성) |265| `configs/korean_3b_sft_1gpu.yaml` | SFT H100 MIG 설정 |266| `configs/dpo_3b_1gpu.yaml` | DPO 학습 설정 |267| `configs/orpo_3b_1gpu.yaml` | ORPO 학습 설정 |268| `scripts/dpo.py` | DPO 학습 코드 |269| `scripts/orpo_native.py` | ORPO 학습 코드 |270| `scripts/sft.py` | SFT 학습 코드 |271| `scripts/evafrill_eval.py` | 벤치마크 평가 코드 |272| `scripts/merge_checkpoints.py` | SLERP 체크포인트 병합 |273 274### 제한사항275 276- **3B 규모 한계**: 사실 정확도·복잡한 추론에 한계가 있으며, 대형 모델 대비 성능이 낮습니다.277- **GGUF/Ollama 불가**: 커스텀 하이브리드 Mamba-2 아키텍처로 표준 변환 툴을 지원하지 않습니다.278- **vLLM 제한적**: 이론상 가능하나 커스텀 weight key 매핑이 필요합니다.279- **반복 생성**: greedy decoding 시 반복률이 높으므로 반드시 `repetition_penalty=1.2` 이상을 설정하세요.280- **언어 편중**: 한국어·영어 외 언어는 성능이 보장되지 않습니다.281 282### 링크283 284- **GitHub**: [pathcosmos/EVAFRILL-Mo](https://github.com/pathcosmos/EVAFRILL-Mo)285- **이전 프로젝트**: [FRANKENSTALLM](https://github.com/pathcosmos/FRANKENSTALLM) — 순수 Transformer 기반 전신 프로젝트286- **참조 논문**: [Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models](https://arxiv.org/abs/2504.03624)287 288### 라이선스289 290MIT License — 상업적 이용·수정·재배포 모두 자유롭습니다.291 292---293 294# English295 296## EVAFRILL-Mo 3B — Hybrid Mamba-2 + Transformer297 298### Introduction299 300EVAFRILL-Mo 3B is a 3-billion-parameter hybrid language model built **entirely from scratch**, inspired by NVIDIA's [Nemotron-H](https://arxiv.org/abs/2504.03624) architecture.301 302- Pretrained on 55B tokens using 7× NVIDIA B200 GPUs (~60 hours)303- Mixed Korean, English, code, and math datasets304- Full SFT → DPO → SLERP pipeline implemented in pure PyTorch — no Transformers Trainer or TRL305- Designed as a Korean-first model with strong multilingual capability306 307### Architecture308 309```310Type:           Hybrid Mamba-2 + Transformer311Parameters:     2.94B (2,975,397,632)312Layers:         26 (24× Mamba-2 SSM + 2× Attention GQA)313d_model:        3,072314Vocabulary:     64,000 (custom SentencePiece)315Max seq length: 4,096316```317 318Mamba-2 SSM blocks handle long-range dependencies efficiently while two GQA Attention blocks provide global context.319Compared to standard Transformers, this architecture significantly reduces KV cache memory during inference.320 321### Development Background & History322 323EVAFRILL-Mo was built through 6 iterative design stages:324 3251. **[FRANKENSTALLM](https://github.com/pathcosmos/FRANKENSTALLM)** — Predecessor project starting as a pure Transformer decoder-only LLM. Built custom SentencePiece tokenizer (64K vocab) on Korean+English+code+math data and established DDP training pipeline.3262. **Nemotron-H Inspiration** — Extracted core design principles from NVIDIA's hybrid Mamba-2 + Transformer architecture and scaled down for constrained hardware.3273. **Systematic Scale Search** — Benchmarked 5 model sizes (1B–3B) on 7×B200 to find the Chinchilla-optimal maximum (3B, 93% achievement).3284. **1B → 3B Transition** — Discovered tok/s was per-GPU, redirecting from 1B over-training (681%) to 3B optimal training (93%).3295. **3B Pretraining** — 319,772 steps, 55B tokens, 60 hours on 7×B200 with FP8.3306. **Post-training** — SFT → DPO → SLERP → ORPO experiments on H100 MIG.331 332### Key Technical Highlights333 334| Technique | Impact |335|-----------|--------|336| **Chunked Cross-Entropy** | Reduces logits memory by 8× for 64K vocabulary |337| **Mamba Memory Cliff Discovery** | Batch 6→7 causes 47GB→183GB+ explosion — structural limitation of selective scan |338| **FP8 Native Training** | TransformerEngine MXFP8BlockScaling delivers ~2× throughput vs BF16 on B200 |339| **LoRA B-zeroing** | Computes DPO reference logprobs without model duplication — 50% VRAM savings |340| **SLERP Checkpoint Merging** | Balances SFT knowledge + DPO alignment via spherical interpolation — mitigates alignment tax |341| **Native DPO/ORPO** | No TRL dependency — implemented from scratch in PyTorch for custom Mamba-2 hybrid |342 343> 📖 **For the complete development journey, architecture design rationale, and hardware optimization details, see the [GitHub README](https://github.com/pathcosmos/EVAFRILL-Mo).**344 345### Model Variants346 347This repository contains **7 checkpoints** representing each stage of the training pipeline.348 349| Variant | Directory | Size | Description | Recommended |350|---------|-----------|------|-------------|:-----------:|351| **SLERP** | `slerp/` | 6.3 GB | Spherical interpolation of SFT + DPO R2 (α=0.5) | ⭐ |352| Pretrain | `pretrain/` | 12.6 GB | Base model (319K steps, 55B tokens) | |353| SFT v2 | `sft-v2/` | 6.3 GB | Instruction-tuned (65K steps) | |354| DPO R1 | `dpo-r1/` | 6.3 GB | Preference-aligned Round 1 (3K steps) | |355| DPO R2 | `dpo-r2/` | 6.3 GB | Conservative fine-tuning Round 2 (2K steps) | |356| ORPO | `orpo/` | 6.3 GB | Simultaneous SFT+alignment experiment (10K steps) | |357| DPO R3 | `dpo-r3/` | 6.3 GB | Repetition-targeted experiment (1K steps) | |358 359### Training Pipeline360 361```362Pretrain (55B tokens, 7×B200, 60h)363  └─► SFT v2 (65K steps, H100 MIG, 5 days)364        ├─► DPO R1 (3K steps) ─► DPO R2 (2K steps)365        │     └─► SLERP Merge (α=0.5) ⭐ Final Recommended366        └─► ORPO (10K steps, experimental)367              └─► DPO R3 (1K steps, repetition experiment)368```369 370Every arrow corresponds to a separate saved checkpoint, enabling reproduction and comparison from any stage.371 372### Benchmark Results373 374**Evaluated on: SLERP model** (0-shot, limit=500)375 376| Benchmark | Accuracy |377|-----------|:--------:|378| HellaSwag | 34.6% |379| ARC-Easy | 32.0% |380| Belebele Korean | 23.6% |381| Global MMLU Korean | 23.7% |382 383**Repetition suppression** (greedy decoding)384 385| Setting | 3-gram repetition rate |386|---------|:----------------------:|387| No rep_penalty | 74.5% |388| rep_penalty=1.2 | **5.5%** |389 390Recommended inference parameters: `temperature=0.7, repetition_penalty=1.2`391 392### DPO vs ORPO Comparison393 394| Metric | SLERP (SFT→DPO) | ORPO | Winner |395|--------|:---------------:|:----:|:------:|396| Greedy repetition | 74.5% | 87.1% | SLERP |397| Chat quality | Fluent | Broken | SLERP |398| HellaSwag | **39.0%** | 35.0% | SLERP |399| Training time | 5d+8h | **12.8h** | ORPO |400 401ORPO's weakness: only 10K steps of training vs SFT's 65K — insufficient base instruction-following before alignment kicks in.402 403### Usage404 405> **GGUF/Ollama not supported**: Custom Mamba-2 hybrid architecture is incompatible with llama.cpp/GGUF/Ollama. PyTorch direct inference only.406 407**Prerequisites:**408 409```bash410# 1. Clone source code (custom architecture modules required)411git clone https://github.com/pathcosmos/EVAFRILL-Mo412cd EVAFRILL-Mo413 414# 2. Install dependencies415pip install torch safetensors tokenizers PyYAML416```417 418**Method 1: Direct safetensors loading (recommended)**419 420```python421import json422import torch423from model.config import LMConfig424from model.transformer import LLM425from tokenizers import Tokenizer426from safetensors.torch import load_file as load_safetensors427 428CKPT = "path/to/EVAFRILL-Mo-3B/slerp"  # slerp/ directory of this repo429 430# Load config & model431with open(f"{CKPT}/config.json") as f:432    data = json.load(f)433for k in ("model_type", "architectures", "_variant", "_description"):434    data.pop(k, None)435cfg = LMConfig(**data)436cfg.use_flash_attn = False437 438model = LLM(cfg)439state = load_safetensors(f"{CKPT}/model.safetensors", device="cpu")440model.load_state_dict(state, strict=False)441model = model.to(device="cuda:0", dtype=torch.bfloat16)442model.eval()443 444tok = Tokenizer.from_file(f"{CKPT}/tokenizer.json")445 446# Generate (recommended: temp=0.7, rep_penalty=1.2)447prompt = "<|user|>\nWhat is artificial intelligence?\n<|assistant|>\n"448ids = torch.tensor([tok.encode(prompt).ids], device="cuda:0")449 450with torch.no_grad():451    for _ in range(256):452        logits, _ = model(ids)453        logits = logits[:, -1, :].float()454        for prev_id in set(ids[0].tolist()):455            if logits[0, prev_id] > 0: logits[0, prev_id] /= 1.2456            else: logits[0, prev_id] *= 1.2457        probs = torch.softmax(logits / 0.7, dim=-1)458        next_id = torch.multinomial(probs, 1)459        ids = torch.cat([ids, next_id], dim=1)460        if next_id.item() == tok.token_to_id("</s>"): break461 462print(tok.decode(ids[0].tolist()))463```464 465**Method 2: Evaluation framework runner**466 467The `evafrill_runner.py` in [frankenstallm_test](https://github.com/pathcosmos/frankenstallm_test) wraps the above into a simple API:468 469```python470from eval_framework.evafrill_runner import generate, unload_model471 472result = generate("Hello, please introduce yourself.")473print(result["response"])474print(f"Speed: {result['tokens_per_sec']:.1f} TPS")475unload_model()476```477 478> Setup instructions: [frankenstallm_test README](https://github.com/pathcosmos/frankenstallm_test#evafrill-mo-모델-설정-pytorch-직접-추론)479 480**System requirements**: GPU VRAM 8GB+ (BF16), CPU inference possible but extremely slow (~0.5 TPS)481 482### Reproducibility483 484| Path | Contents |485|------|----------|486| `data/combined_preference.jsonl` | Preference training data (684K pairs, 2.6 GB) |487| `data/repetition_preference.jsonl` | Repetition-suppression preference data (105 pairs, auto-generated) |488| `configs/korean_3b_sft_1gpu.yaml` | SFT config for H100 MIG |489| `configs/dpo_3b_1gpu.yaml` | DPO training config |490| `configs/orpo_3b_1gpu.yaml` | ORPO training config |491| `scripts/dpo.py` | DPO training code |492| `scripts/orpo_native.py` | ORPO training code |493| `scripts/sft.py` | SFT training code |494| `scripts/evafrill_eval.py` | Benchmark evaluation code |495| `scripts/merge_checkpoints.py` | SLERP checkpoint merging |496 497### Limitations498 499- **3B scale**: Factual accuracy and complex multi-step reasoning are limited compared to larger models.500- **GGUF/Ollama**: Not supported — custom hybrid Mamba-2 architecture cannot be converted with standard tools.501- **vLLM**: Theoretically possible but requires custom weight key mapping.502- **Greedy repetition**: ~74.5% 3-gram repetition rate without `repetition_penalty` — always use `repetition_penalty >= 1.2`.503- **Language coverage**: Performance is not guaranteed for languages other than Korean and English.504 505### Links506 507- **GitHub**: [pathcosmos/EVAFRILL-Mo](https://github.com/pathcosmos/EVAFRILL-Mo)508- **Predecessor**: [FRANKENSTALLM](https://github.com/pathcosmos/FRANKENSTALLM) | [🤗 HuggingFace](https://huggingface.co/pathcosmos/frankenstallm) — Pure Transformer predecessor project509- **Reference paper**: [Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models](https://arxiv.org/abs/2504.03624)510 511### Acknowledgment / 감사의 글512 513이 프로젝트는 **과학기술정보통신부**의 **「첨단 GPU 활용 지원 사업」** (과학기술정보통신부 공고 제2025-1068호)을 통해 제공된 GPU 컴퓨팅 자원을 활용하여 수행되었습니다.514 515> **국가 AI컴퓨팅자원 지원포털**: [https://aiinfrahub.kr](https://aiinfrahub.kr)516>517> - 주관: 과학기술정보통신부 (MSIT), 정보통신산업진흥원 (NIPA)518> - 운영: 한국정보통신진흥협회 (KAIT)519 520대한민국 정부의 AI 인프라 지원 사업 덕분에 7× NVIDIA B200 GPU 환경에서 한국어 3B 하이브리드 Mamba-Transformer 모델을 처음부터 학습할 수 있었습니다. 국가 차원의 AI 컴퓨팅 자원 지원에 깊이 감사드립니다.521 522This project was conducted using GPU computing resources provided through the **"Advanced GPU Utilization Support Program"** (MSIT Notice No. 2025-1068) by the **Ministry of Science and ICT (MSIT)** of the Republic of Korea.523 524> **National AI Computing Resource Support Portal**: [https://aiinfrahub.kr](https://aiinfrahub.kr)525>526> - Organized by: Ministry of Science and ICT (MSIT), National IT Industry Promotion Agency (NIPA)527> - Operated by: Korea Association of Information & Telecommunication (KAIT)528 529We are deeply grateful for the national-level AI computing infrastructure support from the Korean government, which made it possible to train a Korean 3B hybrid Mamba-Transformer model from scratch on 7× NVIDIA B200 GPUs.530 531---532 533### License534 535MIT License — free to use, modify, and distribute commercially.536