BerkayRA/egemen-turkish-124m

Egemen Turkish LM — 124M (from scratch)

A small Llama-style decoder-only language model trained from scratch on Turkish, as the headline run of an A/B pre-training experiment ("Config A"). It is a research / educational base model — a demonstration that a coherent Turkish LM can be trained on a single 8 GB GPU — not an instruction-tuned or aligned assistant. Developed at, and on the infrastructure of, Mega Bilgisayar Tic. Ltd. Şti.

• —Developed by: Berkay Adanalı (@BerkayRA)
• —Developed at: Mega Bilgisayar (Ankara, Turkey)
• —Model type: decoder-only transformer (causal LM), Llama-style
• —Language: Turkish (tr)
• —License: Apache-2.0
• —Trained from scratch (no parent model)
• —Run: u32_124m, stopped at step 300,000 on 2026-08-01
• —Live demo: in-browser Space (runs client-side via transformers.js — no server)

Architecture

The architecture is standard Llama, so it loads directly with AutoModelForCausalLM and is servable by TGI / vLLM / text-generation-inference.

Tokenizer

SentencePiece unigram, vocab 32,000 (sp_unigram_32000). Special tokens: unk=0, bos=1 (<s>), eos=2 (</s>), pad=3 (<pad>). eos (2) doubles as the document separator in the training stream.

Training

• —Data: Turkish web/text corpus, packed into uint16 token shards with eos between documents; ~39.3 B tokens seen over training.
• —Objective: next-token prediction (cross-entropy).
• —Optimizer: AdamW, β = (0.9, 0.95), weight decay 0.1 (norms & embeddings excluded), gradient clip 1.0.
• —LR schedule: peak 6e-4 → cosine floor 6e-5, 2,000-step linear warmup.
• —Batch: 4 micro × 32 grad-accum × 1024 ctx = 131,072 tokens / step.
• —Steps: 300,000 (of a planned 600,000). Precision: fp16 AMP. Init: GPT-2/nanoGPT (std 0.02; residual projections scaled by 1/√(2·n_layer)). Seed 1337.
• —Hardware: single NVIDIA Quadro RTX 4000 (Turing, 8 GB); ~67.5 h wall-clock.

Results

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "BerkayRA/egemen-turkish-124m"   # <- your repo id
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16)

prompt = "Türkiye'nin başkenti"
# The tokenizer does not auto-prepend BOS; the model was trained with id 1 as BOS
# and id 2 as the document separator / EOS.
ids = torch.tensor([[1] + tok(prompt, add_special_tokens=False).input_ids])
out = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8,
                     top_p=0.95, eos_token_id=2, repetition_penalty=1.3)
print(tok.decode(out[0][1:], skip_special_tokens=True))

ONNX / in-browser (transformers.js)

ONNX weights are included under `onnx/`: model.onnx (fp32) and model_quantized.onnx (int8, ~135 MB) — so the model runs in the browser via 🤗 transformers.js with no server (see the live demo above), or under ONNX Runtime in Python.

import { AutoTokenizer, AutoModelForCausalLM, Tensor } from "@huggingface/transformers";
const tok = await AutoTokenizer.from_pretrained("BerkayRA/egemen-turkish-124m");
const model = await AutoModelForCausalLM.from_pretrained("BerkayRA/egemen-turkish-124m", { dtype: "q8" });
const enc = await tok("Türkiye'nin başkenti", { add_special_tokens: false });
const ids = [1n, ...Array.from(enc.input_ids.data, BigInt)];           // bos=1
const input_ids = new Tensor("int64", BigInt64Array.from(ids), [1, ids.length]);
const out = await model.generate({ input_ids, max_new_tokens: 60, do_sample: true,
                                   temperature: 0.8, top_p: 0.95, eos_token_id: 2,
                                   repetition_penalty: 1.3 });
console.log(tok.decode(Array.from(out[0].data, Number).slice(ids.length), { skip_special_tokens: true }));

Intended use & limitations

Intended: research on Turkish LM pre-training, tokenizer/architecture experiments, a small base model for further fine-tuning (SFT/LoRA), education.

Not intended: production use, factual question answering, or any safety-sensitive application. This is a base model — no instruction tuning, no RLHF, no safety alignment.

Limitations:

• —Small (110 M) and trained on a partial schedule (300k of 600k steps) — expect limited factuality, frequent repetition, and hallucination.
• —Short 1024-token context.
• —Trained on web text not filtered for safety; may reproduce biases, errors, or undesirable content present in the corpus.
• —Turkish-only; no meaningful multilingual or code ability.

Citation

@misc{adanali2026turkishllm124m,
  title  = {Egemen Turkish LM 124M (from scratch)},
  author = {Adanalı, Berkay},
  year   = {2026},
  note   = {From-scratch Llama-style Turkish base model, run u32\_124m}
}