tachiwin/tiyat_alpha_arch1

Tiyat Alpha (arch1)

A custom, from-scratch multilingual decoder-only transformer (not a GPT-2/Llama/etc. variant — architecture details below).

Model Architecture

• —Tokenizer: tachiwin/tokenizer_64k
• —Vocabulary Size: 64,000
• —Max Sequence Length: 918
• —Embedding Dimension: 768
• —Number of Layers: 12
• —Number of Heads: 24
• —Feed-Forward Dimension: 3072
• —Tied Embeddings: False
• —Total Parameters: ~184.1M

Training Data

• —Dataset: tachiwin/tiyat-ground-pretrain-m1024

Training Configuration

• —Batch Size: 4
• —Learning Rate: 0.0003
• —Weight Decay: 0.01
• —Gradient Clip: 1.0
• —Warmup Ratio: 0.1
• —Dropout Rate: 0.1
• —Total Steps: 141,672
• —Final Loss: 3.4996705055236816

Usage

This repo ships its own modeling code (configuration_tiyat.py, modeling_tiyat.py) so it loads without any external notebooks — either via trust_remote_code:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("tachiwin/tiyat_alpha_arch1", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("tachiwin/tiyat_alpha_arch1")

ids = tokenizer.encode("Once upon a time", return_tensors="np")[0].tolist()
out = model.generate(ids, max_new_tokens=40, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out))

or by importing the class directly (equivalent, no transformers Auto-dispatch):

from modeling_tiyat import TiyatForCausalLM
model = TiyatForCausalLM.from_pretrained("tachiwin/tiyat_alpha_arch1")

Note: model.generate() is a minimal, hand-implemented generator (argmax / temperature+top-k+top-p sampling with repetition penalty and no-repeat-ngram blocking) — it is not transformers' full GenerationMixin (no beam search, no GenerationConfig). Requires jax, flax>=0.10, safetensors.

Known limitations

• —Inference is currently slow (un-jitted JAX forward pass, no KV cache) — actively being addressed; not yet representative of the architecture's real speed.
• —Multilingual coverage is uneven — quality is noticeably stronger on the higher-resource languages in the training mix than the lower-resource ones.