ismatsamadov/gpt-wiki-az-demo

Azerbaijani GPT - Text Generation Demo

A custom GPT language model trained from scratch on Azerbaijani Wikipedia text, deployed as an interactive Gradio demo.

Model Architecture

The architecture follows the GPT-2 design:

• —Token + positional embeddings – learned positional embeddings up to block_size.
• —Transformer blocks – each block contains pre-norm multi-head self-attention followed by a pre-norm feed-forward MLP (4× expansion, GELU activation).
• —Language-model head – a linear projection from n_embd to vocab_size (no bias).

The causal (autoregressive) mask is baked into the attention scores so the model only attends to past tokens during generation.

Tokenizer

A Byte-Pair Encoding (BPE) tokenizer was trained specifically on Azerbaijani Wikipedia text using the tokenizers library:

• —Vocabulary size: 22 000 merge rules (22 588 total tokens incl. special tokens)
• —Trained on the same corpus as the model – no cross-lingual transfer
• —Stored as az_tokenizer.json (HuggingFace Tokenizers format)

Training

• —Dataset: Azerbaijani Wikipedia dump (cleaned, plain text)
• —Optimiser: AdamW (lr = 3e-4, cosine annealing)
• —Mixed precision: torch.amp (FP16 on GPU)
• —Gradient clipping: 1.0
• —Batch size: 8 sequences × 256 tokens
• —Epochs: up to 50, best checkpoint saved by validation loss

Model weights (best_model.pt) and tokenizer (az_tokenizer.json) are hosted on IsmatS/gpt-wiki-az.

Text Generation

The demo uses nucleus sampling (top-p) for decoding:

1. 1.The prompt is tokenized with the BPE tokenizer.
2. 2.At each step the model produces a distribution over the 22 k vocabulary.
3. 3.Tokens are sorted by probability; the smallest set whose cumulative probability exceeds p = 0.9 is kept.
4. 4.A token is sampled from that reduced distribution, scaled by temperature.
5. 5.Steps 2-4 repeat until max_tokens new tokens are generated.

Interface controls

Example prompts

• —Azərbaycanın tarixi – History of Azerbaijan
• —Bakı şəhəri – City of Baku
• —Neft sənayesi – Oil industry

Limitations

• —The model was trained on encyclopaedic text only; it may struggle with conversational or creative prompts.
• —Output quality depends heavily on the prompt language: prompts in Azerbaijani will yield the best results.
• —The model is relatively small (~85 M parameters) and may produce repetitive or incoherent text for long sequences.

Repository

• —Model weights & tokenizer: IsmatS/gpt-wiki-az
• —Training code: available in the model repository (train.py)