Anurag9817/nepali-llama-590m-380m-english

Nepali LLaMA 590M — 380M English

A 590M-parameter causal language model.

Architecture

• —Parameters: 590,077,440
• —Vocabulary: 40,000
• —Hidden size: 1536
• —Intermediate size: 4096
• —Layers: 18
• —Attention heads: 12
• —KV heads: 6

Training

Base checkpoint:

Anurag9817/nepali-llama-590m-180m-english

Additional training:

• —Previous exposure: 180M English tokens
• —New exposure: 200M English tokens
• —Total exposure: 380M English tokens
• —Dataset: FineWeb-Edu
• —Sequence length: 1024
• —Batch/GPU: 2
• —Gradient accumulation: 8
• —Effective tokens/update: 32,768
• —GPUs: 2 × Tesla T4
• —Learning rate: 1e-5
• —Warmup: 100
• —Scheduler: cosine
• —Optimizer: AdamW 8-bit
• —FP16: enabled
• —Gradient checkpointing: disabled
• —Optimizer steps: 6104
• —Final training loss: 4.196
• —Training time: 8.82 hours

Tokenizer

Raw SentencePiece tokenizer.

File:

tokenizer/nepali_llama_40k.model

Special token IDs:

• —PAD = 0
• —UNK = 1
• —BOS = 2
• —EOS = 3

Do not replace the project tokenizer with AutoTokenizer.

Status

This is a base pretrained language model, not an instruction-tuned chatbot.