kshitijthakkar/loggenix-moe-1b-pretrain

LogGenix MoE 1.4B Pretrained

A 1.4 billion parameter Mixture of Experts (MoE) language model based on the Qwen3MoE architecture, pretrained from scratch on ~328M tokens from NVIDIA Nemotron datasets.

Model Details

Training Details

Training Loss Progression

Dataset

Pretrained on a balanced mix from NVIDIA Nemotron datasets:

• —General text (Nemotron-CC)
• —Mathematics (Nemotron-Math)
• —Code (Nemotron-Code)
• —Scientific content
• —Reasoning tasks

Architecture

This model uses the Qwen3 MoE architecture with:

• —Sparse MoE FFN: 16 experts with top-8 routing per token (8 of 16 experts active)
• —Grouped Query Attention (GQA): 32 query heads, 8 KV heads
• —QK Normalization: RMSNorm on Q and K projections
• —RoPE: Rotary Position Embeddings (base 10,000,000)
• —RMSNorm: Pre-normalization on attention and FFN blocks

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "kshitijthakkar/loggenix-moe-1b-pretrain",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "kshitijthakkar/loggenix-moe-1b-pretrain",
    trust_remote_code=True,
)

inputs = tokenizer("The future of AI is", return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Checkpoints

Intermediate checkpoints are available under checkpoints/:

• —checkpoints/step-2000/
• —checkpoints/step-4000/
• —checkpoints/step-6000/
• —checkpoints/step-8000/
• —checkpoints/step-10000/

Each checkpoint includes model.safetensors, config.json, tokenizer files, and eval inference results under eval/step-{N}/.

Evaluation

Standard Benchmarks (lm-eval harness, 50 samples each)

Synthetic Task Evaluation

Code & Tool Evaluation

Note: These scores are expected for a pretrained-only model with ~328M tokens seen. The model has not been instruction-tuned. PIQA (58%) shows the strongest signal, indicating basic physical commonsense reasoning is emerging. Tool-calling and structured output capabilities require SFT.

Evaluation Charts

[image]

[image]

Full evaluation results are available in eval_outputs/ and on the model page.

Limitations

• —Pretrained only (no instruction tuning) - outputs may be repetitive or incoherent
• —Trained on ~328M tokens (well below Chinchilla-optimal ~28B for 1.4B params)
• —Best suited as a base model for fine-tuning

License

Apache 2.0

Citation

@misc{loggenix-moe-1b-2026,
  title={LogGenix MoE 1.4B: A Mixture of Experts Language Model},
  author={Kshitij Thakkar},
  year={2026},
  url={https://huggingface.co/kshitijthakkar/loggenix-moe-1b-pretrain}
}