WICKED4950/BwETAFv2-130M

🧾 BwETAFv2-130M: Model Card

Boring’s Experimental Transformer for Autoregression (Flax) A 130M parameter autoregressive language model built using a custom Flax pipeline, loads of tuning, and a sprinkle of existential dread.

Trained on determination, fueled by suffering, powered by free TPUs.

📌 Model Summary

• —Model Name: BwETAFv2-130M
• —Parameters: 130,649,088
• —Training Tokens: 3,014,656,000
• —Training Time: 19.32 TPUv2-8 Hours
• —Framework: Flax / JAX
• —Max Context Length: 4096 tokens
• —Tokenizer: GPT-2 BPE (50,257 vocab size)

🧪 Hyperparameters

{
  num_heads=12,
  attention_dim=768,
  vocab_size=50260,
  num_blocks=12,
  ff_dim=2304,
  dropout_rate=0.1,
  max_len = 8192,
  use_fash_attention = False,
  attn_chunks = 1,
  emb_init_range = 0.02,
  use_rope = True,
  emb_scaling_factor = 1,
  res_scale = 1
}

🛠 Optimizer Settings

{
  "peaklr":5e-4,
  "warmup_percent":0.03,
  "min_value":1e-8,
  "training_decay":"cosine",
  "weight_decay": 0.1,
  "min_warmup_value":2e-4,
  "b1": 0.9,
  "b2": 0.95,
  "eps": 1e-7
}

📈 Performance

• —Final Validation Loss: 3.59

• —Validation loss Graphs:

• —Training loss Graphs:

• —For detailed stats, refer to stats.json in the model files.

⚡ Quickstart

pip install BwETAF==0.5.1

import BwETAF

# 🔍 Quick API test
prompt = "The meaning of life is"
output = BwETAF.SetUpAPI(prompt, "WICKED4950/BwETAFv2-130M")
print(output)

# ⬇️ Load from Hugging Face Hub
model = BwETAF.load_hf("WICKED4950/BwETAFv2-130M")

# 📁 Load from local path
BwETAF.load_model("path/to/model")

# 💾 Save to local directory
model.save_model("path/to/save")

# 🔧 Inspect model
params = model.trainable_variables
structure = model.model_struct

☁️ Colab support and examples coming soon!

🧠 BwETAFv2 Architecture Overview

The BwETAFv2 architecture introduces several refinements over its predecessors, improving convergence, training stability, and scalability. Below is an architecture-level overview shared across all models in this series.

🔩 Core Architecture

• —Attention: Standard multi-head self-attention (no GQA, no MQA, no FlashAttention)
• —FFN: Uses Swish-GLU with FF dimension = 3 × model_dim
• —Norm Layer: RMSNorm, pre-layer
• —Positional Encoding: RoPE on embeddings and K/Q vectors
• —Precision:

• —Weights & forward pass: bfloat16 (bf16)
• —Optimizer states: float32
• —Tokenization: GPT-2 BPE (vocab_size = 50257)
• —Bias Terms: No biases in K/Q/V dense projections

⚙️ Training Setup

• —Optimizer: AdamW
• —Batch Size: 32 per step
• —Learning Schedule: Cosine decay with linear warmup
• —Chinchilla Scaling Law: Followed (20× tokens per parameter)
• —Context Window during Training:

• —BwETAFv2-53M: 2048
• —BwETAFv2-130M: 4096

📊 Model Comparison

📬 Contact Me

• —📸 Instagram: boring.\_.wicked
• —💬 Discord: fused_computation.1 (if you spot me lurking in any AI-related servers)