11-47/distilgpt2-supernatural-occult-coder

distilgpt2-supernatural-occult-coder

Model Details

Model Description

This model is a full fine-tuned version of DistilGPT2, specialized in generating a unique blend of text covering the occult, supernatural magic, coding/programming, and mature internet discourse. It was trained comprehensively on a massive merged dataset of over 1.5 million rows to synthesize these distinct themes into a single generative framework.

As a lightweight causal language model (82M parameters), it is optimized for extremely fast text generation across varied esoteric and technical domains.

• —Developed by: GODsStrongestSoldier
• —Model type: Causal Language Model (Transformer Decoder)
• —Language: English
• —License: Apache 2.0
• —Finetuned from model: distilgpt2

Datasets Used for Fine-Tuning

This model was trained on a concatenated corpus consisting of the following datasets:

• —WithinUsAI/high_priest_occult_25k
• —WithinUsAI/gods_universe_codex_distill_god_seed_25k
• —WithinUsAI/Supernatural_25k
• —WithinUsAI/high_priest_supernatural_magic_FACT_BASED_1M
• —acheong08/nsfw_reddit

Training Details

Training Procedure

The model underwent full fine-tuning (no LoRA or adapters). All layers of the base model were globally updated. Datasets were dynamically loaded, stripped of extraneous columns, converted entirely to text, concatenated, and shuffled with a fixed seed to ensure an even distribution of themes throughout the training process.

Texts were grouped into continuous sequences of 512 tokens.

Hardware

• —Environment: Kaggle
• —Accelerators: Dual NVIDIA T4 GPUs (15GB VRAM each)

Hyperparameters

• —Epochs: 1 (Due to the massive 1.5M+ row dataset size)
• —Per-Device Batch Size: 8
• —Gradient Accumulation Steps: 8
• —Effective Global Batch Size: 128
• —Learning Rate: 5e-05
• —Optimizer: Fused AdamW (adamw_torch_fused)
• —Mixed Precision: fp16
• —Gradient Checkpointing: Enabled