Modelpublic

DemonKing1234/gt730-moe-transformer

sourceHugging Facemitupdated 1mo agoView on Hugging Face

0likes28downloads

Model Card

🧠 GT-730 MoE Reasoning Transformer (GGUF)

DemonKing1234/gt730-moe-transformer is a lightweight, edge-optimized Mixture-of-Experts (MoE) Language Model designed for efficient token generation, algorithmic problem-solving, and agentic reasoning on low-resource hardware.

🚀 What This Model Can Do & What It's Used For

1. 🐍 Python Code & Algorithm Generation

Trained on Python data structures, algorithms (binary search trees, graphs, sorting, async operations), and common developer patterns. It can generate small code snippets, logic blocks, and algorithmic skeletons.

2. 🤖 Agentic Problem Solving & Tool-Use

Employs step-by-step reasoning tokens (<think> chains) and action simulation (ACTION: writefile, uncommand), enabling it to simulate autonomous assistant and tool-use behaviors.

3. 🧩 Multi-Step Logic & Math Reasoning

Leverages MoE routing to direct logic and mathematical steps through specialized feed-forward expert networks for structured, multi-step problem solving.

4. ⚡ Ultra-Efficient Edge Inference

Because the model uses a sparse Mixture-of-Experts architecture with top-k gating, only a fraction of the network parameters are activated per token, making it fast and lightweight for edge devices, CPUs, and low-end GPUs.

📊 Model Architecture & Specifications

Parameter	Value
Architecture	Causal Mixture-of-Experts (MoE) Transformer
Layers	5 Transformer Blocks
Routing / Experts	16 Feed-Forward Experts (Top-2 Activated per token)
Attention Heads	4 Heads with Causal Masking
Hidden Size	64
Feed-Forward Dimension	128
Vocabulary Size	2,386 Tokens
Precision	FP32 (GGUF Format)
Weight File	moereasoningtransformer_f32.gguf (8.86 MB)

📈 Training Details & Token Count

Metric	Details
Total Cumulative Tokens Trained	~175,000,000+ Tokens (~175M token exposures across pretraining passes)
Base Corpus Size	~2.1 Million Characters (~400,000 base tokens)
Training Passes / Epochs	300 to 500 iterative optimization epochs
Hardware	NVIDIA GeForce GT 730 (CUDA) + Multi-Core CPU
Frameworks	JAX/Flax & CuPy

📚 Training Dataset (Mega Hybrid Corpus)

Pretrained on a curated dataset combining:

—High-performance Python codebases, data structures, and algorithms
—Agentic scratchpads and structured chain-of-thought demonstrations (<think>)
—Step-by-step mathematical & logical reasoning samples (GSM8K/MATH style)
—Literary knowledge & structured conversational dialogues

⚙️ Intended Use & Limitations

—Intended For: Research into sparse MoE scaling on ultra-constrained edge devices, offline embedded AI experiments, and fast local prototyping.
—Limitations: As a micro-scale model (8.86 MB), it is optimized for compactness and structural reasoning rather than expansive general world knowledge.