Modelpublic

mnnobi/Nexus1-124M-v1

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face

0likes23downloads

Model Card

Nexus1-124M-v1

Nexus1 is the first prototype in the Nexus LLM series. It is a 124M parameter decoder-only transformer pre-trained from scratch on high-quality educational web data.

Model Details

—Architecture: GPT-2 style
—Parameters: 124 Million
—Context Window: 1024 tokens
—Vocabulary Size: 50,257 (Custom BPE)
—Training Stage: Base Pre-training (Complete)

Training Data

Pre-trained on the FineWeb-Edu (10B Sample) dataset, which focuses on high-quality, educational content from the web to ensure better reasoning capabilities in a small model.

Purpose

Nexus1 serves as the proof-of-concept for the Nexus Training Pipeline. It is intended to be used as a reference model for knowledge distillation into the larger Nexus2-1B model.

How to use

python

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("mnnobi/Nexus1-124M-v1")
tokenizer = AutoTokenizer.from_pretrained("mnnobi/Nexus1-124M-v1")