aquilesfd/neo-2-345M-C1

neo-2-345m-c1

This is the first checkpoint of the 'neo-2-345m' model, a next-generation language model developed by aquif AI. This checkpoint is fine-tuned on a diverse dataset including conversational, code, and math data, serving as the initial step in a 5-checkpoint training process designed to create a versatile and capable model.

Model Details

Base Model: gpt2-medium\ Method: LoRA (Low-Rank Adaptation)\ Parameter Count: 355 million params\

Training Information

This checkpoint was trained as the first stage of a multi-checkpoint process. The training was performed using a network-resilient script that includes fallback mechanisms for data loading and model initialization.

Checkpoint Number: 1/5\ Hardware: Trained on a Google Colab T4 GPU.\ Training Duration: Approximately 2.5 hours for this checkpoint.\ Training Framework: PyTorch, Hugging Face Transformers, PEFT, bitsandbytes, TRL.\ Quantization: 8-bit.\

LoRA Configuration:

r=8\ loraalpha=16\ targetmodules: ["qattn", "cattn", "cproj", "cfc", "attn.cattn", "attn.cproj", "mlp.cfc", "mlp.cproj"]\ loradropout=0.05\ bias="none"\ tasktype="CAUSALLM"\ Training Arguments:\ perdevicetrainbatchsize=2\ gradientaccumulationsteps=16\ numtrainepochs=1 (for this checkpoint)\ learningrate=1e-5\ max_steps=400\ \ Optimized for 8-bit training.

Training Loss Data

The following table shows the training loss recorded during the training of this checkpoint:\

Note: Training loss is a metric that indicates how well the model is learning. A decreasing loss generally suggests improvement.\

Intended Use

This checkpoint is an intermediate model in the development of the full 'neo-2'. It is not intended for production use but serves as a foundation for subsequent fine-tuning checkpoints focusing on specific domains and tasks.

How to Load the Model

You can load this model using the Hugging Face 'transformers' library:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "aquiffoo/neo-2-345m-c1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

Future Checkpoints

This is the first of 5 planned checkpoints. Future checkpoints will continue to fine-tune the model on additional data to improve its capabilities across various domains. \ License: Apache 2.0