pvlabs/Chytrej2-90M-Base

Chytrej2-90M-Base

A fully custom pretrained language model built from scratch on the LLaMA architecture trained on FineWeb Edu dataset.

Built by PingVortex Labs.


Model Details

• —Parameters: 90M
• —Context length: 8,192 tokens
• —Language: English only
• —Format: Base model
• —Architecture: LLaMA
• —License: Apache 2.0

Benchmarks

• —No benchmarks - at this scale the benchmarks are more random than based on what the model has learned - we got a checkpoint 55k get better arc easy score than the final model.

Usage

from transformers import LlamaForCausalLM, PreTrainedTokenizerFast

model = LlamaForCausalLM.from_pretrained("pvlabs/Chytrej2-90M-Base")
tokenizer = PreTrainedTokenizerFast.from_pretrained("pvlabs/Chytrej2-90M-Base")

prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, repetition_penalty=1.3)
print(tokenizer.decode(outputs[0]))

Made by [PingVortex](https://pingvortex.com).