croqaz/vintage-LLM-340m-v1-base

Vintage-LLM

This is a 340M params Llama3-based model, trained on text pre-1900.

This is a base model. It has very limited chat capabilities.

Dataset used: https://huggingface.co/datasets/croqaz/vintage-v1

The data was de-duplicated and only the decent quality texts were used.

There are a few checkpoints in the repo:

• —10k steps on short texts: commit/ff3b94efdaccfa312754d3a4328b7c7a05155d36
• —previous model + short ft on vintage-FT-v1: commit/3946781d51e76c7cf0874bd595003bce1755cbd2
• —previous model + 10k steps on long texts: commit/c9b48cc3aaa4b3dc0ea3b846f45e98a05e94c8cb