Modelpublic

halcyon-llm/SmolLM2-360M-japanese_patch-11393

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face

0likes37downloads

Model Card

results

This model is a fine-tuned version of HuggingFaceTB/SmolLM2-360M on the kajuma/training01-09patch dataset. It achieves the following results on the evaluation set:

—Loss: 0.1002

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 0.003
—trainbatchsize: 8
—evalbatchsize: 8
—seed: 42
—gradientaccumulationsteps: 32
—totaltrainbatch_size: 256
—optimizer: Use OptimizerNames.ADAMWBNB with betas=(0.9,0.95) and epsilon=1e-08 and optimizerargs=No additional optimizer arguments
—lrschedulertype: cosinewithmin_lr
—lrschedulerwarmup_steps: 1000
—num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss
3.584	0.0878	1000	0.1139
3.4937	0.1755	2000	0.1080
3.3486	0.2633	3000	0.1059
3.4019	0.3511	4000	0.1044
3.2915	0.4388	5000	0.1033
3.2729	0.5266	6000	0.1023
3.2451	0.6144	7000	0.1015
3.2339	0.7022	8000	0.1009
3.2383	0.7899	9000	0.1005
3.2441	0.8777	10000	0.1003
3.2744	0.9655	11000	0.1002

Framework versions

—Transformers 4.48.0.dev0
—Pytorch 2.5.1+cu124
—Datasets 3.2.0
—Tokenizers 0.21.0