Modelpublic

ambrosfitz/neural-history-chat-v1.0

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face

0likes20downloads

Model Card

Background

Textbook-Yawp-Openstax-merged datasets, is now applied the the Intel/Neural-Chat-7b-v3.1. The training peramters are below.

Key	Value
output_dir	"./results"
numtrainepochs	15
perdevicetrainbatchsize	32
gradientaccumulationsteps	4
optim	"pagedadamw8bit"
save_steps	500
logging_steps	5
learning_rate	1e-4
weight_decay	0.001
fp16	False
bf16	False
maxgradnorm	0.3
max_steps	-1
warmup_ratio	0.3
groupbylength	True
lrschedulertype	"linear"
report_to	"wandb"

Key	Value
train/epoch	▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
train/global_step	▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
train/learning_rate	▁▂▂▃▄▄▅▅▆▇▇███▇▇▇▇▆▆▆▆▆▅▅▅▄▄▄▄▃▃▃▃▂▂▂▂▁▁
train/loss	█▇▆▆▆▅▄▄▃▂▃▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
train/total_flos	▁
train/train_loss	▁
train/train_runtime	▁
train/trainsamplesper_second	▁
train/trainstepsper_second	▁
train/epoch	15.0
train/global_step	270
train/learning_rate	0.0
train/loss	0.4845
train/total_flos	3.550242517150925e+16
train/train_loss	0.89085
train/train_runtime	2501.3387
train/trainsamplesper_second	13.805
train/trainstepsper_second	0.108

https://huggingface.co/datasets/ambrosfitz/textbook-openstax-yawp-merge

https://huggingface.co/ambrosfitz/tinyllama-history-chat-v1.1