Modelpublic

Yiran0924/bk-moo-llama3-8b-cagrad-fineweb

sourceHugging Facellama3updated 5mo agoView on Hugging Face

0likes6downloads

Model Card

BK-MOO-Pretrain · cagrad · fineweb

Continual pretraining checkpoint of `meta-llama/Meta-Llama-3-8B` following the BK-MOO-Pretrain recipe (https://github.com/longhp1618/BK-MOO-Pretrain).

—Method: cagrad
—Training stage: fineweb
—Base model: meta-llama/Meta-Llama-3-8B
—Languages: ar, bn, en, id, it, ko, sw, zh
—Training data: FineWeb-2 (and LIMA+S1 for *_limas1 stages)

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Yiran0924/bk-moo-llama3-8b-cagrad-fineweb"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="bfloat16", device_map="auto")

See the BK-MOO-Pretrain repo for the multi-objective gradient surgery details (MGDA / CAGrad / PCGrad / SGD).