boapps/szurkemarha-rwkv5-3B

finomhangolás folyamatban (steps: 868/1000, ami kb 3 epoch)

magyar nyelvű instrukciókövető modell, lora finomhangolva a szurkemarha korpuszon (több magyar korpusz egyesítése)

használt repo: https://github.com/OpenMOSE/RWKV5-LM-LoRA

inferencehez én ezt használom: https://github.com/RWKV/rwkv.cpp

finomhangolási opciók (gyakorlatilag a default):

{'loadmodel': 'model/RWKV-5-World-3B-v2-20231113-ctx4096.pth', 'wandb': '', 'projdir': 'output', 'randomseed': -1, 'datafile': 'data', 'datatype': 'binidx', 'vocabsize': 65536, 'ctxlen': 4096, 'epochsteps': 200, 'epochcount': 1000, 'epochbegin': 0, 'epochsave': 2, 'microbsz': 1, 'nlayer': 32, 'nembd': 2560, 'dimatt': 2560, 'dimffn': 8960, 'preffn': 0, 'headqk': 0, 'tinyattdim': 0, 'tinyattlayer': -999, 'lrinit': 0.0001, 'lrfinal': 9e-06, 'warmupsteps': 200, 'beta1': 0.9, 'beta2': 0.999, 'adameps': 1e-08, 'gradcp': 1, 'dropout': 0, 'weightdecay': 0, 'weightdecayfinal': -1, 'mypileversion': 1, 'mypilestage': 0, 'mypileshift': -1, 'mypileedecay': 0, 'layerwiselr': 1, 'dsbucketmb': 200, 'mysamplelen': 0, 'myffnshift': 1, 'myattshift': 1, 'headsizea': 64, 'headsizedivisor': 8, 'myposemb': 0, 'loadpartial': 0, 'magicprime': 0, 'myqamask': 0, 'myrandomsteps': 0, 'mytesting': 'r2r3r4', 'myexit': 99999999, 'myexittokens': 0, 'lora': True, 'loraload': '', 'lorar': 8, 'loraalpha': 16.0, 'loradropout': 0.01, 'loraparts': 'att,ffn,time,ln', 'logger': False, 'enablecheckpointing': False, 'defaultrootdir': None, 'gradientclipval': 1.0, 'gradientclipalgorithm': None, 'numnodes': 1, 'numprocesses': None, 'devices': '1', 'gpus': None, 'autoselectgpus': False, 'tpucores': None, 'ipus': None, 'enableprogressbar': True, 'overfitbatches': 0.0, 'trackgradnorm': -1, 'checkvaleverynepoch': 100000000000000000000, 'fastdevrun': False, 'accumulategradbatches': None, 'maxepochs': -1, 'minepochs': None, 'maxsteps': -1, 'minsteps': None, 'maxtime': None, 'limittrainbatches': None, 'limitvalbatches': None, 'limittestbatches': None, 'limitpredictbatches': None, 'valcheckinterval': None, 'logeverynsteps': 100000000000000000000, 'accelerator': 'gpu', 'strategy': 'deepspeedstage2offload', 'syncbatchnorm': False, 'precision': 'bf16', 'enablemodelsummary': True, 'weightssavepath': None, 'numsanityvalsteps': 0, 'resumefromcheckpoint': None, 'profiler': None, 'benchmark': None, 'deterministic': None, 'reloaddataloaderseverynepochs': 0, 'autolrfind': False, 'replacesamplerddp': False, 'detectanomaly': False, 'autoscalebatchsize': False, 'plugins': None, 'ampbackend': 'native', 'amplevel': None, 'movemetricstocpu': False, 'multipletrainloadermode': 'maxsizecycle', 'mytimestamp': '2024-03-25-21-56-16', 'betas': (0.9, 0.999), 'realbsz': 1, 'runname': '65536 ctx4096 L32 D2560'}

lásd a train_log filet több infoért