Modelpublic

tomekkorbak/romantic_bose

sourceHugging Facemitupdated 4y agoView on Hugging Face

0likes

Model Card

romantic_bose

This model was trained from scratch on the tomekkorbak/pii-pile-chunk3-0-50000, the tomekkorbak/pii-pile-chunk3-50000-100000, the tomekkorbak/pii-pile-chunk3-100000-150000, the tomekkorbak/pii-pile-chunk3-150000-200000, the tomekkorbak/pii-pile-chunk3-200000-250000, the tomekkorbak/pii-pile-chunk3-250000-300000, the tomekkorbak/pii-pile-chunk3-300000-350000, the tomekkorbak/pii-pile-chunk3-350000-400000, the tomekkorbak/pii-pile-chunk3-400000-450000, the tomekkorbak/pii-pile-chunk3-450000-500000, the tomekkorbak/pii-pile-chunk3-500000-550000, the tomekkorbak/pii-pile-chunk3-550000-600000, the tomekkorbak/pii-pile-chunk3-600000-650000, the tomekkorbak/pii-pile-chunk3-650000-700000, the tomekkorbak/pii-pile-chunk3-700000-750000, the tomekkorbak/pii-pile-chunk3-750000-800000, the tomekkorbak/pii-pile-chunk3-800000-850000, the tomekkorbak/pii-pile-chunk3-850000-900000, the tomekkorbak/pii-pile-chunk3-900000-950000, the tomekkorbak/pii-pile-chunk3-950000-1000000, the tomekkorbak/pii-pile-chunk3-1000000-1050000, the tomekkorbak/pii-pile-chunk3-1050000-1100000, the tomekkorbak/pii-pile-chunk3-1100000-1150000, the tomekkorbak/pii-pile-chunk3-1150000-1200000, the tomekkorbak/pii-pile-chunk3-1200000-1250000, the tomekkorbak/pii-pile-chunk3-1250000-1300000, the tomekkorbak/pii-pile-chunk3-1300000-1350000, the tomekkorbak/pii-pile-chunk3-1350000-1400000, the tomekkorbak/pii-pile-chunk3-1400000-1450000, the tomekkorbak/pii-pile-chunk3-1450000-1500000, the tomekkorbak/pii-pile-chunk3-1500000-1550000, the tomekkorbak/pii-pile-chunk3-1550000-1600000, the tomekkorbak/pii-pile-chunk3-1600000-1650000, the tomekkorbak/pii-pile-chunk3-1650000-1700000, the tomekkorbak/pii-pile-chunk3-1700000-1750000, the tomekkorbak/pii-pile-chunk3-1750000-1800000, the tomekkorbak/pii-pile-chunk3-1800000-1850000, the tomekkorbak/pii-pile-chunk3-1850000-1900000 and the tomekkorbak/pii-pile-chunk3-1900000-1950000 datasets.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 0.0001
—trainbatchsize: 16
—evalbatchsize: 8
—seed: 42
—gradientaccumulationsteps: 8
—totaltrainbatch_size: 128
—optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
—lrschedulertype: linear
—lrschedulerwarmup_ratio: 0.01
—training_steps: 12588
—mixedprecisiontraining: Native AMP

Framework versions

—Transformers 4.24.0
—Pytorch 1.11.0+cu113
—Datasets 2.5.1
—Tokenizers 0.11.6

Full config

{'dataset': {'datasets': ['tomekkorbak/pii-pile-chunk3-0-50000', 'tomekkorbak/pii-pile-chunk3-50000-100000', 'tomekkorbak/pii-pile-chunk3-100000-150000', 'tomekkorbak/pii-pile-chunk3-150000-200000', 'tomekkorbak/pii-pile-chunk3-200000-250000', 'tomekkorbak/pii-pile-chunk3-250000-300000', 'tomekkorbak/pii-pile-chunk3-300000-350000', 'tomekkorbak/pii-pile-chunk3-350000-400000', 'tomekkorbak/pii-pile-chunk3-400000-450000', 'tomekkorbak/pii-pile-chunk3-450000-500000', 'tomekkorbak/pii-pile-chunk3-500000-550000', 'tomekkorbak/pii-pile-chunk3-550000-600000', 'tomekkorbak/pii-pile-chunk3-600000-650000', 'tomekkorbak/pii-pile-chunk3-650000-700000', 'tomekkorbak/pii-pile-chunk3-700000-750000', 'tomekkorbak/pii-pile-chunk3-750000-800000', 'tomekkorbak/pii-pile-chunk3-800000-850000', 'tomekkorbak/pii-pile-chunk3-850000-900000', 'tomekkorbak/pii-pile-chunk3-900000-950000', 'tomekkorbak/pii-pile-chunk3-950000-1000000', 'tomekkorbak/pii-pile-chunk3-1000000-1050000', 'tomekkorbak/pii-pile-chunk3-1050000-1100000', 'tomekkorbak/pii-pile-chunk3-1100000-1150000', 'tomekkorbak/pii-pile-chunk3-1150000-1200000', 'tomekkorbak/pii-pile-chunk3-1200000-1250000', 'tomekkorbak/pii-pile-chunk3-1250000-1300000', 'tomekkorbak/pii-pile-chunk3-1300000-1350000', 'tomekkorbak/pii-pile-chunk3-1350000-1400000', 'tomekkorbak/pii-pile-chunk3-1400000-1450000', 'tomekkorbak/pii-pile-chunk3-1450000-1500000', 'tomekkorbak/pii-pile-chunk3-1500000-1550000', 'tomekkorbak/pii-pile-chunk3-1550000-1600000', 'tomekkorbak/pii-pile-chunk3-1600000-1650000', 'tomekkorbak/pii-pile-chunk3-1650000-1700000', 'tomekkorbak/pii-pile-chunk3-1700000-1750000', 'tomekkorbak/pii-pile-chunk3-1750000-1800000', 'tomekkorbak/pii-pile-chunk3-1800000-1850000', 'tomekkorbak/pii-pile-chunk3-1850000-1900000', 'tomekkorbak/pii-pile-chunk3-1900000-1950000'], 'issplitbysentences': True, 'skiptokens': 1649999872}, 'generation': {'forcecallon': [25177], 'metricsconfigs': [{}, {'n': 1}, {'n': 2}, {'n': 5}], 'scenarioconfigs': [{'generatekwargs': {'dosample': True, 'maxlength': 128, 'minlength': 10, 'temperature': 0.7, 'topk': 0, 'topp': 0.9}, 'name': 'unconditional', 'numsamples': 2048}], 'scorerconfig': {}}, 'klgpt3callback': {'forcecallon': [25177], 'maxtokens': 64, 'numsamples': 4096}, 'model': {'fromscratch': False, 'gpt2configkwargs': {'reorderandupcastattn': True, 'scaleattnby': True}, 'modelkwargs': {'revision': '9e6c78543a6ff1e4089002c38864d5a9cf71ec90'}, 'pathorname': 'tomekkorbak/nervouswozniak'}, 'objective': {'name': 'MLE'}, 'tokenizer': {'pathorname': 'gpt2'}, 'training': {'dataloadernumworkers': 0, 'effectivebatchsize': 128, 'evaluationstrategy': 'no', 'fp16': True, 'hubmodelid': 'romanticbose', 'hubstrategy': 'allcheckpoints', 'learningrate': 0.0001, 'loggingfirststep': True, 'loggingsteps': 1, 'numtokens': 3300000000, 'outputdir': 'trainingoutput2', 'perdevicetrainbatchsize': 16, 'pushtohub': True, 'removeunusedcolumns': False, 'savesteps': 25177, 'savestrategy': 'steps', 'seed': 42, 'tokensalreadyseen': 1649999872, 'warmupratio': 0.01, 'weight_decay': 0.1}}

Wandb URL:

https://wandb.ai/tomekkorbak/apo/runs/2aqujp4e