raulmodena/leire-corpus
Leire Corpus (EN) Tokenized pretraining corpus for Leire, a 343.7M-parameter Brazilian Portuguese LM trained from scratch on Kaggle T4s. ~15B tokens, 70% PT / 15% code / 8% math / 7% educational English, tokenized with a custom 32,768 BPE vocabulary trained on the same mixture. Shards are uint16 binaries; recipe and stats below (in Portuguese). Corpus de pre-treino da Leire, um LM de 343,7M de parametros em portugues brasileiro, treinado do zero em T4 do Kaggle. O projeto e… See the full description on the dataset page: https://huggingface.co/datasets/raulmodena/leire-corpus.
02.2k
1import os, sys, subprocess2os.environ['HF_HUB_DISABLE_PROGRESS_BARS'] = '1'3os.environ['HF_HUB_DISABLE_XET'] = '1'4try:5 from kaggle_secrets import UserSecretsClient6 os.environ['HF_TOKEN'] = UserSecretsClient().get_secret('HF_TOKEN')7 os.environ['HUGGING_FACE_HUB_TOKEN'] = os.environ['HF_TOKEN']8 print('TOKEN OK', bool(os.environ.get('HF_TOKEN')), flush=True)9except Exception as e:10 print('SEM TOKEN (segue sem publicar heartbeat):', type(e).__name__, flush=True)11subprocess.run([sys.executable, '-m', 'pip', 'install', '-q', '-U', 'huggingface_hub'])12from huggingface_hub import snapshot_download13snapshot_download('raulmodena/leire-corpus', repo_type='dataset', allow_patterns=['_codigo/**'], local_dir='/kaggle/tmpcode')14subprocess.run('cp -r /kaggle/tmpcode/_codigo/. /kaggle/working/', shell=True)15os.chdir('/kaggle/working')16sys.path.insert(0, '/kaggle/working')17print('BOOTSTRAP PISO OK', sorted(os.listdir('/kaggle/working')), flush=True)18r = subprocess.run([sys.executable, '-u', 'scripts/piso_real.py'], cwd='/kaggle/working')19print('PISO SAIDA', r.returncode, flush=True)20