Datasetpublic

ce-lery/corpus-ja-11b

corpus-ja-11b Welcome to this repository.This dataset is japanese corpus that includes wiki, wikibooks, wikiversity, and cc100. Getting Started If you want to use this, please run as follows.This process takes about 3 hours. mkdir -p pretrain/input/ cd pretrain/input/ GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/ce-lery/corpus-ja-11b.git cd corpus-ja-11b git lfs pull bash merge_train.sh

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes15downloads

root

file.gitattributes3 KBdownload

filemerge_train.sh409 Bdownload

fileREADME.md501 Bdownload

filetrain_part_001.jsonl30.00 GBdownload

filetrain_part_002.jsonl30.00 GBdownload

filetrain_part_003.jsonl21.61 GBdownload

ce-lery/corpus-ja-11b · main · files are served by the source, never re-hosted here