Datasetpublic

ce-lery/merged-corpus

Merged Corpus Welcome to this repository.This dataset is japanese corpus that includes wiki, wikibooks, wikiversity, cc100, and oscar2109. Getting Started If you want to use this, please run as follows.This process takes about 3 hours. mkdir -p pretrain/input/ cd pretrain/input/ GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/ce-lery/merged-corpus.git cd merged-corpus git lfs pull bash merge_train.sh

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes31downloads

root

file.gitattributes3 KBdownload

filemerge_train.sh410 Bdownload

fileREADME.md512 Bdownload

filetrain_part_001.jsonl40.00 GBdownload

filetrain_part_002.jsonl40.00 GBdownload

filetrain_part_003.jsonl40.00 GBdownload

filetrain_part_004.jsonl40.00 GBdownload

filetrain_part_005.jsonl40.00 GBdownload

filetrain_part_006.jsonl40.00 GBdownload

filetrain_part_007.jsonl27.15 GBdownload

ce-lery/merged-corpus · main · files are served by the source, never re-hosted here