Datasetpublic

ce-lery/corpus-ja-11b

corpus-ja-11b Welcome to this repository.This dataset is japanese corpus that includes wiki, wikibooks, wikiversity, and cc100. Getting Started If you want to use this, please run as follows.This process takes about 3 hours. mkdir -p pretrain/input/ cd pretrain/input/ GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/ce-lery/corpus-ja-11b.git cd corpus-ja-11b git lfs pull bash merge_train.sh

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes17downloads

Dataset Card

corpus-ja-11b

Welcome to this repository. This dataset is japanese corpus that includes wiki, wikibooks, wikiversity, and cc100.

Getting Started

If you want to use this, please run as follows. This process takes about 3 hours.

bash

mkdir -p pretrain/input/
cd pretrain/input/

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/ce-lery/corpus-ja-11b.git
cd corpus-ja-11b
git lfs pull 
bash merge_train.sh