KeisukeMiyamoto/CleanedFineWeb2Edu-jp

CleanedFineWeb2Edu-jp CleanedFineWeb2Edu-jp is a cleaned Japanese web text dataset. This dataset was created from the sample_10BT subset of hotchpotch/fineweb-2-edu-japanese. The source text was refined with MK0727/corpus-refiner-jp. Purpose The main purpose of this dataset is to provide cleaner Japanese web text for language model pretraining and continued pretraining. This dataset keeps Japanese web documents from FineWeb2-Edu while reducing boilerplate… See the full description on the dataset page: https://huggingface.co/datasets/KeisukeMiyamoto/CleanedFineWeb2Edu-jp.