Datasetpublic

OpenSakura/OpenSakura-DS-260220-LN-ja-zh-PT-Adam

OpenSakura Adam LN Pretrain Dataset OpenSakura-DS-260220-LN-ja-zh-PT-Adam is a large-scale pretraining corpus built from light-novel source shards and filtered to Japanese/Chinese scripts. This export is intended for PT/CPT-style language modeling. Dataset Summary Metric Value Dataset ID OpenSakura/OpenSakura-DS-260220-LN-ja-zh-PT-Adam Total rows 9,515,512 Total parquet files 480 Total size 63,621,025,693 bytes (~63.62 GB, ~59.25 GiB) Languages… See the full description on the dataset page: https://huggingface.co/datasets/OpenSakura/OpenSakura-DS-260220-LN-ja-zh-PT-Adam.

sourceHugging Faceotherupdated 7mo agoView on Hugging Face

4likes309downloads

Dataset Card

OpenSakura Adam LN Pretrain Dataset

OpenSakura-DS-260220-LN-ja-zh-PT-Adam is a large-scale pretraining corpus built from light-novel source shards and filtered to Japanese/Chinese scripts. This export is intended for PT/CPT-style language modeling.

Dataset Summary

Metric	Value
Dataset ID	`OpenSakura/OpenSakura-DS-260220-LN-ja-zh-PT-Adam`
Total rows	9,515,512
Total parquet files	480
Total size	63,621,025,693 bytes (~63.62 GB, ~59.25 GiB)
Languages (BCP-47)	`ja`, `zh-Hans`, `zh-Hant`
Domain	Light Novel (LN)
Training type	PT (pretrain)

Split Information

Split	Rows	Share
train	6,662,783	70.02%
arena	474,135	4.98%
reserve	475,823	5.00%
validation	952,813	10.01%
test	949,958	9.98%

Public Schema

Each row contains:

—uuid (string)
—text (string)
—lang (string)
—token_count (int64)
—glossary_id (string)
—glossary (list[struct{term, translation, comment}])

Usage

python

from datasets import load_dataset

dataset = load_dataset("OpenSakura/OpenSakura-DS-260220-LN-ja-zh-PT-Adam")
train = load_dataset("OpenSakura/OpenSakura-DS-260220-LN-ja-zh-PT-Adam", split="train")

Limitations and Intended Use

—This is a pretraining corpus (text + lang), not aligned translation pairs.
—Language filtering/detection can still make mistakes on short, noisy, or mixed-script lines.
—Light-novel content may include mature/sensitive text (violence/sexual content/profanity).
—Intended for research/model development; evaluate upstream rights and redistribution constraints before commercial use.