OpenSakura/OpenSakura-DS-260220-LN-ja-zh-COT-Lilith

OpenSakura Lilith LN COT Dataset

OpenSakura-DS-260220-LN-ja-zh-COT-Lilith is the COT/segment-level derivative built from the same LN source stream, with reasoning_content preserved. Stats below are computed from the actual generated parquet files.

Dataset Summary

Split Information

Split Policy

This release uses a mapping-based 5-way split with stable labels per uuid:

• —train (70%): training split
• —arena (5%): reserved for human preference collection / comparisons
• —reserve (5%): sealed holdout for future use
• —validation (10%): model selection / early stopping
• —test (10%): final evaluation (do not train on)

Dataset Lineage

• —Source of candidates: Phoenix DB strict-success translation rows (ja → zh*) that pass a strict latest-trace gate.
• —Subset constraint: rows must exist in the Adam mapping (strict Adam subset).
• —Episode/segment mapping: this dataset is built at segment level (with segment_index and line boundaries) and tries to map segments back to episode UUIDs using the Eve HF release and/or mapping metadata.
• —Final filters: drops non-publishable segments (missing mapping, empty fields, line mismatch, length-ratio outliers) before tokenization + sharding.

Public Schema

Each row contains:

• —uuid (string)
• —episode_uuid (string)
• —segment_index (int32)
• —start_line_index (int32)
• —end_line_index (int32)
• —input (string)
• —output (string)
• —reasoning_content (string)
• —source_lang (string)
• —target_lang (string)
• —glossary_uuid (string | null)
• —glossary (list[struct{term, translation, comment}])
• —gen_model, gen_frequency_penalty, gen_max_tokens, gen_temperature, gen_top_p
• —input_token_count (int64), output_token_count (int64), reasoning_content_token_count (int64)

Tokenization and Generation Metadata

• —Token counts are computed with tokenizer model zai-org/GLM-4.7.
• —Token-counted columns: input_token_count, output_token_count, reasoning_content_token_count.
• —Generation metadata for this release:
• —gen_model: GLM-4.7
• —gen_max_tokens: 10240
• —gen_temperature: 0.6
• —gen_top_p: 0.95
• —gen_frequency_penalty: 0.0

Usage

from datasets import load_dataset

dataset = load_dataset("OpenSakura/OpenSakura-DS-260220-LN-ja-zh-COT-Lilith")
train = load_dataset("OpenSakura/OpenSakura-DS-260220-LN-ja-zh-COT-Lilith", split="train")

Acknowledgments

This dataset would not exist without the generous support and contributions of the following individuals:

• —@ixgbe -- Professional guidance and hands-on help with Kubernetes infrastructure that powers the OpenSakura pipeline.
• —An anonymous group member -- Generous sponsorship of H200 compute nodes. Without their support, OpenSakura could not have been founded and this dataset could not have been produced.
• —@Josepha -- Invaluable contributions to dataset cleaning and processing techniques that shaped the quality of this release.
• —@lildub -- LLM API sponsorship during the early experimental stage of the project, enabling the initial research and prototyping.
• —[@neavo](https://github.com/neavo) -- Inspiration and solid implementation of the LinguaGacha open-source project, which informed the design of the translation API used in this project.

Limitations and Intended Use

• —This is machine-generated translation + COT/trace-derived data and may contain errors, inconsistencies, or noise.
• —reasoning_content may include hallucinations or undesirable internal reasoning artifacts; use with care.
• —Light-novel content may include mature/sensitive text (violence/sexual content/profanity).
• —Intended for research/model development; evaluate upstream rights and redistribution constraints before commercial use.