telecomadm1145/esjzone_2024
Dataset Card for telecomadm1145/esjzone_2024 Dataset Details Dataset Description Novels from Esjzone. Check telecomadm1145/creative_writing for instruction finetuning and telecomadm1145/esjzone_2024_chunked_8k for continuation. Curated by: telecomadm1145 Language(s) (NLP): Chinese License: MIT Dataset Sources Source Data: Publicly available online novels (especially light novels). Uses Better using DPP similar to… See the full description on the dataset page: https://huggingface.co/datasets/telecomadm1145/esjzone_2024.
Dataset Card for telecomadm1145/esjzone_2024
Dataset Details
Dataset Description
Novels from Esjzone.
Check telecomadm1145/creative_writing for instruction finetuning and telecomadm1145/esjzone_2024_chunked_8k for continuation.
- Curated by: telecomadm1145
- Language(s) (NLP): Chinese
- License: MIT
Dataset Sources
- Source Data: Publicly available online novels (especially light novels).
Uses
Better using DPP similar to below code:
from datasets import load_dataset
import random
hf_dataset = load_dataset("telecomadm1145/esjzone_2024_chunked_8k", split="train")
def make_prompt_response(example):
text = example["content"]
text = text[:3000] # TODO: Change if needed!
split_ratio = random.uniform(1/6, 1/2)
split_point = int(len(text) * split_ratio)
context = text[:split_point]
continuation = text[split_point:]
user_message = f"""请按照这段文字的大致逻辑和剧情，以及人物发展，续写。直接输出后续的可能文段。{context}
prompt = f"<start_of_turn>user\n{user_message}<end_of_turn>\n<start_of_turn>model\n" # Chat template maybe different
return {
"prompts": prompt,
"responses": continuation,
}
hf_dataset = hf_dataset.map(make_prompt_response)Dataset Structure
- Format: JSON
- Splits:
train- Fields:
book_title: Title of the book.chapter_title: Title of the chapter.content: Content of the chapter.