datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
test-dpo-perplexity
test-dpo-perplexity
This DPO (Direct Preference Optimization) dataset was generated using watermarked text generation
with red/blue token parity sampling. Each prompt has both a red and blue answer for creating
preference pairs.
Watermark Configuration
Sampler Type: soft_watermark
Soft Mode: True
Soft Threshold: 0.95
Sampling Parameters
Temperature: 0.7
Top-p: 0.8
Top-k: 20
Max Tokens: 64
Statistics
Total Prompts: 3
Avg Red Parity Ratio: 0.5521… See the full description on the dataset page: https://huggingface.co/datasets/eac123/test-dpo-perplexity.ko-perplexity-corpus
Lumia101/ko-perplexity-corpus
This dataset was created to measure the perplexity of an LLM trained on a Korean dataset.
Dataset Source
HAERAE-HUB/KOREAN-WEBTEXT
maxidl/FineNews-unfiltered
wikimedia/wikipedia