datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-edu-10BT-mincols
fineweb-edu: 10BT sample
This the "10BT-sample" config of HuggingFaceFW/fineweb-edu with most of the redundant cols removed for efficiency reasons.
token counts
GPT-4 tiktoken token count:
token_count
count 9.672101e+06
mean 1.001188e+03
std 1.834986e+03
min 3.800000e+01
25% 3.380000e+02
50% 6.090000e+02
75% 1.054000e+03
max 1.649670e+05
Total count: 9683.59 M tokens
fineweb-edu-10BT-mincols
fineweb-edu: 10BT sample
This the "10BT-sample" config of HuggingFaceFW/fineweb-edu with most of the redundant cols removed for efficiency reasons.
token counts
GPT-4 tiktoken token count:
token_count
count 9.672101e+06
mean 1.001188e+03
std 1.834986e+03
min 3.800000e+01
25% 3.380000e+02
50% 6.090000e+02
75% 1.054000e+03
max 1.649670e+05
Total count: 9683.59 M tokens
cosmopedia-v2-mincols
cosmopedia-v2: mincols
cosmopedia-v2 with extra cols dropped to make the dataset smaller/easier to use
thinkerA Chain-of-Thought (CoT) dataset that contains traces of complex and sophisticated reasoning, to mimic the "thinking" process of OpenAI's o1. Wrap the contents of the reasoning column in some XML tag (such as <reasoning>).
Raw .jsonl dataset file can be found under the Files and Versions tab.
Thinker-XMLSystem prompt suggestion:
You are a world-class AI system. Always respond in strict XML format with your reasoning steps within the <im_reasoning> XML tag. Each reasoning step should represent one unit of thought. Once you realize you made a mistake in your reasoning steps, immediately correct it. Place your final response outside the XML tag. Adhere to this XML structure without exception.
MemGPT-Functions-DPOI hand crafted these. For MemGPT function calling.
Thinker-JSONUse for whatever you want.
Made to replicate the thought traces of OpenAI's o1, I'll release RL datasets including DPO soon enough.
For fine-tuning smaller models such as Google's google/gemma-2-2b-it with this dataset, I recommend fine-tuning for 2-3 epochs, the loss will be at around 1.6 at the beginning, and 1.3 by the end of the training job with learning rate of 2e-6.
Suggested system prompt:
Always respond in strict JSON format with a reasoning_steps array and a response field. Each… See the full description on the dataset page: https://huggingface.co/datasets/minchyeom/Thinker-JSON.Thinker-XML-2Suggested system prompt:
Respond to each user instruction in an XML format, using <step> tags to document your logical reasoning process step-by-step, while the <output> tag should be reserved for your final communication with the user. Incorporate self-correction by reflecting on prior steps; if a previous thought requires adjustment, add a new <step> to refine your reasoning without altering the original. Include self-reflection by periodically assessing your thought process and noting any… See the full description on the dataset page: https://huggingface.co/datasets/minchyeom/Thinker-XML-2.