datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
function-calling-chatml
Dataset Card for "function-calling-chatml"
Converted glaiveai/Glaive-function-calling-v2 to chatml format.
Example entry
[ { "from": "system", "value": "You are a helpful assistant with access to the following functions. Use them if required -{\n \"name\": \"create_contact\",\n \"description\": \"Create a new contact\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\n \"type\": \"string\",\n \"description\": \"The name of the contact\"\n }… See the full description on the dataset page: https://huggingface.co/datasets/Locutusque/function-calling-chatml.ultrachat-10k-chatmldolly-chatml-sftTiny-Short-R1-CoT-Distill-HF-ChatMLOnly Short CoT
Type: 'tiny' (Very Small CoT)
Type: 'short' (Small CoT)
Each CoT Should be less than 2048 tokens
Subset of glaiveai/reasoning-v1-20m
tldr-17-ChatMLKVQA-ChatML
KVQA-ChatML Dataset
이 데이터셋은 SKT의 KVQA 데이터셋을 ChatML 형식으로 변환한 것입니다.
데이터셋 정보
원본 데이터셋: skt/KVQA
총 샘플 수: 약 100,000개
파일 수: 100개 parquet 파일
포맷: ChatML (user/assistant 대화 형식)
데이터 구조
{
"image": PIL.Image,
"conversations": [
{"role": "user", "content": "질문"},
{"role": "assistant", "content": "답변"}
],
"id": "unique_id",
"source": "KVQA",
"answer_type": "answer_type",
"answerable": boolean
}
사용 방법
from datasets… See the full description on the dataset page: https://huggingface.co/datasets/ko-vlm/KVQA-ChatML.unified-uncensored-qwen-chatml-sft
Unified Uncensored Qwen SFT Dataset
This dataset is a mixed-license compilation of instruction/chat datasets converted into a single Qwen/ChatML-style text JSONL format.
Format
Each row has:
{
"text": "<|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n...<|im_end|>",
"source": "dataset/repo",
"source_format": "alpaca|sharegpt|messages|human_bot_text|prompt_response",
"source_license": "apache-2.0|mit|cc-by-4.0|cc-by-nc-4.0|other|unknown",
"source_family":… See the full description on the dataset page: https://huggingface.co/datasets/usamakenway/unified-uncensored-qwen-chatml-sft.medical-sci-instruct-100k-sharegpt-chatmlThis is a reformat of the database formatted for Mistral training with chatml tokens added to the tokenizer, and '<|endoftext|>' as an end of sequence token.
The max_length setting may need to be adjusted to include the additional tokens for training.
This max_length was set to 2030 from 2048 for extra headroom.
iv4-chatml-4k
Dataset Card for "iv4-chatml-4k"
More Information needed
OpenOrca-Traditional-Chinese-ChatML-Formatmlabonne-chatml-dpo-pairs-copyThis is a copy and unmaintained version of mlabonne/chatml_dpo_pairs that we use in TRL CI for testing purpose. Please refer to the original dataset for usage and more details
glaive-function-calling-v2-chatmli1_chatmlVietnamese-Locutusque-function-calling-chatml-gg-translateddetails_anthracite-org__magnum-v3-9b-chatml
Dataset Card for Evaluation run of anthracite-org/magnum-v3-9b-chatml
Dataset automatically created during the evaluation run of model anthracite-org/magnum-v3-9b-chatml.
The dataset is composed of 136 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest… See the full description on the dataset page: https://huggingface.co/datasets/OALL/details_anthracite-org__magnum-v3-9b-chatml.gpt-roleplay-realm-chatml
Follow me
HuggingFace: https://huggingface.co/AlekseyKorshuk
GitHub: https://github.com/AlekseyKorshuk
Twitter / X: https://x.com/alekseykorshuk
synthetic-dialog-summaries-processed-clean-chatmlsoda-chatmlFollow me:
HuggingFace: https://huggingface.co/AlekseyKorshuk
GitHub: https://github.com/AlekseyKorshuk
Twitter / X: https://x.com/alekseykorshuk
sharegpt_chatml_compressed
Dataset Card for "sharegpt_chatml_compressed"
Removed the chat template, also since the original datasest was a multiround conversation, I removed other rounds and kept only the first one to save context.
chain-of-thoughts-chatml
Follow me
HuggingFace: https://huggingface.co/AlekseyKorshuk
GitHub: https://github.com/AlekseyKorshuk
Twitter / X: https://x.com/alekseykorshuk
hhrlhf_evol_chatml
Dataset Card for "hhrlhf_evol_chatml"
More Information needed
reasoning_tulu_chatml_580k_all_lengthschatml_dpo_pairs
ChatML DPO Pairs
This is a preprocessed version of Intel/orca_dpo_pairs using the ChatML format.
Like the original dataset, it contains 12k examples from Orca style dataset Open-Orca/OpenOrca.
Here is the code used to preprocess it:
def chatml_format(example):
# Format system
if len(example['system']) > 0:
message = {"role": "system", "content": example['system']}
system = tokenizer.apply_chat_template([message], tokenize=False)
else:
system = ""… See the full description on the dataset page: https://huggingface.co/datasets/mlabonne/chatml_dpo_pairs.nemotron-sft-code-focused-stage1-2-ChatML
Nemotron SFT Dataset (Chat Template Formatted)
Overview
This dataset is a curated supervised fine-tuning (SFT) dataset built from NVIDIA's Nemotron-Cascade-SFT-Stage-1 and Stage-2 datasets.
Important: This dataset uses the tokenizer's apply_chat_template() method to properly format conversations from the original messages/conversations fields.
Statistics
Total Samples: 50,000
Total Tokens: 415,605,764
Average Tokens per Sample: 8312.1
Tokenizer:… See the full description on the dataset page: https://huggingface.co/datasets/kshitijthakkar/nemotron-sft-code-focused-stage1-2-ChatML.iv4-chatml-16k
Dataset Card for "iv4-chatml-16k"
More Information needed
sharegpt-chatml
Dataset Card for "sharegpt-chatml"
Data preprocessing pipeline: https://github.com/AlekseyKorshuk/chat-data-pipeline
nemotron-sft-advanced-stage1-2-ChatML-V1
Nemotron SFT Dataset (Chat Template Formatted)
Overview
This dataset is a curated supervised fine-tuning (SFT) dataset built from NVIDIA's Nemotron-Cascade-SFT-Stage-1 and Stage-2 datasets.
Important: This dataset uses the tokenizer's apply_chat_template() method to properly format conversations from the original messages/conversations fields.
Statistics
Total Samples: 50,000
Total Tokens: 306,946,919
Average Tokens per Sample: 6138.9
Tokenizer:… See the full description on the dataset page: https://huggingface.co/datasets/kshitijthakkar/nemotron-sft-advanced-stage1-2-ChatML-V1.arc_cot-chatmlnemotron-sft-benchmark-focused-stage1-2-ChatML-V1
Nemotron SFT Dataset (Chat Template Formatted)
Overview
This dataset is a curated supervised fine-tuning (SFT) dataset built from NVIDIA's Nemotron-Cascade-SFT-Stage-1 and Stage-2 datasets.
Important: This dataset uses the tokenizer's apply_chat_template() method to properly format conversations from the original messages/conversations fields.
Statistics
Total Samples: 50,000
Total Tokens: 428,330,639
Average Tokens per Sample: 8566.6
Tokenizer:… See the full description on the dataset page: https://huggingface.co/datasets/kshitijthakkar/nemotron-sft-benchmark-focused-stage1-2-ChatML-V1.orca-gpt4-chatml
Dataset Card for "orca-gpt4-chatml"
As of 7/21/23, the OpenOrca dataset has something malformed, causing a crash when you try to load it in dataset.
The GPT-4 data looks good though, so I preprocess it and push it up here in ChatML format.