MasonMac/WildChat-4M-English-Semantic-Deduplicated
ALERT: This dataset has been superseded by https://huggingface.co/datasets/MasonMac/WildChat-4.8M-EN-Semantic-Deduplicated This is a dataset of all English prompts from the WildChat-4M dataset. It was created by checking that at least 80% non-punctuation characters were in the English alphabet (to remove some more non-English entries). Then, it was deduplicated (ignoring punctuation/whitespace differences) by collecting to a HashSet. Another deduplication pass was done with MinHash. And… See the full description on the dataset page: https://huggingface.co/datasets/MasonMac/WildChat-4M-English-Semantic-Deduplicated.
ALERT: This dataset has been superseded by https://huggingface.co/datasets/MasonMac/WildChat-4.8M-EN-Semantic-Deduplicated
This is a dataset of all English prompts from the WildChat-4M dataset. It was created by checking that at least 80% non-punctuation characters were in the English alphabet (to remove some more non-English entries). Then, it was deduplicated (ignoring punctuation/whitespace differences) by collecting to a HashSet. Another deduplication pass was done with MinHash. And finally, it was deduplicated semantically using Qwen-4B-Embedding using HNSW with HCS where entries were clustered based on >=0.75 similarity.
Currently, this only includes prompts that are <= ~2000 tokens. The rest will be done separately with 8B and added at a later time.
(Note that source_idx does not correspond to the source WildChat-4M dataset, but to an intermediary in the data cleaning process.)