Datasetpublic

ManBib/DUE-v2-quality-filtered

removed markdown formatting, links, emojis, multiple white spaces replaced user and channel tags with @user and #channel deduplicated (not fuzzy only exact) basic filtering: word count filter(min_words=3, max_words=200) mean word length filter(min_mean_word_length=2.0, max_mean_word_length=12.0) long word filter(max_word_length=200) whitespace filter (max_white_space_ratio=0.25)

sourceHugging Facemitupdated 1y agoView on Hugging Face

0likes299downloads

Dataset Card

—removed markdown formatting, links, emojis, multiple white spaces
—replaced user and channel tags with @user and #channel
—deduplicated (not fuzzy only exact)
—basic filtering:
—word count filter(minwords=3, maxwords=200)
—mean word length filter(minmeanwordlength=2.0, maxmeanwordlength=12.0)
—long word filter(maxwordlength=200)
—whitespace filter (maxwhitespace_ratio=0.25)