Datasetpublic

pszemraj/midjourney-messages-cleaned

midjourney-messages-cleaned This is vivym/midjourney-messages but with the following cleaning steps: remove most columns (keep id columns for reference vs. original) Apply clean-text to all rows (keep casing) rename content to text (ffs) remove intermediate ID/tag (???) in angle brackets at the end, remove double asterisks ** remove exact duplicate rows dataset structure overall: DatasetDict({ train: Dataset({ features: ['id', 'channel_id', 'text']… See the full description on the dataset page: https://huggingface.co/datasets/pszemraj/midjourney-messages-cleaned.

sourceHugging Faceapache-2.0updated 9mo agoView on Hugging Face

8likes141downloads

Dataset Card

midjourney-messages-cleaned

This is vivym/midjourney-messages but with the following cleaning steps:

—remove most columns (keep id columns for reference vs. original)
—Apply clean-text to all rows (keep casing)
—rename content to text (ffs)
—remove intermediate ID/tag (???) in angle brackets at the end, remove double asterisks **
—remove exact duplicate rows

dataset structure

overall:

python

DatasetDict({
    train: Dataset({
        features: ['id', 'channel_id', 'text'],
        num_rows: 19738964
    })
})

A single example looks like this:

python

random.choice(dataset['train'])

{'id': '1108635049391308879',
 'channel_id': '1008571088919343124',
 'text': 'Warhammer 40k Chaos Space Marine with pink Armor and a guitar'}

details

585M GPT-4 tiktoken tokens.

token_count
count  1.971668e+07
mean   2.971651e+01
std    3.875208e+01
min    1.000000e+00
25%    1.000000e+01
50%    1.900000e+01
75%    3.400000e+01
max    2.077000e+03