CoolFace
Datasetpublic

BEE-spoke-data/upvoteweb-posts

upvoteweb: posts Posts in upvoteweb. configs [!IMPORTANT]There are several configs representing different permutations of this dataset. Load the relevant config for the task you are interested in. Overview of configs: default: largely unfiltered/unprocessed original data eduscored: the "eduscore" predicted on the text column with huggingface's trained classifier en-clean: filter language for en and language_score for > 0.6. Run clean-text on the text col… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/upvoteweb-posts.

sourceHugging Faceodc-byupdated 9mo agoView on Hugging Face
1likes172downloads
Dataset Card

upvoteweb: posts

Posts in upvoteweb.

configs

[!IMPORTANT] There are several configs representing different permutations of this dataset. Load the relevant config for the task you are interested in.

Overview of configs:

  • —default: largely unfiltered/unprocessed original data
  • —eduscored: the "eduscore" predicted on the text column with huggingface's trained classifier
  • —en-clean: filter language for en and language_score for > 0.6. Run clean-text on the text col, preserving casing
  • —image-dataset-sample: filter for posts with score > 2000, at least one URL that points to an image. Load the first image URL into a new image col, and resize it to have a max dim of 2048

Pass the config name as a second fn arg when loading the dataset to load that config. For example, to load eduscored:

py
from datasets import load_dataset

ds = load_dataset("BEE-spoke-data/upvoteweb-posts", "eduscored")
ds

lang details

Some simple counts in default:

language
en     14745361
de       271905
es       176108
pt       138175
fr       126898
         ...   
kv            2
tyv           1
myv           1
hif           1
lez           1
Name: count, Length: 171, dtype: int64