CoolFace
Datasetpublic

Bingsu/laion-translated-to-en-korean-subset

laion-translated-to-en-korean-subset About dataset a subset data of laion/laion2B-multi-joined-translated-to-en and laion/laion1B-nolang-joined-translated-to-en, including only korean Lisence CC-BY-4.0 Data Structure Data Instance >>> from datasets import load_dataset >>> dataset = load_dataset("Bingsu/laion-translated-to-en-korean-subset") >>> dataset DatasetDict({ train: Dataset({ features: ['hash', 'URL'… See the full description on the dataset page: https://huggingface.co/datasets/Bingsu/laion-translated-to-en-korean-subset.

sourceHugging Facecc-by-4.0updated 4y agoView on Hugging Face
7likes190downloads
Dataset Card

laion-translated-to-en-korean-subset

Dataset Description

  • —Homepage: laion-5b
  • —Download Size 1.40 GiB
  • —Generated Size 3.49 GiB
  • —Total Size 4.89 GiB

About dataset

a subset data of laion/laion2B-multi-joined-translated-to-en and laion/laion1B-nolang-joined-translated-to-en, including only korean

Lisence

CC-BY-4.0

Data Structure

Data Instance

py
>>> from datasets import load_dataset
>>> dataset = load_dataset("Bingsu/laion-translated-to-en-korean-subset")
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['hash', 'URL', 'TEXT', 'ENG TEXT', 'WIDTH', 'HEIGHT', 'LANGUAGE', 'similarity', 'pwatermark', 'punsafe', 'AESTHETIC_SCORE'],
        num_rows: 12769693
    })
})
py
>>> dataset["train"].features
{'hash': Value(dtype='int64', id=None),
 'URL': Value(dtype='large_string', id=None),
 'TEXT': Value(dtype='large_string', id=None),
 'ENG TEXT': Value(dtype='large_string', id=None),
 'WIDTH': Value(dtype='int32', id=None),
 'HEIGHT': Value(dtype='int32', id=None),
 'LANGUAGE': Value(dtype='large_string', id=None),
 'similarity': Value(dtype='float32', id=None),
 'pwatermark': Value(dtype='float32', id=None),
 'punsafe': Value(dtype='float32', id=None),
 'AESTHETIC_SCORE': Value(dtype='float32', id=None)}

Data Size

download: 1.40 GiB<br> generated: 3.49 GiB<br> total: 4.89 GiB

Data Field

  • —'hash': int
  • —'URL': string
  • —'TEXT': string
  • —'ENG TEXT': string, null data are dropped
  • —'WIDTH': int, null data are filled with 0
  • —'HEIGHT': int, null data are filled with 0
  • —'LICENSE': string
  • —'LANGUAGE': string
  • —'similarity': float32, CLIP similarity score, null data are filled with 0.0
  • —'pwatermark': float32, Probability of containing a watermark, null data are filled with 0.0
  • —'punsafe': float32, Probability of nsfw image, null data are filled with 0.0
  • —'AESTHETIC_SCORE': float32, null data are filled with 0.0

Data Splits

train
# of data12769693

polars

sh
pip install polars[fsspec]
py
import polars as pl
from huggingface_hub import hf_hub_url

url = hf_hub_url("Bingsu/laion-translated-to-en-korean-subset", filename="train.parquet", repo_type="dataset")
# url = "https://huggingface.co/datasets/Bingsu/laion-translated-to-en-korean-subset/resolve/main/train.parquet"
df = pl.read_parquet(url)

pandas broke my colab session.