Datasetpublic

zidcenek/GLAMI-Entity-Matching-Dataset

GLAMI Duplication Detection Product-duplicate detection over GLAMI e-commerce listings: ~1.3M product images plus multilingual titles, descriptions and attributes, with labelled groups of items that do or do not refer to the same physical product. Released under the Apache License 2.0 — see LICENSE. TODO: describe how the labels were produced. Structure Config Files Contents images images/shard-*.parquet itemId → image bytes, one row per product image… See the full description on the dataset page: https://huggingface.co/datasets/zidcenek/GLAMI-Entity-Matching-Dataset.

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes230downloads

Dataset Card

GLAMI Duplication Detection

Product-duplicate detection over GLAMI e-commerce listings: ~1.3M product images plus multilingual titles, descriptions and attributes, with labelled groups of items that do or do not refer to the same physical product.

Released under the Apache License 2.0 — see LICENSE.

TODO: describe how the labels were produced.

Structure

Config	Files	Contents
`images`	`images/shard-*.parquet`	`itemId` → image bytes, one row per product image
`items`	`items_train.csv`, `items_test.csv`	`itemId`, `price`, `colorTagIdsString`, `departmentIds`, `brandEditionTagId`, `title`, `description`, `geo`, and `label` on train
`splits`	`train_split.csv`, `val_split.csv`	train/validation partition of the training items
`groups`	`groups_*.csv`	`item1..itemN`, `label` — candidate groups of 5–10 items, plus `hard_` and `category_` adversarial variants

Images are keyed by itemId alone and are not pre-split. Every other file references itemId, so all of them join against the same image table and no picture is stored twice.

Usage

python

from datasets import load_dataset

images = load_dataset("zidcenek/GLAMI-Entity-Matching-Dataset", "images", split="train")
items  = load_dataset("zidcenek/GLAMI-Entity-Matching-Dataset", "items",  split="train")
groups = load_dataset("zidcenek/GLAMI-Entity-Matching-Dataset", "groups", split="g5")

images[0]["image"]    # PIL.Image, decoded lazily
images[0]["itemId"]

The image config is ~TODO GB, so stream it if you do not want a full local copy:

python

images = load_dataset(
    "zidcenek/GLAMI-Entity-Matching-Dataset", "images", split="train", streaming=True
)

To attach images to items, build the itemId → row-index map once and index into it:

python

index = {item_id: i for i, item_id in enumerate(images["itemId"])}
row = images[index[items[0]["itemId"]]]

Reproducing the image shards

bash

pip install -U datasets Pillow "huggingface_hub[hf_xet]"

python scripts/build_image_shards.py --inspect    # check the filename -> itemId mapping
python scripts/build_image_shards.py              # glami_images.tar.gz -> images/shard-*.parquet
hf auth login
python scripts/upload_to_hub.py                   # push to the Hub over HTTP

Both scripts are resumable: rerun the same command after an interruption. See the header of each for the full options.

Citation

TODO