lihicarmeli/fashion-stylist-multimodal

👗 Fashion Stylist Multimodal Dataset

A synthetic multimodal dataset pairing structured fashion metadata, styled outfit text, and AI-generated portraits.

🎯 Overview

This dataset contains ~1,000 synthetic fashion-styling profiles, each combining:

• —🧬 Structured demographic & style metadata
• —📝 A styled outfit description with e-commerce search queries
• —🖼️ A generated 512×512 studio-style portrait of a fictional person wearing the outfit

The dataset was built end-to-end using pre-trained Hugging Face models, inference-only — no fine-tuning. It was developed as a full data-science pipeline exercise: schema design → generation → validation → EDA → publishing, under real-world constraints (a single Colab T4 GPU running two models simultaneously).

A key goal of this card is full pipeline transparency — every generation choice below is traceable directly to the source code, including where the pipeline behaved differently than originally intended. ---

📊 Quick Facts

🗂️ Dataset Fields

⚙️ Generation Pipeline

Schema Definition ──▶ Attribute Sampling ──▶ TinyLlama Text Gen (+ Fallback) ──▶ SDXL Portrait Gen ──▶ Checkpointing ──▶ EDA ──▶ HF Hub

1. Schema definition — 6 skin tones × 3 undertones × 8 styles × 2 genders × 4 age groups define the attribute space (1,152 combinations).

2. Attribute sampling — for each of the 1,000 rows, attributes are drawn uniformly at random (with replacement).

3. Outfit text generation (TinyLlama) — a chat-formatted prompt requests a structured JSON outfit description (top, bottom, shoes, accessory, description, search queries).

• —If parsing fails, a rule-based fallback template is used instead ("A stylish {style} outfit in {colors} tones, perfect for {skin_tone} skin."), with shoes/accessory chosen randomly from a small per-style list.
• —If the whole call raises an exception, a second, simpler fallback is used (distinguishable by wording — it says "pants" instead of "trousers").

4. Portrait generation (SDXL) — a template prompt combining a fixed skin-tone phrase, a fixed style phrase, and the gender word renders a studio portrait, paired with a negative prompt to suppress common artifacts (cartoon look, blur, deformities, NSFW).

5. Checkpointing — progress saved to CSV every 100 rows for resilience against Colab disconnects.

6. Publishing — only rows with a successfully generated image are kept; images are cast to a Hugging Face Image feature and the dataset is pushed via push_to_hub.

✍️ Prompt Engineering

Text prompt (TinyLlama) — Role Prompting + Zero-Shot Structured Output

Uses TinyLlama's native chat template, assigning the model a persona and requesting direct JSON output with no examples:

prompt = (
    f"<|system|>\n"
    f"You are a fashion stylist. Output JSON only.\n</s>\n"
    f"<|user|>\n"
    f"Outfit for {gender}, {skin_tone} skin, {undertone} undertone, "
    f"{style} style, colors: {colors_str}.\n</s>\n"
    f"<|assistant|>\n"
)

Image prompt (SDXL) — Template-Based Positive/Negative Prompting

A flat descriptive prompt (not chat-formatted, matching SDXL's non-instruction-tuned nature):

prompt = (
    f"professional portrait photo of a {gender}, {skin_desc}, "
    f"{style_desc}, studio lighting, clean background, "
    f"high quality, realistic, fashion photography, 4k"
)

negative_prompt = (
    "cartoon, anime, blurry, low quality, "
    "deformed, extra limbs, nsfw, nude"
)

🤖 Models Used

Model sizes were chosen to fit both models in a single T4's VRAM simultaneously — a deliberate trade-off favoring pipeline feasibility over maximal text-generation quality.

🔍 Data Quality & Transparency Report

Rather than only listing generic caveats, this section documents specific, code-verified findings from auditing the generation pipeline — a mini data-quality audit rather than a disclaimer:

This audit was performed by comparing the published dataset statistics directly against the generation notebook, and is intended to give downstream users an accurate, verifiable picture of what the data actually contains — not just what the pipeline was designed to produce.

💻 Loading the Dataset

from datasets import load_dataset

ds = load_dataset("lihicarmeli/fashion-stylist-multimodal", split="train")
print(ds[0])
ds[0]["image_path"]  # PIL Image

Project Narrative

This project builds a personal color-styling app: a user uploads a selfie (or enters their features manually), and the app determines their color "season" (spring/summer/autumn/winter) and recommends matching outfits with real shopping links.

Data generation (Part 1) started with a multimodal dataset (fashion-stylist-multimodal, 1,000 rows) combining AI-generated outfit text (TinyLlama-1.1B-Chat) with AI-generated portraits (Stable Diffusion XL). A second, larger dataset (fashion-color-stylist, 10,000 rows) was then built on established seasonal color-theory rules rather than a language model — a deliberate choice, since color-season matching is a well-defined function of skin tone and undertone, and TinyLlama's raw output frequently failed to parse into valid structured data.

EDA (Part 2) on the multimodal dataset was iterative: an automated image-quality check (facebook/detr-resnet-50, counting people per photo) found only 68.6% of portraits usable, so we regenerated the failing images over several rounds, reaching 96.7% good. Missing values, duplicates, and class balance were checked on both datasets.

Recommendation (Part 3) compared three embedding models (all-MiniLM-L6-v2, all-mpnet-base-v2, all-MiniLM-L12-v2) on load time, encoding speed, and retrieval quality, and selected the best one. That model was then fine-tuned with Contrastive Loss on season-labeled sentence pairs drawn from both datasets — with the season name explicitly stripped from the input text beforehand to prevent label leakage. The fine-tuned model's embeddings were indexed with FAISS for fast retrieval. We also explored CLIP-based image embeddings on the multimodal dataset's portraits as a multimodal search path; this is documented here but is not wired into the deployed app, which serves recommendations from the text-based FAISS index only.

Generation (Part 4) in the live app uses BLIP (image captioning) and BLIP-VQA (visual question answering) together: BLIP-VQA answers direct questions about hair and eye color, with pixel-based skin-tone detection and a captioning fallback for hair, resolving disagreements with a pixel-darkness tie-break. This combination was chosen after finding that plain captioning rarely mentions eye color at all.

Application (Part 5) is a Gradio app on a Hugging Face Space, with both a photo-upload flow and a manual-selection flow feeding the same recommendation pipeline, plus three one-click quick-start examples. Beyond the core requirements, the app adds: a fine-tuned domain-specific embedding model; live shopping results via SerpAPI (Google Shopping), strictly filtered by gender; and a Telegram bot (@Fashion_look_match_bot) running the identical pipeline over chat, served via webhook when deployed and via polling when run locally.

<div align="center">

</div>