OrSabbach/food-delivery-support-tickets

Food Delivery Support Tickets (synthetic)

10,153 synthetic English customer-support conversations for a food delivery platform (à la Wolt / Uber Eats / DoorDash). Each record is a realistic customer message with structured labels and a professional agent resolution + reply.

Built for the Food Delivery Support Copilot — an assistant that classifies an incoming ticket, retrieves similar resolved cases, and drafts a reply.

How it was made

Generated locally with the Hugging Face model Qwen/Qwen2.5-3B-Instruct-GGUF (Q4KM) via llama-cpp-python (Apple Metal). The strategy is fully spec-conditioned: we sample each ticket's specification (category, urgency, sentiment, order context, and diversity levers — restaurant, city, tone, dish, issue sub-variant), and the model writes only the free text. Batches are category-homogeneous with category-specific few-shot exemplars to keep the small model on-topic.

Because the labels are sampled rather than annotated, category is ground truth by construction — the model was asked to write about that topic, and it did. sentiment and urgency are not: the model was asked to express them and largely did not. Both are measured and quantified below; please read that section before using either column.

See notebooks/01_data_generation.ipynb (generation) and src/ for the code.

Fields

EDA & cleaning

The dataset was cleaned in notebooks/02_eda.ipynb: exact + near-duplicate messages (TF-IDF cosine), length outliers, and sentiment/label mismatches (the main class of model mistake observed) were detected and removed. Category balance is preserved after cleaning. Figures are under reports/.

[image] [image]

Category distribution

Sentiment distribution

Urgency distribution

Known limitations — which labels you can trust

Short version: `category` is reliable. `sentiment` and `urgency` are not, and both fail for the same structural reason — they were sampled as part of the spec rather than derived from the text the generator produced. We measured both rather than assuming.

sentiment contradicts its own text ~84% of the time

We audited the cleaned data with an independent classifier (distilbert-base-uncased-finetuned-sst-2-english, disagreements counted only at ≥ 0.90 confidence). It is not part of the generation or cleaning pipeline, so its verdict is evidence rather than a restatement of our own heuristic.

About 84% of rows labeled `positive` read as negative to an independent model. This is an instruction-following failure, not random noise: generation is spec-conditioned, but all 8 categories are complaint categories, so a 3B model asked to write a "positive" message about being double-charged writes a complaint anyway. It follows the category and drops the sentiment instruction.

category is unaffected — the text follows the category faithfully, which is why a classifier on it reaches 98.8%.

If you use this dataset: treat sentiment as the requested sentiment rather than the expressed one. Our own app derives sentiment from the message text at inference time instead of trusting the column, and retrieval relevance is defined by category only.

[image]

See notebooks/02_eda.ipynb section 7b for the full analysis.

urgency has the same problem, less severely

A logistic-regression classifier trained on message embeddings reaches 44.7% accuracy against a 45.8% majority-class baseline — it is worse than always guessing `medium`. Supporting evidence: delay_minutes runs backwards with respect to the label (tickets marked high average a shorter delay than tickets marked low).

The cause is the same as for sentiment: urgency was sampled from category-conditional priors independently of the text the model then wrote, so the message never had to express it. The only recoverable structure is weak and category-level.

Treat `urgency` as a category-correlated random draw, not as a property of the message. See notebooks/04_classifier.ipynb section 3.

Embedding model selection

Three Hugging Face models were scored on the identical 3,000-row stratified sample, leave-one-out, relevance = same category:

Winner: `BAAI/bge-small-en-v1.5` — best Recall@5, statistically level with all-mpnet-base-v2 on the other metrics while being ~3x faster and ~3x smaller. Silhouette score against the category labels: 0.203.

Precomputed embeddings for all rows are in embeddings/embeddings.npy (float32, L2-normalized, row order identical to data/dataset.parquet).

[image]

See notebooks/03_embeddings_recsys.ipynb.

Reproducibility

To regenerate a dataset with identical structure, category balance, and label distributions:

1. 1.Open notebooks/colab_generate.ipynb in Google Colab.
2. 2.Runtime → Change runtime type → T4 GPU.
3. 3.Runtime → Run all (~30–40 min).
4. 4.Download food_support_raw.jsonl and run notebooks/02_eda.ipynb.

Reproducibility notes. The spec sampler uses seed=42, so every run produces the exact same sequence of (category, urgency, sentiment, ordercontext, dish, variant) tuples. The free-text fields (`customermessage, resolution, agentreply`) are LLM-generated with `temperature=0.8, topp=0.95`, so their exact wording varies run-to-run — this is intentional (deterministic decoding collapses to near-identical tickets and hurts recsys evaluation). Structural guarantees hold across runs: same 8 categories, same balance, same label distributions, same schema, same overall quality.

Model: Qwen/Qwen2.5-3B-Instruct (Hugging Face); Colab uses full fp16, local runs use the Q4KM GGUF via llama-cpp-python.

Intended use

Recommendation (retrieve similar resolved tickets via embeddings) and generation (draft agent replies). Synthetic data — not real customers.