marcowxm/order-ingestion-service

Order Ingestion Service

Automates custom-order intake from Slack email forwards or Gmail into monday.com Pipeline cards, with team notifications in a second Slack channel.

Flows:

• —Slack (default): #sales-custom-orders webhook → parse → monday New Prospects + Files tab → notify #product-custom-orders
• —Gmail (optional): poll custom@weatherman.com (or any inbox) → same pipeline → same monday + Slack destinations

Full specification: automation_spec.md

Architecture overview

flowchart LR
  subgraph slackIn [Slack - Source]
    SC["#sales-custom-orders"]
  end

  subgraph gmailIn [Gmail - optional]
    GM["Inbox e.g. custom@weatherman.com"]
  end

  subgraph svc [Order Ingestion Service]
    WH["POST /webhooks/slack/new-order"]
    GJ["POST /jobs/gmail/ingest"]
    CRON["GitHub Actions cron"]
    HF["Hugging Face Space :7860"]
    LLM["AI Parser"]
    PIPE["process_order_pipeline"]
    WH --> PIPE
    GJ --> PIPE
    CRON --> GJ
    HF --> PIPE
    PIPE --> LLM
  end

  subgraph monday [monday.com]
    BRD["Pipeline board"]
    GRP["New Prospects group"]
    ITM["Item + Update"]
    FILES["Files tab"]
    BRD --> GRP --> ITM --> FILES
  end

  subgraph slackOut [Slack - Destination]
    PC["#product-custom-orders"]
  end

  SC -->|email forward + files| WH
  SC -->|Events API| HF
  GM -->|Gmail API poll| GJ
  PIPE -->|create item, update, upload file| ITM
  PIPE -->|post message + file thread| PC

What it does

When a mockup or proof handoff is ingested (explicit create/revise request in the latest reply):

1. 1.Extracts order number and company name (Groq openai/gpt-oss-20b by default).
2. 2.Creates or updates item #3278429 Valley Brook CC on Pipeline board, group New Prospects.
3. 3.Sets Owner (Sajjad Hussain / Sunny), Designer (Paula Bacolod), and monday column defaults (Status NEW, etc.).
4. 4.Adds an Update with the latest reply text (quoted thread stripped).
5. 5.Uploads logo/files to the item Files tab.
6. 6.Slack product posts are disabled by default (SLACK_NOTIFY_ENABLED=false) — monday is the source of truth for Paula.

Gmail path: when any @weatherman.com staff member (Sunny, Marco, etc.) emails Paula with uploadable attachments on that same email and the latest reply explicitly asks for a mock-up/proof. Accepted wording includes:

• —“please create / prepare / update / revise a mockup”
• —“can you please mock-up with…” (mock-up used as the verb)
• —“the client requested mockup…”

Historical mentions such as “I found the design mockup file” do not trigger by themselves. Links alone do not trigger ingest.

Company names are resolved in this order (first usable wins):

1. 1.Custom-order form Company/organization field in the thread
2. 2.Email signature / footer (e.g. mea-group, LifeSecure Insurance Company)
3. 3.Customer email domains (meagroup.net → MEA Group, properbrands.com → Proper Brands)
4. 4.Logo / attachment filenames (IndianSprings_Logo_CMYK.pdf → Indian Springs)
5. 5.Body / subject / LLM guess

Email-wrapper ZIP stems (re3612117youreceivedamessagevia…) and product lines (Walk, Travel & Trek, colors) are rejected as company names. Threads without a Weatherman # order are titled with the company name only and deduped by company on later handoffs.

monday update format (Slack notify off by default)

#3346526 Metro Squash
Status: NEW

Hi Paula,

The customer, Allyson Pooley from MetroSquash, would like to see a mock-up of their logo on white golf umbrellas...

• —No Priority, Design, Date Requested, or Mock-Up Due Date in monday updates.
• —Set SLACK_NOTIFY_ENABLED=true on HF only if product Slack posts are needed again.
• —Logo files appear in the Slack thread under the main message.

Pipeline sequence

sequenceDiagram
  autonumber
  participant Email as Email integration
  participant Sales as #sales-custom-orders
  participant API as FastAPI service
  participant LLM as Groq / OpenAI
  participant Monday as monday.com
  participant Product as #product-custom-orders

  Email->>Sales: Forward order email + attachment
  Sales->>API: Webhook or test_order.py
  API->>API: Keyword gate (mockup / proof)
  API->>LLM: Parse order_number, company, instructions
  LLM-->>API: Structured OrderExtraction
  API->>Sales: Fetch email body + resolve logo file
  Sales-->>API: Ruscittio Logo.eps (bytes)
  API->>Monday: create_item (Pipeline / New Prospects)
  API->>Monday: create_update (status + email text)
  API->>Monday: add_file_to_update (Files tab)
  API->>Product: chat.postMessage (summary + mentions)
  API->>Product: files v2 upload (logo in thread)

Attachment resolution (email + logo)

Slack email integration often uploads two files seconds apart. The service links them by timestamp:

flowchart TD
  A[Find email wrapper files matching order #] --> B[Get newest wrapper timestamp]
  B --> C["files.list ±30s in #sales-custom-orders"]
  C --> D{Filter}
  D -->|drop| E[HTML email wrappers]
  D -->|drop| F[Other orders / signatures]
  D -->|keep| G[Companion assets e.g. .eps logo]
  G --> H[Download via files.info]
  H --> I[Upload to monday Files tab]
  H --> J[Share in #product-custom thread]

Quick start

Prerequisites

• —Python 3.11+
• —Slack app with bot token (see Slack setup)
• —monday.com API token with boards:write
• —Groq or OpenAI API key

Install

git clone <repo-url>
cd order-ingestion-service
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env

Edit .env with your secrets (see .env.example for all variables).

Run the API (local)

uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

• —Health: GET http://localhost:8000/health
• —Webhook: POST http://localhost:8000/webhooks/slack/new-order
• —Gmail job (optional): POST http://localhost:8000/jobs/gmail/ingest with header X-Job-Secret if configured

For parity with production (Docker / Hugging Face), use port 7860 instead of 8000.

Run with Docker (local)

docker build -t order-ingestion-service .
docker run --rm -p 7860:7860 --env-file .env order-ingestion-service

• —Health: GET http://localhost:7860/health

Deploy to Hugging Face Spaces

The repo includes a production `Dockerfile` for a Docker Space on Hugging Face. The container runs Uvicorn on port 7860 (required by HF).

flowchart LR
  SLACK[Slack Events API] -->|HTTPS POST| HF["HF Space<br/>your-space.hf.space"]
  HF --> UV[Uvicorn :7860]
  UV --> APP[app.main:app]
  APP --> PIPE[process_order_pipeline]
  PIPE --> M[monday.com]
  PIPE --> P[#product-custom-orders]

Create the Space

1. 1.Create a new Space → SDK: Docker.
2. 2.Push this repository (or connect GitHub).
3. 3.Open Settings → Variables and secrets and add the same variables as .env (see Environment variables). At minimum:
4. 4.SLACK_SIGNING_SECRET
5. 5.SLACK_BOT_TOKEN
6. 6.MONDAY_API_KEY
7. 7.GROQ_API_KEY (or OPENAI_API_KEY)
8. 8.Wait for the Space to build from the Dockerfile.

Slack webhook URL

Point your Slack app Event Subscriptions request URL to:

https://<your-username>-<your-space>.hf.space/webhooks/slack/new-order

Verify with:

https://<your-username>-<your-space>.hf.space/health

Dockerfile summary

Secrets are not copied into the image — configure them in the Space settings or HF secrets UI.

Run a manual test

Exercises the full pipeline against live Slack + monday (creates a real item):

.venv/bin/python test_order.py

Expected result includes:

"detail": "Uploaded 1 file(s) to monday.com item Files tab; Shared 1 file(s) in Slack #product-custom-orders"

Gmail ingestion (recommended when Slack email display is incomplete)

The Slack “email” app often posts only the subject in #sales-custom-orders; the full body and PDFs live in the side pane and may not reach the Events API webhook. Gmail reads the full MIME message (body + attachments) directly.

Setup

1. 1.In Google Cloud Console, create an OAuth Desktop client for the workspace account that receives orders (e.g. custom@weatherman.com).
2. 2.Enable the Gmail API for the project.
3. 3.Obtain a refresh token:

export GMAIL_CLIENT_ID=...
export GMAIL_CLIENT_SECRET=...
pip install google-auth-oauthlib
python scripts/gmail_oauth_setup.py

1. 1.On Hugging Face (or .env), set:

1. 1.Schedule polling with `.github/workflows/ingest_gmail.yml` (every 10 minutes). Add Actions secrets:
2. 2.ORDER_INGESTION_URL — e.g. https://marcowxm-order-ingestion-service.hf.space
3. 3.GMAIL_JOB_SECRET — same value as on the Space

Manual trigger:

curl -X POST "https://<your-space>.hf.space/jobs/gmail/ingest" \
  -H "X-Job-Secret: $GMAIL_JOB_SECRET"

Processed messages receive the Gmail label order-ingestion-processed so they are not ingested twice.

Who triggers Gmail ingest

All of the following must be true on the same Gmail message:

Customer emails and internal notes to other staff (e.g. Hi Weslie, Hi Connor) are marked processed and skipped.

Upsert by order number

Gmail ingest looks up an existing Pipeline item by order number with or without a leading # (e.g. #3320030 and 3320030 Company both match):

• —If found: follow-up only when there are new attachments on the staff handoff (e.g. revised logo/vector files). Text-only thread replies are skipped.
• —If not found: creates a new item when the handoff criteria above are met.
• —Unnumbered threads (#UNNUMBERED) upsert by company name instead.

Slack/monday body format: #order Company, Status: NEW, latest staff handoff text. Slack adds: @Paula, cc @Sunny, @Marco Gastelum, @Mollie Cutillo.

Default GMAIL_QUERY matches custom-order subjects or any mail to Paula, within the last 2 days. Staff handoff gates still require internal sender, mock-up wording, and an attachment.

Tuning GMAIL_QUERY on Hugging Face (optional)

If HF has an older GMAIL_QUERY variable, update or delete it so the new default applies. Old queries that only matched Custom order form subjects will miss Paula handoffs on threads like OPAL Fuels.

Steps:

1. 1.Open Space Settings → Variables.
2. 2.Click New variable → name GMAIL_QUERY, paste the query above.
3. 3.Save and Restart the Space (Factory → Restart this Space).

The from:weatherman.com clause limits the Gmail poll to staff senders so each cron run sees fewer messages. Ingest still requires mockup/proof in the latest reply; customer mail is skipped even without this filter.

Other useful fragments (combine with spaces):

• —from:sajjad@weatherman.com — only one mailbox
• —newer_than:2d — temporary catch-up window
• —label:inbox — exclude archived mail

Test in Gmail search first; whatever matches there is what the API returns.

Scheduled runs (GitHub Actions)

For local smoke tests, run python test_order.py manually (creates a real Pipeline item).

Repository secrets (Gmail cron)

All monday/Slack/LLM secrets live on the Hugging Face Space, not in the Gmail workflow.

Cron timing

GitHub Actions schedules use UTC. To change frequency, edit the cron expression in the workflow (e.g. */10 * * * * for every 10 minutes).

Deployment modes

Note: test_order.py runs a fixed sample order (#3278429). Production Gmail ingest upserts by order number (one Pipeline row per #) and labels processed mail so it is not ingested twice.

Slack app setup

1. 1.Create or use an existing Slack app at api.slack.com/apps.
2. 2.OAuth scopes (Bot Token):
3. 3.channels:history
4. 4.files:read
5. 5.files:write
6. 6.chat:write
7. 7.Install the app to your workspace and copy the Bot User OAuth Token → SLACK_BOT_TOKEN.
8. 8.Invite the bot to:
9. 9.#sales-custom-orders
10. 10.#product-custom-orders
11. 11.For Events API (production webhook), subscribe to message.channels (or relevant events) and point the request URL to your deployed /webhooks/slack/new-order (e.g. Hugging Face Space URL). Set SLACK_SIGNING_SECRET.

Optional: SLACK_USER_TOKEN for a user who is in #sales-custom-orders if the bot cannot read files.

Environment variables

See .env.example for optional variables (SLACK_USER_TOKEN, SLACK_TEST_FILE_ID, Gmail).

Channel reference

flowchart TB
  subgraph tokens [Credentials]
    BOT["SLACK_BOT_TOKEN"]
    MON["MONDAY_API_KEY"]
    GROQ["GROQ_API_KEY / OPENAI_API_KEY"]
  end

  subgraph read [Read only]
    SALES["#sales-custom-orders<br/>C7Z5NEDRB"]
  end

  subgraph write [Write]
    PROD["#product-custom-orders<br/>C9HG8MM8E"]
  end

  subgraph mdc [monday.com]
    PIPE["Pipeline 3706090324<br/>group: topics"]
  end

  BOT -->|history, files:read| SALES
  BOT -->|chat:write, files:write| PROD
  MON --> PIPE
  GROQ --> SVC[Order Ingestion Service]
  SVC --> PIPE
  SALES -.->|ingest| SVC
  SVC -.->|notify| PROD

Project structure

flowchart TB
  subgraph app [app/]
    MAIN[main.py<br/>webhook + pipeline]
    CFG[config.py]
    SCH[schemas.py]
    subgraph services [services/]
      AI[ai_parser.py]
      GML[gmail_client.py]
      GATE[ingestion_gate.py]
      BODY[email_body.py]
      SLK[slack_client.py]
      MON[monday_client.py]
      NOT[notification.py]
      ATT[email_attachments.py]
    end
  end

  TEST[test_order.py]
  GHA[.github/workflows/ingest_gmail.yml]
  KA[.github/workflows/keep_alive.yml]
  DOCK[Dockerfile]
  ENV[.env / HF secrets]

  MAIN --> AI & SLK & MON & NOT
  SLK --> ATT
  MAIN --> CFG
  CFG --> ENV
  TEST --> MAIN
  GHA --> TEST
  DOCK --> MAIN

app/
  main.py              # FastAPI routes and pipeline orchestration
  config.py            # Environment settings
  constants.py         # Status defaults and Slack @mention line
  services/
    ai_parser.py       # LLM structured extraction (Groq)
    gmail_client.py    # Gmail API poll, MIME body, attachments
    ingestion_gate.py  # Paula handoff + mock-up request gate
    email_body.py      # HTML strip, latest-reply extraction
    company_name.py    # Form / signature / domain / logo company refine
    slack_client.py    # Channel history, downloads, notifications
    monday_client.py   # Items, updates, file uploads, upsert lookup
    notification.py    # Slack/monday message templates
    email_attachments.py
test_order.py
tests/                 # Unit tests (email_body, company_name, ingestion_gate, notification)
Dockerfile
requirements.txt
.github/workflows/ingest_gmail.yml   # Gmail poll every 10m
.github/workflows/keep_alive.yml     # HF /health ping every 10m
automation_spec.md

Troubleshooting

License

Internal use — Weatherman.