malakgaballa/marketsync

<!-- The YAML block above configures the Hugging Face Space (Docker); it must be the very first thing in this file. Ignore it on GitHub. -->

<div align="center">

🧭 MarketSync

AI-powered labor-market intelligence for Egypt & the GCC — a two-sided career platform.

Upload a CV and see the real jobs that fit you. Paste a role and rank a stack of CVs. Explore what the regional market actually wants — all from one live, explainable engine.

<br/>

      

</div>

[!NOTE] New here? Read the non-technical [Product Guide →](GUIDE.md) · the deep [Technical Report →](TECHNICAL_REPORT.md) · the [Architecture & ERD →](ARCHITECTURE.md) · or the portfolio [Case Study →](CASE_STUDY.md).

<!-- 📸 HERO SCREENSHOT PLACEHOLDER — landing page + a Pursuit results view side by side. Replace with: docs/assets/hero.png --> <div align="center"><em>📸 Hero screenshot — <code>docs/assets/hero.png</code></em></div>

Overview

Job seekers across MENA scroll a dozen boards with no idea which roles fit them or what to learn next. Recruiters burn hours hand-screening CV piles with no defensible read on fit. And nobody has a single, trustworthy picture of what the regional market wants.

MarketSync turns ~15K live, de-duplicated Egypt & Gulf postings (7 countries, 6K+ employers, refreshed weekly) into an intelligence platform with two flagship products over a shared analytics layer:

Both products run on one matching engine so a "fit %" means the same thing on each side. Everything is local and free by default — matching needs no API key; the LLM layer is strictly opt-in.

Features

🎯 Pursuit — the career copilot for seekers

• —Explainable fit % per real, open Egypt/GCC job — not a raw model score.
• —Skill-gap analysis: matched skills you have vs. missing skills the job wants.
• —Learning roadmap aggregated across matches — "learn X → unlocks N of your roles."
• —Still accepting applications? — a free age-based caution plus an on-demand live check against the source board (GET /job/status), so a top match isn't a dead link.
• —Career copilot grounded in the live corpus + CV coach that writes as a senior practitioner in your target role — both fall back to deterministic output with no LLM configured.
• —CV readout with a shareable match card; reads PDF, Word .docx, and plain text.
• —Guided wizard: Upload → Review skills & target → Results (Best Fits · Level Up · Copilot · CV Readout).

🧑‍💼 Prospect — the screening engine for recruiters

• —Rank many candidate CVs against one job description — pairwise, at request time.
• —Each candidate card: fit %, required skills they have (green) vs. lack (red), and a "Why this score?" breakdown.
• —Coverage-led weighting — a partial-skills candidate outranks a same-industry candidate with none of the must-haves.
• —Candidate profiles parsed from the CV — employment history with month-precision tenure, education, certifications, languages, industries — powering 12 insight badges (Job Hopper · Loyal · Gap Detected · Career Switch · Fresh Grad · Top Employer · Academic · Freelancer · Highly Certified · Scholarship · Award Winner · Different Industry). A badge never moves the fit % — it says what the score can't.
• —Instant faceted filters built from the uploaded pile itself (university, degree, field, languages & certificates with AND/OR, industry all-time vs. last job, graduation year) — every option offered matches at least one candidate.
• —Smart search over the shortlist — ask "who has applied a machine-learning model?" and the CV that says "built a churn classifier" is a hit, with the exact passages it matched on. Keyword and semantic modes, both fully local and free.
• —Paste or upload the JD (PDF / Word / .txt); upload CVs as PDFs, .docx, .txt, or a `.zip` (unpacked server-side). No pipeline run required.
• —Side-by-side candidate compare ("only A / both / only B") computed entirely client-side, plus a Discuss copilot grounded in this screening run and a Role readout of the parsed JD.

📊 Market Intelligence — shared by both audiences

• —Overview — Egypt + GCC composition, seniority mix, remote share, top industries & employers, region-locked map.
• —Jobs — searchable corpus, newest first, filterable by region, source, and title.
• —Skills & Trends — one tab, two reads: Explore (click any skill for demand, who hires for it, and what it pairs with) and Over time (By role / By skill series, dual-handle date-range slider, 🔥 rising & ❄️ cooling, next-week projection).
• —Weekly Report — generated key takeaways + a role-aware insight lens (momentum + leverage/scarcity).

Demo

The Space is live and refreshed weekly (see scripts/SCHEDULING.md). On the free tier it sleeps after ~48 h idle — the first visit wakes it, and the first CV match downloads the models, so give it a few seconds before it's fast.

<!-- 📸 DEMO GIF PLACEHOLDER — 20s: land → pick seeker → upload CV → ranked results. Replace with: docs/assets/demo.gif --> <div align="center"><em>📸 Demo GIF — <code>docs/assets/demo.gif</code></em></div>

Architecture

A batch pipeline produces files; a FastAPI service reads them live. The two halves are decoupled by artifact files — the API never recomputes, so re-running the pipeline is picked up without a restart (mtime-aware cache).

flowchart LR
  subgraph SRC["Live sources · Egypt / GCC only"]
    S["Jooble · JobSpy<br/>Wuzzuf · Tanqeeb<br/>LinkedIn (opt-in) · Muse · Remotive …"]
  end

  subgraph PIPE["pipeline.py · batch (9 steps)"]
    P["ingest → clean → extract skills →<br/>features → embed → graph → report → DB"]
  end

  subgraph ART["Artifacts on disk · the contract"]
    A["parquet corpus · match_index (vectors)<br/>knowledge_graph.json · marketsync.db"]
  end

  subgraph API["api/main.py · FastAPI · single origin"]
    R1["Analytics endpoints"]
    R2["/match/cv → matcher.py · Pursuit"]
    R3["/screen → screener.py · Prospect<br/>+ profiles · insights · facets"]
    R4["/screen/search → smart_search.py<br/>(in-memory run_store, 1h TTL)"]
    DASH["/app/ → React dashboard"]
  end

  subgraph UI["React dashboard · single file"]
    U["Landing → Pursuit | Prospect + shared analytics"]
  end

  SRC --> PIPE --> ART --> R1 --> DASH --> UI
  ART --> R2 --> DASH
  CV["Recruiter uploads JD + CVs live"] -. no pipeline needed .-> R3 --> DASH
  R3 -. run_id .-> R4 --> DASH

Full end-to-end flow, request sequence diagrams, and the database ERD live in [ARCHITECTURE.md](ARCHITECTURE.md) and [TECHNICAL_REPORT.md](TECHNICAL_REPORT.md).

System Pipeline

python pipeline.py runs 9 ordered steps. Each writes an artifact the next step reads.

flowchart TD
  T["1 · Skill taxonomy<br/><i>ESCO ▸ O*NET ▸ Wikidata + auto-discovery</i>"]
  I["2 · Ingest + geo-tag + delta<br/><i>Egypt/GCC filter, cross-source dedup</i>"]
  C["3 · Clean & normalise<br/><i>HTML, dates, seniority, remote</i>"]
  K["4 · NLP skill extraction<br/><i>dictionary + fuzzy match</i>"]
  F["5 · Feature engineering<br/><i>counts · co-occurrence · trends</i>"]
  E["6 · CV-match index<br/><i>sentence-transformers embeddings</i>"]
  G["7 · Knowledge graph<br/><i>skill co-occurrence · NetworkX</i>"]
  R["8 · Weekly report<br/><i>market insights</i>"]
  D["9 · Database sync<br/><i>SQLite / Neon</i>"]
  T --> I --> C --> K --> F --> E --> G --> R --> D

Flags: --skip-ingest (reuse last snapshot) · --reset-delta (force full re-fetch) · --debug (verbose).

Tech Stack

Repository Structure

MarketSync/
├─ pipeline.py            # orchestrates the 9-step batch pipeline
├─ config.py             # all paths + settings (loaded from .env)
├─ api/main.py           # FastAPI: REST endpoints + serves the dashboard at /app/
├─ dashboard_react/      # single-file React SPA (landing → Pursuit / Prospect + analytics)
├─ src/
│  ├─ ingestion/         # per-source fetchers · geo-tagging · delta sync · dedup
│  ├─ preprocessing/     # cleaning & normalisation
│  ├─ taxonomy/          # skill dictionary (ESCO/O*NET/Wikidata + discovery)
│  ├─ nlp/               # skill extraction
│  ├─ features/          # aggregate tables for the dashboard
│  ├─ matching/          # RAG engine: matcher.py (Pursuit), screener.py (Prospect),
│  │                     #   embeddings, cross_encoder, cv_parser, cv_llm, llm_rerank
│  │                     #   + Prospect's ATS layer: cv_profile, insights,
│  │                     #     smart_search, run_store, screen_chat
│  ├─ analytics/         # knowledge graph
│  ├─ career/            # chat · cv_feedback · courses
│  ├─ reporting/         # weekly report + LLM brief
│  ├─ storage/db.py      # SQLite / Neon loader (all artifacts → SQL tables)
│  └─ filters/           # job-search filter engine
├─ data/                 # raw → interim → processed → outputs (+ match_index)
├─ sample_data/          # synthetic CV .zip + a sample JD, for trying Prospect
├─ scripts/              # weekly refresh (launchd) · run monitor (Streamlit) · backfills
├─ tests/                # pytest suite
├─ Dockerfile            # single-container deploy (HF Space, port 7860)
└─ marketsync.db         # relational mirror of every artifact

Quick Start

# 0) Environment (tested on conda, Python 3.12)
pip install -r requirements.txt
playwright install chromium            # one-time, for Wuzzuf/browser scraping

# 1) Configure — create .env in the repo root (copy .env.example) with at least:
#    JOOBLE_API_KEY=...     (free: https://jooble.org/api/about)

# 2) Build data artifacts (ingest → … → index → database)
python pipeline.py
#   python pipeline.py --skip-ingest   # reuse last snapshot (fast iteration)

# 3) Run the app — ONE command serves BOTH the dashboard and the API (single origin)
uvicorn api.main:app                   # open http://127.0.0.1:8000/app/
                                        # interactive API docs at /docs

The API auto-reloads data when the pipeline writes new files — no restart needed. Add --reload while editing Python code. Prospect works immediately; it needs no pipeline run — try it with sample_data/sample_jd_senior_backend.txt + sample_data/sample_cvs.zip.

python -m pytest                       # test suite (conftest.py puts the repo root on PYTHONPATH)
streamlit run scripts/monitor_app.py   # operator-only: pipeline run + dedup monitor (not public)

Optional LLM layer (free, no card). Set GROQ_API_KEY in .env and the backend selects itself — that enables the copilots, CV-coach tips, smart-search notes and the market brief. Left unset, each of those falls back to deterministic offline output; matching and analytics never need an LLM.

Deploying? See [DEPLOY.md](DEPLOY.md) for the $0 Hugging Face Space path (this repo's hf-space branch is what's live), and [scripts/SCHEDULING.md](scripts/SCHEDULING.md) for the weekly auto-refresh.

API Overview

Thin wrappers over src/ modules. Full interactive docs at /docs.

Future Roadmap

Prioritized from two "voice of the customer" exercises (recruiter + seeker). Full detail in [ROADMAP.md](ROADMAP.md).

• —Focus-tailored analytics — an optional "targeting a role? / hiring for which field?" on the landing that becomes a lens over the shared analytics (never a filter — the full market stays visible).
• —Explainability / evidence layer — quote the exact CV line behind each matched skill (shared by both products).
• —Bias & compliance — blind-screening mode, adverse-impact reports, audit trail (Prospect).
• —Feedback → learning loop — capture outcomes and re-fit scoring weights per role family.
• —Calibration & confidence — turn raw blends into trustworthy, comparable percentages.
• —Workflow fit — ATS integrations, one-click shortlist export.
• —Performance — precompile the JSX, bake models into the image for faster cold starts.

Constraint — no salary data, ever. MarketSync never collects, stores, estimates, or displays pay/compensation. "Value" is always expressed as demand (roles a skill unlocks).

License

Released under the MIT License.

⚠️ Ethical use of scrapers. The LinkedIn path is off by default and intended for personal / educational use only — LinkedIn's User Agreement prohibits automated scraping. Do not run it in production or redistribute its data.

<div align="center">_{MarketSync — intelligence for both sides of the Egypt & Gulf job market.}</div>