DrystanGovender/crs-competitive-intelligence

CRS Competitive Intelligence Dashboard

Internal Streamlit dashboard for Cyber Retaliator Solutions (CRS) — a South African cybersecurity VAD and IBM / Red Hat / SUSE / CompTIA training partner. It surfaces African government and private-sector tenders, scores them as channel-partner opportunities, recommends reseller companies, and assists with B2B contact discovery and outreach.

The nightly GitHub Action owns scraping and AI scoring. This Space is read-mostly: it displays that data, provides on-demand AI analysis, and lets you push opportunities and leads to Monday.com CRM.

Quick Start

1. 1.Make the Space private — Settings → Visibility. The app writes to Supabase and spends Apollo credits; public access is a billing risk. Optionally add APP_PASSWORD to gate it behind a password prompt.
2. 2.Add secrets — Settings → Variables and secrets. See the Secrets Reference below.
3. 3.Push to the Space — Docker builds automatically; the app comes up on port 7860.

Local development:

pip install -r requirements.txt
# Create .streamlit/secrets.toml with the required keys, then:
streamlit run app/streamlit_app.py

Navigation Pages

The sidebar has nine pages. Each page supports live filtering, detail cards, and Monday.com push buttons.

Nightly Pipeline

A GitHub Action (daily-ingest.yml) runs at 01:00 UTC (03:00 SAST) and:

1. 1.Scrapes open tenders from OCDS publisher APIs (South Africa, Kenya, Nigeria, Tanzania, Zimbabwe, Uganda, and more). Optionally includes World Bank and UNDP non-OCDS sources.
2. 2.AI-scores each new tender as a CRS channel-partner opportunity on a 1–10 scale with rationale, proposed solutions, and outreach angle.
3. 3.Runs partner analysis over awarded tender data — identifies the top 12 ICT / security companies CRS should approach as channel partners, with urgency ranking and deal-size estimate.
4. 4.Writes results to Supabase: sa_tenders, awarded_tenders, tender_score_history, partner_recommendation_history, ai_usage_log, pipeline_runs.

The pipeline can also be triggered manually from the Actions tab with overrideable parameters:

The sidebar shows the last run status. Runs stuck beyond 2 hours are auto-expired in both the cleanup step and the app UI.

AI Provider Cascade

Both the app and the nightly Action share the same cascade:

Groq → Cerebras → OpenRouter → GitHub Models → NVIDIA → DeepSeek → Gemini → HF

Rate-limited or failed calls fall through automatically. At least one key is required for AI features (tender scoring, partner analysis, contact classification).

Session-scoped daily call limits:

Provider status is shown at the bottom of the sidebar (🟢 active / ⚪ key not configured).

Secrets Reference

Set each secret as a Space environment variable (Settings → Variables and secrets). make_secrets.py converts them to ~/.streamlit/secrets.toml at container startup — no code changes needed to add or remove keys.

Required

AI providers (at least one required)

Integrations (optional)

To add a new secret without editing make_secrets.py, set EXTRA_SECRET_KEYS="MY_KEY1,MY_KEY2" in the Space. For permanent additions, append the name to ALLOWED_KEYS in make_secrets.py.

Supabase Tables

Monday.com Integration

Board IDs and column IDs are verified and hardcoded in app/monday_client.py. Reuse its exported functions — do not re-derive IDs.

Key functions in monday_client.py:

push_tender_to_monday(row)       → Leads 2.0
push_partner_to_companies(rec)   → Companies
sync_lead_to_monday(contact)     → Leads 2.0
push_to_contacts_board(contact)  → Contacts
lookup_monday_crm(contact)       → checks all CRM boards
lookup_monday_company(name)      → checks Companies board

Deploy Flow

GitHub push to main
  └─ sync-to-hf.yml ──► HF Space: DrystanGovender/crs-competitive-intelligence
                              └─ Docker build from root Dockerfile
                                    └─ entrypoint.sh
                                          ├─ make_secrets.py  (env → secrets.toml)
                                          └─ streamlit run app/streamlit_app.py :7860

GitHub cron 01:00 UTC
  └─ daily-ingest.yml ──► ubuntu runner
                              └─ python app/Daily_ingest.py
                                    └─ app/ingest_core.py (scrape + score + partner)
                                          └─ writes Supabase tables

Hard Conventions

• —*.sh files must use LF line endings — enforced by .gitattributes (*.sh text eol=lf). CRLF breaks the container entrypoint.
• —Secrets via st.secrets.get(...) only — never hardcoded, never committed.
• —No APScheduler, no `streamlit-autorefresh` — the GitHub Action owns scheduling.
• —Repo layout: Dockerfile, entrypoint.sh, make_secrets.py, requirements.txt at the repo root; all app code under app/.
• —This README.md HF front-matter (--- … ---) must be the first lines of the file.
• —New Python dependencies: pin in requirements.txt (app) and/or requirements-ingest.txt (Action only).
• —Do not add scrapers or schedulers to app/streamlit_app.py — that file is UI-only.