Darkweb007/CONCORD_DATA_AGENTS

Concord Data Agents — Legacy System Interoperability Engine

[View the project landing page](docs/index.html) (enable GitHub Pages on this repo to host it live — see below) · [Live interactive demo](https://huggingface.co/spaces/Darkweb007/CONCORD_DATA_AGENTS)

Portfolio project 3 of 5 — a demo response to the "garbage data" problem insurance and healthcare giants like UnitedHealth Group face: decades-old mainframe tables with no documentation, inconsistent formats, and mixed missing-value conventions, where data scientists reportedly spend up to 80% of their time just cleaning data before any modeling can start.

⚠️ All data in this project is synthetic. data/legacy_claims_raw.csv is generated by generate_synthetic_data.py with randomized names, IDs, and diagnosis codes. No real patient, member, or claims data is used anywhere in this repo.

Why this exists

Legacy tables like this one are common in large, decades-old healthcare and insurance systems: cryptic column names (PT_NM, CLM_AMT_STR), no schema documentation, missing values encoded six different ways ("", NULL, N/A, -999, UNK, 9999-99-99), dates in four different formats in the same column, and categorical codes that were never standardized (M, F, 1, 2, Male, female, ...). Standard ML pipelines break on this kind of input, and hand-cleaning it doesn't scale. This project automates the first two stages of that cleanup with a small multi-agent pipeline.

Architecture

legacy CSV (undocumented, garbled, inconsistent)
    |
    v
Agent 1 — Schema Scanner        <- agents/schema_agent.py
    infers, per column: canonical field name, likely data type,
    and the specific quality issues present (mixed date formats,
    inconsistent categorical spellings, missing-value sentinels, ...)
    |
    v
Agent 2 — Cleaning Agent         <- agents/cleaning_agent.py
    dispatches a cleaning rule per column based on Agent 1's schema map
    (date parsing across 4 legacy formats incl. 2-digit-year pivot,
    currency coercion, categorical normalization, name/ID standardization),
    executes it, drops exact duplicate rows
    |
    v
Quality Dashboard                <- quality_metrics.py
    scores completeness/validity before and after, so the improvement
    is measured, not just asserted

In production this two-agent handoff is implemented with LangGraph or CrewAI, and Agent 2's rule generation would come from a fine-tuned 7B/8B model (validated before execution) rather than a fixed dispatch table -- same architecture, same agent boundaries, this demo just keeps the actual transform logic deterministic so it runs anywhere with zero GPU and zero API keys.

Try it

pip install -r requirements.txt
streamlit run app.py

Click Run pipeline in the sidebar. Four tabs walk through the full pipeline: Agent 1's inferred schema (with detected issues per column), Agent 2's cleaning log (what rule was applied and why), a before/after data quality dashboard, and a side-by-side of the raw vs. cleaned tables.

To regenerate the synthetic dataset with a different sample size or random seed, edit and rerun generate_synthetic_data.py.

Project structure

legacy-etl-agents/
├── app.py                        # Streamlit UI
├── pipeline.py                   # orchestrates Agent 1 -> Agent 2 -> quality scoring
├── quality_metrics.py            # before/after completeness & validity scoring
├── generate_synthetic_data.py    # produces data/legacy_claims_raw.csv
├── agents/
│   ├── schema_agent.py           # Agent 1: infers schema from undocumented columns
│   └── cleaning_agent.py         # Agent 2: generates + executes cleaning rules
├── data/
│   └── legacy_claims_raw.csv     # SYNTHETIC messy legacy claims table
└── requirements.txt

Production upgrade path

Project landing page

docs/index.html is a standalone, single-file static landing page (no build step) summarizing the project's results, method, and findings. To host it live on GitHub Pages: repo Settings → Pages → Source: Deploy from a branch → Branch: main, folder: /docs → Save. It'll be live within a minute or two at https://data-geek-astronomy.github.io/CONCORD_DATA_AGENTS/.