nappingCommodore/bihar-procurement-analysis
0
1"""Human-facing documentation for the anomaly rules and the investigative2workflow. Kept as data so the /guide page can render it and merge live counts3from v_anomaly_summary. Descriptions mirror bihar_ingestion/analysis/anomaly_scorer.py.4"""5 6from __future__ import annotations7 8# How the app is used, step by step.9WORKFLOW = [10 ("Start on the Integrity Monitor",11 "The overview's six signal sections show where the strongest patterns are — vendor "12 "capture, overpricing, splitting, geography and data integrity. Every bar links straight "13 "to the entity or case behind it."),14 ("Triage in the Worklist",15 "Every flag from every detector, ranked by score. Filter by rule, family, status or "16 "minimum score, or search a vendor / department. A higher score means bigger magnitude "17 "AND higher confidence — work from the top down."),18 ("Verify in the Case file",19 "Open a tender to see its awards, line-items-vs-Schedule-of-Rates, existing flags and "20 "documents. This is where you confirm a flag against the actual numbers instead of taking "21 "it on faith."),22 ("Follow the entities",23 "Click any vendor, department or officer to see their whole footprint: concentration, "24 "their flags, related parties, and an interactive ego-network of who they transact with."),25 ("Explore and export",26 "The Data Explorer filters the full tender register and exports CSV — build your own view, "27 "or assemble an evidence pack for a specific department, vendor or time window."),28 ("Record a decision",29 "On a case, mark each flag confirmed or dismissed with a note. Your decision survives "30 "re-scoring, so the worklist becomes an audit trail rather than a throwaway list."),31]32 33# How to read the score / status of a flag.34SCORE_NOTES = [35 ("Score = severity × confidence",36 "Severity (0–1) grows with the anomaly's magnitude — how many times the SOR rate, how "37 "large the value vs peers. Confidence (0–1) is how sure we are it is a real anomaly rather "38 "than noise. A mathematically-provable data artifact scores high confidence; a soft "39 "heuristic like contract-splitting scores lower on purpose."),40 ("Severity bands",41 "As a rule of thumb: ≥ 0.60 is a strong signal (red), 0.30–0.60 is worth a look (amber), "42 "below 0.30 is a weak pointer. Sort by score, but always read the evidence line."),43 ("Status lifecycle",44 "Every flag starts open. As you review it you set confirmed (a real red flag worth "45 "escalating) or dismissed (explained / benign). Re-running the scorer never overwrites "46 "your status."),47 ("Disposition: quarantine vs review",48 "Quarantine = an impossible value we set aside and repair (data quality). Review = an "49 "internally-consistent value that needs a human to judge. We never delete anything — the "50 "raw payload is always preserved."),51 ("Raw vs trusted value",52 "Where a stored amount is corrupt, we keep the raw figure and compute a trusted (repaired) "53 "one. All money in the app uses trusted values; the raw is shown on the case so you can "54 "see exactly what was changed."),55]56 57# Practical playbook: how a person actually finds red flags.58PLAYBOOK = [59 ("Rank by score, but read the evidence",60 "Score orders the queue; the one-line evidence tells you what actually fired and on which "61 "entity. Never act on the number alone."),62 ("Separate corruption from signal first",63 "Artifact flags (the A-family) are usually data-entry problems. Clear or repair them first "64 "so a doubled digit doesn't masquerade as a ₹2-crore overrun."),65 ("One flag is a lead — stacked flags are a case",66 "The strongest cases combine signals: a department with vendor capture whose dominant "67 "vendor is also a related-party cluster and shows monthly splitting bursts is far more "68 "compelling than any single large number."),69 ("Verify at the source",70 "Open the case and check line items, the SOR comparison and documents before calling "71 "something real. The raw payload is preserved for exactly this cross-check."),72 ("Look for repetition and regionality",73 "The same item awarded at 100×+ its SOR across many vendors in one area, or award bursts "74 "that stop just under a round approval ceiling, are stronger than a lone big value."),75 ("Decide, and write down why",76 "Confirm or dismiss with a short rationale. That turns the worklist into a defensible "77 "trail and stops the same lead being re-litigated later."),78]79 80FAMILY_DOCS = {81 "artifact": ("Data integrity", "#e0a53d",82 "A recorded value that is internally impossible or malformed. We never delete it: "83 "the raw value is preserved and a repaired value is used for analysis. Usually a "84 "data-quality issue — but it can hide a real transaction, so it is still flagged."),85 "reasonableness": ("Cost & value", "#f0553d",86 "The value is internally consistent but suspicious on its economics — priced "87 "far above the Schedule of Rates, far from the estimate, or a statistical "88 "extreme for its procurement category."),89 "network": ("Relationships", "#5aa9e6",90 "Patterns across vendors, departments and officers — one vendor dominating, distinct "91 "records sharing an identity, or many small awards that look like one split contract."),92}93 94# One entry per rule_code, grouped by family for display (family, [rule_codes]).95GROUPS = [96 ("reasonableness", ["SOR_overprice", "V3_award_vs_estimate", "S1_value_outlier"]),97 ("network", ["N1_vendor_capture", "P2_tender_splitting", "N4_related_party",98 "N3_vendor_district_concentration"]),99 ("artifact", ["A1_digit_doubled", "A2_pct_tax_line", "A3_scale_error",100 "A4_repeated_digit", "A5_epoch_as_value"]),101]102 103FLAG_DOCS = {104 "SOR_overprice": {105 "title": "Overpricing vs Schedule of Rates",106 "what": "An item was awarded at 2× or more the tender's OWN Schedule-of-Rates estimate for "107 "that item. The sharpest cost signal in the dataset, and it works even on lump-sum "108 "tenders because it compares awarded vs estimated for the same line.",109 "how": "awarded_rate ÷ sor_rate ≥ 2, matched on (tender, item code); Re-1 placeholder SOR "110 "rates are excluded. Severity scales with the multiple.",111 "verify": "Open the case → the line-item table shows awarded vs SOR and the ×SOR multiple. "112 "Read the item name — is it genuinely the same scope? A 100–300× soak-pit / "113 "tubewell cluster recurring across vendors in one region is a strong lead. Dismiss "114 "if the SOR rate is a placeholder or the items differ.",115 },116 "V3_award_vs_estimate": {117 "title": "Award far from estimate",118 "what": "The awarded value is a large multiple of the pre-tender estimate (overrun ≥ 3×) or "119 "far below it (lowball ≤ 0.4×). Either can signal manipulation, poor estimation, or "120 "a corrupt value.",121 "how": "trusted award value ÷ estimate (PAC); overruns and lowballs are flagged separately. "122 "Confidence 0.5 — many legitimate causes exist.",123 "verify": "First rule out data corruption (A-family). Then check line items and SOR. A clean, "124 "itemised award many times the estimate with no artifact flag is a genuine review "125 "case; a deep lowball can be a front-loaded or loss-leader bid.",126 },127 "S1_value_outlier": {128 "title": "Statistical value outlier",129 "what": "An award that is both in the top 1% of its procurement category by value AND at "130 "least 10× the category's median — a statistical extreme for its peer group.",131 "how": "Per category (with ≥ 100 awards): value in the top 1% percentile AND ≥ 10× the "132 "category median. Confidence 0.4; severity scales with the ×median multiple.",133 "verify": "Open the case: is it a legitimate large project, an artifact, or an unexplained "134 "multiple of its peers? A clean ₹49 Cr award at 200× its category median is a real "135 "review case; always rule out corruption first.",136 },137 "N1_vendor_capture": {138 "title": "Vendor capture (department)",139 "what": "One vendor won 60% or more of a department's awarded value across at least five "140 "awards — competition that may exist only on paper.",141 "how": "Per department: top-vendor value share ≥ 0.60 over ≥ 5 awards. Severity = the share "142 "itself; the department page also shows the HHI concentration index.",143 "verify": "Open the department profile: look at the vendor bar chart and HHI. Is the work "144 "specialised (few possible suppliers) or a commodity that should be competitive? "145 "Persistent 80–93% capture on commodity work is a strong red flag. Cross-check the "146 "dominant vendor's related-party cluster.",147 },148 "P2_tender_splitting": {149 "title": "Contract splitting (structuring)",150 "what": "Four or more sub-₹1 Cr awards to the SAME vendor from the SAME department in a "151 "SINGLE month, together exceeding ₹50 L — the classic pattern of splitting one job "152 "to stay under an approval threshold.",153 "how": "(department, vendor, month) with ≥ 4 awards, every award < ₹1 Cr, and total > ₹50 L. "154 "A heuristic (no official ₹ thresholds), so confidence is a deliberate 0.4; severity "155 "grows with the count.",156 "verify": "Filter the Data Explorer by that department and vendor: are these really one "157 "divisible job? Repeated 15–24 award bursts just under a round ceiling are a strong "158 "lead. Sharpen it if you know the real approval-tier ₹ thresholds.",159 },160 "N4_related_party": {161 "title": "Related parties (shared PAN / GSTIN)",162 "what": "Two distinct vendor records share a PAN or GSTIN — the same or a related legal "163 "entity appearing under more than one name.",164 "how": "Vendors joined on an identical PAN or GSTIN; pairs are grouped into clusters on the "165 "Network page. Confidence 0.7.",166 "verify": "Network page → open the cluster. If the members jointly win in the SAME "167 "department, that is possible bid-rigging or splitting under one identity. Benign "168 "if it is a genuine parent/branch with separate legitimate roles.",169 },170 "N3_vendor_district_concentration": {171 "title": "District concentration (weak signal)",172 "what": "The single biggest vendor holds 15% or more of a district's awards (over ≥ 20 "173 "awards). Deliberately low-confidence: districts pool many vendors, so shares dilute "174 "and this rarely proves capture on its own.",175 "how": "Top-vendor share ≥ 0.15 over ≥ 20 district awards. Confidence 0.3 — real capture "176 "shows at DEPARTMENT level (see vendor capture).",177 "verify": "Use as a pointer, not proof. It matters when the district's top vendor also shows "178 "department capture or a related-party cluster; on its own, 15–24% is often normal.",179 },180 "A1_digit_doubled": {181 "title": "Doubled amount (digit-doubling)",182 "what": "The amount was stored with its digits written twice — e.g. ₹19,68,925 recorded as "183 "19689251968925. A data-capture corruption, not a real payment.",184 "how": "Fires when the stored value equals its own first half repeated. The repaired (true) "185 "value is kept beside the raw one. Confidence 0.95.",186 "verify": "On the case, compare raw vs trusted value against the estimate and line items — the "187 "trusted figure should sit near them. Quarantine as an artifact; it is only a red "188 "flag if the repaired value is itself unreasonable.",189 },190 "A2_pct_tax_line": {191 "title": "Impossible tax / percentage line",192 "what": "A line labelled as GST / tax / percentage whose amount is larger than the base it "193 "applies to — impossible for a true percentage.",194 "how": "Line matches gst / % / tax and its total exceeds the real work lines (base > 0). The "195 "trusted PO value recomputes that line as percentage × base. Confidence 0.95.",196 "verify": "In the line-item table, compare the flagged line to the work lines. If the 'tax' "197 "dwarfs the works it is a corrupt cell — use the repaired total. A red flag only if "198 "the award is still unreasonable after repair.",199 },200 "A3_scale_error": {201 "title": "Possible scale / paise error",202 "what": "The PO value is a suspicious round multiple of the estimate, as if units slipped "203 "(paise vs rupees, or a ×100 / ×1000 scale error).",204 "how": "po_value ÷ estimate lands in a scale-error band. Confidence 0.6 — it can also be a "205 "genuine overrun.",206 "verify": "Compare against line items and SOR. If only the PO header is inflated by a round "207 "factor while the items are sane, it is a scale artifact; if the items agree with "208 "the big number, treat it as a real cost anomaly.",209 },210 "A4_repeated_digit": {211 "title": "Repeated-digit placeholder",212 "what": "The amount is all identical digits (e.g. 1111111 or 9999999) — a placeholder, not a "213 "real figure.",214 "how": "Raw value is a run of a single repeated digit. Confidence 0.8.",215 "verify": "Check whether a real value exists in the line items or documents. Usually a "216 "placeholder to quarantine; escalate only if it hides a real transaction.",217 },218 "A5_epoch_as_value": {219 "title": "Timestamp used as an amount",220 "what": "A value in the millisecond-epoch range, suggesting a date was mistakenly stored as "221 "an amount. No matches in the current dataset.",222 "how": "Value falls in the ms-epoch band. Confidence 0.5.",223 "verify": "Cross-check the field against the tender's dates. A data-hygiene issue rather than "224 "corruption of money.",225 },226}227