arikw/intervention-learning-framework

1# intervention-learning-framework (v1, milestone 1)2 3A recursive intervention-learning framework for a real-time sales-call4assistant, built **simulator-first**: every estimator is validated by5recovering known ground truth from the generative simulator in `intervene/sim/`.6No production data exists yet; nothing in this repo claims a result from real7data, and no estimate is reported without an uncertainty interval.8 9Milestone 1 scope: **simulator + detection + offline effect estimation**, with10the confounding parameter-recovery test passing. `policy/`, `ope/`,11`discover/` and `monitor/` are documented skeletons for later milestones;12`planning/` is implemented (it is small and gates runnability).13 14## Install & run15 16```bash17pip install -e ".[dev]"18pytest                       # parameter-recovery suite19python scripts/run_experiment.py --n-calls 25000 --seed 2026091020```21 22Determinism: every stochastic component takes one master seed; child RNGs are23spawned from it via `numpy.random.SeedSequence`.24 25## Layout26 27| Module | Contents |28|---|---|29| `intervene/domain/` | Versioned pydantic v2 models (`MomentDefinition`, `MomentInstance`, `Intervention`, `Decision`, `Outcome`), version-immutability registry |30| `intervene/sim/` | Seeded generative simulator + analytic ground truth (quadrature, no Monte-Carlo truth) |31| `intervene/detect/` | Calibrated per-definition classifiers (isotonic/Platt), PR curves, ECE, cost-based thresholds (C_fp/C_fn default 3:1) |32| `intervene/effects/` | Pooled, stratified, IPW (Horvitz-Thompson) estimators, all with 95% Wald intervals, confounding warning |33| `intervene/planning/` | Two-proportion sample size + calendar-time calculator (proxy and objective) |34| `intervene/policy/`, `ope/`, `discover/`, `monitor/` | Milestone-2 skeletons with documented interfaces |35 36## Estimator assumptions and where they break37 38**Pooled difference** (`effects/estimators.py`)39- Assumes: treatment randomised within the moment population.40- Breaks: under any confounding — biased, sometimes badly (the confounding41  test demonstrates a ~10σ bias). Only ever report it next to an adjusted one.42 43**Stratified / size-weighted difference**44- Assumes: confounding operates only through the strata; independent strata;45  Wald variance within stratum.46- Breaks: confounders varying within strata (only partial adjustment — see47  the confounding test: bias shrinks but does not vanish); sparse strata where48  an arm is empty (those strata are dropped, changing the estimand).49 50**IPW (Horvitz–Thompson) using logged propensities**51- Assumes: propensities logged at decision time and treated as *known*52  (correct, since the logging policy logged them); positivity (0 < e < 1).53- Breaks: propensities near 0/1 (variance explodes; the estimator refuses54  e ∉ (0,1)); any decision logged without a propensity — the domain model55  raises a hard validation error at construction, so such records cannot56  enter the pipeline.57 58**Detector calibration (isotonic / Platt)**59- Assumes: training and scoring distributions match.60- Breaks: distribution shift (moment base-rate drift / feedback61  contamination — monitored by `intervene/monitor/`, milestone 2); isotonic62  needs large samples per fold, Platt degrades with extreme class imbalance.63 64**Doubly robust (milestone 2)** will require *both* a correct propensity65model and a correct outcome model; it is listed here because its failure mode66(error when either model is wrong, provided not both) is the reason it is67preferred over IPS.68 69## The confounding test (the core of milestone 1)70 71`tests/test_effects_confounding.py` builds a moment whose intervention has a72**true-zero effect** but whose logging policy assigns treatment increasingly73to high-quality accounts, which also convert better. On 25k simulated calls:74 75- the pooled estimator is biased upward with a confidence interval excluding76  zero;77- the IPW estimator using the logged propensities covers zero;78- `flag_confounding` fires because the two intervals do not overlap;79- on a companion moment with a real effect, IPW covers the analytically80  computed true ATE.81 82See `DECISIONS.md` for every modelling choice and its rejected alternative.