A feature drift detector whose alert thresholds are calibrated from your own reference data; in the demo, it flags a categorical drift (PSI 0.183) that the folklore 0.25 threshold calls "no significant drift".
CI coverage thresholds license
- Drift monitors shipping the 1990s credit-scoring constants (PSI 0.1/0.25) are deaf to real shifts: with 2,000-record windows the no-drift PSI of the demo features is 0.01 to 0.04, an order of magnitude below the folklore alert line.
- Alerts nobody can defend get disabled; here every threshold is the p99 of the no-drift PSI distribution computed by resampling your own reference window, so an alert means "larger than anything your baseline produces by chance".
- Drift numbers hidden in logs never page anyone; PSI, calibrated thresholds, KS p-values, and drift flags are Prometheus gauges with a provisioned Grafana dashboard showing distance-to-alert.
Models degrade silently. The input distribution moves (a deploy changes a latency profile, a campaign changes the customer mix), predictions drift off-calibration, and the first detection is a business metric weeks later. Monitoring teams know this and deploy PSI dashboards, but almost every one of them uses the same hardcoded 0.1/0.25 thresholds regardless of window size, bin count, or feature shape, and those constants determine everything: too high and drift is missed, too low and on-call learns to ignore the channel.
drift-sentinel makes the threshold an output of the data instead of an input from folklore. When a reference window is posted, the detector splits it in half at random 200 times, computes PSI between halves (drift-free by construction), and sets each feature's alert threshold at the 99th percentile of that distribution. Production records stream into a rolling window; on each report the service computes PSI (quantile-binned) and a tie-correct two-sample KS test per numeric feature, compares against the calibrated thresholds, and exposes everything as Prometheus gauges.
The demo (make demo, also run in CI) seeds three known conditions: a +30% latency shift, a plan-mix shift, and an untouched control feature. Result: both drifts flagged, control clean. The plan-mix drift scores PSI 0.183, which the folklore scale calls "moderate, below significant"; its calibrated threshold is 0.017, making it a 10x-over-threshold unambiguous alert.
flowchart LR
REF[POST /reference<br/>2000 records] --> CAL[Calibration<br/>200 self-splits, p99]
CAL --> THR[(Per-feature<br/>PSI thresholds)]
OBS[POST /observe<br/>rolling deque] --> REP[GET /report<br/>PSI + KS per feature]
THR --> REP
REP --> PROM[GET /metrics<br/>Prometheus gauges]
PROM --> GRAF[Grafana dashboard<br/>PSI vs threshold]
REP -->|drifted features| LOGS[JSON logs / alerting]
Failure boundaries: undersized reference or current windows are rejected with explicit errors (422/409), mixed-type features fail schema inference loudly, and a restart empties only the rolling window, never the caller's data.
| Technology | Role in this project | Why chosen here |
|---|---|---|
| Python 3.10+ | statistics and service | PSI and tie-correct two-sample KS implemented from the formulas (about 120 lines), so the math is reviewable |
| FastAPI + uvicorn | ingestion and report API | Validation at the edge via pydantic models; async server measured at 355+ rps of 100-record batches on one worker |
| prometheus-client | metrics exposition | Thresholds exported next to live PSI so dashboards show distance-to-alert |
| Prometheus + Grafana | history, dashboard, alerting | The service stays stateless-ish; monitoring infra owns history (ADR-002) |
| Docker + compose | one-command stack | make up starts service, Prometheus, and a provisioned dashboard |
| pytest + pytest-cov | 37 tests, 98% measured coverage | The KS tie bug was caught by this suite before first release |
Prerequisites: Python 3.10+, git. Docker only for the full stack.
git clone https://github.com/tkgo1599-max/drift-sentinel.git cd drift-sentinel pip install -e ".[dev]" pytest # 37 tests make demo # calibration + healthy window + seeded drift, printed make serve # API on :8000 make up # full stack: service + Prometheus + Grafana (:3000, anonymous viewer)
API in 30 seconds: POST /reference {"records": [...]} calibrates thresholds; POST /observe {"records": [...]} feeds the rolling window; GET /report returns per-feature PSI, KS p-value, thresholds, and drift flags; GET /metrics is for Prometheus. All knobs are DRIFT_* env vars (window size, splits, quantile, KS alpha, bins).
Methodology: python benchmark/bench.py starts a single uvicorn worker on the same host, seeds a 2,000-record reference, then drives /observe (100-record batches) and /report with an async httpx client at increasing concurrency. Linux container, shared vCPUs. Raw output: benchmark/results/results.json.
| endpoint | concurrency | rps | p50 (ms) | p95 | p99 |
|---|---|---|---|---|---|
| /observe | 1 | 376.7 | 2.55 | 3.06 | 3.56 |
| /observe | 8 | 392.0 | 13.15 | 32.41 | 146.21 |
| /observe | 32 | 354.9 | 89.30 | 101.79 | 113.30 |
| /report | 1 | 70.1 | 14.20 | 15.02 | 16.54 |
| /report | 8 | 85.8 | 88.61 | 135.17 | 178.62 |
Ingestion sustains roughly 37k records/s (376 batches x 100 records) at 2.5 ms p50. Throughput plateaus rather than grows with concurrency because drift math is CPU-bound Python under the GIL: latency stretches while rps stays near 380. The knee is visible at /observe c=8 p99 (146 ms), where report computations contend with ingestion. Scaling path: more uvicorn workers with per-worker windows, which ADR-002 discusses.
- ADR-001: Thresholds calibrated from reference self-splits, not folklore constants
- ADR-002: In-memory rolling window with Prometheus pull, not a metrics database
Multivariate drift (correlation shifts with stable marginals) is deliberately absent: univariate PSI/KS already covers the dominant production failure modes, and multivariate methods need labeled incidents to validate against before they earn on-call trust. Trigger for adding: the first incident post-mortem where marginals stayed clean while joint behavior moved. Also out: concept drift (needs labels or proxy feedback) and persistence of raw windows (a PII decision, ADR-002).
Raw feature values live only in process memory and are never logged; log lines carry feature names and statistics, not values. The /metrics endpoint exposes aggregates only. No credentials are required anywhere in the demo stack; in production, put the service behind the same authn as any internal API and keep Grafana's anonymous mode off. Feature names themselves can be sensitive: treat dashboard access accordingly.
| Failure | Detection | Behavior | Recovery |
|---|---|---|---|
| Reference too small | Explicit check vs min_window | 422 with required count | Post a bigger reference |
| Report before reference | State check | 409, never a fabricated report | Post /reference first |
| Mixed types in one feature | Schema inference | 422 naming the feature | Fix the producer |
| Service restart | window_fill in /healthz | Window empties; reports 409 until refilled | Re-post reference; observe refills |
| Feature missing from current window | Per-feature presence check | Feature skipped, others still reported | Investigate the producer |
| Prometheus scrape fails | Prometheus target down alert | Service unaffected | Standard prom on-call playbook |
The test suite failed on what should have been the most boring assertion in the file: the KS statistic of a sample against an identical copy of itself came back as 0.0005 instead of 0. The two-pointer merge that walks both sorted samples advanced one side per step, so at a tied value the empirical CDFs showed a transient gap of 1/n, and the supremum happily took it.
For 2,000 continuous records that error is invisible noise, which is what makes it dangerous: the implementation looks correct, passes eyeball tests, and then materially inflates D on discrete, heavily tied features (integer counts, rounded amounts), where ties are the norm. The fix (commit in history: "fix(stats): handle ties correctly") advances both pointers past every value equal to the current minimum before measuring the gap, which is the textbook treatment. Two regression tests lock it in: identical samples must give exactly 0, and a constructed heavy-tie case must give exactly 0.3. The takeaway I keep from it: statistical code needs exact-value tests, not just "roughly right" tests, because its failure mode is being plausibly wrong.
- Init container for the compose stack that re-posts the reference on restart, removing the one manual step.
- Per-worker window ownership for multi-worker uvicorn, with per-replica Prometheus labels.
- Chi-square test for categorical features alongside PSI, with the same calibration treatment.
- Alertmanager rules shipped next to the dashboard (PSI over threshold for 3 consecutive scrapes).
- First metric to watch after deploying: alert precision in the first month (alerts acknowledged as real vs dismissed), the number that decides whether on-call trusts the tool.