Baseline anomaly detectors on 8 real sensor and telemetry streams from the Numenta Anomaly Benchmark, evaluated under three protocols — including a random control that has no ability to detect anything.
The headline result: under the widely-used "point-adjust" protocol, the random control scores 0.956 F1 — second of five, ahead of every method that genuinely detects anything, and within 0.001 of the top score. Under strict point-wise scoring it ranks near the bottom, where it belongs.
Mean over 8 streams. Both F1 columns are best-achievable over a threshold sweep, applied identically to every detector.
| Detector | point-wise F1 | point-adjusted F1 | inflation | window recall | alerts / 1000 pts |
|---|---|---|---|---|---|
| isolation_forest | 0.310 | 0.9504 | 3.07x | 0.969 | 54 |
| rolling_mad | 0.211 | 0.9299 | 4.41x | 0.969 | 115 |
| rolling_zscore | 0.209 | 0.9450 | 4.52x | 0.969 | 127 |
| random_control | 0.179 | 0.9557 | 5.35x | 0.969 | 211 |
| rolling_diff | 0.177 | 0.9564 | 5.39x | 0.969 | 183 |
Point-adjusted F1 is shown to four decimals deliberately: at three, rolling_diff
and random_control both read 0.956 and the ordering looks like a tie.
Three things fall out of this:
- Point-adjust destroys the ranking. It inflates F1 by 3-5.4x and lifts pure noise above every detector that actually works — Isolation Forest, which is comfortably best under honest scoring, drops below the random control. Any result reported only under point-adjust tells you almost nothing about whether a detector works. This is not a new observation — it replicates the critique in Kim et al., Towards a Rigorous Evaluation of Time-series Anomaly Detection (AAAI 2022) — but it is worth reproducing on data you can download.
- Point-wise, everything is mediocre. The best detector reaches F1 = 0.31. NAB's anomalies are labelled as broad windows while these detectors fire on individual points, so precision is structurally capped. Difficulty here is in precision, not recall.
- The useful comparison is recall against alert cost. Every detector catches the same ~97% of labelled windows. Isolation Forest does it with 54 alerts per 1000 points against the random control's 211 — a ~4x lower false-alarm burden for the same catch rate. That, not F1, is the number an operator would care about.
A benchmark without a floor cannot tell you whether a method is working. Including a detector that provably knows nothing makes any protocol that ranks it first immediately suspect. It cost about ten lines and it is the most informative row in the table.
- The F1 columns are optimistic upper bounds. The threshold is chosen with knowledge of the labels (best-F1 over a sweep). This is applied identically to every detector so the comparison is fair, but no number here is what you would get choosing a threshold in advance. Stated plainly because this is frequently left implicit in published tables.
- No deep-learning baseline. LSTM/autoencoder/transformer detectors are not included, so this does not show classical methods beating them. It shows that the protocol has to be pinned down before any such comparison means anything.
- 8 streams, one benchmark. NAB's labelling convention (wide windows around a root cause) directly shapes conclusion 2. A point-labelled dataset would behave differently.
- Single seed (1337), no confidence intervals. Isolation Forest and the random control are stochastic; the rolling detectors are deterministic.
- Anomaly rate is ~10% by construction of the NAB windows — much denser than most real deployments, which flatters precision relative to a production setting.
All are streaming-safe: rolling statistics are shifted so a point is never part of its own baseline, and the fitted model only ever sees the first 15% (NAB's probationary period). No detector sees the future.
rolling_zscore |
trailing mean/std, score = abs z |
rolling_mad |
trailing median/MAD — robust to the anomalies it is looking for |
rolling_diff |
z-score of the first difference; catches abrupt level shifts |
isolation_forest |
sklearn, fit on the probationary prefix, 5 windowed features |
random_control |
uniform noise. The floor. |
point_wise |
every timestamp judged on its own. Strict. |
point_adjusted |
one hit anywhere in a window marks the entire window correct. Optimistic. |
window_recall |
fraction of windows containing >=1 detection, paired with alert rate. |
A single lucky sample inside a 100-point window scores F1 = 0.02 point-wise and 1.00 point-adjusted. That one line explains the whole table.
uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -r requirements.txt .venv/bin/python scripts/fetch_data.py # downloads NAB streams + labels .venv/bin/python scripts/run.py # ~1 min .venv/bin/python scripts/plot.py
Outputs results/per_stream.csv, results/summary.csv, results/summary.json,
results/protocols.png.
scripts/ fetch_data.py detectors.py evaluate.py run.py plot.py
data/ NAB streams + combined_windows.json (downloaded, not vendored)
results/ per_stream.csv summary.csv summary.json protocols.png
Code: Apache-2.0. Data is NAB (Numenta, Apache-2.0), downloaded at run time rather
than vendored — see scripts/fetch_data.py.