Skip to content

Navigation Menu

Sign in
Sign up

Latest commit

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

Sensor anomaly detection: what the evaluation protocol is really measuring

Baseline anomaly detectors on 8 real sensor and telemetry streams from the Numenta Anomaly Benchmark, evaluated under three protocols — including a random control that has no ability to detect anything.

The headline result: under the widely-used "point-adjust" protocol, the random control scores 0.956 F1 — second of five, ahead of every method that genuinely detects anything, and within 0.001 of the top score. Under strict point-wise scoring it ranks near the bottom, where it belongs.

Protocol comparison

Results

Mean over 8 streams. Both F1 columns are best-achievable over a threshold sweep, applied identically to every detector.

Detector point-wise F1 point-adjusted F1 inflation window recall alerts / 1000 pts
isolation_forest 0.310 0.9504 3.07x 0.969 54
rolling_mad 0.211 0.9299 4.41x 0.969 115
rolling_zscore 0.209 0.9450 4.52x 0.969 127
random_control 0.179 0.9557 5.35x 0.969 211
rolling_diff 0.177 0.9564 5.39x 0.969 183

Point-adjusted F1 is shown to four decimals deliberately: at three, rolling_diff and random_control both read 0.956 and the ordering looks like a tie.

Three things fall out of this:

  1. Point-adjust destroys the ranking. It inflates F1 by 3-5.4x and lifts pure noise above every detector that actually works — Isolation Forest, which is comfortably best under honest scoring, drops below the random control. Any result reported only under point-adjust tells you almost nothing about whether a detector works. This is not a new observation — it replicates the critique in Kim et al., Towards a Rigorous Evaluation of Time-series Anomaly Detection (AAAI 2022) — but it is worth reproducing on data you can download.
  2. Point-wise, everything is mediocre. The best detector reaches F1 = 0.31. NAB's anomalies are labelled as broad windows while these detectors fire on individual points, so precision is structurally capped. Difficulty here is in precision, not recall.
  3. The useful comparison is recall against alert cost. Every detector catches the same ~97% of labelled windows. Isolation Forest does it with 54 alerts per 1000 points against the random control's 211 — a ~4x lower false-alarm burden for the same catch rate. That, not F1, is the number an operator would care about.

Why the random control is in here

A benchmark without a floor cannot tell you whether a method is working. Including a detector that provably knows nothing makes any protocol that ranks it first immediately suspect. It cost about ten lines and it is the most informative row in the table.

Honest limitations

  • The F1 columns are optimistic upper bounds. The threshold is chosen with knowledge of the labels (best-F1 over a sweep). This is applied identically to every detector so the comparison is fair, but no number here is what you would get choosing a threshold in advance. Stated plainly because this is frequently left implicit in published tables.
  • No deep-learning baseline. LSTM/autoencoder/transformer detectors are not included, so this does not show classical methods beating them. It shows that the protocol has to be pinned down before any such comparison means anything.
  • 8 streams, one benchmark. NAB's labelling convention (wide windows around a root cause) directly shapes conclusion 2. A point-labelled dataset would behave differently.
  • Single seed (1337), no confidence intervals. Isolation Forest and the random control are stochastic; the rolling detectors are deterministic.
  • Anomaly rate is ~10% by construction of the NAB windows — much denser than most real deployments, which flatters precision relative to a production setting.

Detectors

All are streaming-safe: rolling statistics are shifted so a point is never part of its own baseline, and the fitted model only ever sees the first 15% (NAB's probationary period). No detector sees the future.

rolling_zscore trailing mean/std, score = abs z
rolling_mad trailing median/MAD — robust to the anomalies it is looking for
rolling_diff z-score of the first difference; catches abrupt level shifts
isolation_forest sklearn, fit on the probationary prefix, 5 windowed features
random_control uniform noise. The floor.

Protocols

point_wise every timestamp judged on its own. Strict.
point_adjusted one hit anywhere in a window marks the entire window correct. Optimistic.
window_recall fraction of windows containing >=1 detection, paired with alert rate.

A single lucky sample inside a 100-point window scores F1 = 0.02 point-wise and 1.00 point-adjusted. That one line explains the whole table.

Reproducing

uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -r requirements.txt
.venv/bin/python scripts/fetch_data.py # downloads NAB streams + labels
.venv/bin/python scripts/run.py # ~1 min
.venv/bin/python scripts/plot.py

Outputs results/per_stream.csv, results/summary.csv, results/summary.json, results/protocols.png.

Layout

scripts/ fetch_data.py detectors.py evaluate.py run.py plot.py
data/ NAB streams + combined_windows.json (downloaded, not vendored)
results/ per_stream.csv summary.csv summary.json protocols.png

Licence and data

Code: Apache-2.0. Data is NAB (Numenta, Apache-2.0), downloaded at run time rather than vendored — see scripts/fetch_data.py.

About

Anomaly detection baselines on 8 real NAB sensor streams — including a random control that beats every working detector under the common point-adjust protocol. Three evaluation protocols, honest upper bounds, reproducible.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

AltStyle によって変換されたページ (->オリジナル) /