forecast-backtest-kit

A rolling-origin backtesting kit for demand forecasting whose feature pipeline is provably leak-free: corrupting every post-origin actual changes zero predictions, a property enforced as a CI test.

CI coverage leakage license

• Most forecast backtests leak: per-target-day lag features quietly feed actuals from inside the forecast window to the model; this repo's first implementation did exactly that (193 of 210 predictions were contaminated) and the fix plus its regression test are the centerpiece.
• MAPE lies on intermittent demand (division by zero-demand days); the kit reports WAPE as the headline, MAPE only over nonzero days with the exclusion count shown, and signed bias so over-forecasting cannot hide.
• A single aggregate error hides where the model helps; results split by SKU segment (head, torso, tail, cold-start) against two honest baselines, so "the model wins" becomes "the model wins on head and torso, and here is the tail story".

Demand forecasting drives inventory dollars: over-forecast and capital sits in a warehouse; under-forecast and you stock out on payday weekend. Model improvements are claimed via backtests, so a backtest that flatters the model is worse than no backtest, and the most common flattery is temporal leakage, which produces beautiful offline numbers and a production launch that mysteriously underperforms them.

fbk is a small kit built around one discipline: forecasts for origin O may use nothing after day O-1. Features (lags, rolling means, same-weekday naive) are computed in SQL, anchored at the as-of day with the horizon step h as a model feature (direct multi-horizon forecasting, ADR-001), so the leakage boundary is arithmetic on day indexes in one reviewable file (sql/features.sql). The protocol is verified by a corruption probe that runs in CI: replace every actual at day >= origin with garbage and require that not a single prediction changes.

The included panel generator produces a synthetic but structured demand history (60 SKUs, 540 days: volume tiers, weekly seasonality, promos, cold-start launches, integer noise), labeled clearly as synthetic. On it, LightGBM with as-of features beats seasonal-naive by a wide margin on high-volume SKUs and by much less on the intermittent tail, which is the honest, decision-relevant shape of most real demand forecasting results.

flowchart LR
 G[Panel generator<br/>tiers, promos, cold starts] --> DB[(SQLite panel)]
 DB --> FT[features_train_h<br/>as-of anchored, h in 1,4,8,14]
 DB --> FX[features_test_asof<br/>anchored at origin-1]
 FT --> M1[LightGBM<br/>poisson objective]
 FT --> M2[seasonal-naive]
 FT --> M3[moving-average-28]
 FX --> M1 & M2 & M3
 M1 & M2 & M3 --> BT[Rolling-origin backtest<br/>6 origins, horizon 14]
 BT --> R[Per-segment WAPE, MAPE, bias<br/>results JSON + leaderboard]
 LT[Leakage corruption test] -.verifies.-> FX

Prerequisites: Python 3.10+, git.

git clone https://github.com/tkgo1599-max/forecast-backtest-kit.git
cd forecast-backtest-kit
pip install -e ".[dev]"
pytest # 17 tests, including the leakage probe
make run # full backtest: 60 SKUs, 540 days, leaderboard
./scripts/leak_probe.sh # just the leakage regression, verbose

To backtest your own panel: load rows (sku, day, dow, promo, tier, demand) into PanelStore, keep your feature changes inside sql/features.sql, and re-run the leak probe before believing any new number.

Backtest on the reference panel (40 SKUs, 400 days, 6 monthly origins, 14-day horizon), abridged to the "all" and extreme segments; full table from make run:

The segment split is the point: the model's advantage concentrates in high-volume SKUs where per-SKU patterns are learnable; on the intermittent tail the gap narrows toward the moving average. A team using this kit would deploy the model for head and torso and keep a cheap baseline for the deep tail, which is a decision an aggregate WAPE could never justify.

Methodology: python benchmark/bench.py runs the complete backtest (SQL features at 4 training horizons per origin, 3 models, 6 origins, horizon 14) at three panel scales. Linux container, shared vCPUs. Raw: benchmark/results/results.json.

Wall time scales with panel rows times the training-horizon mix (4x), dominated by the correlated-subquery rolling means; ADR-002 names the window-frame rewrite as the first optimization if panels grow 10x.

• ADR-001: Direct multi-horizon with h as a feature, not recursive
• ADR-002: Feature engineering in SQL, not pandas

Hierarchical reconciliation (SKU-store-region coherence) is deliberately absent: it matters once forecasts feed a hierarchy of decisions, and the trigger is the first consumer that aggregates these forecasts. Also out: probabilistic forecasts (quantile objectives are a LightGBM parameter away, but honest evaluation needs pinball loss and calibration plots, a project of its own), stockout censoring correction, and hyperparameter search (the kit measures protocols, not leaderboard squeezing).

The kit is offline and needs no credentials. Real demand panels are commercially sensitive: treat any repo containing one as private, and note that the SQLite database is in-memory by default so no panel data lands on disk unless you choose a file path.

The first backtest looked great and was wrong. Multi-step forecasts were consuming actual demand from inside their own forecast window: features were computed per target day, so predicting day origin+5 used the real demand at origin+4, information no forecaster has. Offline error was flattered in exactly the way that later disappoints in production.

What makes this story worth telling is the detection method, because eyeballing the feature code had already failed once: corrupt every actual at day >= origin with garbage and re-predict. A leak-free pipeline must produce bit-identical predictions. The probe changed 193 of 210 predictions. The fix (commit "fix(features): anchor all features at the as-of day") rebuilt the features as origin-anchored SQL with h as a model feature, and turned the probe into a permanent regression test that now passes at zero changed predictions and runs in CI. Notably, on this panel the honest numbers came out close to the leaky ones, which is its own lesson: leakage does not announce itself with implausible scores; you have to test for it structurally.

• Quantile forecasts (LightGBM pinball objectives) with calibration reporting, for safety-stock decisions.
• Warehouse SQL port of the feature queries (the reason they are plain SQL).
• Stockout-aware evaluation: mask censored days from error, report separately.
• Feature importance drift across origins, to catch regime changes in what the model uses.
• First metric to watch when adopting: the gap between backtest WAPE and the first month of production WAPE; that gap is the leakage-and-drift detector for the whole system.