-
Notifications
You must be signed in to change notification settings - Fork 0
Evaluation Workflow
The evaluation stage scores a fitted ClassifierEnsemble against ground truth, never re-fitting.
It reuses the upstream TrainingModel or PredictionModel (in-memory or from a .pkl) and
writes per-classifier (n_samples, n_seeds) y_proba / y_pred CSV matrices plus aggregate
metric plots, with cross-classifier ranking via ClassifierComparator. Per-classifier per-seed
metric tables stay in memory on self.metrics β no per-seed JSON tree is produced.
Driver: EvaluationModel (src/eruption_forecast/model/evaluation_model.py). Wrapped by ForecastModel.evaluate(...).
EvaluationModel dispatches on model.kind:
fm.evaluate(model="...")
β
βββββββββββββββ΄ββββββββββββββ
βΌ βΌ
model.kind == "training" model.kind == "prediction"
β β
ββββββββββββββββ΄βββββββββββββ βββββββββββββ΄ββββββββββββββββββββββ
β Training reuse β β Prediction reuse β
β β β β
β y_true β TrainingModel β β y_true β fresh LabelBuilder β
β .labels β β over prediction window grid, β
β (already ground truth on β β joined to PredictionModel β
β the labelled grid) β β .labels by datetime β
β β β β
β eruption_dates: optional β β eruption_dates: REQUIRED β
ββββββββββββββββ¬βββββββββββββ βββββββββββββββββ¬ββββββββββββββββββ
β β
βΌ βΌ
output to evaluation/training/ output to evaluation/prediction/
Both modes share the same per-seed scoring engine (MetricsEnsemble) and aggregation step.
| Mode | When to use | eruption_dates |
|---|---|---|
model="training" |
In-sample / training-window diagnostics | optional - embedded in training labels |
model="prediction" |
Forecast-window evaluation after predict()
|
required to build truth on the prediction grid |
For each classifier in ClassifierEnsemble:
MetricsEnsemble.compute()
βββ ClassifierEnsemble.predict_proba(features_df)
β β (n_samples, n_seeds) probability matrix
βββ threshold at 0.5
β β (n_samples, n_seeds) prediction matrix
βββ persist predictions/{y_proba,y_pred}.csv
βββ per-seed metric loop in joblib (compute_seed)
β β in-memory metrics: dict[classifier, pd.DataFrame(seed Γγ°γ€ metric)]
βββ idempotent fast-path: re-running compute() with populated
y_probas is a no-op
The result of evaluate() is a dict[classifier_name, pd.DataFrame] β one DataFrame per classifier, one row per seed, one column per metric. The dict is also cached on self.metrics for downstream use.
Available metric columns (per seed):
accuracy balanced_accuracy precision recall
f1_score roc_auc pr_auc g_mean
true_positives true_negatives false_positives false_negatives
sensitivity specificity optimal_threshold
f1_at_optimal recall_at_optimal precision_at_optimal
em.evaluate( plot_aggregate=True, # ROC, PR, threshold, g-mean, MCC per classifier plot_per_seed=False, # same dispatcher, one plot file per seed compare_classifiers=True, # also run ClassifierComparator at the end use_cache=True, # skip the load path when False ) -> dict[str, pd.DataFrame]
fm.evaluate(...) forwards plot_per_seed, plot_aggregate, and use_cache from its own kwargs. SHAP plots are produced by the dedicated Explanation Workflow via ExplanationModel.explain(). use_cache=False skips the internal EvaluationModel.load(...) short-circuit even when a cached pickle exists on disk β independent of overwrite, which additionally controls plot regeneration.
comparator = em.compare(metrics=["recall", "roc_auc"]) # or em.compare() for defaults comparator.get_ranking() # β comparison/metrics/ranking_recall.csv comparator.plot_all() # β comparison/figures/*.png
ClassifierComparator works on the metrics already computed by MetricsEnsemble - repeat em.compare()
calls reuse the cached MetricsEnsemble, so the per-classifier predict_proba pass is only paid once.
Outputs land under {evaluation_dir}/comparison/:
| Artefact | Content |
|---|---|
metrics/ranking_{metric}.csv |
Classifiers sorted by mean of {metric}
|
figures/metric_bar_{metric}.png |
Bar chart per metric (mean Β± std) |
figures/metric_bar_all.png |
All metrics in one figure |
figures/seed_stability_{metric}.png |
Violin + strip plot per metric across seeds |
figures/comparison_grid.png |
Classifier Γγ°γ€ metric grid |
figures/comparison_roc.png |
Overlaid mean ROC curves with Β± std bands |
When invoked through fm.EvaluationModel.compare(), the live (ClassifierEnsemble, features_df, y_true)
triple is forwarded as ensemble_source so the ROC overlay computes from in-memory probabilities rather than re-reading CSVs.
{station_dir}/evaluation/{training|prediction}/
βββ classifiers/
β βββ {classifier-name}/
β βββ predictions/
β β βββ y_proba.csv # (n_samples, n_seeds)
β β βββ y_pred.csv # (n_samples, n_seeds)
β βββ figures/
β βββ aggregate/{plot_name}.{png,csv} # plot_aggregate=True
β βββ {plot_name}/{seed:05d}.png # plot_per_seed=True
βββ labels/y_true.csv # prediction-reuse mode only
βββ MetricsEnsemble.pkl # optional, via em.MetricsEnsemble.save()
βββ comparison/ # populated when em.compare() runs
βββ metrics/ranking_*.csv
βββ figures/*.png
Per-classifier folder names use the unslugified sklearn class name (e.g. RandomForestClassifier),
distinct from the slug used by TrainingModel (random-forest-classifier). Aggregate / per-seed
plot names come from evaluation_plots.AGGREGATE_PLOT_DISPATCHER and
PER_SEED_PLOT_DISPATCHER (roc_curve, precision_recall, threshold_analysis, g_mean_curve, mcc_curve, confusion_matrix).
EvaluationModel participates in the BaseModel content-addressable cache layer. Cache identity is built from an upstream-model fingerprint (model kind, classifier list, features shape and column set, eruption_dates, evaluation window) plus the evaluate() knobs (plot_aggregate, plot_per_seed, compare_classifiers). On a cache hit self.metrics, self.MetricsEnsemble, and self.comparator are restored from {evaluation_dir}/{hash}.EvaluationModel.pkl without re-running the per-classifier predict_proba pass. use_cache=True (default) gates both the load and the write; pass use_cache=False to skip the load path even when a cached pickle exists β independent of overwrite, which additionally controls plot regeneration. When the cache misses and computes fresh, in-process re-runs are further gated by MetricsEnsemble.compute()'s in-memory idempotency fast-path (once self.y_probas is populated, repeated compute() calls short-circuit).
from eruption_forecast import EvaluationModel # training-window evaluation em = EvaluationModel.from_file( "output/VG.OJN.00.EHZ/TrainingModel_2025εΉ΄01ζ01ζ₯_2025εΉ΄07ζ26ζ₯.pkl", ) metrics = em.evaluate(plot_aggregate=True) # forecast-window evaluation - eruption_dates required em = EvaluationModel.from_file( "output/VG.OJN.00.EHZ/PredictionModel_2025εΉ΄07ζ27ζ₯_2025εΉ΄08ζ22ζ₯.pkl", eruption_dates=["2025-08-02", "2025-08-18"], ) metrics = em.evaluate(plot_aggregate=True) comparator = em.compare() print(comparator.get_ranking()) comparator.plot_all()
rf_metrics = metrics["RandomForestClassifier"] print(rf_metrics["recall"].describe()) # mean std min max # 0.81 0.06 0.69 0.92
ranking = em.compare(metrics="balanced_accuracy").get_ranking(metric="balanced_accuracy")
em.save_config() # β {evaluation_dir}/evaluation.config.yaml
evaluate() already auto-calls save_config() once self.metrics is set, so
a standalone evaluation always leaves a YAML snapshot at
{output_dir}/evaluation/{training|prediction}/evaluation.config.yaml. The
path is already mode-namespaced, so a training-reuse and a prediction-reuse
run sharing the same output_dir never collide. The upstream model
parameter is intentionally omitted from the config (live model instances are
not serializable); the captured fields are eruption_dates, overwrite,
output_dir, root_dir, n_jobs, and verbose.
See Configuration.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β EvaluationModel (BaseModel cache layer) β
β β
β ββββββββββββββββββββββββββββββββββββββββββββ β
β β MetricsEnsemble.compute() β β
β β per-classifier predict_proba β β
β β (n_samples Γγ°γ€ n_seeds) y_proba / y_pred β β
β β per-seed metrics β self.metrics (mem) β β
β β idempotent once y_probas is populated β β
β ββββββββββββββββββββββ¬ββββββββββββββββββββββ β
β β cached on self.MetricsEnsemble β
β βΌ β
β em.compare() β ClassifierComparator β
β ranking CSV + comparison plots β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ