Static Badge Static Badge Coverage

Multi-country energy price forecasting system with event awareness and data quality guards.

Logo

• Multi-country support: Extensible architecture for any country/market
• Event-aware forecasting: Incorporates holidays, DST transitions, and manual events
• Data quality guards: Cleaning and basic validation for electricity, weather, and gas data
• Advanced Model Ensembling: Supports XGBoost, LightGBM, Scikit-learn baselines, and Weighted/Stacking ensembles
• Robust Validation: Multi-year Time Series Split Walk-Forward Validation with expanding/sliding windows
• Operational monitoring: Daily health summaries, threshold alerts, and runbooks
• Uncertainty-aware forecasts: Stable p10/p50/p90 intervals with scorecard-driven calibration
• Country abstraction: Add new countries with minimal code changes

• Fully Implemented MVP: End-to-end pipeline (Fetch → Clean → Features → Train → Forecast) works for Mock Country (XX) and Portugal (PT).
• Core Capabilities:
  • Inference: Day-ahead price forecasting with runtime data guards.
  • Validation: Walk-Forward Validation with expanding/sliding windows to analyze seasonal drift over multi-year datasets (2022-2026).
  • Modelling: Advanced ensembling (Weighted/Stacking) combining XGBoost, LightGBM, and Ridge Regression.
  • Quality: Windows-compatible automation, CI/CD guardrails, and daily ops monitoring.
• Phases 4-6: Advanced Ensembling and Multi-Year Walk-Forward Validation (Complete).
• Phase 7: Macro Integration (TTF Gas) and Model Adaptability (In Progress).
• Phases 8–10: CI/CD (Complete). Monitoring and Deployment (Next).

• Portugal (PT): Full implementation with ENTSO-E, Open-Meteo, and TTF data.
• Mock Country (XX): Synthetic data for testing and fast training demos.

A quick guide to get started with PriceSentinel.

# Clone repository
git clone https://github.com/JDPS/pricesentinel.git
cd pricesentinel
# Create and activate a virtual environment (example with venv)
python -m venv .venv
# Windows:
.venv\Scripts\activate
# Linux/Mac:
source .venv/bin/activate
# Install runtime dependencies
pip install .
# (Optional) Install development extras (tests, linting, docs)
pip install ".[dev,test,docs]"
# Configure environment
copy .env.example .env
# Edit .env with your API keys

1. Add your ENTSO-E API key to .env:

   ENTSOE_API_KEY=your_key_here

2. (Optional) Download TTF gas prices and place in:

   data/manual_imports/ttf_gas_prices.csv
   

# Test with mock country (no API keys required) – fetch only
python run_pipeline.py --country XX --fetch --start-date 2024年01月01日 --end-date 2024年01月07日
# Run full pipeline for mock country (fetch → clean → features → train)
python run_pipeline.py --country XX --all --start-date 2024年01月01日 --end-date 2024年01月07日
# Fetch data for Portugal
python run_pipeline.py --country PT --fetch --start-date 2024年01月01日 --end-date 2026年01月16日
# Run full pipeline for Portugal (training assumes sufficient data and configuration)
python run_pipeline.py --country PT --all --start-date 2024年01月01日 --end-date 2024年12月31日
# Generate forecasts (after training)
python run_pipeline.py --country PT --forecast --forecast-date 2024年01月08日

These commands are shell-agnostic. Only the env var syntax changes by shell.

Set ENTSO-E key:

• PowerShell: $env:ENTSOE_API_KEY="your_key"
• Bash: export ENTSOE_API_KEY="your_key"

Then run:

# 1) Backfill/fetch (long ranges are chunked automatically for PT ENTSO-E)
uv run python run_pipeline.py --country PT --fetch --start-date 2024年01月01日 --end-date 2026年01月16日
# 2) Reproducible training
uv run python experiments/run_training.py \
 --country PT \
 --model-name baseline \
 --train-start 2024年01月01日 \
 --train-end 2025年12月31日 \
 --holdout-start 2026年01月01日 \
 --holdout-end 2026年01月16日
# 3) Champion selection (will skip unavailable optional models gracefully)
uv run python experiments/select_champion.py --country PT --start 2024年01月01日 --end 2026年01月16日
# 4) Daily forecast for D+1
uv run python experiments/daily_ops.py forecast --country PT --target-date 2026年02月17日
# 5) Evaluate D (when actuals are complete)
uv run python experiments/daily_ops.py evaluate --country PT --target-date 2026年02月16日

Evaluate model performance using Time Series Split Cross-Validation:

python experiments/run_cv.py --country PT --start 2023年01月01日 --end 2023年12月31日 --splits 5

This generates a detailed report in outputs/reports/.

Run a full backtest benchmark (Train 2023 / Test Jan 2024):

python experiments/benchmark_pt.py

pricesentinel/
 config/ # Configuration files
 countries/ # Country-specific configs
 PT.yaml # Portugal configuration
 XX.yaml # Mock country configuration
 country_registry.py # Country registry and factory
 validation.py # Pydantic validation schemas
 core/ # Core pipeline logic
 abstractions.py # Abstract base classes
 data_manager.py # Data directory management
 logging_config.py # Logging setup
 cleaning.py # Data cleaning and verification
 features.py # Feature engineering
 pipeline.py # Main pipeline orchestration
 cross_validation.py # CV logic
 guards.py # Runtime data guards
 data_fetchers/ # Data source adapters
 mock/ # Mock country (synthetic data)
 portugal/ # Portugal-specific fetchers
 shared/ # Reusable fetchers (Open-Meteo, TTF)
 models/ # Model trainers and saved artefacts
 run_forecast.py # Forecasting CLI
 experiments/ # Experiments and Benchmarks
 run_cv.py # Cross-Validation CLI
 benchmark_pt.py # Portugal Benchmark
 data/ # Data storage (gitignored)
 PT/ # Portugal data
 XX/ # Mock country data
 tests/ # Test suite
 run_pipeline.py # Main CLI entry point
 setup_country.py # Country setup utility
 tasks.py # Invoke-based automation
 pyproject.toml # Project and dependency metadata

PriceSentinel uses an adapter pattern to remain country-agnostic while supporting country-specific data sources:

1. Abstract Base Classes: Define interfaces for all data fetchers
2. Country Registry: Maps country codes to specific implementations
3. Factory Pattern: Creates appropriate fetchers for each country
4. Country Configuration: YAML files define country-specific parameters

See dev_ws/RevisedPhase0_and_Phase1.md (country extension guide) for detailed instructions.

Quick summary:

1. Create config/countries/{CODE}.yaml

2. Implement country-specific fetchers (if needed)

3. Register in data_fetchers/__init__.py

4. Test with:

   python run_pipeline.py --country {CODE} --info

# Run all tests
pytest
# Run with coverage_enable (HTML and terminal)
pytest --cov=. --cov-report=html --cov-report=term-missing

The coverage badge at the top of this README (coverage.svg) can be regenerated from the coverage tools.

# Create directory structure
python setup_country.py ES
# This creates:
# - data/ES/ directories
# - Prompts for next steps

• Architecture Overview: docs/ARCHITECTURE.md
• Training & Features: docs/TRAINING.md
• Feature Toggles (YAML): docs/FEATURE_TOGGLES.md
• Implementation Phases 0-1: .dev_ws/RevisedPhase0_and_Phase1.md
• Extended Roadmap Phases 2-10: .dev_ws/RevisedPhases2-10_CountryAbstraction.md
• Consolidated Assessment & Refactoring Plan: .dev_ws/CONSOLIDATED_ASSESSMENT.md

For quick health checks of a trained run and its forecasts, you can use:

uv run python inspect_run.py --country XX --model-name baseline

This prints metrics from metrics.json and basic stats for any forecast CSVs under data/{country}/processed/forecasts/ for the given model.

• Core abstractions, Registry, CLI
• Portugal implementation
• Data verification and cleaning
• Quality checks and guards

• Advanced Feature engineering (lags, rolling windows)
• Advanced Ensembling (XGBoost, LightGBM, Weighted Ensembles)
• Walk-Forward Validation Engine
• Runtime guards (Input validation)

• Phase 6: Multi-Year Validation & Model Drift Analysis (Complete)
• Phase 7: Macroeconomic Integration (TTF Gas) & Sliding Window Adaptability (In Progress)

• CI/CD Guardrails (GitHub Actions, Pre-commit) - Complete
• Extended testing (100% pass rate) - Complete
• Monitoring and Alerting
• Deployment Pipelines

• Python 3.13+
• See pyproject.toml for dependencies and optional extras

• ENTSO-E Transparency Platform (for EU electricity data)

  • Register at: https://transparency.entsoe.eu/
  • Free API key required

• Open-Meteo (for weather data)

  • Free tier: https://open-meteo.com/
  • No API key required

• TTF Gas Prices (manual download for MVP)

  • Future: API integration planned

This is currently a development project. Contribution guidelines will be added in Phase 10.

This project is licensed under the Apache Licence 2.0 – see the LICENSE file for details.

For questions or issues:

• Technical issues: joaosoarex@gmail.com
• API access: Check respective API provider documentation

Note: This project is in active development. Many features are planned but not yet fully implemented. See the roadmap above for current status.