An automated machine learning pipeline for model training, evaluation, and deployment with 75% cloud cost reduction through spot instance orchestration.
| Feature | Details |
|---|---|
| Automated HPO | Optuna TPE sampler + Hyperband pruning, 50 trials, parallel |
| Multi-Model | Random Forest, XGBoost, LightGBM, Logistic Regression |
| Spot Instances | Dynamic bid pricing, interruption handling, checkpointing |
| Cost Reduction | ~75% savings vs on-demand through spot orchestration |
| Model Registry | SQLite-backed versioning + auto-promotion to production |
| Blue/Green Deploy | Health checks, automatic rollback on degradation |
| REST + WebSocket API | FastAPI with real-time pipeline updates |
| Premium Dashboard | Glassmorphism dark UI with live Chart.js visualizations |
ML-pipeline/
βββ config/
β βββ pipeline_config.yaml # Master configuration
βββ src/
β βββ pipeline/
β β βββ orchestrator.py # Main pipeline coordinator (async)
β β βββ data_ingestion.py # Load, preprocess, split data
β β βββ hyperparameter_tuner.py # Optuna HPO engine
β β βββ trainer.py # Multi-model trainer
β β βββ evaluator.py # Metrics & reporting
β β βββ model_registry.py # SQLite registry
β βββ resources/
β β βββ spot_orchestrator.py # Spot instance lifecycle
β β βββ cost_monitor.py # Real-time cost tracking
β β βββ resource_allocator.py # Bin-packing scheduler
β βββ deployment/
β β βββ deployer.py # Blue/green deployment
β β βββ model_server.py # FastAPI inference layer
β βββ api/
β βββ main.py # FastAPI app (REST + WS)
β βββ websocket_manager.py # Live broadcast manager
βββ dashboard/
β βββ index.html # Single-page dashboard
β βββ style.css # Glassmorphism dark theme
β βββ app.js # Chart.js + WebSocket client
βββ scripts/
β βββ run_pipeline.py # CLI entry point
β βββ demo_dataset.py # Synthetic dataset generator
βββ tests/
β βββ test_pipeline.py # Pytest suite
βββ requirements.txt
βββ Dockerfile
βββ docker-compose.yml
cd "c:\Users\Bennerdo\OneDrive\Documents\PROJECTS\ML-pipeline" pip install -r requirements.txt
# Built-in datasets (iris, wine, breast_cancer, diabetes) python scripts/run_pipeline.py --dataset iris # Synthetic classification dataset python scripts/run_pipeline.py --dataset synthetic_classification # Skip HPO (faster) python scripts/run_pipeline.py --dataset iris --no-tune # Custom CSV file python scripts/run_pipeline.py --dataset data/mydata.csv --target price --task-type regression
set PYTHONPATH=.
uvicorn src.api.main:app --reload --port 8000Then open the dashboard: dashboard/index.html in your browser, or visit http://localhost:8000/docs for the Swagger API.
docker compose up --build # API: http://localhost:8000 # Dashboard: http://localhost:3000
| Method | Endpoint | Description |
|---|---|---|
GET |
/health |
Health check |
POST |
/pipeline/run |
Trigger pipeline run |
GET |
/pipeline/status |
Current run status |
GET |
/registry/models |
List registered models |
POST |
/registry/promote |
Promote model to stage |
POST |
/models/{name}/predict |
Single prediction |
POST |
/models/{name}/batch_predict |
Batch predictions |
GET |
/cost/summary |
Cost savings report |
WS |
/ws |
Real-time updates |
The pipeline achieves ~75% cost reduction by:
- Using spot instances for HPO (most expensive stage) at 25-35% of on-demand price
- Using spot instances for training and evaluation
- Only using on-demand for ingestion and deployment (reliability-critical)
- Checkpointing every 60s to survive spot interruptions
- Fallback to on-demand automatically after max retries
| Stage | Spot? | Savings |
|---|---|---|
| Data Ingestion | β | 0% |
| HPO Tuning | β | 70-75% |
| Model Training | β | 70-75% |
| Evaluation | β | 70-75% |
| Deployment | β | 0% |
| Overall | ~75% |
Edit config/pipeline_config.yaml to tune:
hyperparameter_tuning.n_trialsβ number of Optuna trialsresources.profiles.*.spot_max_price_ratioβ spot bid as fraction of on-demandmodels.enabledβ which model families to trainregistry.promotion.min_improvementβ score delta required to auto-promote
python -m pytest tests/ -v --tb=short