Smart Dispatch

Delivery Time Prediction and Late Delivery Classification using PySpark ML.

Smart Dispatch live prediction demo

• Linear Regression
• Logistic Regression

Synthetic Delivery Dataset

• PySpark
• Spark ML
• Hadoop HDFS
• Jupyter Notebook

Run in order — each notebook reads artifacts the previous one saved. All notebooks expect to be run with the project root (smart_dispatch/) as the working directory (VS Code: set via .vscode/settings.json's jupyter.notebookFileRoot), using the venv (3.12.13.final.0) / smart_dispatch kernel, with SPARK_HOME set (see ~/.bashrc).

1. scripts/generate_dataset.py — run once first (python scripts/generate_dataset.py) to create data/raw/delivery_data.csv (10,000 synthetic rows). Regenerate any time you want a fresh dataset — it's deterministic (seed=42).
2. 01_Data_Preparation.ipynb — loads the raw CSV, validates it (nulls, duplicates, feature ranges, label balance), builds and fits the preprocessing pipeline (categorical encoding + scaling), and saves:
   • data/processed/prepared_data.parquet
   • models/preprocessing_pipeline/
3. 02_Model_Training.ipynb — loads prepared_data.parquet, splits train/test (80/20, seed=42), trains a LinearRegression (target: actual_time_mins) and a LogisticRegression (target: is_late), and saves:
   • data/processed/train_data.parquet, data/processed/test_data.parquet
   • models/regression_model/, models/classification_model/
4. 03_Model_Evaluation.ipynb — loads both models and the test set, evaluates them (RMSE/R2/MAE, AUC/accuracy/precision/recall/F1, confusion matrix, feature importance, error analysis), and saves output/metrics/metrics_<timestamp>.json.
5. 04_Model_Inference.ipynb — loads all three saved models, defines single-record and batch prediction functions, runs deployment-readiness checks (edge cases, latency, memory), and saves:
   • scripts/predict.py — a dependency-free (no Spark/JVM) scoring function for low-latency serving
   • output/predictions/predictions_<timestamp>.csv

Each notebook can also be run non-interactively, e.g.:

jupyter nbconvert --to notebook --execute --inplace \
 --ExecutePreprocessor.kernel_name=smart_dispatch \
 notebooks/01_Data_Preparation.ipynb

run from the project root (so relative paths and config.py imports resolve correctly).

api/app.py is a Flask app built on top of scripts/predict.py (the dependency-free scorer from 04_Model_Inference.ipynb — no Spark/JVM per request, so it's actually suited to real-time traffic; the Spark models themselves take ~850ms/call and shouldn't be called directly per-request). Regenerate scripts/predict.py by re-running that notebook's Section 5 if the models are retrained.

From the project root:

# dev (foreground)
python api/app.py
# production (foreground)
gunicorn -w 4 -b 0.0.0.0:8000 api.app:app
# production, backgrounded
nohup gunicorn -w 4 -b 0.0.0.0:8000 api.app:app > /tmp/smart_dispatch_api.log 2>&1 &
disown

/predict only accepts POST requests with a JSON body, so you can't view a prediction by just navigating to it — but GET / serves a small HTML form for exactly that:

1. With the server running (step 1), open http://127.0.0.1:8000/ in a browser. On WSL2 this is reachable directly from Windows (localhost auto-forwards); VS Code may also pop up a "port 8000 available" notification you can click instead.
2. The form is pre-filled with sample values (distance, packages, traffic index, weather, driver experience, time of day). Click Predict.
3. The JSON result renders below the form — green background on success (delivery_time, is_late, probability_late), red on error (e.g. a missing/invalid field).

Each submission calls POST /predict under the hood via fetch(), so it's the same response you'd get from curl — just rendered instead of printed:

curl -X POST http://127.0.0.1:8000/predict \
 -H "Content-Type: application/json" \
 -d '{"distance_km": 18.5, "num_packages": 6, "traffic_index": 6.2, "weather_condition": "Rain", "driver_experience_yrs": 4.0, "time_of_day": "Rush_Hour"}'

{"delivery_time": 71.04, "is_late": 1, "probability_late": 0.989}

Returns 400 with an {"error": ...} body for missing fields or invalid value types; unknown weather_condition/time_of_day values are accepted (mapped to the pipeline's reserved "unseen category" index) rather than rejected.

• Foreground (python api/app.py or a plain gunicorn ... command in a terminal): press Ctrl+C in that terminal.

• Backgrounded (started with nohup ... &, or you lost track of the terminal): find and stop the gunicorn master process —

  pkill -f "gunicorn.*api.app:app"

  or find its PID first and send it a graceful shutdown signal:

  ps aux | grep "gunicorn.*api.app:app"
  kill <PID> # SIGTERM: graceful, finishes in-flight requests

  Avoid kill -9 unless a plain kill doesn't stop it — that skips gunicorn's graceful worker shutdown.

This project is licensed under the MIT License — you are free to use, modify, and distribute it. See the LICENSE file for details.