Goal: Learn the workflow FAANG expects after you have a model: diagnose errors, find slices that fail, run feature ablations, and iterate safely.

Outcome: Students can:

• build a baseline churn classifier,
• compute slice metrics and identify failure cohorts,
• perform error analysis (FP vs FN),
• run feature ablation and interpret results,
• tune threshold for business cost.

1. Fork this repository.
2. Open debug_student_lab.ipynb in Google Colab.
3. Complete all TODO sections.
4. Restart runtime → Run All cells.
5. Push changes and submit a Pull Request.

⚠️ Do NOT edit notebooks directly on GitHub.

• ✅ Start with a baseline, then debug systematically
• ✅ Slice analysis must be on a held-out set
• ✅ Keep preprocessing leakage-safe (Pipeline/ColumnTransformer)
• ✅ Separate ranking metrics from threshold metrics

Expected path:

• data/churn/churn.csv

Common schema (Telco churn style):

• target: Churn (Yes/No) or churn (0/1)

If the file is missing, the notebook uses a small synthetic churn-like dataset so it still runs.

• numeric + categorical preprocessing
• LogisticRegression baseline

Checkpoint Questions:

• Why is a pipeline required for trustworthy debugging?

Interview Angle:

• Which is worse: FP or FN? (Depends on business cost.)

Examples:

• Contract
• InternetService
• PaymentMethod
• tenure bucket

FAANG Gotcha:

• "Overall accuracy looks fine" can hide severe cohort failures.

• remove all service columns
• remove price columns
• remove contract columns

Example: FN costs 5x FP.

• Baseline + holdout metrics
• At least 3 slice tables with findings
• Feature ablation summary
• Threshold choice explained

• Add calibration + ECE
• Add SHAP/permutation importance (conceptual or sklearn permutation)
• Add drift simulation by shifting tenure distribution