Model Monitoring & Drift Detection

ML models degrade silently in production. Unlike software bugs, model failures are gradual and statistical — accuracy drops over weeks without any error in the code. Causes: • Data drift — input distribution shifts from training distribution • Concept drift — relationship between X and y changes over time • Upstream data pipeline changes — new null values, schema changes • Covariate shift — feature distributions change

Data Drift (Covariate Shift) — P(X) changes but P(Y|X) stays the same. Feature statistics diverge from training. Detectable without labels. Label Drift — P(Y) changes. Class distribution shifts. Need labels to detect. Concept Drift — P(Y|X) changes. The underlying relationship changes. Hardest to detect — needs labels and time. Prediction Drift — Distribution of model outputs changes. Can be detected without labels by comparing score distributions.

KS Test (Kolmogorov-Smirnov) — Tests if two continuous distributions are different. Sensitive to all types of distributional changes. Population Stability Index (PSI) — Industry standard for credit scoring. PSI < 0.1: stable. 0.1-0.2: some shift. >0.2: major drift. Chi-Square Test — For categorical features. Tests if frequencies differ significantly. MMD (Maximum Mean Discrepancy) — Statistical test for high-dimensional data. Used in deep learning drift detection.