MLOps Interview Questions
Q: What is training-serving skew and how do you prevent it?
Training-serving skew is when the data distribution or feature computation during serving differs from training. It's the most common and dangerous production ML bug. Causes: • Feature computed differently in training pipeline vs serving code • Serving uses stale cached values while training used fresh data • Preprocessing order differs (e.g., fill nulls before/after normalization) Prevention: • Feature store — single source of truth for feature computation • Identical preprocessing pipeline in training and serving (shared Python module) • Shadow mode testing — run new model alongside old, compare outputs • Distribution monitoring on serving features vs training features
Q: How do you monitor ML models in production?
Four layers of monitoring: 1. Infrastructure: CPU/memory/GPU utilization, API latency, error rates. Standard DevOps. 2. Data quality: null rates, schema changes, out-of-range values, cardinality shifts. 3. Data drift: compare input feature distributions (KS test, PSI) between training baseline and current production data. 4. Model performance: prediction drift (output distribution), accuracy/F1/AUC when ground truth labels arrive (may be delayed). Alert thresholds: set alerts at 2σ deviation. Trigger retraining when drift PSI > 0.2 or accuracy drops > 5%.
Q: Explain A/B testing for ML models.
A/B testing for ML models splits traffic between the old model (control) and new model (treatment). Key principles: • Statistical significance — run until p-value < 0.05 (at least 1-2 weeks) • Minimum detectable effect — define minimum meaningful improvement before starting • No peeking — don't stop early when you see a good result (inflates false positives) • Segment analysis — check model doesn't degrade for subgroups even if aggregate improves Common pitfalls: novelty effect (users behave differently with a new experience), network effects (users interact with each other), cannibalization.
Q: What is a feature store and why use one?
A feature store is a centralized data platform for storing and serving ML features. Problem it solves: without a feature store, each team recomputes the same features differently, leading to inconsistency between training and serving (skew) and duplicated engineering effort. Components: • Offline store: historical feature values for training dataset generation (data warehouse) • Online store: low-latency serving for real-time inference (Redis/DynamoDB, <10ms) • Materialization job: computes features and syncs offline → online • Feature registry: metadata, ownership, lineage When to use: when you have 5+ models sharing features, or when training-serving skew is a problem.
Q: How do you decide when to retrain a model?
Scheduled retraining: retrain on a fixed schedule (weekly, monthly). Simple but wasteful if data is stable. Performance-based: retrain when model accuracy drops below a threshold (requires labels — may be delayed). Drift-based: retrain when data drift exceeds a threshold (PSI > 0.2, KS p-value < 0.05). Can act before performance degrades. Event-based: retrain after a known distribution shift event (product launch, seasonal change, external shock). Best practice: combine drift monitoring (fast signal) with performance monitoring (ground truth) and schedule periodic safety retrains regardless.
Finished reading? Mark it complete to earn your XP.