ML System Design

Framework for any ML system design question: 1. Clarify requirements (online/batch? latency? scale? accuracy vs speed?) 2. Define the ML problem (formulation, labels, metrics) 3. Data collection & pipeline 4. Feature engineering 5. Model selection & training 6. Evaluation & offline testing 7. Serving infrastructure 8. Online A/B testing 9. Monitoring & retraining

Problem: recommend items to users (Netflix, YouTube, Amazon). Formulation: rank items by predicted engagement (click, watch, purchase). Data: user-item interaction matrix, item features (genre, price), user features (age, history). Two-stage architecture: 1. Candidate Generation (recall): fast retrieval of ~1000 relevant items from millions. Matrix factorization, two-tower neural network. 2. Ranking (precision): score and rank 1000 candidates with a richer model. Wide & Deep, gradient boosting. Key features: user history, item popularity, collaborative filtering signals, recency. Metrics: Precision@K, NDCG, click-through rate, watch time (business metric).

Problem: classify each transaction as fraudulent in <50ms. Challenges: extreme class imbalance (0.1% fraud), concept drift (fraud patterns change), adversarial (fraudsters adapt). Architecture: • Feature store: pre-computed user behavior features (avg spend, velocity, location history) • Real-time features: time since last transaction, transaction amount deviation • Two-model ensemble: fast heuristic rules + gradient boosted model • Human review queue: borderline cases flagged for analyst review Metrics: precision/recall (tune threshold), $ saved, false positive rate (customer experience).

Problem: return relevant documents for natural language queries. Architecture: Offline (indexing): 1. Crawl & clean documents 2. Chunk (512 tokens, 10% overlap) 3. Embed with bi-encoder (e.g., BGE-large) 4. Index in vector DB (Qdrant/Pinecone) + BM25 for hybrid search Online (query time): 1. Query expansion (synonyms, HyDE) 2. Hybrid retrieve (dense + BM25, RRF fusion) → top 50 3. Cross-encoder rerank → top 5 4. Optional: LLM generates answer citing retrieved docs (RAG) Metrics: NDCG@10, MRR, latency P50/P99.