Model Deployment & Serving
Deployment Patterns
Online (Real-time) Serving — REST API or gRPC endpoint. Responds to single requests with low latency (<100ms). Use when users are waiting. Batch Inference — Run predictions on large datasets periodically (nightly, hourly). No latency constraint. Cheapest option. Streaming Inference — Process records from a message queue (Kafka, Kinesis) as they arrive. Near-real-time. Edge Deployment — Model runs on device (phone, IoT, browser). No server round-trip. Private by design.
Production FastAPI Server
FastAPI is the standard choice for wrapping ML models as REST APIs — async, fast, auto-documentation.
Containerizing with Docker
Use multi-stage Docker builds for ML models to keep images small — copy only the model file and inference code, not training dependencies.
Finished reading? Mark it complete to earn your XP.