Support Vector Machines (SVM)
Core Idea
SVM finds the hyperplane that maximally separates two classes — the decision boundary with the largest margin. Points closest to the boundary are called Support Vectors. Maximum margin = 2 / ||w||, so we minimize ||w||² subject to correct classification.
The Kernel Trick
When data is not linearly separable, the kernel trick implicitly maps data to a higher-dimensional space where a linear separator exists — without explicitly computing the transformation. Common kernels: • Linear: K(x,z) = xᵀz • RBF (Radial Basis Function): K(x,z) = exp(−γ||x−z||²) • Polynomial: K(x,z) = (xᵀz + c)ᵈ RBF kernel is the default — works well in most cases.
Soft Margin (C parameter)
Real data has noise and outliers. Soft margin SVM allows misclassifications with a penalty: Minimize: ½||w||² + C·Σξᵢ High C → narrow margin, low training error, risk of overfitting. Low C → wide margin, more misclassifications allowed, better generalization.
SVM Example
Always scale features before SVM. SVMs are slow on large datasets (>100K samples) — use linear SVM or switch to Random Forest.
Finished reading? Mark it complete to earn your XP.