SVM

Given a labeled data set, is able to classify data points by seeing on what side of the decision boundary they lie on within an N dimensional space, where N is the number of features. The decision boundary is tuned during training.

Topics

• Support Vector
• Support Vector Margin
• Kernel Method
• Hinge Loss

${\text{min}}_{(w,b)}\sum _n{\ell }^{\text{Hinge}}(y_n,w^T\varphi (x_n)+b)+\frac{\lambda }{2}{\Vert w\Vert }_2^2$

• Primal formulation of SVM

$\frac{1}{2}{\text{argmin}}_{(w,b)}(\Vert w{\Vert }^2):y_n[w^T\varphi (x_n)+b]\ge 1$

• Hard margin SVM
• This is how to find the parameters $w,b$ while under some constraints to create a support vector margin
• $w$ is the support vector
• $b$ is the bias term

$\frac{1}{2}{\text{argmin}}_{(w,b)}(\Vert w{\Vert }^2)+\lambda \sum _{n=1}^N{\xi }_n:y_n[w^T\varphi (x_n)+b]\ge 1-{\xi }_n$

• Soft margin SVM
• $\lambda$ is a regularization parameter
• $N$ is the number of data points that are on the wrong side of the decision boundary
• ${\xi }_n$ is the distance from the $n$th wrong point to the decision boundary