Logistic Regression

A model that predicts the probability of a data point belonging to a certain class. This probability may then be used to classify (e.g. $P(y=1|x)>0.5\to y=1$) if desired

Topics

• Log Likelihood
• Softmax

$L({\beta }_0,{\beta }_1)=-\log (L(p|Y))-\sum _i\left(y_i\ln \left(p_i\right)+(1-y_i)\ln \left(1-p_i\right)\right)$

• Negative log of the product of log likelihood…

$p_i=\frac{1}{1+e^{-({\beta }_0+{\beta }_1{x}_i)}}$

Training

Use gradient descent to minimize the negative log likelihood of the dataset (same as finding the Maximum Likelihood Estimate) by modifying $\theta$, or the weights of the model

$J(\theta )=-\sum _n\left\{y_n\log h_{\theta }(x_n)+(1-y_n)\log [1-h_{\theta }(x_n)]\right\}$

• The log likelihood of the dataset
• $h_{\theta }(x_n)$ is the same as $p_i$

$\frac{\partial J(\theta )}{\partial \theta }=\sum _n\left\{h_{\theta }(x_n)-y_n\right\}{x}_n$

• $\left\{h_{\theta }(x_n)-y_n\right\}$ is the error of the $n$th training sample

Testing

$h_{\theta }(x)=P(y=1|x)=\sigma (a(x))$

• $\sigma (x)$ is the sigmoid function
• $a(x)$ is the activation function
• If $h_{\theta }(x)>0.5$
  • Classify $y=1$
• Else
  • Classify $y=0$

$O(d)$

• $d$ is the dimensionality of the data