Simple Linear Regression

Topics

• Coefficient of Determination

$\hat{Y}={\hat{\beta }}_{1}X+{\hat{\beta }}_0+\epsilon$

• The hats are considered estimators, or values we have to calculate to predict $Y$
• ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$ are regression coefficients
• $\epsilon$ is random noise

$L({\beta }_0,{\beta }_1)=\frac{1}{n}\sum _{i=1}^n[y_i-({\beta }_{1}X+{\beta }_0{)}^2]=J(\mathbf{\theta })={\Vert \mathbf{y}-\mathbf{X\theta }\Vert }_2^2$

• The loss function for simple linear regression
• Minimizing this by varying ${\beta }_0$/${\beta }_1$ or $\mathbf{\theta }$ gives the linear model that best fits with the data $\mathbf{y}$
• $\mathbf{y}$ is the target vector
• $\mathbf{X}$ is the design matrix
• $\mathbf{\theta }$ is the Parameter Matrix

$\nabla J(\mathbf{\theta })=2({\mathbf{X}}^{{}^{\top }}\mathbf{X}\mathbf{\theta }-{\mathbf{X}}^{{}^{\top }}\mathbf{y})$

• Gradient of loss function

${\hat{\beta }}_1=w=\frac{{\sum }_i(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_i(x_i-\bar{x}{)}^2}$

• Coefficient of regression that minimizes $L$
• Same as $\frac{\text{cov}(X,Y)}{\text{var}(X)}$

${\hat{\beta }}_0=b=\bar{y}-{\hat{\beta }}_1\bar{x}$

• Coefficient of regression that minimizes $L$

$\hat{\beta }=\theta =\underset{\beta }{\mathrm{argmin}}\text{MSE}(\beta )=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{Y}$

• If ${\mathbf{X}}^{{}^{\top }}\mathbf{X}$ is not invertible, this happens both/either because
  • $N<D$, or there are fewer data points than features
  • $\mathbf{X}$ is not linearly independent, in which case

$SE({\hat{\beta }}_0)=\sigma \sqrt{\frac{1}{n}+\frac{{\bar{x}}^2}{{\sum }_i(x_i-\bar{x}{)}^2}}$

• $\sigma$ is the variance in $\hat{y}$
• $\bar{x}$ is the average value of our sample

$SE({\hat{\beta }}_1)=\frac{\sigma }{\sqrt{{\sum }_i(x_i-\bar{x}{)}^2}}$

$\sigma \approx \sqrt{\sum _i\frac{(\hat{f}(x)-y_i{)}^2}{n-2}}$

• Used for when the noise $\epsilon$ is unknown