PCA

Projects data onto the dimension that maximizes variance of the data along that dimension, which so happens to be along the eigenvector with the largest eigenvalue when looking at the sample covariance matrix of the data

$\mathbf{Z}=\mathbf{XP}$

• $\mathbf{Z}\in {\mathbb{R}}^{n\times k}$ the lower-dimensional representation of the data, where $n$ is the number of datapoints and $k$ is the number of principal components
• $\mathbf{X}\in {\mathbb{R}}^{n\times d}$ is the data, where $d$ is the dimensionality of the data
• $\mathbf{P}\in {\mathbb{R}}^{d\times k}$ is the transformation matrix

$\text{Fraction of retained variance}=\frac{{\sum }_{i=1}^k{\lambda }_i}{{\sum }_{i=1}^d{\lambda }_i}$

• If you choose the first $k$ highest eigenvalues, this is proportion of variance you retain after PCA

${\text{max}}_p{p}^{{}^{\top }}{C}_{X}p,{\Vert p\Vert }_2=1$

• The goal of PCA is to maximize the variance represented by $p^{{}^{\top }}{C}_{x}p$ by varying $p$ with the constraint that $p$ is normal
• $C_X$ is the covariance matrix

Training