Gaussian Mixture Models

Clustering algorithm that models each cluster as a Gaussian

$p(x)=\sum _{k=1}^K{\omega }_k\mathcal{N}(x|{\mu }_k,{\Sigma }_k):\forall k,{\omega }_k>0\wedge \sum _k{\omega }_k=1$

• The probability density function for observing a data point $x$ in our feature space
• $K$ is the number of Gaussians/classes/clusters

$p(x_n|z_n=k)=\mathcal{N}(x_n|{\mu }_k,{\Sigma }_k)$

• The conditional distribution
• $\mathcal{N}(x_n|{\mu }_k,{\Sigma }_k)$ is the Multivariate Gaussian Distribution
• $x_n$ is the position of point $n$

Parameter Estimation

$${\gamma }_{nk}=\{\begin{array}{ll}1,& z_n=k\\ 0,& \text{otherwise}\end{array}$$

• Indicator function for whether data point $x_n$ belongs to cluster $k$.
• Kind of like the Kronecker Delta in this case

${\gamma }_{nk}=p(z_n=k|x_n)=\frac{p(x_n|z_n=k)p(z_n=k)}{{\sum }_{k^{\prime }=1}^{K}p(x_n|z_n=k^{\prime })}=\frac{\mathcal{N}(x_n|{\mu }_k,{\Sigma }_k)\cdot {\omega }_k}{{\sum }_{k^{\prime }=1}^K\mathcal{N}(x_n|{\mu }_k,{\Sigma }_k)}$

• Non-binary “soft” version, used when actually training the Gaussian Mixture Model
• Expanded by Bayes’ Theorem

${\omega }_k=p(z=k)=\frac{{\sum }_n{\gamma }_{nk}}{{\sum }_{k^{\prime }}{\sum }_n{\gamma }_{n{k}^{\prime }}}$

• The weight associated with a particular Gaussian
• The probability of a particular class
• Proportion of points with $z_n=$ to a particular $k$

${\mu }_k=\frac{1}{{\sum }_n{\gamma }_{nk}}\sum _n{\gamma }_{nk}{x}_n$

• The mean position of all points in cluster $k$

${\Sigma }_k=\frac{1}{{\sum }_n{\gamma }_{nk}}\sum _n{\gamma }_{nk}(x_n-{\mu }_k)(x_n-{\mu }_k{)}^{{}^{\top }}$

• The covariance matrix for all points with $z_n=k$

Training

Called expectation-maximization

• Very similar to K-Means, except that it is considered a “soft” version and also with a non zero $\sigma$
• The E step corresponds to step 1 while the M step corresponds to step 2