Gaussian Discriminant Analysis

GDA is a generative learning algorithm, where it learns P(x|y) rather than, learning the mapping function between features(x) and labels(y) i.e. what discriminative algorithms do. In this type of algorithm, we try to model the distribution of the features knowing their source(labels) assuming they come from a Gaussian distribution.

Getting Started

What is Gaussian Distribution?
It is a classic distribution over single scalar random variable 'x', parameterized by mean(mu) and standard deviation(sigma). Which looks like a typical bell curve. It's probability density function is as follows:

What is Multivariate Gaussian?
A multivariate gaussian is a generalization of the gaussian defined over one dimensional random variable, to multiple random variable at the same time. These are vector valued random variable rather than univariate random variable.
It's probability density function is as follows:

A multivariate gaussian in 2 dimension would look something like this, where the right hand side image shows the contour plot of the gaussian.

Maximum Likelihood Estimates

Multivariate gaussian is parameterized by mean(mu) which controls the location of the gaussian and covariance matrix(sigma) which controls the shape of the gaussian.

How to fit the training set?
In order to fit these parameters, we need to maximize the joint likelihood. Once we do this we would have the maximum likelihood estimates for mu and sigma.
Refer this link for the derivation of maximizing the joint likelihood.

Results

In the above code, GDA is used to perform bi-class classification, by modelling class A and class B separately. Once we fit Multivariate Gaussians to each class individually, we can get the probability of any new data point from the probability density function.

Scatter plot of dataset

A two dimensional dataset with 100 data points from class A and 100 data points from class B, where the red points show their mean.

Contour plot with decision boundary

After fitting Gaussians to both class independently, we can get an approximation of the decision boundary as show in the contour plot.

3D visualization of Gaussians

Better visualization of how GDA fits gaussians to the dataset would be as follows.

Note: This plot is not for the provided dataset. This is to get an picture as of how GDA fits gaussians to distribution.