Principal Component Analysis(PCA)

Introduction

Goal

To apply dimension deduction on input data set. e.g. to project original input data matrix M(n*m) into a lower dimension matrix N(n*p) where p is a number smaller than m. Why we need this? Cause some data features are not such usable in current learning problem and it will increase the calculate complexity if we add them into our learning process. So in this way, we could deduct original dimension into a smaller without losing import feature spaces.

PCA(Maximum Projection Variance)

where S is the sample variance matrix of X:$$S = \frac{1}{N}\sum\limits_n(X-\bar{X})^{\top}(X-\bar{X})$$The first principal component $w_1$ is simply the eigenvector of $S$ corresponding to its largest eigenvalue. The kth principal component $w_k$ is the eigenvector of $S$ with respect to $k$th largest eigenvalue.