PCA & Dimensionality Reduction

High-dimensional data causes the 'curse of dimensionality': data becomes sparse, distances lose meaning, models overfit, and visualization is impossible. Reducing dimensions helps with speed, visualization, and noise removal.

PCA finds the directions (principal components) of maximum variance in the data and projects onto a lower-dimensional subspace. Steps: 1. Standardize the data (zero mean, unit variance) 2. Compute covariance matrix Σ 3. Compute eigenvectors and eigenvalues of Σ 4. Sort eigenvectors by eigenvalue (descending) 5. Select top-k eigenvectors as principal components 6. Project data: Z = X·W_k

Each principal component explains a fraction of total variance. Cumulative explained variance tells you how many components you need to retain X% of information. Typical target: 95%.