close
Skip to content
ML Visualization

Principal Component Analysis

Unsupervised & Dim. ReductionIntermediate~8 min

Principal Component Analysis — Project data onto the directions of greatest variance.

PCA finds the directions in which your data varies most and projects onto them, compressing many correlated features into a few meaningful axes while keeping as much information as possible.

The data, with the axis under test
  • Data
  • Axis under test
  • First PC (and best-fit plane)
  • Residual thrown away

PCA controls

Data
Dataset

Two correlated features. One direction holds nearly all the spread — find it.

Shape80 × 2
Model
Explained variance (click a bar to select that component)
Captured by this axis76%
Plane keeps100%
Playback
Step 0 / 71
Speed
  1. Rotate the axis
  2. Measure the spread
  3. Keep the maximum

Step 0 of 71 — axis at 0° — it captures 76% of the spread in this plane; keep turning and watch the number climb

Or drag anywhere on the projection to swing the axis by hand.

Break it

The idea in plain words

PCA finds the directions along which your data varies the most and projects onto them. Rotate the projection axis by hand and the “variance captured” meter peaks exactly at the first principal component — you discover PCA instead of being told it.

It’s a linear method, though. Hand it a curved manifold like a swiss roll and it can only flatten by projection — it can’t unroll the sheet. That limitation is what motivates t-SNE and UMAP.

Now, the math

The principal components are the eigenvectors of the covariance matrix:

Σ vk=λk vk,Σ=1n∑i(xi−μ)(xi−μ)⊤\Sigma\, v_k = \lambda_k\, v_k,\qquad \Sigma = \tfrac{1}{n}\sum_i (x_i - \mu)(x_i - \mu)^\top
vkv_k
the k-th principal component (a direction).
λk\lambda_k
its eigenvalue — the variance captured along that direction.
Σ\Sigma
the data covariance matrix.
▸ Show the derivation

The direction of maximum variance is the top eigenvector of the covariance matrix; the explained variance ratio is its eigenvalue over the total. Projecting onto the first few components keeps the most information for the fewest dimensions — but only along straight axes, so curved structure is lost.

Trace it by hand

Four 2-D points: (0,0), (2,1), (1,2), (3,3). We build the covariance matrix, extract its eigenvectors with the repo's Jacobi eigensolver, and read off the explained variance (eigenvector entries rounded to 3 decimal places).

  1. Step 1 — center the data at its mean

    μ=(0+2+1+34, 0+1+2+34)=(1.5, 1.5)\mu = \left(\tfrac{0+2+1+3}{4},\ \tfrac{0+1+2+3}{4}\right) = (1.5,\ 1.5)

    PCA measures spread around the mean, so everything below uses deviations from (1.5, 1.5).

  2. Step 2 — the covariance matrix

    Σ=14∑i(xi−μ)(xi−μ)⊤=(1.25111.25)\Sigma = \tfrac{1}{4}\sum_i (x_i - \mu)(x_i - \mu)^\top = \begin{pmatrix} 1.25 & 1 \\ 1 & 1.25 \end{pmatrix}

    The large positive off-diagonal says the two features move together — the cloud is stretched along the diagonal.

  3. Step 3 — its eigenvalues and eigenvectors

    λ1=2.25,    v1=(0.707, 0.707)λ2=0.25,    v2=(0.707, −0.707)\lambda_1 = 2.25,\;\; v_1 = (0.707,\ 0.707) \qquad \lambda_2 = 0.25,\;\; v_2 = (0.707,\ -0.707)

    0.707 is 1 over root 2 — the first principal component points exactly along the 45 degree diagonal, matching the covariance's hint.

  4. Step 4 — explained variance ratio

    λ1λ1+λ2=2.252.50=0.90,λ22.50=0.10\frac{\lambda_1}{\lambda_1 + \lambda_2} = \frac{2.25}{2.50} = 0.90, \qquad \frac{\lambda_2}{2.50} = 0.10

    Projecting onto v1 alone keeps 90 percent of the variance while halving the dimensions.

What just happened: The covariance matrix's eigenvector (0.707, 0.707) is the diagonal direction the four points visibly stretch along, and its eigenvalue says that one axis holds 90 percent of the variance — compression with a known, computed cost.

Now Break It

Try this: Dropping to too few components loses the structure — reconstruction becomes a blur.

Control: Components-to-keep slider (set to 1)

What happens: Too much compression! Keeping only one component throws away the structure — reconstruction fails.

Where principal component analysis is used

Principal Component Analysis finds the directions along which data varies most and projects onto them, making it a staple for compression, visualization, and noise reduction. Analysts use it to squeeze dozens of correlated features into a handful of components before feeding them to a model, which speeds training and reduces overfitting. It underlies eigenfaces in face recognition, helps genomics researchers visualize population structure from thousands of genetic markers, and denoises signals by discarding low-variance components that often carry mostly noise. Because the components are ordered by how much variance they explain, PCA also gives a clear way to decide how many dimensions to keep, and projecting onto the first two or three components produces a quick, interpretable map of an otherwise unwieldy high-dimensional dataset.

The biggest misconception is that PCA finds the features most useful for prediction; it actually finds directions of maximum variance, which are not always the directions that matter for a label, so a low-variance component can still be the discriminative one. PCA is also strictly linear: it captures linear correlations and cannot unfold curved manifolds, where kernel PCA or methods like UMAP do better. Two practical pitfalls matter. First, PCA is sensitive to feature scaling, so features with large units dominate unless you standardize first. Second, the components are linear combinations of all original features, which makes them powerful but hard to interpret, and reifying a component as a single real-world concept is usually a mistake.

Frequently asked questions

What exactly is a principal component?
A principal component is a new axis formed as a weighted combination of the original features, chosen so that the data spread along it is as large as possible. The first component captures the most variance, the second captures the most of what remains while being perpendicular to the first, and so on. Each component is orthogonal to the others.
How many components should I keep?
A common approach is to keep enough components to explain a target share of total variance, such as ninety or ninety-five percent, read off a cumulative variance plot. The scree plot's elbow, where added components explain little extra, is another guide. The right number depends on whether you are compressing, visualizing, or denoising.
Do I need to standardize my data before PCA?
Usually yes, because PCA is driven by variance and a feature measured in large units will dominate the components purely because of its scale. Standardizing each feature to unit variance puts them on equal footing. You would skip standardization only when all features share the same meaningful units and you want scale differences to count.
Is PCA a feature selection method?
No. Feature selection keeps a subset of the original features, while PCA creates entirely new features that are combinations of all the originals. This means PCA can reduce dimensionality effectively but sacrifices the direct interpretability of the raw variables.
Why does PCA fail on nonlinear data?
PCA only captures linear relationships, so it cannot unfold structures like a spiral or an S-shaped surface where the meaningful directions curve. Projecting such data linearly collapses distinct regions together. Nonlinear techniques such as kernel PCA, t-SNE, or UMAP are designed for those cases.
← Back to all visualizationsContinue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated .