Intuition for SVD

What is Singular Value Decomposition?

Singular Value Decomposition (SVD) is a way of breaking down a matrix into simpler parts.

If we have a matrix $A$. We can take the matrix A and write it as

$$A=U\mathrm{\Sigma }{V}^T$$

Way to think about what this is doing are:

$U$: the matrix contains the directions of the input space

$\mathrm{\Sigma }$: A diagonal matrix containing singular values, so basically just a number. Which tells you How important that direction is. The bigger the more important

$V^T$: This matrix contains the directions of the output space.

Essentially SVD rotate and stretches the data.

Visual Intuition

We can think of SVD transformation as a series of steps. We will use the metaphor as hshown in the Figure:

1. $V^T$ is the initial rotation that aligns the principle directions of the transformations.

2. The $\mathrm{\Sigma }$ scales the transformation. The amount of scaling is determined by the singular values inside of the $\mathrm{\Sigma }$.

3. The $U$ does the rotation.

SVD digram credit: KDNuggets