Dtyjlui

On page 576 in Bishop's PRML, it is stated that

$$
(mathbf{W}_{ML}^Tmathbf{W}_{ML})^{-1}mathbf{W}^T_{ML}(mathbf{x} - mathbf{bar{x}})
$$

represents an orthogonal projection of the data point $mathbf{x}$ onto the latent space.

$mathbf{W}$ is a $Dtimes M$ matrix. $mathbf{x}$ is $Dtimes 1$. The latent space is $M$-dimensional.

What does it mean that the expression represents an orthogonal projection (and how do we know that it is one) onto the latent space and why is it important?

This question arises in the context of principal component analysis. To keep our notation simple, let's assume that our dataset is centred at the origin, i.e. $mathbf {bar x }= 0$. The goal is to approximate a datapoint $mathbf x in mathbb R^n$ as closely as possible using a point chosen from the $d$-dimensional subspace spanned by the columns of an $n times d$ matrix $mathbf W$. In other words, we wish to find
$$ mathbf x_star := mathbf Wmathbf z_star,$$

where
$$ mathbf z_{star}:= {rm argmin}_{mathbf z in mathbb R^d}| mathbf x - mathbf W mathbf z|^2.$$

[This $mathbf x_star$ is called the "orthogonal projection" of $mathbf x$ onto the hyperplane spanned by the columns of $mathbf W$, because, if computed correctly, $mathbf x_star$ will be orthogonal to the displacement vector from $mathbf x_star$ to $mathbf x$. Geometrically, this is quite intuitive.]

Let's go ahead and compute this approximation. First, let's find $mathbf z_star$, by differentiation:
$$ mathbf 0 = left( frac{partial}{partial mathbf z} | mathbf x - mathbf W mathbf z|^2 right)vert_{{mathbf z = mathbf z_star}} = -2mathbf W^T(mathbf x - mathbf Wmathbf z_star) implies mathbf z_star = (mathbf W^T mathbf W)^{-1}mathbf W^T mathbf x.$$

In machine learning, this $mathbf z_{star}$ is the latent vector for this datapoint, and corresponds to the expression in your question (assuming $mathbf {bar x} = 0$). The approximation $mathbf x_star$ is then given by $mathbf x_star = mathbf W mathbf z_star$.

[Just for fun, let's verify that $mathbf x_star$ and $mathbf x - mathbf x - mathbf x_star$ are orthogonal, justifying the phrase "orthogonal projection":
begin{align} mathbf x_star . (mathbf x - mathbf x_star) &= mathbf xmathbf W(mathbf W^Tmathbf W)^{-1}mathbf W^Tleft( mathbf x-mathbf W(mathbf W^Tmathbf W)^{-1}mathbf W^T mathbf xright) \ &= mathbf xmathbf W(mathbf W^Tmathbf W)^{-1}mathbf W^T mathbf x- mathbf xmathbf W(mathbf W^Tmathbf W)^{-1}mathbf W^T mathbf x \ &= 0. end{align}
]

I don't know what machine learning or a latent space is, but I understand orthogonal projections. x is a data point and you want to find the closest point y in the latent space. This is done by orthogonal projection. Geometrically, you can write x as a linear combination of ($M$+1) orthogonal vectors, $M$ of which are in the latent space. The remaining vector, call it z is the orthogonal complement of x.

Note $midmid$z$midmid$ is the distance from x to the latent space.