Dtyjlui

I'm self-learning the book "The Elements of Statistical Learning", and I've got a little problem on deriving equation 2.27 in this book. I would appreciate if anyone could help me with this. In order to make this question consistent, I'll list all the information needed for the deriving.

Suppose we have two random variables $Y$ and $X$ with the following relation:

$Y = X^Tbeta + epsilon$ where $epsilon sim N(0, sigma^2)$.

and we fit data in a training set $mathcal{T}$ with linear regression by least squares:

$hat{Y} = X^T hat{beta}$, where $hat{beta}$ is the parameter and $hat{Y}$ is our prediction, $X$ is the input.

we define the Expected Prediction Error (EPE) of a record ($x_0, y_0$) in test data as

$EPE(x_0) = E_{y_0|x_0}E_{mathcal{T}}(y_0 - hat{y_0})^2$ where $hat{y_0}$ is our prediction w.r.t. $x_0$.

According to the book ,
$EPE(x_0) = Var(y_0|x_0) + Var_{mathcal{T}}(hat{y_0}) + Bias^2(hat{y_0})$

where $Bias^2(hat{y_0}) = (E_{mathcal{T}}hat{y_0} - y_0)^2$.

Here is my derivation:

$EPE(x_0) = E_{y_0|x_0}E_{mathcal{T}}(y_0 - hat{y_0})^2$

$=E_{y_0|x_0}E_{mathcal{T}}(y_0^2 - 2y_0hat{y_0} + hat{y_0}^2)$

$=E_{y_0|x_0}(y_0^2) -2E_{y_0|x_0}y_0E_{mathcal{T}}hat{y_0} + E_{mathcal{T}}hat{y_0}^2$

By the lemma

$Var(X) = E(X^2) - (E(X))^2$ we have

$EPE(x_0) = Var(y_0|x_0) + (E_{y_0|x_0}(y_0))^2 -2E_{y_0|x_0}y_0E_{mathcal{T}}hat{y_0} + Var_{mathcal{T}}(hat{y_0)} + (E_{mathcal{T}}(hat{y_0}))^2$

$= Var(y_0|x_0) + Var_{mathcal{T}}(hat{y_0}) + (E_{mathcal{T}}(hat{y_0}) - E_{y_0|x_0}(y_0))^2$ which is really close to the conclusion given in the book, as long as we can prove $E_{y_0|x_0}(y_0) = y_0$ the proof is completed.

But I've no clue how to prove $E_{y_0|x_0}(y_0) = y_0$ since I've no idea about $p(y_0 | x_0)$. I'm wondering if anyone could help me on that.

My guess is that, since $y_0 = x_0^Tbeta + epsilon$, $E_{y_0|x_0}(y_0) = E_{y_0|x_0}(x_0^Tbeta) + E_{y_0|x_0}(epsilon) = E_{y_0|x_0}(x_0^Tbeta) + epsilon$, and $x_0^Tbeta$ is deterministic given $x_0$, so $E_{y_0|x_0}(y_0) = x_0^Tbeta + epsilon = y_0$, but I'm not sure if this is right.

Thank you very much for your reading!

Possible relative questions:

1. What is Expected Prediction Error (EPE) a function of?




2. Expected squared prediction error conditioned on training set