3
$\begingroup$

Suppose you have a regular set up for a regression problem. You have response vector $y$ and a matrix $X$, and you want to find $\hat{\beta}$ that minimizes $||y-X\hat{\beta}||$

The hat matrix $H=X(X'X)^{-1}X'$ maps vectors onto the column space of $X$.

From Linear Models and Generalizations pg.323:

We can break $X$ into submatrices $X=[X_1 \mid X_2]$ and then rewrite $H=H_1+(I-H_1)X_2(X_2'(I-H_1)X_2)^{-1}X_2'(I-H_1)$ where $H_1=X_1(X_1'X_1)^{-1}X_1'$, which is essentially saying the hat matrix $H$ equals the hat matrix of $X_1$ plus the projection of $X_2$ onto orthogonal complement of $H_1$.

The proof in the back of the book skips a lot of steps so I am having trouble understanding why I can decompose $H$ this way.

$\endgroup$

1 Answer 1

1
$\begingroup$

I have no idea what the proof in the book does. But, here is an argument.

Note that $H$ is an orthogonal projection ($H^2=H'=H$); as you say, it projects onto the column space of $X$. Similarly, $H_1$ is the orthogonal projection onto the column space of $X_1$, and $I-H_1$ is the projection onto the columns space of $X_2$. Let us call $$ K=H_1+(I-H_1)X_2(X_2'(I-H_1)X_2)^{-1}X_2'(I-H_1). $$ Since $H_1(I-H_1)=0$, it is very easy to check that $K^2=K$, and that $K'=K$. So $K$ is an orthogonal projection. Its range contains the column space of $X_1$, because if $x$ is in the column space of $X_1$ then $H_1x=x$, and so $Kx=x$. Note that $$ (I-H_1)X_2(X_2'(I-H_1)X_2)^{-1}X_2'(I-H_1) $$ is the orthogonal projection onto the column space of $(I-H_1)X_2$. So $K$ is the orthogonal sum of two orthogonal projections; to conclude that $K=H$ we need to show that the span of the column spaces of $X_1$ and $(I-H_1)X_2$ agrees with the column space of $X$.

If $y$ is in the column space of $X$, and $c_1,\ldots,c_m$ are the columns of $X$, we can write $$ y =(\alpha_1c_1+\cdots+\alpha_rc_r)+(\alpha_{r+1}d_{1}+\cdots+\alpha_sd_s), $$ where $c_1,\ldots,c_r$ are the columns of $X_1$ and $d_{1},\ldots,d_s$ is an orthonormal basis obtained by applying Gram-Schmidt to those of $c_{r+1},\ldots,c_m$ that are linearly independent with $c_1,\ldots,c_r$. Then $$ H_1y=\alpha_1c_1+\cdots+\alpha_rc_r $$ and $$ (I-H_1)y=\alpha_{r+1}d_{1}+\cdots+\alpha_sd_s. $$ That is, $H_1y$ belongs to the span of the columns of $X_1$ and $(I-H_1)y$ belongs to the span of the columns of $(I-H_1)X_2$. Then \begin{align} Ky&=K(H_1y+(I-H_1)y)\\ \ \\ &=H_1^2y+(I-H_1)X_2(X_2'(I-H_1)X_2)^{-1}X_2'(I-H_1)\,(I-H_1)y\\ \ \\ &=H_1y+(I-H_1)y=y. \end{align} This shows that $K$ is the identity on the column space of $X$.

Conversely, if $y$ is orthogonal to the column space of $X$, then in particular $y$ is orthogonal to the column spaces of $X_1$ and $X_2$. Then $H_1y=0$. But then $$ (I-H_1)X_2(X_2'(I-H_1)X_2)^{-1}X_2'(I-H_1)y=(I-H_1)X_2(X_2'(I-H_1)X_2)^{-1}X_2'y=0, $$ the second equality since $X_2'y=0$ (because $y$ is orthogonal to the column space of $X_2$). Then we have that $Ky=0$.

We have shown that $K$ is the identity on the range of $H$ and zero on the orthogonal complement of the range of $H$. That is, for any $y$, $$ Ky=K(Hy+(I-H)y)=Hy. $$ so $K=H$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .