Properties of ridge regression hat matrix and ridge residuals

Question

I'm referencing https://arxiv.org/pdf/1509.09169.pdf on ridge regression. On page 34 question 1.5 we need to prove :
Ridge fit $\widehat{Y}(\lambda)=X(X^{\top}X+\lambda I_p)^{-1}X^{\top}Y$ is not orthogonal to ridge residual $Y − \widehat{Y}(\lambda)$.

To how this since I think we can use that the hat matrix for ridge regression is not a projection matrix but that does not give me anything useful. In the OLS case we show that the residual is not orthogonal to $X$ since $\widehat{Y}(\lambda)$ is linear combination of $X$, but I do not think we can use this here as the linear combination property might not hold here due to the term $\lambda I_p$. Please tell how to show this.

Answer 1

Set for clarity

$$B \equiv (X^{\top}X+\lambda I_p)^{-1}$$

and you are asked to examine

$$(Y - \widehat{Y})^{\top}XBX^{\top}Y = (Y - XBX^{\top}Y)^{\top}XBX^{\top}Y.$$

Doing the algerba

$$...=Y^{\top}XBX^{\top}Y - Y^{\top}XBX^{\top}XBX^{\top}Y.$$

If $B$ was equal to $X^{\top}X$ the second component would simplify and become equal to the first, hence the zero result. But in Ridge regession this is not the case, so the expression does not equal zero.

Continuing with the manipulations,

$$...=Y^{\top}XBX^{\top}\big[I-XBX^{\top}]Y$$.

If $B$ was equal to $X^{\top}X$, then the term in brackets would become the ("complementary") projection matrix of $X$, and would make the expression zero.It is not, so no zero result.