We would like to minimize the quantity
$E_{in}(\vec{w})=\frac{1}{N}\sum_{i=1}^N(\vec{w}^{T}\vec{x_n}-y_n)^2$
under the constraint $\vec{w}^T\Gamma^T\Gamma\vec{w}\leq C$ where $\Gamma$ is a matrix, $C$ is a positive constant.
Let $\vec{w}_{reg}$ be the solution to the above problem, further suppose $\vec{w}_{lin}$, the linear regression solution satisfies $\vec{w}^T\Gamma^T\Gamma\vec{w}\leq C$.
What is the relationship between $\vec{w}_{reg}$ and $\vec{w}_{lin}$?
.
I am not sure how to approach this. My idea is that the constraint can be rewritten as $(\Gamma\vec{w})^T\Gamma\vec{w}\leq C$. And since $\vec{w}_{lin}$ satisfies it, and linear regression has a unique solution. If I could show $\Gamma \vec{w}_{lin}$ minimizes our quantity, then we have $\vec{w}_{reg}=\Gamma \vec{w}_{lin}$. However, I'm unsure how to begin.
Any pointers?
