3
$\begingroup$

I was wondering if you could help me out. I am quite confused about the difference between sieve estimators (Ulf Grenander) and structural risk minimization (SRM) (Vladimir Vapnik). Could anyone give more information on the difference between these procedures? Both seem to trade off the considered model complexity with the available sample size.

Thanks in advance :D

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

A sieve estimator is a type of semi-nonparametric procedure in which we have a parameter that lives in a high-dimensional or infinite-dimensional space. Since it is difficult to maximize an objective function over an infinite-dimensional space we instead maximize over a sequence of finite-dimensional spaces, these are called sieves (hence, the name). We are essentially making a "promise" to infinitely increase these spaces so that they are dense in the space in which our parameter lives. There are many assumptions we need for this to work well and this description is meant to be heuristic for better explanations take a look at the linked references below.

SRM is essentially a regularization procedure to avoid overfitting. Think about how we add a regularization term in the SVM estimator. Basically, we are trying to find a reasonable trade-off between how well the model fits the data and how complicated the model is. Interestingly, there is a relationship here. In both cases, we are essentially looking to approximate a complicated model in a less complicated space.

Some references:

https://www.sciencedirect.com/science/article/pii/S157344120706076X

http://www.cnel.ufl.edu/courses/EEL6814/srm.pdf

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for responding to this, quite old, question. I am not satisfied with the way I phrased this question. See, I know about both theories already a decent amount. What is still not crystal clear to me is what (fundamentally) differentiates these theories. I do like your answer but it does not help me with this differentiation. In the near future I will probably edit the question and try to give a decent answer myself. Cheers, $\endgroup$
    – vshas
    Commented May 27, 2021 at 12:53
  • $\begingroup$ Glad I could help, sieve is a fascinating estimator that I work a lot on and so I love answering questions about it. Perhaps reading this old paper on SRM might help a bit: /papers.nips.cc/paper/1991/file/… I think one way to think about this is that sieve is the slightly more general method since you are approximating a function using ever-increasing function spaces. The sieves allow us to employ many different ways to restrict the "geometric" complexity of the spaces - including using VC-type classes. $\endgroup$
    – Ariel
    Commented May 27, 2021 at 14:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.