5
$\begingroup$

I was studying parameter estimation from Sheldon Ross' probability and statistics book. Here the task of parameter estimation is described as follows:

enter image description here

Is this task the same of density estimation in machine learning contexts?

Mathematics for Machine Learning book describes density estimation as follows:

enter image description here

My question arises from the fact that a probability density function is fully described by its parameters (e.g. a Gaussian or normal distribution is a density function which is fully described by its mean $\mu$ and its variance $\sigma^2$).

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
$\begingroup$

I understand this argument and can buy it as being technically true. However, the goal of language is to communicate ideas, and statistics has decided that “density estimation”, for better or for worse, refers to doing density estimation with minimal assumptions about the density as to keep from being restricted to a particular parametric family.

Perhaps this means that the use of English words is not perfect. However, you are likely to elicit confusion (or at least strange looks) in statistics circles if you deviate from the established terminology.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I agree with this broadly. We can, I hope, all agree that what the problems have in common is indeed estimation, with a key flavour in both cases that results will vary if the data do -- or if you use a different method. That said, I agree with the signal here, as density estimation is not in my reading usually regarded or reported as a parameter estimation problem. One reason is that there's not a fixed number of parameters. In some problems estimating the density at say 30 points could be enough in practice, in others 300 or 3000 points might or might not work well. $\endgroup$
    – Nick Cox
    Commented Apr 20, 2023 at 10:44
  • 1
    $\begingroup$ (Density estimation can be almost trivial with one variable, and almost impossible with several!) $\endgroup$
    – Nick Cox
    Commented Apr 20, 2023 at 10:46
  • 1
    $\begingroup$ So what's the key difference between the two? $\endgroup$
    – tail
    Commented Apr 20, 2023 at 20:47
  • 1
    $\begingroup$ @tail It’s a matter of the “feel” for the terminology. You’re right that estimating a parameter and going with a distribution with that parameter is one way to do density estimation, yet this tends not to be what the field means by “density estimation”. $\endgroup$
    – Dave
    Commented Apr 20, 2023 at 20:49
6
$\begingroup$

No, it's not the same. Density estimation is about estimating the distribution of the data. This can be achieved with a parametric model, for example, fitting a Gaussian mixture to the data. In such a case, to find the distribution means to estimate its parameters since the distribution is defined by its parameters. But there are also non-parametric approaches to density estimation, like Kernel density estimation, using histograms, etc where we are not estimating any parameters but rather finding the density itself.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ So, we can say that density estimation methods exploit parameter estimation (or non-parametric methods) for estimating a suitable probability distribution for some given data? $\endgroup$
    – tail
    Commented Apr 20, 2023 at 10:49
  • 1
    $\begingroup$ @tail You can say that, but I wouldn't usually do so, Estimating a suitable probability distribution implies that you are going to use that distribution henceforth. Often density estimation (which in turn usually means kernel density estimation) is a descriptive method that is directly informative about the data, so that graphs can be used to show what is going on. It can also be used to guide decisions e.g. on whether to work with data as they come or to apply some transformations. But there can be other uses. $\endgroup$
    – Nick Cox
    Commented Apr 20, 2023 at 10:57
  • $\begingroup$ A bit confusing, in my opinion. $\endgroup$
    – tail
    Commented Apr 26, 2023 at 13:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.