If you use a point estimate that maximizes $P(x | \theta)$, what does that say about your philosophy? (frequentist or Bayesian or something else?)

Question

If somebody said

"That method uses the MLE the point estimate for the parameter which maximizes $\mathrm{P}(x|\theta)$, therefore it is frequentist; and further it is not Bayesian."

would you agree?

• Update on the background: I recently read a paper that claims to be frequentist. I don't agree with their claim, at best I feel it's ambiguous. The paper does not explicitly mention either the MLE (or the MAP, for that matter). They just take a point estimate, and they simply proceed as if this point estimate was true. They do not do any analysis of the sampling distribution of this estimator, or anything like that; the model is quite complex and therefore such analysis is probably not possible. They do not use the word 'posterior' at any point either. They just take this point estimate at face value and proceed to their main topic of interest - inferring missing data. I don't think there is anything in their approach which suggests what their philosophy is. They may have intended to be frequentist (because they feel obliged to wear their philosophy on their sleeve), but their actual approach is quite simple/convenient/lazy/ambiguous. I'm inclined now to say that the research doesn't really have any philosophy behind it; instead I think their attitude was more pragmatic or convenient:

  "I have observed data, $x$, and I wish to estimate some missing data, $z$. There is a parameter $\theta$ which controls the relationship between $z$ and $x$. I don't really care about $\theta$ except as a means to an end. If I have an estimate for $\theta$ it will make it easier to predict $z$ from $x$. I will choose a point estimate of $\theta$ because it's convenient, in particular I will choose the $\hat{\theta}$ that maximizes $\mathrm{P}(x|\theta)$."

The idea of an unbiased estimator is clearly a Frequentist concept. This is because it doesn't condition on the data, and it describes a nice property (unbiasedness) which would hold for all values of the parameter.

In Bayesian methods, the roles of the data and the parameter are sort of reversed. In particular, we now condition on the observed data and proceed to make inferences about the value of the parameter. This requires a prior.

So far so good, but where does the MLE (Maximum Likelihood Estimate) fit into all this? I get the impression that many people feel that it is Frequentist (or more precisely, that it is not Bayesian). But I feel that it is Bayesian because it involves taking the observed data and then finding the parameter which maximizes $P(data | parameter)$. The MLE is implicitly using a uniform prior and conditioning on the data and maximizing $P(parameter | data)$. Is it fair to say that the MLE looks both Frequentist and Bayesian? Or does every simple tool have to fall into exactly one of those two categories?

The MLE is consistent but I feel that consistency can be presented as a Bayesian idea. Given arbitrarily large samples, the estimate converges on the correct answer. The statement "the estimate will be equal to the true value" holds true for all values of the parameter. The interesting thing is that this statement also holds true if you condition on the observed data, making it Bayesian. This interesting aside holds for the MLE, but not for an unbiased estimator.

This is why I feel that the MLE is the 'most Bayesian' of the methods that might be described as Frequentist.

Anyway, most Frequentist properties (such as unbiasedness) apply in all cases, including finite sample sizes. The fact that consistency only holds in the impossible scenario (infinite sample within one experiment) suggests that consistency isn't such a useful property.

Given a realistic (i.e. finite) sample, is there a Frequentist property that holds true of the MLE? If not, the MLE isn't really Frequentist.

Answer 1

Or does every simple tool have to fall into exactly one of those two categories?

No. Simple (and not so simple tools) can be studied from many different viewpoints. The likelihood function by itself is a cornerstone in both Bayesian and frequentist statistics, and can be studied from both points of view! If you want, you can study the MLE as an approximate Bayes solution, or you can study its properties with asymptotic theory, in a frequentist way.

Answer 2

When you're doing Maximum Likelihood Estimation you consider the value of the estimate and the sampling properties of the estimator in order to establish the uncertainty of your estimate expressed as a confidence interval. I think this is important regarding your question because a confidence interval will in general depend on sample points that were not observed, which is seem by some as an essentially unbayesian property.

P.S. This is related to the more general fact that Maximum Likelihood Estimation (Point + Interval) fails to satisfy the Likelihood Principle, while a full ("Savage style") Bayesian analysis does.

Answer 3

The likelihood function is a function that involves the data and the unknown parameter(s). It can be viewed as the probability density for the observed data given the value(s) of the parameter(s). The parameters are fixed. So by itself the likelihood is a frequentist notion. Maximizing the likelihood is just to find the specific value(s) of the parameter(s) that makes the likelihood take on its maximum value. So maximum likelihood estimation is a frequentist method based solely on the data and the form of the model that is assumed to generate it. Bayesian estimation only enters in when a prior distribution is placed on the parameter(s) and Bayes' formula is used to obtain an aposteriori distribution for the parameter(s) by combining the prior with the likelihood.

Answer 4

Assuming that by "Bayesian" you refer to subjective Bayes (a.k.a. epistemic Bayes, De-Finetti Bayes) and not the current empirical Bayes meaning-- it is far from trivial. On the one hand, you infer based on your data alone. There are no subjective beliefs at hand. This seems frequentist enough... But the critique, expressed even at Fisher himself (a strict non (subjective) Bayesian), is that in the choice of the sampling distribution of the data subjectivity has crawled in. A parameter is only defined given our beliefs of the data generating process.

In conclusion-- I believe the MLE is typically considered a frequentist concept, albeit it is a mere matter of how you define "frequentist" and "Bayesian".

Answer 5

The point estimator that maximises $P(x|\theta)$ is the MLE. This is a commonly used point estimator in frequentist statistics, but it is less commonly used in Bayesian statistics. In Bayesian statistics it is usual to use a point estimator which is either the posterior expected value, or the value minimising the expected-loss (risk) in a decision problem. There are certainly some cases where the Bayesian estimator will correspond with the MLE (e.g., if we have a uniform prior, or in some special cases of minimising loss), but this is not a common occurrence. Hence, as a general rule, the MLE is usually a frequentist estimator.

Answer 6

(answering own question)

An estimator is a function that takes some data and produces a number (or range of numbers). An estimator, by itself, isn't really 'Bayesian' or 'frequentist' - you can think of it as a black box where numbers go in and numbers come out. You can present the same estimator to a frequentist and to a Bayesian and they will have different things to say about the estimator.

(I'm not happy with my simplistic distinction between frequentist and Bayesian - there are other issues to consider. But for simplicity, let's pretend that are just two well-defined philosophical camps.)

You cannot tell whether a researcher is frequentist of Bayesian just by which estimator they choose. The important thing is to listen to what analyses they do on the estimator and what reasons they give for choosing that estimator.

Imagine you create a piece of software that finds that value of $\theta$ which maximizes $\mathrm{P}(\mathbf{x}|\theta)$. You present this software to a frequentist and ask them to make a presentation about it. They will probably proceed by analyzing the sampling distribution and testing whether the estimator is biased. And maybe they'll check if it is consistent. They will either approve of, or disapprove of, the estimator based on properties such as this. These are the types of properties that a frequentist is interested in.

When the same software is presented to a Bayesian, the Bayesian might well be happy with much of the frequentist's analysis. Yes, all other things being equal, bias isn't good and consistency is good. But the Bayesian will be more interested in other things. The Bayesian will want to see if the estimator takes the shape of some function of posterior distribution; and if so, what prior was used? If the estimator is based on a posterior, the Bayesian will wonder whether the prior is good one. If they are happy with the prior, and if the estimator is reporting the mode of the posterior (as opposed to, say, the mean of the posterior) then they are happy to apply this interpretation to the estimate: "This estimate is the point estimate which has the best chance of being correct."

I often hear is said that frequentists and Bayesian "interpret" things differently, even when the numbers involved are the same. This can be a little confusing, and I don't think it's really true. Their interpretations don't conflict with each other; they simply make statements about different aspects of the system. Let's put aside point estimates for the moment and consider intervals instead. In particular, there are frequentist confidence intervals and Bayesian credible intervals. They will usually give different answers. But in certain models, with certain priors, the two types of interval will give the same numerical answer.

When the intervals are the same, how can we interpret them differently? A frequentist will say of an interval estimator:

Before I see the data or the corresponding interval, I can say there is at least a 95% probability that the true parameter will be contained within the interval.

whereas a Bayesian will say of an interval estimator:

After I see the data or the corresponding interval, I can say there is at least a 95% probability that the true parameter is contained within the interval.

These two statements are identical, apart from the words 'Before' and 'After'. The Bayesian will understand and agree with the former statement and also will acknowledge that its truth is independent of any prior, thereby making it 'stronger'. But speaking as a Bayesian myself, I would worry that the former statement mightn't be very useful. The frequentist won't like the latter statement, but I don't understand it well enough to give a fair description of the frequentist's objections.

After seeing the data, will the frequentist still be optimistic that the true value is contained within the interval? Maybe not. This is a bit counterintuitive but it is important for truly understanding confidence intervals and other concepts based on the sampling distribution. You might presume that the frequentist would still say "Given the data, I still think there is a 95% probability that the true value is in this interval". A frequentist would not only question whether that statement is true, they would also question whether it is meaningful to attribute probabilities in this way. If you have more questions on this, don't ask me, this issue is too much for me!

The Bayesian is happy to make that statement: "Conditioning on the data I have just seen, the probability is 95% that the true value is in this range."

I must admit I'm a little confused on one final point. I understand, and agree with, the statement made by the frequentist before the data is seen. I understand, and agree with, with the statement made by the Bayesian after the data is seen. However, I'm not so sure what the frequentist will say after the data is seen; will their beliefs about the world have changed? I'm not in a position to understand the frequentist philosophy here.