0
$\begingroup$

Consider an urn from which we sample with replacement. Let $\pi$ represent the proportion of the urn's balls that are black, with the remainder being white.

From a frequentist perspective, each observation is treated as an independent and identically distributed (IID) variable. However, under a Bayesian perspective, do these observations still retain their independence? It seems each draw updates our estimate of $\pi$, potentially affecting the probability of the next draw being black.

Can someone clarify where I might be misunderstanding this?

Edit - a concrete example: Let's assume a very simple prior. There is a $q$ probability that the urn is completely black ($\pi=1$) and $1-q$ that it is completely white ($\pi=0$).

We draw the first ball, and it is black. Therefore, the second ball must be black as well: $p(X_2=black\mid X_1=black)=1$. This is in contrast to the marginal probability, $p(X_2=black)=p(X_1=black)=q$.

The same logic would apply to any prior (not necessarily dichotomous). If the parameter is viewed as a random variable, doesn't it automatically render the observations dependent?

Edit 2: Wikipedia's article on conditional independence put forward a very similar thought experiment (using a poll instead of an urn), and suggests that under the Bayesian perspective "the random variables $X_1$, ..., $X_n$ are not independent, but they are conditionally independent given the value of $p$" ($p$ is the probability of answering "yes" in the poll).

If we treat an unknown parameter as a random variable, are observations that depend on the parameter conditionally independent instead of simply independent?

Improve this question
$\endgroup$
9
  • 2
    $\begingroup$ Both frameworks use the same mathematical theory of probability. The observations do not depend on what anybody's estimate of $\pi$ might be. $\endgroup$
    – whuber ♦
    Commented Apr 28 at 18:16
  • $\begingroup$ Hi: As whuber stated, the independence of the trials can be ( and is assumed to be ) uneffected by the number of blacks that appear. But you can still use a bayesian approach. You would need to start out with some prior distribution for $\pi$ and an assumed likelihood for the number that were black. Then, given the number of blacks, there will be posterior for $\pi$ which can be updated etc. Just the standard bayesian approach. $\endgroup$
    – mlofton
    Commented Apr 28 at 18:39
  • $\begingroup$ @whuber, please look at the example I added. You are right, but I'd like to understand why the argument I made above is wrong. $\endgroup$
    – Trisoloriansunscreen
    Commented Apr 29 at 5:28
  • $\begingroup$ Your example looks like it continues to confound the value of a parameter with estimates of its value. $\endgroup$
    – whuber ♦
    Commented Apr 29 at 13:41
  • 1
    $\begingroup$ Trisoloriansunscreen: As soon as you treat the parameter as a random variable you are in bayesian land !!!!! In that land, you can update the parameter as new information comes in and you are then correct that the number of balls observed "effects" the parameter update. But that is unrelated to the standard concept of independence in the classical ( non-bayesian ) framework. $\endgroup$
    – mlofton
    Commented May 2 at 3:08

1 Answer 1

Reset to default
3
+50
$\begingroup$

This is an example of confusing logical and causal independence. The number of black balls in the urn, and hence the empirical distribution of the color of the next draw ($X_2$) is causally independent from $X_1$, i.e. your draw does not influence it in any way. However, it is not logically independent: By looking at the color of your current draw, your information about $\pi$, and therefore the plausibility you assign to $X_2=black$ changes.

Edwin Jaynes talks about this distinction between logical and causal (in)dependence extensively in his book "Probability Theory: The Logic of Science". For example, look at the section "6.11 A simple Bayesian estimate: quantitative prior information".

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you @Feri. Do you agree that if $\pi$ is treated as a random variable, then $p(X_2 = b \mid X_1 = b) = p(X_2 = b \mid \pi =1) p(\pi = 1 \mid X_1 = b) + p(X_2 = b \mid pi = 0) p(\pi=0 \mid X_1 = b) = 1$, which may be unequal to $p(X_2 = b) = q$, and hence only conditional independence holds? To me, it seems that you (and @whuber) are answering from a frequentist perspective, assuming there is no uncertainty about $\pi$. $\endgroup$
    – Trisoloriansunscreen
    Commented May 4 at 8:25
  • $\begingroup$ @Trisoloriansunscreen My answer is written from a Bayesian perspective, assumes uncertainty about $\pi$, and admits $Pr(X_2=b|X_1=b)\neq Pr(X_2=b)$ in the general case. $\endgroup$
    – Feri
    Commented May 5 at 0:08

Not the answer you're looking for? Browse other questions tagged or ask your own question.