Sufficient statistics - how can the conditional pdf not depend on $\theta$ when $\theta$ is in the equation?

Question

I understand the heuristic definition: say you know a statistic, $T$, of some sample that you want to use to estimate the corresponding population parameter - but you don't know the data points of the sample themselves. 'We say $T$ is a sufficient statistic if the statistician who knows the value of $T$ can do just as good a job of estimating the unknown parameter $\theta$ as the statistician who knows the entire random sample' - that's the definition of a sufficient statistic I've read online, and understand.

But then comes the factorisation theorem, which I'm struggling with: A statistic $T$ is a sufficient one for a sample $\boldsymbol{X} = (X_1,X_2,\ldots,X_n)$ if $f(\boldsymbol{X} \mid\theta)$, the conditional pdf for $\boldsymbol{X} $ given the parameter $\theta$ and stat. $T$, does not depend on $\theta$. This is equivalent to factorising $f(\boldsymbol{X} \mid\theta)$ into two functions:

$$f(\boldsymbol{X} \mid\theta) = h(X_1,X_2,\ldots,X_n) \cdot g(T(X_1,X_2,\ldots,X_n),\theta).$$

$T$ would then be a sufficient statistic, as the conditional probability $f(\boldsymbol{X} \mid\theta)$ now does not depend on $\theta$. But here's my question - how can the new factorised $f(\boldsymbol{X}\mid\theta)$ not depend on $\theta$ when $\theta$ is still in the final equation? In the examples I've seen, the final equations still have $\theta$ in them, as well as the statistic as some function of $X_1,X_2,\ldots,X_n$ - so how can the conditional probability depend on $T$ alone?

If $T$ is supposed to be all you need to know to know the conditional distribution, how can $\theta$ be a variable in the equation that you need the data of? I think I've gone wrong in some basic understanding of what's supposed to be going on here, so apologies if this is elementary.

Answer 1

Perhaps a specific example [similar to one in Bain & Englehardt, 2e (1992); Example 10.2.1, p338] will help by showing the required functional independence.

Let data $\mathbf{X} = (x_1, \dots, x_n)$ be a random sample from $\mathsf{Exp}(\lambda),$ an exponential distribution with rate $\lambda;$ and let $t = \sum_i x_i.$ We wish to show that $t$ is sufficient for $\lambda.$

First, the joint density function is

$$f_{\mathbf{X};\lambda}(x_1, \dots, x_n;\lambda) = \lambda^ne^{-\lambda t},\; \text{for}\; x_i > 0.$$

Also, one can show using moment generating functions that $t \sim \mathsf{Gamma}(n, \lambda),$ so that $$f_{t;\lambda}(t;\lambda) = \frac{\lambda^n}{\Gamma(n)}t^{n-1}e^{-\lambda t},\; \text{for}\; t > 0.$$

Thus

$$f_{\mathbf{X}|t}(x_i,\dots,x_n|t) = \frac{f_{\mathbf{X};\lambda}(x_1, \dots, x_n;\lambda)}{f_{t;\lambda}(t;\lambda)} = \frac{\Gamma(n)}{t^{n-1}},$$ which is functionally independent of the parameter $\lambda,$ so that the statistic $t$ is sufficient for $\lambda.$

---

Note: The simulation below illustrates (for $n=5$ and $\lambda = 3)$ that $\hat \lambda = \frac{n-1}{t}$ is an unbiased estimator of $\lambda$ and that $t \sim \mathsf{Gamma(n, \lambda)}.$

set.seed(1884);  m = 10^6;  n=5;  lam = 3
t = replicate( m, sum(rexp(n, lam)) )
mean((n-1)/t)
[1] 2.999782  # aprx E(4/t) = 3

hist(t, prob=T, col="skyblue2", br=30, main="Simulated Total with GAMMA(5, 3) Density")
  curve(dgamma(x,n,lam), add=T, lwd=2)

Answer 2

You have slightly misunderstood sufficiency and the factorisation theorem here. You can see from the form of the factorisation theorem that the conditional density $f(\boldsymbol{X}|\theta)$ does depend on $\theta$. (You are right - because the value $\theta$ is in the equation, this density does indeed depend on $\theta$.) However, if the factorisation in the factorisation theorem holds, then it can be shown that:

$$f(\boldsymbol{X}|T(\boldsymbol{X}), \theta) = f(\boldsymbol{X}|T(\boldsymbol{X})) = \text{Function depending on }T \text{ but not }\theta,$$

and thus, the conditional density $f(\boldsymbol{X}|T(\boldsymbol{X}), \theta)$ does not depend on $\theta$. That latter property is what is required for sufficiency of $T$. The factorisation theorem just says that if the density $f(\boldsymbol{X}|\theta)$ has a certain form, then the required condition for sufficiency will emerge from this.

Remember that sufficiency means that if you already know the sufficient statistic (i.e., when you conditional on $T$), then the parameter has no further influence on the density of the observed data. If you don't condition on the sufficient statistic then nothing happens - the data is still dependent on the parameter of the underlying distribution.