Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
607
questions
2
votes
1
answer
49
views
In Expectation-Maximization, in the maximization step, do we maximize expectation of the log likelihood (wikipedia) or evidence lower bound (cs 229)?
From cs 229 page 6:
Intuitively, the EM algorithm alternatively updates Q and θ by a) setting
Q(z) = p(z|x; θ) following Equation (8) so that ELBO(x; Q, θ) = log p(x; θ)
for x and the current θ, and ...
- 2,603
0
votes
0
answers
19
views
How to evaluate this conditional expectation for the E-step in expectation-maximisation?
I'm trying to devise an expectation-maximisation algorithm for a certain problem but I'm unable to derive the conditional expectation in the E-step. For the purpose of this question I'll simplify the ...
- 43
2
votes
2
answers
62
views
Why does Variational Inference work?
ELBO is a lower bound, and only matches the true likelihood when the q-distribution/encoder we choose equals to the true posterior distribution. Are there any guarantees that maximizing ELBO indeed ...
- 143
0
votes
0
answers
16
views
For EM algorithm, why we assign an independent distribution $Q_i$ for each sample index
I'm learning the expectation maximization algorithm with Andrew's CS229 lecture notes https://pillowlab.princeton.edu/teaching/statneuro2020/notes/notes18_LatentVariableModels.pdf
The derivation and ...
- 11
0
votes
0
answers
26
views
Can complicated $\mathcal{Q} (\theta; \theta^{\text{old}})$ function be replaced by log-likelihood when implementing/coding EM algorithm?
I am working on a MLE problem where one of the parameters does not have a closed-form solution. I have a proposal for $\theta^{\text{new}}$, but reject it if it does not improve $\mathcal{Q} (\theta; \...
3
votes
2
answers
48
views
Does Factor Analysis completely mitigate the singular covariance matrix problem?
Background
I have been trying to understand Stanford CS 229’s lecture about Factor Analysis and the accompanying lecture notes. The lecturer introduced Factor Analysis as a way to mitigate the ...
- 131
1
vote
2
answers
41
views
Tractability of the E Step of the Expectation Maximization Algorithm
To optimize a probabilistic model with latent variables, I was trying to convince myself that there are situations where computing the marginalization $p(x)$ is intractable and computing the $Q(\theta ...
- 61
1
vote
0
answers
80
views
Datasets with multiple maximum likelihood estimators
There is a sizeable body of literature on the issue of multiple maximizers in maximum likelihood estimation, such as
https://projecteuclid.org/journals/statistical-science/volume-15/issue-4/...
- 139
0
votes
0
answers
31
views
Under What Conditions Does a Gaussian Mixture Model (GMM) Have Maximum Entropy?
Introduction
I'm delving into Gaussian Mixture Models (GMMs) within unsupervised learning frameworks and am particularly interested in their statistical properties, with a focus on entropy. Entropy ...
- 113
0
votes
0
answers
17
views
Expected Variance of EM Estimator in GMM with Respect to Observations
Title:
Variance of EM Estimator in GMM with Respect to Observations
Body:
I'm estimating a parameter S from observations X and <...
- 113
1
vote
0
answers
40
views
Understanding Variational inference and EM in relation to each other
I have read several answers like here but, somehow I still have a few doubts. I hope to present my understanding and ask a few questions to clear my doubts
EM:
A maximization maximization algorithm
E-...
- 2,603
4
votes
3
answers
159
views
Justification of independence assumption for latent variables in Expectation Maximization algorithm
When deriving the ELBO/free energy in the EM algorithm, it is often done in a "general" case of observed and latent variables and then an assumption of independent (or iid) variables is ...
- 101
0
votes
0
answers
24
views
Can I assume that this is a GMM?
I'm trying to find the MLE for the parameters of the following distribution:
$$f(x) = a \ \mathcal{N}(\mu_a, 1) + \beta \ \mathcal{N}(\mu_\beta, 1)$$
Taking the log likelihood of this complicates ...
- 121
0
votes
0
answers
32
views
Definition of expectation with condition variables
I am having a hard time of digesting this, which is part of EM algorithm that I borrowed Equation 3.2.7 from https://www.informit.com/articles/article.aspx?p=363730&seqNum=2#:~:text=3.2%...
- 9
3
votes
0
answers
20
views
Feature importance in expectation maximization
The context is using EM algorithm for a mixture model - more precisely Dirichlet Multinomial Mixture, as discussed in Dirichlet Multinomial Mixtures: Generative Models for Microbial Metagenomics. One ...
- 4,439