Is Gaussian Process just a Multivariate Gaussian Distribution?

Question

Besides the fact that The covariance is constructed using kernels. I am not sure if I am as clear on how MVN is much different from GP.

Answer 1

The multivariate Gaussian distribution is a distribution that describes the behaviour of a finite (or at least countable) random vector. Contrarily, a Gaussian process is a stochastic process defined over a continuum of values (i.e., an uncountably large set of values). Usually the process is defined over all real time inputs, so it is a process of the form $\{ X(t) | t \in \mathbb{R} \}$. The Gaussian process is fully defined by a mean function and covariance function, which respectively describe the mean of the process at any point, and the covariance of the process at any two points.

Now, one of the central properties of the Gaussian process is that any finite vector of points has a multivariate Gaussian distribution with mean vector and variance matrix described by the mean function and covariance function of the process. Specifically, for any time points $\mathbf{t}=(t_1,...,t_n)$ we have:

$$[X(t_1),...,X(t_n)] \sim \text{N}(\boldsymbol{\mu}(\mathbf{t}), \boldsymbol{\Sigma}(\mathbf{t})).$$

where $\boldsymbol{\mu}(\mathbf{t}) = [\mu(t_i)]_{i=1,...,n}$ is the mean vector composed of values of the mean function over these time points, and $\boldsymbol{\Sigma}(\mathbf{t}) = [\sigma(t_i, t_j)]_{i,j=1,...,n}$ is the variance matrix composed of values of the covariance function over pairs of time points. The stochastic behaviour of the Gaussian process can be regarded as an extension of the multivariate Gaussian distribution to stochastic process defined on a continuum.

Answer 2

A Gaussian process is a generalization of the Gaussian probability distribution. Whereas a probability distribution describes random variables which are scalars or vectors (for multivariate distributions), a stochastic process governs the properties of functions. Leaving mathematical sophistication aside, one can loosely think of a function as a very long vector, each entry in the vector specifying the function value f (x) at a particular input x. It turns out, that although this idea is a little naı̈ve, it is surprisingly close what we need. Indeed, the question of how we deal computationally with these infinite dimensional objects has the most pleasant resolution imaginable: if you ask only for the properties of the function at a finite number of points, then inference in the Gaussian process will give you the same answer if you ignore the infinitely many other points, as if you would have taken them all into account! And these answers are consistent with answers to any other finite queries you may have. One of the main attractions of the Gaussian process framework is precisely that it unites a sophisticated and consistent view with computational tractability.

– C. E. Rasmussen & C. K. I. Williams, Gaussian Processes for Machine Learning, the MIT Press, 2006. (Emphasis is my own.)

Answer 3

I had the same question so I found the following on Stan user guide which I think has both a sort answer and a more detailed answer if you want to read further (well, by now you probably have nailed it but it might be interesting for other)