Upon close reading, this appears to be 3 (interesting) questions, not one. I'm not sure if the mods have the tools needed to split it up, so I'm just going to write down the three questions as I see them and then deal with them one by one. Note, it is simpler for me to talk about variance instead of volatility. This has no material impact on the answer.
Question 1: Is the variance of the one-day ahead return forecastable?
Question 2: Is the distribution of the one-day ahead return forecastable?
Question 3: What methods are available for ex post assessment of a forecasting model for variance or a forecasting model for the return distribution?
My answers will use a common notation, so I begin with that:
Let $p_{n,t}$ denote the $n^{th}$ transaction on the $t^{th}$ day on some risky asset in a financial market. For simplicity let us assume the market is open all 24 hours of the day, e.g. the FX market. I'm only making this assumption so I can skip dealing with pesky institutional details when talking about high frequency data. Let $N+1$ denote the number of transactions each day, and, again for simplicity, assume one of those transactions occurs at the exact roll-over time from one day to the next. The sequence of intraday continuously compounding returns on day $t$ can thus be defined $r_{n,t} = \log(p_{n,t}) -\log(p_{n-1, t})$. By construction, the daily return is thus:
\begin{equation}
r_t = \log(p_{N,t}) - \log(p_{0,t}) = \sum_{n=1}^N r_{n,t}
\end{equation}
Okay, here we go.
Answer 1: Assume we're at time $t$. Then the variance of the one-day ahead return is $\mathbb{V} r_{t+1}$. This is an unconditional variance. Before we do anything else, let's ask a very simple question: Does this quantity exist? The unconditional distribution of daily financial returns are known to be heavily fat-tailed, and this has led several authors to attempt to test for the existence of the unconditional second moment of daily returns. In general, this is a hard thing to test for (in a rigorous statistical framework), so rather than point you toward hefty academic literature, have a look at some of the more heuristic evidence presented by authors like Nassim Taleb or Benoit Mandelbrot.
However, based on your question, I suspect that you are more interested in a conditional distribution of daily returns. Probably the most well-known literature dealing with this question is the ARCH/GARCH strand which was partially responsible for netting the original author (Rob Engle) a Nobel prize (well, the Nobel-equivalent presented by the Royal Swedish Academy of Sciences).
This strand of the literature proposed a set of models for volatility at time $t+1$ based on time $t$ information. The most famous is probably the GARCH model: $\sigma_{t+1}^2 = \omega + \alpha \epsilon_t^2 + \beta \sigma_t^2$, where $\epsilon_t$ is the source of randomness, sometimes set equal to $r_t$. Does this model have any predictive ability?
There were quite a few papers in the 1990's that suggested, via the methodology of Mincer-Zarnowitz regressions (Mincer, Zarnowitz (1969) "The Evaluation of Economic Forecasts"), that it has almost no predictive ability. These regressions involve regressing the forecast on the quantity that you are attempting to forecast. Of course, the quantity that we are attempting to forecast here is unobservable. So the authors used squared daily returns as a proxy. This proved to be a poor choice, because although it is unbiased for true variance, it is also a very noisy proxy. In the classic paper, Andersen, Bollerslev (1998) "Answering the Skeptics: Yes, Standard Volatility Models do Provide Accurate Forecasts", it was shown that the poor forecast evaluation results of the early 90's were purely a result of the noise in the proxy. Andersen and Bollerslev used a more accurate proxy, the "realised variance" which was first proposed in Merton (1980) "On Estimating the Expected Return Of the Market" (in an appendix). The realised variance is just the sum of squared intraday returns, i.e.
\begin{equation}
RV_t = \sum_{n}^N r_{n,t}^2
\end{equation}
and under ideal modelling assumptions, it can be shown that $RV_t$ converges to a quantity known as quadratic variation, which, for the purposes of answering this question we will pretend is equal to $\mathbb{V} r_t | \sigma_t = \sigma_t^2$ (a deeper discussion of this is fraught with pit-falls and low-level assumptions and is not suitable for answering this question). Thus, Andersen and Bollerslev simply used a proxy that had a lot less noise in it. They were able to show that, in fact, the conditional volatility models actually have a quite good predictive ability for daily variance.
More recently, other classes of variance forecasting models have been proposed, e.g. stochastic volatility models, or, perhaps the most interesting, forecasting models that use estimators like realised variance as inputs. These models have been shown to have predictive ability that is at least as good as (and possibly much better than) the conditional volatility models.
The other candidate is, as you mention, the VIX. I have mixed feelings about this. First, it is fairly well-known in the literature (can't remember the reference off the top of my head) that there are periods where the VIX will be a significantly biased forecast of true volatility (that is, ex post, one can show periods of several months where VIX forecasts were consistently too large or too small). On the other hand, there are solid theoretical reasons for using VIX, although they are all based on modelling the price process as a continuous-time semi-martingale... I am approaching another discussion here fraught with dangers and pitfalls, so best to leave it at that. Certainly I would expect the VIX to massively outperform a monkey throwing darts at a dartboard.
So in conclusion, there are lots of methods for forecasting variance, and ample evidence that they work quite well. I'll talk a little bit more about the specific methodologies of forecast evaluation in answer 3.
Answer 2: Under certain modelling the assumptions, the answer to this question is identical to the answer for the previous question. Specifically, if:
\begin{equation}
r_t | \sigma_t \backsim \mathcal{N}(0, \sigma_t^2)
\end{equation}
then a forecast model for variance is equivalent to a forecast model for the distribution.
So what evidence do we have to support the above model for the conditional distribution of daily returns?
If we knew $\sigma_t^2$ ex post, then such a model would be trivial to test for. Simply apply a Kolmogorov-Smirnov test to the sequence $r_t / \sigma_t$ with null of the sequence being generated by a standard Normal distribution. Unfortunately, as mentioned, $\sigma_t$ is unobservable. However, as mentioned previously in this answer, we do know that $RV_t \rightarrow \sigma_t^2$ under certain modelling assumptions. So in Andersen, Bollerslev, Diebold, and Ebens (2001) "The Distribution of Realized Stock Return Volatility", the authors examine the sequence $r_t / \sqrt{RV_t}$ and find that it is very close to a standard Normal (see Figure 1 of that paper). The slight deviation from the standard Normal that is observed could easily be due to the noise present in $RV_t$ as a proxy for true variance, and so the evidence quite strongly supports the suggestion that $r_t | \sigma_t \backsim \mathcal{N}(0, \sigma_t^2)$.
Of course the assumption that $\mathbb{E} r_t = 0$ is clearly false, however, over horizons as short as one day, the expected value of $r_t$ is often assumed to be small enough relative to the (conditional) variance of $r_t$ that assuming $\mathbb{E} r_t = 0$ is relatively harmless (and infinitely preferable to substituting in a very noisy estimator for $\mathbb{E} r_t$). In practice, this is not much use to many practitioners, who are very interested in days in which the conditional mean of $r_t$ is not zero. For now (and possibly forever) this is an open issue in financial economics and financial econometrics.
So what about non-normal distributions $r_t$? We have already discussed that the unconditional distribution is clearly fat-tailed, and possibly does not even have finite variance. Regarding other modelling assumptions, there has been plenty of literature, e.g. autoregressive models for distribution functions e.t.c., but it is all fairly heavy-going and probably not what you're really after here.
The only exception to the above is the forecasting of a specific parameter of the distribution, namely, the quantile. This has received a lot of attention due to the widespread use of Value-at-Risk. Off the top of my head, forecasting models for quantiles that do not make any assumption of Normality include those based on Extreme Value Theory, and the CAViaR class of models first proposed in Engle, Manganelli (2004) "CAViaR: Conditional Autoregressive Value-at-Risk by Regression Quantiles".
So in summary, yes, the conditional probability distribution of $r_{t+1}$ can be predicted with some accuracy if one is willing to assume that $r_t | \sigma_t \backsim \mathcal{N}(0, \sigma_t^2)$. This predictive accuracy comes from the predictive accuracy discussed in answer 1 above.
Answer 3: We can split this answer into two parts: 1) methods for evaluating forecast procedures for daily variance, and 2) methods for evaluating forecast procedures for the distribution of a daily return.
There are a couple of ways one can evaluate forecast models for daily variance. The most popular in the extant literature is via loss-based methods, e.g. Diebold, Mariano (1995) "Comparing Predictive Accuracy", West (1996) "Asymptotic Inference About Predictive Ability", White (2000) "A Reality Check For Data Snooping", Hansen (2005) "A Test for Superior Predictive Ability" or Hansen, Lunde and Nason "The Model Confidence Set". All these methods, one way or another, are simply evaluating the distance (for some suitably chosen metric) between the forecast, and the thing we are attempting to forecast. Of course, the forecast target is unobservable, so we use the same trick mentioned above of substituting in a proxy. There are some surprising tricks lurking here for the unwary, see e.g. Patton (2011) "Volatility Forecast Comparison Using Imperfect Volatility Proxies". In summary, if your proxy is a fairly good one, e.g. Realised Variance for a highly-traded asset, then you can use whatever loss function you want, but if you use a noisier proxy, e.g. squared daily returns, you'll need to restrict your analysis to a class of "robust" loss functions.
Next up is methods for evaluating forecasts of probability distributions. This hasn't received as much attention in the literature. Why? Well, A large part of financial economics is based on the continuous-time semi-martingale model. In particular, the following specific assumption is often made:
\begin{equation}
dp_t = \mu_t dt + \sigma_t dW_t
\end{equation}
where $\mu_t$ and $\sigma_t$ obey some set of boundedness conditions, and $W_t$ is a Wiener process. It is readily apparent that this model is going to suggest that daily returns should be modelled as conditionally Normal (conditioning on $\mu_t$ and $\sigma_t$). So other distributional assumptions simply don't arise that much.
Don't get me wrong, I am not particularly enamoured of this assumption (or even the above model). Nonetheless, that is the current state of affairs.
To conclude this answer, you mentioned in the question some sort of ad hoc procedure for looking at ex post daily returns and seeing if they fit some sore of distributional forecast model. Such a procedure could probably be made to work for the unconditional distribution without too much additional work, but for the conditional distribution it is a bit more difficult. As in answer 2, some work has been done in this area for one specific aspect of the conditional distribution, namely, the quantile. You might be interested in having a look at either Christoffersen (1998) "Evaluating Interval Forecasts" or the Dynamic Quantile (DQ) test in Engle, Manganelli (2004) "CAViaR: Conditional Autoregressive Value-at-Risk by Regression Quantiles". Both tests look at ex post daily returns and analyse the number of VaR violations relative to those predicted by the VaR forecasting model.
If you wanted to compare ex post returns to forecasts of the entire conditional distribution, I think you would need a lot of daily returns to do this with any accuracy. I can definitely think of a way to do it using intraday data, but that is quite a long story and this is already quite a long answer.
Cheers all, I hope the readers who made it to this point don't feel like they've wasted their time.
Colin
