1
$\begingroup$

If I run $n$ samples of a physical experiment, I expect to see roughly similar time vs. position plots but with slight variations run-to-run. What are good statistical metrics to quantify the consistency of run-to-run variations of this time series data? Ideally, they would be unitless and easy to understand, perhaps similar conceptually to temporally averaging the relative standard deviation (RSD) across all runs, but I wasn't sure if there's a more statistically commonplace method to average variance across a timespan.

As an example, off the top of my head I can think of an error metric for temporally averaged RSD, like: $$\frac{1}{T} \sum_{t=1}^T \left| \frac{\sigma(x_{1...n}(t))}{\overline{x_{1...n}}(t)} \right|$$ Saying the temporally averaged relative standard deviation is 3.8% (via the arbitrary metric above) is IMO more understandable if I said that average run-to-run standard deviation was 0.0415mm (see sample data below). I'm perfectly open to other methods of calculating consistency, but I've never seen metrics for measuring variance across multiple time series runs so I don't know what anything would even be called, or if it's statistically valid to even average the standard deviation across all time points.

Just to clarify, I'm specifically looking for metrics on run-to-run consistency, not a regression error metric, which IMO makes this question different from this one, which got answers assuming the poster had a model to compare against/which suggested Dynamic Time Warping as a curve-similarity metric, which has no physical meaning from my understanding unless you're comparing DTW metrics against each other?

Here's an example of sample data. For visualization purposes, I can plot the 95% percentile ($y(t) \pm 1.96\sigma_i(x(t))/\sqrt{n}$), but if I want to quantify my run-to-run variance I'm not sure what any of the formal names are/common practices, or if there's a commonplace statistic metric for expressing this in percentage variance.

Sample data pattern of 5 runs with slight noise in between themSample data with 95% percentile and mean shown.

$\endgroup$

0