How should I analyze data that is paired and also repeatedly measured?

Question

I have data collected from the same subjects (n=10) over 11 days, with measurements taken under two conditions each day: immediately after training(D0) and 30 minutes after training(D30). The data is ordinal, scaled from 1 to 10. I want to learn if there is a different between measurements taken immediately after trainings and 30 minutes after trainings. How should I analyze this data, which test should I use?

Answer 1

I don't have a named test for you, but here is how I would try modelling this data.

Let $X^{(i,n,t)}=1\dots 10$ be the results of $i$-th individual on $n$-th day observed either before ($t=0$) or after ($t=1$) training. These can clearly be modeled as multinomial variables, so the question is how to model the probability of each result, for each subject.

Since you do not have that much data you can try going with:

$$ \mbox{logit} P\left[X^{(i,n,t)}\ge k\right]=\left(1-t\right)\cdot a^{(i,n)}+t\cdot b^{(i,n)}-d_{k},\quad d_{k+1}\ge d_k $$

So the log-odds of $X^{(i,n,t)}\ge k$ has a common threshold for each level $k$, but where each subject starts relative to each threshold is specific to the subject and to the day, and where the subject ends after test is also specific to the subject. Your measurable effect can be something like $b^{(i,n)}-a^{(i,n)}$ for each subject. You can pool the effect amongst all individuals by letting $a^{\left(i,n\right)}=a^{\left(n\right)}$ and the same for $b^{(i,n)}$. You can further try to isolate the averaged effect over all days by letting $b^{(n)}=B+a^{(n)}$, i.e. each day the population may start at a different point each day, but the change before-after test ($B$) is the same for all days.

You will probably need to have some special treatment around $k=1$since $P\left[X^{(i,t)}\ge 1\right]=1$ irrespective of the statistical model.

You will need to fit these parameters to the data. In terms of significance, your null hypothesis could be that $a^{(i,n)}=b^{(i,n)}$.

I would probably try modelling it as a Bayesian model with the following priors: $a^{(i,n)}$ are $b^{(i,n)}$ normally distributed and $d_k=f+\sum_{r=1}^k g_r$ where $f$ can be normally distributed, whilst $g_{\dots}$ have to be non-negative (e.g. Gamma distributed).

Answer 2

The answer will differ depend on whether or not the subjects are receiving training on the same topic each day. For instance, if the questions is about training retention and the topics differ each day, than you can safely ignore the effect of time (day) in the model. However, if subjects receive training on the same topic each day, than time becomes very relevant (e.g., imagine a Subject that struggles early on but studies the topic daily vs one that never studies).

If the topic is new every day, a mixed-effects model that uses the ordinal score as a response variable, time since training (immediate vs delayed) as a categorical treatment, and subject as a random effect seems reasonable.

If the topic never changes, you would want to include Day as a covariate in the model.