1
$\begingroup$

The independent variable in my data is verb class: I have got four verb classes, each consisting of ten verbs. The dependent variables are accuracy (either 0 - incorrect or 1 - correct), and reaction time. I would like to perform separate analyses to see if the accuracy scores are significantly different between verb classes, and if the reaction times are significantly different. Due to the relatively small sample size of participants (10), I would have to rely on non-parametric tests.

My supervisor has told me to make use of Friedman's ANOVA to assess this (with Wilcoxon signed-rank tests as post-hoc testing).

However, some accuracy scores are missing (due to technical errors or other issues), and I have only got reaction times for the items that have been answered correctly (since there is no reaction time for a wrong answer). Therefore, the groups differ in size. Would this be an issue for carrying out a Friedman's ANOVA, and if so, what would be a good workaround or alternative?

Improve this question
$\endgroup$
2
  • $\begingroup$ Nonparametric/semiparametric methods are efficient and much more flexible than a gamma distribution. See here. $\endgroup$
    – Frank Harrell
    Commented Jul 4 at 11:51
  • $\begingroup$ It doesn’t pay to assess normality, because semiparametric models are so efficient. And of course if normality doesn’t hold semiparametric / nonparametric methods can be much more powerful than parametric procedures. $\endgroup$
    – Frank Harrell
    Commented Jul 4 at 15:50

2 Answers 2

Reset to default
0
$\begingroup$

In general, Friedman's test is used for complete designs. The Skillings-Mack test might be a solution for you: it is a Friedman-type test for designs with missing data. More info about this test can be found in this article.

If for some reason you really want to evade this alternative and rely on Friedman, another approach I can think of is to try multiple imputation to impute the data, but this will introduce some bias of course.

Improve this answer
$\endgroup$
1
$\begingroup$

It may give misleading results to omit reaction times for wrong answers. Instead this may be considered as right-censoring of reaction times for which a time-to-event (survival) analysis such as the Cox proportional hazards semiparametric model can do.

To account for the repetitions within subject, which create intra-cluster correlation, you can use random subject effects in the Cox model, or possibly use GEE with a cluster sandwich variance estimator.

Note that seeking ‘statistical significance’ is to have an extremely low bar and to be interested in arbitrary thresholds such as $\alpha=0.05$.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi Frank, thank you for your answer! To clarify why there are no reaction times for wrong answers: besides incorrect answers (e.g., participant says 'to clean' instead of 'to paint'), 'wrong answers' often consisted of a lack of response; therefore, it isn't an option to include reaction times for wrong answers. $\endgroup$
    – Lotte Nijs
    Commented Jul 4 at 14:32
  • $\begingroup$ I was thinking that you would right censor reaction time at the last scheduled observation duration so that your estimate of the reaction time distribution would reflect the probability of a response that is correct. Otherwise you are estimating a perhaps hard-to-interpret conditional quantity: time to react if correct. $\endgroup$
    – Frank Harrell
    Commented Jul 4 at 15:48

Not the answer you're looking for? Browse other questions tagged or ask your own question.