Using multilevel modeling to assess gender differences in intervention effects of an RCT, and how to look at mechanisms of change

Question

I'm looking for some help here - so I'm a grad student and stats newbie (but trying to move on from newbie status!) and attempting to figure out how to conduct the analyses I'm interested in but I'm feeling very lost.

I'm looking to do MLM with a large RCT (N=434) with the active treatment vs. services as usual (i.e., 2 treatment conditions), and I have four total timepoints - baseline, posttreatment, 6 months follow-up, and 12 months follow-up. The main purpose of the study is to look at gender as a moderator of treatment outcomes, and a secondary goal is to look at whether there are differences in mechanisms/mediators of treatment effects by gender as well. I spoke with a professor in the quant psychology department about my project and he mentioned that the strength of my project is that I have follow-up data with such a large sample (and high retention rates) with a marginalized, hard-to-reach population - so he suggested using MLM. Typically, when I've used MLM in the past, I've done it with looking at clusters or within/between individual change across multiple timepoints. But is there a way to use MLM to address my research questions - 1) to look at gender as a moderator of intervention outcomes in this large-scale RCT, and 2) to look at whether hypothesized, female-specific mediators of change are indeed more predictive of improved treatment outcomes for females relative to males? I'm feeling a little lost as to what would be my level 1 vs level 2 variables here.

Any help would be greatly appreciated!! Thank you so much!!

Answer 1

It doesn't make sense to use MLM merely because you have a lot of data, you use it because you have data with variance at different levels. That said, I very much prefer the terminology of "mixed effects models" rather than "multilevel" models for the sort of question you're asking, where you actually don't seem to be interested in variance at different hierarchical levels, but rather about the fixed effects. I'd recommend reading about mixed effects modeling and starting from there, though if you are familiar with MLM you will probably quickly discover they are exactly the same thing. There are also blogs out there talking about the terminology/approach differences that you might find helpful, like here: https://www.theanalysisfactor.com/multilevel-hierarchical-mixed-models-terminology/

In your case, it sounds like you have repeated measures in some subjects; therefore, it would make sense to have a random effect for subject, otherwise your data are violating assumptions of independence. However, your research questions are all about the fixed effects of gender and gender interactions with other parameters. The random effects here would seem to be more of a nuisance parameter that you include in your model but aren't likely to discuss at all.

I can't give you a complete answer of how you should structure your models with the information provided (and don't think I really should, unless you want to collaborate and give authorship - setting up the model to answer questions like this is pretty much the main intellectual effort for a project like this), and there's a whole separate world of mediation analysis, but I think this should be enough to get you working towards a solution.