Psychotherapy trials frequently generate multilevel longitudinal data with 3 levels. This type of hierarchy exists in all trials in which therapists deliver the treatment and patients are repeatedly measured. Unfortunately, researchers often ignore the possibility that therapists could differ in their performance and instead assume there is no difference between therapists in their average impact on patients’ rate of change. In this new article, just published in Journal of Consulting and Clinical Psychology, we focused on scenarios in which therapists are fully and partially nested within treatments and investigate the consequences of ignoring even small therapist effects in longitudinal data.
We first derived the factors leading to increased Type I errors for the Time × Treatment effect in a balanced study. Scenarios with an unbalanced allocation of patients to therapists and studies with missing data were then investigated in a comprehensive simulation study, in which the correct 3-level linear mixed-effects model, which modeled therapist effects using a random slope at the therapist level, was compared with a misspecified 2-level model.
Results: Type I errors were strongly influenced by several interacting factors. Estimates of the therapist-level random slope suffer from bias when there are very few therapists per treatment.
Conclusion: Researchers should account for therapist effects in the rate of change in longitudinal studies. To facilitate this, we developed an open source R package powerlmm, which makes it easy to investigate model misspecification and conduct power analysis for these designs.
The paper has public health significance since it shows that the prevalent statistical practice of ignoring therapist effects when analyzing psychotherapy trials with longitudinal measures is likely to substantially increase the risk of finding a treatment effect when none exists. We show the factors most likely to lead to spurious findings and provide software tools that make it easy to perform our calculations.
Read the full paper:
Magnusson, K., Andersson, G., & Carlbring, P. (2018). The consequences of ignoring therapist effects in trials with longitudinal data: A simulation study. Journal of Consulting and Clinical Psychology, 86(9), 711-725. doi:10.1037/ccp0000333