Reproducibility in small-N treatment research (Cavanaugh et al., 2022)

Purpose: Small-N studies are the dominant study design supporting evidence-based interventions in communication science and disorders, including treatments for aphasia and related disorders. However, there is little guidance for conducting reproducible analyses or selecting appropriate effect sizes in small-N studies, which has implications for scientific review, rigor, and replication. This tutorial aims to (a) demonstrate how to conduct reproducible analyses using effect sizes common to research in aphasia and related disorders and (b) provide a conceptual discussion to improve the reader’s understanding of these effect sizes.

Method: We provide a tutorial on reproducible analyses of small-N designs in the statistical programming language R using published data from Wambaugh et al. (2017). In addition, we discuss the strengths, weaknesses, reporting requirements, and impact of experimental design decisions on effect sizes common to this body of research.

Results: Reproducible code demonstrates implementation and comparison of within-case standardized mean difference, proportion of maximal gain, tau-U, and frequentist and Bayesian mixed-effects models. Data, code, and an interactive web application are available as a resource for researchers, clinicians, and students.

Conclusions: Pursuing reproducible research is key to promoting transparency in small-N treatment research. Researchers and clinicians must understand the properties of common effect size measures to make informed decisions in order to select ideal effect size measures and act as informed consumers of small-N studies. Together, a commitment to reproducibility and a keen understanding of effect sizes can improve the scientific rigor and synthesis of the evidence supporting clinical services in aphasiology and in communication sciences and disorders more broadly.

Supplemental Material S1. Code needed to reproduce the analysis in Section 1 of the article.

Supplemental Material S2. Code batch calculate effect sizes.

Cavanaugh, R., Quique, Y. M., Swiderski, A. M., Kallhoff, L., Terhorst, L., Wambaugh, J., Hula, W. D., & Evans, W. S. (2022). Reproducibility in small-N treatment research: A tutorial using examples from aphasiology. Journal of Speech, Language, and Hearing Research, 66(6), 1908–1927. https://doi.org/10.1044/2022_JSLHR-22-00333

Publisher Note: This article is part of the Forum: Promoting Reproducibility for the Speech, Language, and Hearing Sciences.