ASHA journals
3 files

Reproducibility in small-N treatment research (Cavanaugh et al., 2022)

Download all (1.49 MB)
online resource
posted on 2022-12-21, 22:10 authored by Robert Cavanaugh, Yina Quique, Alexander M. Swiderski, Lydia Kallhoff, Lauren Terhorst, Julie Wambaugh, William D. Hula, William S. Evans

Purpose: Small-N studies are the dominant study design supporting evidence-based interventions in communication science and disorders, including treatments for aphasia and related disorders. However, there is little guidance for conducting reproducible analyses or selecting appropriate effect sizes in small-N studies, which has implications for scientific review, rigor, and replication. This tutorial aims to (a) demonstrate how to conduct reproducible analyses using effect sizes common to research in aphasia and related disorders and (b) provide a conceptual discussion to improve the reader’s understanding of these effect sizes.

Method: We provide a tutorial on reproducible analyses of small-N designs in the statistical programming language R using published data from Wambaugh et al. (2017). In addition, we discuss the strengths, weaknesses, reporting requirements, and impact of experimental design decisions on effect sizes common to this body of research.

Results: Reproducible code demonstrates implementation and comparison of within-case standardized mean difference, proportion of maximal gain, tau-U, and frequentist and Bayesian mixed-effects models. Data, code, and an interactive web application are available as a resource for researchers, clinicians, and students.

Conclusions: Pursuing reproducible research is key to promoting transparency in small-N treatment research. Researchers and clinicians must understand the properties of common effect size measures to make informed decisions in order to select ideal effect size measures and act as informed consumers of small-N studies. Together, a commitment to reproducibility and a keen understanding of effect sizes can improve the scientific rigor and synthesis of the evidence supporting clinical services in aphasiology and in communication sciences and disorders more broadly.

Supplemental Material S1. Code needed to reproduce the analysis in Section 1 of the article.

Supplemental Material S2. Code batch calculate effect sizes.

Cavanaugh, R., Quique, Y. M., Swiderski, A. M., Kallhoff, L., Terhorst, L., Wambaugh, J., Hula, W. D., & Evans, W. S. (2022). Reproducibility in small-N treatment research: A tutorial using examples from aphasiology. Journal of Speech, Language, and Hearing Research, 66(6), 1908–1927.

Publisher Note: This article is part of the Forum: Promoting Reproducibility for the Speech, Language, and Hearing Sciences.


Research reported in this article was supported by the National Institute on Deafness and Other Communication Disorders under Award F31DC019853-01 awarded to Robert Cavanaugh. The contents of this article were also developed under a grant from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) awarded to Yina M. Quique.