Reliability of expert CAS diagnosis (Murray et al., 2023)

Purpose: The current standard for clinical diagnosis of childhood apraxia of speech (CAS) is expert clinician judgment. The psychometric properties of this standard are not well understood; however, they are important for improving clinical diagnosis. The purpose of this study is to determine the extent to which experts agree on the clinical diagnosis of CAS using two cohorts of children with mixed speech sound disorders (SSDs).

Method: Speech samples of children with SSDs were obtained from previous and ongoing research from video recordings of children aged 3–8 years (n = 36) and audio recordings of children aged 8–17 years (n = 56). A total of 23 expert, English-speaking clinicians were recruited internationally. Three of these experts rated each speech sample to provide a description of the observed features and a diagnosis. Intrarater reliability was acceptable at 85% agreement.

Results: Interrater reliability on the presence or absence of CAS among experts was poor both as a categorical diagnosis (κ = .187, 95% confidence interval [CI] [0.089, 0.286]) and on a continuous “likelihood of CAS” scale (0–100; intraclass correlation = .183, 95% CI [.037, .347]). Reliability was similar across the video-recorded and audio-only samples. There was greater agreement on other diagnoses (such as articulation disorder) than on the diagnosis of CAS, although these too did not meet the predetermined standard. Likelihood of CAS was greater in children who presented with more American Speech-Language-Hearing Association CAS consensus features.

Conclusions: Different expert raters had different thresholds for applying the diagnosis of CAS. If expert clinician judgment is to be used for diagnosis of CAS or other SSDs, further standardization and calibration is needed to increase interrater reliability. Diagnosis may require operationalized checklists or reliable measures that operate along a diagnostic continuum.

Supplemental Material S1. Qualtrics Rating Form.

Supplemental Material S2. Rating instructions.

Supplemental Material S3. Rating definitions.

Supplemental Material S4. Correlations between features and the likelihood of CAS.

Supplemental Material S5. Data sheets and read me file.

Murray, E., Velleman, S., Preston, J. L., Heard, R., Shibu, A., & McCabe, P. (2023). The reliability of expert diagnosis of childhood apraxia of speech. Journal of Speech, Language, and Hearing Research. Advance online publication. https://doi.org/10.1044/2023_JSLHR-22-00677

Publisher Note: This article is part of the Special Issue: Selected Papers From the 2022 Apraxia Kids Research Symposium.