ASHA journals
Browse
DOCUMENT
S1_JSLHR-22-00677murray.pdf (945.1 kB)
DOCUMENT
S2_JSLHR-22-00677murray.pdf (137.77 kB)
DOCUMENT
S3_JSLHR-22-00677murray.pdf (183.5 kB)
DOCUMENT
S4_JSLHR-22-00677murray.pdf (378.28 kB)
ARCHIVE
S5_JSLHR-22-00677murray.zip (393.76 kB)
1/0
5 files

Reliability of expert CAS diagnosis (Murray et al., 2024)

Version 2 2024-10-04, 19:13
Version 1 2023-08-29, 13:25
online resource
posted on 2024-10-04, 19:13 authored by elizabeth murray, Shelley Velleman, Jonathan L. Preston, Rob Heard, Akhila Shibu, Patricia McCabe

Purpose: The current standard for clinical diagnosis of childhood apraxia of speech (CAS) is expert clinician judgment. The psychometric properties of this standard are not well understood; however, they are important for improving clinical diagnosis. The purpose of this study is to determine the extent to which experts agree on the clinical diagnosis of CAS using two cohorts of children with mixed speech sound disorders (SSDs).

Method: Speech samples of children with SSDs were obtained from previous and ongoing research from video recordings of children aged 3–8 years (n = 36) and audio recordings of children aged 8–17 years (n = 56). A total of 23 expert, English-speaking clinicians were recruited internationally. Three of these experts rated each speech sample to provide a description of the observed features and a diagnosis. Intrarater reliability was acceptable at 85% agreement.

Results: Interrater reliability on the presence or absence of CAS among experts was poor both as a categorical diagnosis (κ = .187, 95% confidence interval [CI] [0.089, 0.286]) and on a continuous “likelihood of CAS” scale (0–100; intraclass correlation = .183, 95% CI [.037, .347]). Reliability was similar across the video-recorded and audio-only samples. There was greater agreement on other diagnoses (such as articulation disorder) than on the diagnosis of CAS, although these too did not meet the predetermined standard. Likelihood of CAS was greater in children who presented with more American Speech-Language-Hearing Association CAS consensus features.

Conclusions: Different expert raters had different thresholds for applying the diagnosis of CAS. If expert clinician judgment is to be used for diagnosis of CAS or other SSDs, further standardization and calibration is needed to increase interrater reliability. Diagnosis may require operationalized checklists or reliable measures that operate along a diagnostic continuum.

Supplemental Material S1. Qualtrics Rating Form.

Supplemental Material S2. Rating instructions.

Supplemental Material S3. Rating definitions.

Supplemental Material S4. Correlations between features and the likelihood of CAS.

Supplemental Material S5. Data sheets and read me file.

Murray, E., Velleman, S., Preston, J. L., Heard, R., Shibu, A., & McCabe, P. (2024). The reliability of expert diagnosis of childhood apraxia of speech. Journal of Speech, Language, and Hearing Research, 67(9S), 3309–3326. https://doi.org/10.1044/2023_JSLHR-22-00677

Publisher Note: This article is part of the Special Issue: Selected Papers From the 2022 Apraxia Kids Research Symposium.

Funding

This study was funded by a grant from Apraxia Kids (formerly CASANA) to Elizabeth Murray (principal investigator [PI]). Primary data collection was also supported by grants from the National Institutes of Health (R03DC012152 and R15DC016426) to Jonathan L. Preston (PI). The authors thank Apraxia Kids (formerly CASANA) for funding for this study.

History