Measuring change in picture naming ability (Walker et al., 2021)

Purpose: Meaningful changes in picture naming responses may be obscured when measuring accuracy instead of quality. A statistic that incorporates information about the severity and nature of impairments may be more sensitive to the effects of treatment.

Method: We analyzed data from repeated administrations of a naming test to 72 participants with stroke aphasia in a clinical trial for anomia therapy. Participants were divided into two groups for analysis to demonstrate replicability. We assessed reliability among response type scores from five raters. We then derived four summary statistics of naming ability and their changes over time for each participant: (a) the standard accuracy measure, (b) an accuracy measure adjusted for item difficulty, (c) an accuracy measure adjusted for item difficulty for specific response types, and (d) a distance measure adjusted for item difficulty for specific response types. While accuracy measures address the likelihood of a correct response, the distance measure reflects that different response types range in their similarity to the target. Model fit was assessed. The frequency of significant improvements and the average magnitude of improvements for each summary statistic were compared between treatment groups and a control group. Effect sizes for each model-based statistic were compared with the effect size for the standard accuracy measure.

Results: Interrater and intrarater reliability were near perfect, on average, though compromised somewhat by phonological-level errors. The effects of treatment were more evident, in terms of both frequency and magnitude, when using the distance measure versus the other accuracy statistics.

Conclusions: Consideration of item difficulty and response types revealed additional effects of treatment on naming scores beyond those observed for the standard accuracy measure. The results support theories that assume naming ability is decomposable into subabilities rather than being monolithic, suggesting new opportunities for measuring treatment outcomes.

Supplemental Material S1. Directed acyclic graphs of the variables, dependencies, and prior assumptions for the cross-sectional and longitudinal Bayesian MPT models.

Supplemental Material S2. Directed acyclic graphs of the variables, dependencies, and prior assumptions for the cross-sectional and longitudinal Bayesian IRT models.

Supplemental Material S3. Database identifiers, demographic information, and naming data for control participants from the MAPPD database.

Supplemental Material S4. Detailed methods and results for Bayesian model fitting, including justification of prior assumptions, and checks for convergence, sensitivity, and posterior prediction.

Supplemental Material S5. Distributions on the probability scale for default and alternative prior specifications for Bayesian model fitting.

Supplemental Material S6. PNT scoring reliability, including Cohen’s kappa, confusion matrices, and Jaccard indices for all response types.

Supplemental Material S7. Scatterplot illustrating the correlation between unidimensional item difficulty estimates obtained by Fergadiotis et al. (2015) and the current study.

Supplemental Material S8. AUC analysis of magnitude change differences between treatment and control groups, examining partial credit scores, rather than %C, alongside model-based statistics.

Supplemental Material S9. All naming data, model code, analysis code, and posterior samples generated from this study.

Supplemental Material S10. All naming data, model code, and analysis code generated from this study, without posterior samples.

Walker, G. M., Basilakos, A., Fridriksson, J., & Hickok, G. (2021). Beyond percent correct: Measuring change in individual picture naming ability. Journal of Speech, Language, and Hearing Research. Advance online publication. https://doi.org/10.1044/2021_JSLHR-20-00205