For speech-language pathologists
How much do SLPs agree when scoring speech sounds?
Published · Updated
The short answer
Agreement is good for clear productions and much weaker for distortions and in-between sounds. In conversational samples from children with speech delay, transcribers agreed on about 90% of consonants in broad transcription but about 74% in narrow transcription. For children’s /r/, trained listeners split on acoustically intermediate tokens. That matters for progress data: a small change in percent correct can fall within rater error, so check your own reliability before relying on it.
Inter-rater and intra-rater agreement: the numbers
Inter-rater reliability is how closely two listeners agree on the same productions; intra-rater reliability is how closely one listener agrees with themselves on a second scoring. Both depend heavily on what’s being judged.
- Broad versus narrow transcription. In the 1997 PCC reliability study, for a subset of 7 speech-delayed children, transcribers agreed on about 90% of consonants in broad transcription and about 74% in narrow transcription. The authors describe a nearly 20-percentage-point gap when most speakers have speech involvement (Shriberg et al., 1997).
- Within versus between raters. In a 2010 study of 10 speakers, with two transcribers who had 15 and 7 years of experience, narrow transcription agreement for consonants was 91.8% within a transcriber and 86.7% between transcribers; for vowels it was 85.9% and 77.6%. Narrow agreement averaged 6.1 points below broad (Shriberg et al., 2010).
- Children’s /r/. When 25 phonetically trained listeners rated 100 words from children with /r/ errors as correct or incorrect, 10 acoustically intermediate items drew less than 60% agreement, and samples of up to 15 trained listeners still didn’t reach 80% agreement on them (McAllister Byun et al., 2015).
- Clinicians in therapy. Across 3,776 rated /ɹ/ attempts from 5 participants aged 10 to 19, mean pairwise agreement between clinicians (F1) was .66 (Benway & Preston, 2023).
Percent consonants correct: how reliable is it?
PCC was introduced as a severity metric computed from a continuous speech sample (Shriberg & Kwiatkowski, 1982). The 1997 extension paper reported reliability and standard error of measurement (SEM) estimates for PCC and related metrics from 5- to 10-minute conversational samples (Shriberg et al., 1997). Three findings matter clinically:
- Distortions drive disagreement. SEMs were generally highest for PCC, which counts all distortions as errors, and lowest for PCC-Revised, which scores all distortions as correct (Shriberg et al., 1997). For a child whose errors are mainly distortions, such as a lateral /s/ or a derhotacized /r/, expect more rater noise.
- Point-by-point agreement isn’t score reliability. In the most speech-delayed subset, two transcribers’ narrow agreement on consonants and vowels ranged from 66.0% to 84.2%, yet their speech-status classifications agreed almost perfectly. The authors conclude that agreement percentages alone can’t establish the reliability of a score or classification (Shriberg et al., 1997).
- Transcriber drift. Transcribers who work alone over long periods tend to drift, so the authors recommend consensus transcription where feasible and periodic recalibration, especially in clinical settings where transcription skills may rest on one undergraduate phonetics class (Shriberg et al., 1997).
How to take and score a sample is covered in conversational speech samples for articulation.
What affects agreement
- The production. Agreement is highest for clearly correct productions. Inexperienced listeners agreed more on correct /r/ than on distorted or non-rhotic /r/ (Klein et al., 2012), and 24 graduate student clinicians agreed with expert SLPs most on accurate productions and least on inaccurate ones (Jung et al., 2022).
- The sound and context. All listeners were more consistent rating vocalic than consonantal /r/ (Klein et al., 2012), and vowels were transcribed less reliably than consonants (Shriberg et al., 2010).
- The speaker. Narrow transcription agreement was 90.7% for typically developing speakers but 80.3% for speakers with motor speech disorders (Shriberg et al., 2010). Ten practicing SLPs’ judgments of phonetic accuracy were significantly less reliable for 5-year-olds with dysarthria than for typically developing children (Allison et al., 2021).
- Listener experience. Compared with 21 untrained listeners, 21 experienced SLPs had higher intra-rater reliability, less bias toward the more frequent sound, and ratings more closely tied to the acoustics (Munson et al., 2012). Experienced listeners also separated intermediate from fully incorrect /r/ better (Klein et al., 2012).
- Training and clear categories. Graduate students reached substantial agreement with expert ratings on a 3-point scale when given detailed descriptions of each rating category (Jung et al., 2022).
- More listeners. Pooling the binary ratings of 9 untrained online listeners matched the performance of 3 trained listeners, reaching 80% agreement with the trained-listener consensus on 86% of the items analyzed (McAllister Byun et al., 2015).
How to check your own reliability
You don’t need a research protocol to get a working estimate.
- Record a probe. With consent, audio-record a probe list or a short sample. See recording speech therapy sessions.
- Rescore it blind. A week or two later, rescore the recording without your original sheet, and ideally without knowing which session it came from. Blinded ratings are considered essential for valid assessment of interventions (McAllister Byun et al., 2015).
- Swap with a colleague to get inter-rater agreement. Single-case research standards ask for agreement checks on at least 20% of data points in each phase (What Works Clearinghouse criteria, summarized by Byiers et al., 2012).
- Calculate point-by-point agreement: items both raters scored the same, divided by all items, times 100. Many texts treat 80% as the minimum acceptable (McHugh, 2012).
- Add Cohen’s kappa, which corrects for agreement expected by chance: (observed agreement − chance agreement) ÷ (1 − chance agreement). McHugh suggests any kappa below .60 indicates inadequate agreement, and that kappa matters most when raters are likely to be guessing (McHugh, 2012).
A worked example. Two raters score 20 words. Both mark 12 correct and 4 incorrect, and they disagree on 4, each marking 14 correct in total. Agreement is 16 ÷ 20 = 80%. Chance agreement is (0.7 × 0.7) + (0.3 × 0.3) = 0.58, so kappa is (0.80 − 0.58) ÷ (1 − 0.58) ≈ .52, below McHugh’s .60 threshold even though raw agreement hits 80%. The more often both raters say “correct”, the higher chance agreement climbs.
Why it matters for progress data and automated scoring
Rater error limits what change you can detect. The 1997 authors stress that SEM is central to interpreting effect sizes (Shriberg et al., 1997). A rise of a few points on a 20-item probe scored by one listener may sit inside your own rescoring disagreement, so keep the same rater, list and conditions where you can, and look for change larger than that disagreement. See progress monitoring in speech sound therapy.
The same logic applies to automated scoring. A tool can only be judged against human raters who themselves disagree, so ask for comparisons with several clinicians and for accuracy on error productions specifically. A 2018 systematic review found tools were typically less than 80% accurate on words containing mispronunciations (McKechnie et al., 2018). In the /ɹ/ study above, agreement was .57 between the classifier and clinicians, against .66 between clinicians (Benway & Preston, 2023). More in automated speech scoring for SLPs.
Questions SLPs ask
What is good inter-rater reliability for speech sound scoring?
Many texts treat 80% point-by-point agreement as the minimum, and one widely cited guide suggests a kappa below .60 shows inadequate agreement. Expect lower agreement for distortions and intermediate productions such as distorted /r/.
What is the difference between inter-rater and intra-rater reliability?
Inter-rater reliability compares two listeners scoring the same productions. Intra-rater reliability compares one listener with themselves on a second scoring, and in transcription studies it is usually higher.
How do I calculate Cohen’s kappa?
Subtract chance agreement from observed agreement, then divide by one minus chance agreement. Chance agreement depends on how often each rater uses each score, so 80% raw agreement can still give a kappa near .5.
Is percent consonants correct reliable?
It depends on transcription. PCC’s standard error is generally higher than PCC-Revised’s because PCC counts distortions as errors, and distortions are the hardest productions to transcribe reliably.
Sources
- Shriberg, L. D., Austin, D., Lewis, B. A., McSweeny, J. L., & Wilson, D. L. (1997). The percentage of consonants correct (PCC) metric: Extensions and reliability data. Journal of Speech, Language, and Hearing Research, 40(4), 708–722.
- Shriberg, L. D., Fourakis, M., Hall, S. D., Karlsson, H. B., Lohmeier, H. L., McSweeny, J. L., Potter, N. L., Scheer-Cohen, A. R., Strand, E. A., Tilkens, C. M., & Wilson, D. L. (2010). Perceptual and acoustic reliability estimates for the Speech Disorders Classification System (SDCS). Clinical Linguistics & Phonetics, 24(10), 825–846.
- McAllister Byun, T., Halpin, P. F., & Szeredi, D. (2015). Online crowdsourcing for efficient rating of speech: A validation study. Journal of Communication Disorders, 53, 70–83.
- Benway, N. R., & Preston, J. L. (2023). Prospective validation of motor-based intervention with automated mispronunciation detection of rhotics in residual speech sound disorders. Proceedings of Interspeech 2023, 4558–4562.
- Shriberg, L. D., & Kwiatkowski, J. (1982). Phonological disorders III: A procedure for assessing severity of involvement. Journal of Speech and Hearing Disorders, 47(3), 256–270.
- Klein, H. B., Grigos, M. I., McAllister Byun, T., & Davidson, L. (2012). The relationship between inexperienced listeners’ perceptions and acoustic correlates of children’s /r/ productions. Clinical Linguistics & Phonetics, 26(7), 628–645.
- Jung, S., Jing, L., & Grigos, M. (2022). Graduate student clinicians’ perceptions of child speech sound errors. Perspectives of the ASHA Special Interest Groups, 7(4), 1275–1283.
- Allison, K. M., Russell, M., & Hustad, K. C. (2021). Reliability of perceptual judgments of phonetic accuracy and hypernasality among speech-language pathologists for children with dysarthria. American Journal of Speech-Language Pathology, 30(3S), 1558–1571.
- Munson, B., Johnson, J. M., & Edwards, J. (2012). The role of experience in the perception of phonetic detail in children’s speech: A comparison between speech-language pathologists and clinically untrained listeners. American Journal of Speech-Language Pathology, 21(2), 124–139.
- Byiers, B. J., Reichle, J., & Symons, F. J. (2012). Single-subject experimental design for evidence-based practice. American Journal of Speech-Language Pathology, 21(4), 397–414.
- McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282.
- McKechnie, J., et al. (2018). Automated speech analysis tools for children’s speech production: A systematic literature review. International Journal of Speech-Language Pathology.