For speech-language pathologists

How much do SLPs agree when scoring speech sounds?

Published · Updated

The short answer

Agreement is good for clear productions and much weaker for distortions and in-between sounds. In conversational samples from children with speech delay, transcribers agreed on about 90% of consonants in broad transcription but about 74% in narrow transcription. For children’s /r/, trained listeners split on acoustically intermediate tokens. That matters for progress data: a small change in percent correct can fall within rater error, so check your own reliability before relying on it.

Inter-rater and intra-rater agreement: the numbers

Inter-rater reliability is how closely two listeners agree on the same productions; intra-rater reliability is how closely one listener agrees with themselves on a second scoring. Both depend heavily on what’s being judged.

Percent consonants correct: how reliable is it?

PCC was introduced as a severity metric computed from a continuous speech sample (Shriberg & Kwiatkowski, 1982). The 1997 extension paper reported reliability and standard error of measurement (SEM) estimates for PCC and related metrics from 5- to 10-minute conversational samples (Shriberg et al., 1997). Three findings matter clinically:

How to take and score a sample is covered in conversational speech samples for articulation.

What affects agreement

How to check your own reliability

You don’t need a research protocol to get a working estimate.

A worked example. Two raters score 20 words. Both mark 12 correct and 4 incorrect, and they disagree on 4, each marking 14 correct in total. Agreement is 16 ÷ 20 = 80%. Chance agreement is (0.7 × 0.7) + (0.3 × 0.3) = 0.58, so kappa is (0.80 − 0.58) ÷ (1 − 0.58) ≈ .52, below McHugh’s .60 threshold even though raw agreement hits 80%. The more often both raters say “correct”, the higher chance agreement climbs.

Why it matters for progress data and automated scoring

Rater error limits what change you can detect. The 1997 authors stress that SEM is central to interpreting effect sizes (Shriberg et al., 1997). A rise of a few points on a 20-item probe scored by one listener may sit inside your own rescoring disagreement, so keep the same rater, list and conditions where you can, and look for change larger than that disagreement. See progress monitoring in speech sound therapy.

The same logic applies to automated scoring. A tool can only be judged against human raters who themselves disagree, so ask for comparisons with several clinicians and for accuracy on error productions specifically. A 2018 systematic review found tools were typically less than 80% accurate on words containing mispronunciations (McKechnie et al., 2018). In the /ɹ/ study above, agreement was .57 between the classifier and clinicians, against .66 between clinicians (Benway & Preston, 2023). More in automated speech scoring for SLPs.

Questions SLPs ask

What is good inter-rater reliability for speech sound scoring?

Many texts treat 80% point-by-point agreement as the minimum, and one widely cited guide suggests a kappa below .60 shows inadequate agreement. Expect lower agreement for distortions and intermediate productions such as distorted /r/.

What is the difference between inter-rater and intra-rater reliability?

Inter-rater reliability compares two listeners scoring the same productions. Intra-rater reliability compares one listener with themselves on a second scoring, and in transcription studies it is usually higher.

How do I calculate Cohen’s kappa?

Subtract chance agreement from observed agreement, then divide by one minus chance agreement. Chance agreement depends on how often each rater uses each score, so 80% raw agreement can still give a kappa near .5.

Is percent consonants correct reliable?

It depends on transcription. PCC’s standard error is generally higher than PCC-Revised’s because PCC counts distortions as errors, and distortions are the hardest productions to transcribe reliably.

Sources

Join the Early Adopter Program