For speech-language pathologists

Can automated speech scoring be trusted? What SLPs should know

Published 21 October 2026 · Updated 21 October 2026

The short answer

Automated scoring of children’s speech is improving, but it isn’t reliable enough to replace clinical judgment. A 2018 systematic review found tools were typically less accurate on exactly the words children mispronounced. The best-studied /ɹ/ classifier agreed with clinicians in therapy about as well as clinicians agreed with each other for most participants, but in small samples of older children. ASHA says efficacy claims for AI tools must be backed by evidence.

What the research shows

A systematic review of 32 studies and 18 tools found tools met an 80% agreement threshold with human judgment for intelligibility, severity or error category, but accuracy was typically under 80% for words containing mispronunciations (McKechnie et al., 2018).
A classifier for /ɹ/ using age-and-sex normalized formants reached a mean participant-specific F1 of .81 (median .83) on 48 held-out participants (Benway et al., 2023).
Used prospectively in therapy with 5 participants aged 10–19 (3,776 rated tokens), mean pairwise F1 was .66 between clinicians and .57 between the classifier and clinicians; classifier agreement was within the clinician range for 4 of 5 participants. Expert raters themselves agreed only moderately on ambiguous /ɹ/ (Benway & Preston, 2023; full study in Benway & Preston, 2024).

Lab accuracy dropped in real therapy, and the /ɹ/ evidence comes from one research group working with older children who could already produce the sound. Don’t read it as “clinician-level” in general.

Questions to ask about any tool

A 2025 tutorial gives clinicians a framework for appraising clinical AI on validity, reliability, and ethical and equitable use (Benway & Preston, 2025). In practice:

Who was it tested on: what ages, dialects and kinds of speech sound errors?
Against how many SLP raters, and how did their agreement compare with each other?
How accurate is it on error attempts specifically, not just overall?
What does it do with attempts it’s unsure about?
Can you adjust its strictness, override its judgments, or control its feedback schedule?
Does it record children? If so, where is audio kept, for how long, and with what consent?
What does it claim, and is there published evidence for those claims?

Your responsibility

ASHA’s guidance on generative AI says that “any claims about the efficacy of an AI tool must be substantiated with evidence,” ties clinicians’ duty to evaluate technology to the Code of Ethics, and is clear that AI doesn’t replace clinical judgment (ASHA). The same standard is a sensible bar for any automated scoring tool.

Disclosure

Saylo, which publishes this page, is building a device that uses automated scoring. It’s in development and hasn’t been validated with children. We plan to publish validation against several SLP raters before making accuracy claims.

Questions SLPs ask

Can automated scoring replace perceptual judgment?

Not on current evidence. Agreement with clinicians varies by child, and accuracy tends to be lowest on mispronounced words, which are the ones that matter most clinically.

Why is agreement hard to judge for /ɹ/?

Even expert clinicians agree only moderately on ambiguous /ɹ/ productions, so a tool should be compared against several raters, not one.

What should I ask a vendor?

Who it was tested on, against how many raters, how accurate it is on errors specifically, how it handles uncertainty, whether you can override it, and whether it keeps recordings.