Untrained listeners who are crowdsourced to assess voice treatment for people with Parkinson's can reach conclusions that agree with reliable measurement tools, finds a team of speech-language pathology researchers.

"Studies of speech and voice treatment methods for individuals with Parkinson's Disease typically evaluate progress using fine-grained acoustic measures or highly trained listeners' perceptual ratings," says lead author Tara McAllister, NYU Steinhardt Associate Professor. "However, this approach leaves some uncertainty about the real-world impact of these changes, since it is not always clear they would be apparent to untrained listeners that patients interact with in their daily life."

In a study published in the Journal of Speech, Language, and Hearing Research, McAllister and her co-authors successfully replicated a 2022 study (Moya-Galé et al) in the Journal of Voice that showed improved voice quality for people with Parkinson's after receiving the Lee Silverman Voice Treatment (LSVT LOUD). The results of the original study were determined using the Acoustic Voice Quality Index (AVQI), a metric that combines multiple acoustic measures of speech to arrive at a single value that correlates highly with expert listeners' ratings.

In crowdsourcing, researchers use platforms such as Amazon's Mechanical Turk to connect with a large number of individuals who perform tasks online for monetary compensation. The researchers in this study used an initial task to determine the reliability of crowdsourced listeners in assessing audio samples. Thirty eligible raters were then asked to evaluate the 250 voice recordings from the Moya-Galé study, which were divided into blocks of 50 recordings each (25–27 listeners for each block).

The authors found a statistically significant correlation between the listeners' ratings and the AVQI scores. They also replicated the original study's finding that significant gains in voice quality were observed after LSVT LOUD treatment but not after two comparison treatments.

"Overall, these results support the use of crowdsourcing to evaluate speech samples collected from clinical populations, even for constructs such as voice quality that may be less familiar to untrained listeners," they write.

"We think that crowdsourcing has exciting potential to make it easier for researchers to understand the real-world relevance of the treatment methods they study, and ultimately support better alignment of priorities between researchers and patients."