WithYou: An Interactive Shadowing Coach with Speech Recognition

Speech shadowing is a proven way of practicing speaking skills when learning foreign languages. Though effective, however, it is very difficult doing because the sound playback is irrelevant to learner’s speech. When a learner fails to catch up, he/she will have to reset the playback or jumps directed to where is being played, resulting in a waste of time and learner’s resilience in doing it. In this research, we propose a shadowing support system called WithYou. By comparing on-the-fly speech recognition results and the words currently being played, WithYou can detect shadowing failure while a student is practicing, and then rewind the playback position to the corresponding time point when the failure occurred. Moreover, based on the recognition tailored for shadowing practice, WithYou is capable of recognizing pauses in learner’s speech during trial-and-error. In addition, it supports repetitive practice with certain extent of error tolerance so that the feedback it provides is neither too frequent nor too late.