High-tech test for spoken English

By Robert Brumfield, Assistant Editor, eSchool News

March 22nd, 2005

Ordinate Corp., a subsidiary of Harcourt Assessment, is offering what it says is the world’s only technology-enabled, spoken-language proficiency test. Ordinate says its test uses speech recognition technology to assess the listening and speaking skills of non-native English speakers.

The company’s Spoken English Test (SET) is designed to analyze speech components from native and non-native speakers of English. The system locates and evaluates relevant segments, syllables, and phrases in speech. It then uses statistical modeling techniques to score the spoken performance.

The test is available in five- and 10-minute English language versions. It also is available in a junior English language version for children. It is delivered over the phone and scored by computer. The test-taker reportedly can view his or her score online within minutes of having completed the examination.

Independent studies have shown that SET exams are more objective and reliable than the current tests rated by humans, including one-on-one oral proficiency interviews, Ordinate president and test developer Jared Bernstein said. “If I have you evaluate a non-native speaker of English, you’ll give that person a score, say a 4.1. A second human rater does an evaluation and gives that person a 5.4, a third gives that person a seven, et cetera,” Bernstein said.

“As you collect … information from more and more independent raters rating someone’s English, a median emerges. If I take the human rater median scores, the machine [score] has been shown to be a better estimate of the central tendency of the human rater.”

Though the technology also has implications for use in foreign-language courses, such as French or Spanish, Bernstein cautioned that Ordinate has no immediate plans to offer such assessments to students. “It’s under study,” he said. “But that’s as far as it goes at this point.” Using criteria developed by linguists, the SET reportedly assesses English proficiency, fluency, and pronunciation. An overall score ranging from 20 to 80 is gleaned from the combined average of each of the testing categories. These categories are sentence mastery, vocabulary, fluency, and pronunciation.

Part A of the demonstration test available on the company’s web site asks the examinee to read aloud from a list of 12 sentences. If the testing voice orders the examinee to read sentence number 12, then the examinee would read aloud, “I just wish she could get along better with their cat.” Part B asks the examinee to repeat unrelated sentences in the time given. “His car is being repaired” and “War broke out” were among them. Part C is a short-answer format. “How many sides does a triangle have?” Three. “What season comes before spring?” Winter. “Does a tree usually have fewer trunks or branches?” Trunks. “Kim’s jewelry business is very lucrative. Does it make money or lose money?” It made money.

Part D, sentence building, builds upon comprehension with syntax. The examinee is asked to rearrange a series of subjects, verbs, and objects into a logical sentence. For instance, “we didn’t / the movie / enjoy” becomes “we didn’t enjoy the movie”; “to wait here / we didn’t want / very long” becomes “we didn’t want to wait here very long”; and so on.

In Part E, the final scored segment, the examinee is given 20 seconds to answer each of three open-ended questions. “If your family received a large amount of money, how should this money be used? Please explain.”

A final optional segment collects demographic data.

While this might not sound like an exhaustive proficiency test, the SET appears to get the job done quickly and well. The company’s growing clientele includes universities and community colleges from around the world. The testing service also is used by the U.S. government and several major corporations, including Nike and Samsung. SETs are available in Spanish and Dutch (used by the government to test candidates for citizenship), and Japanese, French, and German versions of the test are currently in development. Kathleen Bowen, director of educator preparation services for Region 4 Education Service Center in Houston, Texas, said her organization has had great success using the SET in determining the English language proficiency of potential K-12 teachers and teaching assistants for 54 of the state’s school districts.

In 2003, Bowen explained, the Texas state government made English language proficiency mandatory for teachers and teaching assistants working in the state. “We were going to develop our own instrument, and then I found this,” she said. Since then, the SET has been “one of the preferred methods [for testing English language proficiency]. It’s convenient, quick, [and] it can be done at home,” Bowen said.

Other ways to prove English language proficiency in Texas are to take and pass the Test of English as a Foreign Language (TOEFL) or the Test of Spoken English (TSE) exams, or earn a post-secondary degree in the United States or at an English-speaking institution abroad. All of these measures take somewhere between several hours and several years longer than the SET to complete. Bowen said she was skeptical about the effectiveness of the SET at first. But her institution piloted the SET with subjects whose English proficiency levels and native languages varied widely, and “it was amazing,” she said. “I’m so impressed with what we saw.”

Bowen said the SET scores were consistent with proficiency as determined by other relevant standards. She added, “We definitely see a relationship between [examinees’] performance on the test and their performance in our program.” Bowen said that her potential teachers do not have to wait to take the test like they might with the TOEFL or the TSE. She said candidates to take those tests “can get backed up for months. It delays their ability to take part in the training program. It delays their ability to start teaching.”

She also noted that feedback from the SET is available almost immediately online. “If they take one of the other tests, their feedback could take several weeks,” she said.

Ordinate’s Bernstein claims the SET is fairer than other similar exams. “A test can’t be fair unless it’s reliable and accurate,” he said, adding that the SET is more reliable and accurate because it is not subject to the caprices and prejudices of human judgment.

Bowen agreed. “There’s a lot of subjectivity involved in grading these kinds of tests,” she said. “With [the SET], there is none.”