April 8, 2015

Third-grade teacher Annie Long grappled with a new problem when her students reviewed the online practice test to prepare for the Common Core-aligned assessments they will be taking in a few weeks. The instructions were too complicated for many of them to understand.

The practice test instructions included words such as “scan” and “sources” – words that were unfamiliar to the majority of her class at Bayshore Elementary School in Daly City.

But they were not the only students struggling with these elements of the practice test. High school students in San Jose interviewed by EdSource Today also had difficulty understanding the instructions and how to navigate different sections of the test, as did 3rd-graders in Fresno.

The online practice test includes about 30 sample exercises of varying difficulty for each grade level in math and English language arts, according to the website of the Smarter Balanced Assessment Consortium, which developed it and the end-of-the year assessments. Over 3 million students in grades 3 to 8 and 11 will take the required assessments this spring to measure their performance in these two subject areas.

Educators like Long worry that overly complicated instructions could interfere with students’ ability to get through the full battery of tests they will be taking over the next several weeks unless they have previously been given substantial help to understand them.

Students won’t receive a score on the practice test. But it gives them a preview of the types of test items they will see on the required Smarter Balanced assessments, which will be completely online for the first time.

Long asked her students to look at the instructions on the English Language Arts exercises and read them aloud.

“After you’ve reviewed the sources briefly scan the sources and questions that follow, then go back and review,” the instructions read.

She then tried to explain the instructions to the students. “‘Scan’ means to read something quickly to get the main idea,” she said, as she wrote “scan” on the board. “You’re just reading it quickly just to go, ‘What’s this about? Oh, it’s about dogs,’” she said as an example.

The practice testasked students to imagine that they’re a student researching the job of an astronaut. “You think an astronaut is an interesting job. You have found two sources about being an astronaut,” the instructions explained.

“What does the word ‘sources’ mean?” Long asked. The room of 17 students fell silent, with many eyes peering intently at their computer screens.

“The student wants to be a what?” Long asked. “Astronaut!” came a chorus of responses. “So the student went out and found two sources. What do we think the word ‘source’ means,” Long asked again. “Facts?” asked one student.

“Sources are reading materials,” Long said. “What are some sources where we can learn about topics?” she asked a boy. “Books?” he asked. “What are some other sources?” Long asked. “Dogs,” said one student, as another suggested “Animals.”

“Dogs and animals are topics,” Long replied. She also asked the students to practice typing in the “notes” section of the online practice test, a section where students can take notes on the texts they read to remember the main points.

Her step-by-step review still eluded some students.

“I thought it was a little challenging to write the notes because it’s new,” 3rd-grader Catherine Posadas told a reporter at the end of class, referring to typing in the online “notes” section of the test. “I wasn’t sure what to do.”

At the end, Long asked students to give a thumbs up, sideways or down to show how they felt about the instructions. “I saw a lot of thumbs sideways, which means (students had a) ‘so-so understanding,’” she said afterwards.

“They’re not going to get it in one session,” said Long, who added that she had to go slowly through a lot of new vocabulary just so her students would understand the instructions.

Bayshore Elementary School District Interim Superintendent Audra Pittman said that it’s challenging enough for 3rd- graders to be able to put their thoughts into words to communicate verbally. The added chore of typing “is throwing another wrench in their thought process,”Pittman said, referring to the challenges they face taking the practice test.

In spring 2014, about 3 million California students took the Smarter Balanced Assessment Consortium’s online “field test.” The scores and results were not released to schools or the public. But the consortium used data from the field tests to make improvements, including tweaking the instructions, according to communications director Kelli Gauthier. She said she couldn’t describe the specific changes the consortium had made in the instructions, because a report about them has not been completed.

Laura Bolton, a teacher at William Saroyan Elementary in Fresno, who spoke to EdSource earlier about her students’ keyboard challenges, recently gave the Smarter Balanced midterms to her 3rd-graders. She said her students struggled with the instructions and added that they were not appropriate for the age of her students.

“It’s using vocabulary that we just don’t use,” she said. The language, she said, should be much more “kid-friendly.”

Going Deeper

Vito Chiala, principal of Overfelt High School in San Jose, sought feedback from students to find out how the school could better prepare them for the required Smarter Balanced Assessments.

Three students who took the English and math practice test also found the instructions challenging. Ninth-grader Maria Vargas, for example, said she didn’t know how to make the online calculator work on the test.

Photo by Laurie Udesky/EdSource Today

Overfelt High School students (from left to right) Maria Vargas, Jazmin Martinez and Esmerelda Corona share their experiences of taking the Smarter Balanced practice test.

“I needed someone to explain the directions to me,” she said. “I had to keep raising my hand and asking ‘how do you do this?’”

Chiala gave students an hour to complete each subject area. But 11th-grader Esmerelda Corona said she could not decipher from the test instructions how many sections she had to complete, and she didn’t realize there were more sections until after finishing the first one.

“I went ‘Oh, I’m done,’” Corona said. “That was after 20 minutes, and I probably had two sections to go.”

Long, the Bayshore Elementary teacher, said her students spent a previous class learning how to log in to the practice test. So when they sat down to take it again, they were able to log in quickly and didn’t have problems. Her district is planning more training for teachers to help students take the practice test and she’s since developed more lessons to mirror the format and content of both the math and English language arts sections of the practice test.

“They were really having to remember many different things (from the instructions) in order to do just one task,” she said. To make sure they are fully prepared for the Common Core assessments, “I’m going to have to break it down and show them, so they’re not going to get overwhelmed.”

john3 years ago3 years ago

Katt4 years ago4 years ago

So, it takes about a month of teaching how to take this test for it to be taken. Wouldn’t that time be better spent actually teaching things like math, art, history, writing etc…?

cheryl4 years ago4 years ago

This process is too much for 3rd graders. They can be taught the process, but it should be scaffolded over a little longer time. The process and the content are so different from each other and brand new. Take time to build this process so by the time they are in 5th grade they might be able to fully do this process and content. I won’t even approach the typing skill thought process..

Gary Ravani4 years ago4 years ago

It should not be too surprising that when a new set of standards, emphasizing more complex thinking, is implemented that the assessments designed to align with tese standards are going to get more sophisticated in the questioning strategies.
That said, Doug's inquiries into the "appropriateness" of the language are certainly in order. As well, and even more certainly, detailed feedback from teachers should be collected on how the assessments are working, or not, with their students. … Read More

It should not be too surprising that when a new set of standards, emphasizing more complex thinking, is implemented that the assessments designed to align with tese standards are going to get more sophisticated in the questioning strategies.

That said, Doug’s inquiries into the “appropriateness” of the language are certainly in order. As well, and even more certainly, detailed feedback from teachers should be collected on how the assessments are working, or not, with their students. And that feedback should immediately be integrated by SBAC (and CDE and the SBE) into the assessment protocols and the assessments themselves.

CA, and the rest of the nation, have undertaken a huge challenge. Implementing standards that are asserted to be more in line with what students really need, as the old standards seemed too skill and recall based, as well as implementing a new tech based assessment system that includes performance based assessment components.

The lesson here is it will all take some time to roll out in any meaningful fashion and be adjusted to fit the needs of real classrooms. This is in contrast to the prior pseudo-accountability system where classrooms, teachers and students had to adjust to the standards and tests under draconian threats of being labeled as failures, closed, retained-in-grade, fired, or be subject to a hostile-takeover by private sector charter management companies that are underwritten by the super-wealthy trying at all costs to protect their super-wealth.

The question, for CA, will be if the policy makers at all levels pay attention to the needs of kids and teachers in real classrooms and provide necessary resources, necessary time, and necessary “room” in new accountability systems to allow all of the multiple parts of the new systems to spool up to speed and then coordinate in some kind of system that advances true learning and makes sense. For a change!

Michele Trompeter4 years ago4 years ago

Our state creates their own tests. One of my third grade students raised his hand and asked me what a word was that was in the directions, and I had never heard of it. Neither had anyone else I talked to except the person who had a son who is a junior in high school. It just happened to be in his assignment the night before. So a large percentage … Read More

Our state creates their own tests. One of my third grade students raised his hand and asked me what a word was that was in the directions, and I had never heard of it. Neither had anyone else I talked to except the person who had a son who is a junior in high school. It just happened to be in his assignment the night before. So a large percentage of my evaluation is to be based on a test with directions my 3rd graders cannot understand because the test uses vocabulary most adults do not know. This has got to stop.

Gary Ravani4 years ago4 years ago

Doug McRae4 years ago4 years ago

Gary — It would be a violation of CAASPP regulations approved by the State Board for a test administrator to reveal vocabulary from an operational Smarter Balanced test.

Gary Ravani4 years ago4 years ago

Doug:
When Michelle said "our state" I assumed she was not speaking CA, where we do not "create our own tests." Anyway, I never interpreted the "confidentiality agreements" teachers are required to sign to cover "one word" or test directions for that matter, just "content." Read More

Doug:

When Michelle said “our state” I assumed she was not speaking CA, where we do not “create our own tests.” Anyway, I never interpreted the “confidentiality agreements” teachers are required to sign to cover “one word” or test directions for that matter, just “content.”

Doug McRae4 years ago4 years ago

OK, Gary, fair enough. I interpreted the comment to apply to California's new Smarter Balanced tests since those were the focus of the post upon which the comment was made. IF the comment did apply to CA's new Smarter Balanced tests, then revealing the vocabulary in question would be a violation of CAASPP test security regulations. The regulations [Sec 859 (d) (1), part of the Test Security Affidavit to be signed by all … Read More

OK, Gary, fair enough. I interpreted the comment to apply to California’s new Smarter Balanced tests since those were the focus of the post upon which the comment was made. IF the comment did apply to CA’s new Smarter Balanced tests, then revealing the vocabulary in question would be a violation of CAASPP test security regulations. The regulations [Sec 859 (d) (1), part of the Test Security Affidavit to be signed by all CAASPP test administrators] state “I will not divulge the contents of the tests and test materials to any other person through verbal, written, or any other means of communication.” The “contents of the tests” are not defined within the regulations, but the “test materials” are defined to include test administration manuals, which contain specific directions for the tests. Also, on CDE’s website, one cannot access the operational test administration manuals unless one has a username and password, which are restricted to individuals who have signed a Test Security Affidavit. [Note: These regulations apply to operational Smarter Balanced tests, not practice tests which are not secure material. Thus, EdSource did not violate CAASPP regulations when the post included specific questionable vocabulary from the Smarter Balanced practice tests.] Getting away from the green eyeshades compliance orientation associated with regulations, I’d mention that specific directions for large scale K-12 tests have been considered secure material for the many years I’ve been in the K-12 test development enterprise. The situation is similar to any teacher-made test . . . . if a teacher-made test had problems such as inappropriate vocabulary in the directions for the test, and the teacher wanted to use the scores from that test for grading purposes, that teacher would get pushback from students and parents . . . . justifiably so. Been there, done that, from both the test maker and the student and the parent point of view . . . .

navigio4 years ago4 years ago

So I guess the only pressure that can legally attempt to straighten that crooked timber of humanity is the students.

Gary Ravani4 years ago4 years ago

Doug:

This all sounds like a topic ripe for debate between the NSA and Edward Snowdon. It also shows the ridiculous levels that “security” has risen to in order to protect, not the tests so much, but the “proprietary rights” of the testing companies involved. In the end, the students are the victims with the teachers and learning following at a close, and tied, second.

Doug McRae4 years ago4 years ago

Gary -- The "proprietary rights" for STAR tests belong to the people of California since CA paid for those tests and CA owns the copyright, and the "proprietary rights" for Smarter Balanced tests are now public domain since federal dollars paid for those tests. The testing companies have no "proprietary rights" for any of the main components of CA statewide testing program. Test security requirements are recommended by CDE and approved by SBE . . … Read More

Gary — The “proprietary rights” for STAR tests belong to the people of California since CA paid for those tests and CA owns the copyright, and the “proprietary rights” for Smarter Balanced tests are now public domain since federal dollars paid for those tests. The testing companies have no “proprietary rights” for any of the main components of CA statewide testing program. Test security requirements are recommended by CDE and approved by SBE . . . . the testing companies have no ownership for test security regulations either. Testing companies are only contracted vendors to carry out what CA leadership wants carried out on tests owned by CA and the consortium of states CA joined.

Gary Ravani4 years ago4 years ago

Doug:

Really? When I’ve heard of teachers being accosted for alleged “security breaches” it has been reps from the test company that have been involved. I accept your explanation as technically accurate; however, from the trenches things don’t appear that clear cut.

Doug McRae4 years ago4 years ago

Checking for grade level appropriate vocabulary for directions for both practice tests and operational tests is a standard element for K-12 large scale test development programs. If it wasn't done for Smarter Balanced tests, that is an error by the test developers. Correcting this error for practice tests is relatively simple -- replace the inappropriate vocabulary and send an apology/alert to the user base with if needed a rebate or refund of some sort [just … Read More

Checking for grade level appropriate vocabulary for directions for both practice tests and operational tests is a standard element for K-12 large scale test development programs. If it wasn’t done for Smarter Balanced tests, that is an error by the test developers. Correcting this error for practice tests is relatively simple — replace the inappropriate vocabulary and send an apology/alert to the user base with if needed a rebate or refund of some sort [just like an automobile recall for repair process]. Correcting this error after-the-fact for an operational test is impossible; it leads to invalid uncorrectable test results with potential retesting as the only remedy.

Manuel4 years ago4 years ago

I am sorry, Doug, that those who appear to be in charge of the SBAC are giving a black-eye to your profession.
It is worth asking, had not the SBoE declared the 2015 administration yet another "test of the test" would this issue with language have been caught? I doubt.
Hence, this incident buttresses my belief that this whole testing thing is not ready for prime time because nobody is really minding the store.
It also … Read More

I am sorry, Doug, that those who appear to be in charge of the SBAC are giving a black-eye to your profession.

It is worth asking, had not the SBoE declared the 2015 administration yet another “test of the test” would this issue with language have been caught? I doubt.

Hence, this incident buttresses my belief that this whole testing thing is not ready for prime time because nobody is really minding the store.

It also highlights the need for an independent commission of some sort to review tests and any materials to ensure that amateurish (or maybe not) mistakes like this one are prevented.

Manuel4 years ago4 years ago

Sigh...
I hope that you all noticed what happened here: an adult with a particular level of formal education wrote questions meant for third-graders (and high-schoolers!) who have never ever heard, read or used the vocabulary this adult normally uses. How could the students even understand what is expected of them when they can't get the questions?
Given this, isn't it a surprise that there is an achievement gap between poor students and wealthy students? It is … Read More

Sigh…

I hope that you all noticed what happened here: an adult with a particular level of formal education wrote questions meant for third-graders (and high-schoolers!) who have never ever heard, read or used the vocabulary this adult normally uses. How could the students even understand what is expected of them when they can’t get the questions?

Given this, isn’t it a surprise that there is an achievement gap between poor students and wealthy students? It is well-known that there is a wide gulf in language use between poor and wealthy households. I’ve heard of study after study that looks into how vocabulary is developed in the absence of books and with a heavy dosage of television. Again, why are we surprised that poor students do less well in standardized tests than better-off children?

Sure, even if you are poor you can surround your children with books and turn off the tv. But how many can do it given that their socioeconomic status does not allow them to indulge in book buying or alternate activities for their children?

If a test is written from a different socioeconomic POV, how can we expect all children to do well? If the tests are written in the language used by only one segment of the population, how can we expect the rest to do well? This is a cruel joke we are playing on our children.

Double sigh…

Roxana Marachi4 years ago4 years ago

The SmarterBalanced Math test has been invalidated in a detailed 30+ page report. In the analysis, Steve Rasmussen writes that "question after question, the tests:
* Violate the standards they are supposed to assess;
* Cannot be adequately answered by students with the technology they are required to use;
* Use confusing and hard-to-use interfaces; or
* Are to be graded in such a way that incorrect answers are identified as correct and correct answers … Read More

The SmarterBalanced Math test has been invalidated in a detailed 30+ page report. In the analysis, Steve Rasmussen writes that “question after question, the tests:
* Violate the standards they are supposed to assess;
* Cannot be adequately answered by students with the technology they are required to use;
* Use confusing and hard-to-use interfaces; or
* Are to be graded in such a way that incorrect answers are identified as correct and correct answers as incorrect.”

“No tests that are so flawed should be given to anyone. Certainly, with stakes so high for students and their teachers, these Smarter Balanced tests should not be administered. The boycotts of these tests by parents and some school districts are justified. Responsible government bodies should withdraw the tests from use before they do damage.”… Full report at mathedconsulting.com.

For anyone interested in current updates, including optout policies according to each state, please visit: http://bit.ly/testing_testing. Fairtest.org also has excellent information. According to this document (http://sco.lt/8ImSB7) “State laws in California allow children to opt out of state assessments for any reason.”

CarolineSF4 years ago4 years ago

Don, are you opting your kid(s) out? It’s not really a thing in San Francisco yet, so I’m interested to see how that goes. Please share if there’s any drama (or not, for that matter) around your doing that.

(EdSource, are you still moderating my posts, and why? Nobody has said a thing to me.)

CarolineSF4 years ago4 years ago

NVM -- that post wasn't moderated. Not sure why others were. Anyway, Don, please tell us what happens when you opt out! Parents in other states (where opting out is a much bigger thing) have alarming tales. One Illinois parent posted a picture of her son's arm, on which she had written in marker, on the test days: "You may not speak to my son without my permission" and her signature -- because there are … Read More

NVM — that post wasn’t moderated. Not sure why others were. Anyway, Don, please tell us what happens when you opt out! Parents in other states (where opting out is a much bigger thing) have alarming tales. One Illinois parent posted a picture of her son’s arm, on which she had written in marker, on the test days: “You may not speak to my son without my permission” and her signature — because there are so many tales of kids being harassed after their parents opted them out of the various Common Core tests.

Don4 years ago4 years ago

Caroline, to answer your question, no drama at all. Opting out has been smooth. I gave the charter school an opt-out form which was signed by them and photocopied. I then received an email acknowledgement from both the principal and the teacher to verify that instead my son would have alternative activities. They have followed through and immediately stopped giving him any practice tests or test related homework. As far as I … Read More

Caroline, to answer your question, no drama at all. Opting out has been smooth. I gave the charter school an opt-out form which was signed by them and photocopied. I then received an email acknowledgement from both the principal and the teacher to verify that instead my son would have alternative activities. They have followed through and immediately stopped giving him any practice tests or test related homework. As far as I know we are the only ones to have opted out. Today is the first day of testing.

In general, my experience with parents here in SFUSD is that they are a very acquiescent lot, doing what they are told to do. Schools are not required by law to inform parents of their right to opt-out – CA EC 60615 which might explain in part this sheepish behavior. I also believe the mentality that would result in opting out has a political component. That said, I did not opt-out my older son who will take only the science test this year.

A little more on this “acquiescence” – last night I went to Presidio Middle School (a school my older son attended) for a discussion on the change in the math sequence. I was previously informed that many parents are upset about the fact that their gate students will lose honors math classes in 8th grade. In addition this school gets very low funding so it also has very large class sizes making differentiated teaching practically impossible which contrasts with lower performing school with class sizes of about half. True to form the district set up the meeting by strangling off any opportunity to speak . They spent the first 20 minutes having parents do math. When I complained that I didn’t come to do math but to discuss the math sequence I was told that the program was set. Then some high school students gave a presentation about the high school sequence. Parents were given index cards to ask questions. Of course there were more questions than time permitted so any unwanted questions went unanswered at the bottom of the stack. Parents went right along with this dog and pony show. This the par for the course in progressive San Francisco where grassroots are given lip-service followed with the cold shoulder by the supposedly sensitive-minded school leadership.

FloydThursby19414 years ago4 years ago

I was at the same meeting and helped organize it and was promised I could ask one question but had to write it on a card. The person asking took several challenging questions and merged them into one softened, vanilla question. When I complained after the event, the woman flipped me off and cursed at me. She was a PTSA member, but somehow they convinced her to go soft. She dishonestly … Read More

I was at the same meeting and helped organize it and was promised I could ask one question but had to write it on a card. The person asking took several challenging questions and merged them into one softened, vanilla question. When I complained after the event, the woman flipped me off and cursed at me. She was a PTSA member, but somehow they convinced her to go soft. She dishonestly relayed my question and made it have no teeth, let them off the hook. 2 Lowell Students and one parent challenged it well but were told not to be too political and weren’t, then a defender was very political. A former Principal and an 8th Grader tried to challenged the dog and pony show presenters and both were blown off and left. It was really designed to make it look like they seek community input while completely ignoring it, very similar to what happened with the neighborhood schools issue. They did things to take up most of the time without being challenged, limited the time, by doing a problem with cups that no one understood for ten minutes.

Don4 years ago4 years ago

Is it a flight of fancy to assume test developers can write decent instructions children can understand? Something maybe age- appropriate? What does this say about the rest of the test? After much consideration ranging between acquiescence and doubt I finally decided to let my son take the test if for no other reason than everyone else was doing it even though he was complaining about how confused he was. Then a few days … Read More

Is it a flight of fancy to assume test developers can write decent instructions children can understand? Something maybe age- appropriate? What does this say about the rest of the test? After much consideration ranging between acquiescence and doubt I finally decided to let my son take the test if for no other reason than everyone else was doing it even though he was complaining about how confused he was. Then a few days ago he came home with a test prep worksheet that changed my mind. It reminded me some tax software I threw in the garbage. There’s nothing I can do about Common Core but at least I can save him a few hours of his time so he can work on something else at school.

Thanks to Laurie Udesky for giving us the other side of the story. I was beginning to think Ed Source had a missionary zeal for the wholesome goodness of SBAC. Soon enough they’ll no end to the outrage.