Saturday, March 17, 2012

NAEP Exclusion Rates and Kentucky Success

There's been some talk recently about the number of Kentucky students
excluded from NAEP participation based on disabilities. The chart [below], drawn from the NAEP "Nation's Report Card" documents for each
subject, shows those rates. To me, it offers two clear messages.

First, Kentucky has been doing something different from the rest of the
country with NAEP reading...

Second, NAEP science and mathematics have worked differently.
Kentucky's exclusion rate has been a perfect match to the country in
science, and it has differed by a single point in mathematics.

So, what if we look at reading, and leave out students with disabilities
for a moment? ... All the excluded students in the chart above above were in the
group with disabilities. Even if the disability exclusion rate made a difference, it wouldn't change the scores for the students without disabilities. Here's that comparison.

For both grades, Kentucky students scored above the national average by a statistically significant margin in reading.

That is, Kentucky's recent record of relative NAEP success does not, in
fact, evaporate when exclusion rates are considered...the notion of Kentucky as distinctively weak among the
states is one we need to lay to rest...

4 comments:

Skip Kifer
said...

The issue is testing accommodations. A student may have a plan that changes how he or she takes a test. The most common accommodation is allowing more time to take the test. Others include, for example, a human reader for blind students.

In reading, Kentucky provides accommodations that NAEP does not allow. That is why Kentucky's exclusion rate is higher than average.

Varied exclusion rates for different states produce problems in interpreting scores. If Kentucky's 5% difference from the national rate contains only low scores, the Kentucky mean goes down. If all of those scores fall in the bottom tail of the distribution, the downward adjustment would be substantial.

It is not clear, however, how one would determine where the scores might be. For any of a number of reasons, it is unlikely that each of those hypothetical 5% scores is extremely low.

Dr. Kifer’s somewhat confusing post is correct, though incomplete, about one point – Kentucky’s nation-leading exclusion rates for students with learning disabilities are definitely related to our state’s use of an unreasonable testing accommodation not allowed on NAEP reading assessments.

Such inflation has been present in Kentucky’s NAEP reading since 1998 and makes comparison of Kentucky’s scores to other states problematic.

As Kifer points out, no-one knows exactly how the excluded students would score if they took NAEP reading. However, there are some pretty interesting clues regarding this, information that Kifer and Prichard both overlook. Given the ready availability of the information, such oversight is surprising.

During the past six months the Kentucky Board of Education has engaged in extensive and very public discussions about NAEP exclusion and why it is so excessive in Kentucky’s reading samples (which were the highest for any state in 2011 NAEP for both 4th and 8th grade). Those board discussions make it clear: our high rate of NAEP exclusion involves a long-running testing accommodation used in Kentucky that allows providing special education students with readers for the state’s reading assessments.

Keep this in mind: learning disabled students who get readers on Kentucky’s state tests must routinely receive the same accommodation on all classroom tests they take. The provision must be written into their Individual Education Plans (IEP). Clearly, the vast majority of the learning disabled students who receive this accommodation are very poor readers, perhaps mostly even non-readers. These students never get a real, printed text decoding and comprehension test, which is exactly what the NAEP administers. Their IEPs don’t allow real reading assessment.

If students who routinely get readers on all of their tests suddenly were to take the challenging NAEP reading assessment without readers, it would be very surprising to see anything other than a very, very low overall group score. Many of these students would probably react very emotionally if they were suddenly required to demonstrate a skill they previously never had to demonstrate. Probably, many of these students would produce scores at or very close to a non-performance level.

By the way, let’s clarify Kifer’s confusing comment about how excessive exclusion impacts scores. NAEP reading scores actually being reported for Kentucky students who do take this federal test are HIGHER, not lower, than they would be if the excluded students also were tested and included in the average. Exclusion creates inflated, not depressed, scores.

So, there is very good reason for concern that Kentucky’s reported NAEP reading scores have notable inflation. You just have to know all the facts.

Speaking of knowing all the facts, I’ll have more to say about Prichard’s very weak post later this evening.

Now that a couple of supporting posts are on line at www.bipps.org/bipps-blog, let’s talk about some really serious problems with Prichard’s blog.

For one thing, Prichard’s blog doesn’t contain any mention of the very well-known major cause of Kentucky’s excessive exclusion rates on NAEP reading assessments. Lots of learning disabled students in Kentucky have IEPs that stipulate all tests, including reading tests, are to be read to those students. That accommodation is totally incompatible with the construct tested by the NAEP, which is true printed text decoding and comprehension. This leads to Kentucky’s nation-leading levels of exclusion on NAEP.

This accommodation is also hostile to the impacted students getting much, if any, real instruction in reading, as well. The accommodation creates negative inducements to teach reading to these special students.

The situation is no secret; the Kentucky Board of Education has been talking about the problem for half a year.

Prichard’s NAEP analysis of scores for students who are not learning disabled also ignores years of advice on how NAEP score comparisons need to be conducted. As the 2011 NAEP Reading Report Card puts it:

“Differences in states’ demographic makeup should be taken into consideration when interpreting state results.”

Prichard undertakes no such demographic analysis.

As soon as the NAEP 2011 reading results are disaggregated by race, Kentucky’s supposed advantage in NAEP reading evaporates for the state’s dominant racial population. Instead of scoring statistically significantly higher than the national average, Kentucky’s white, non-learning disabled students score statistically significantly lower than the national average for whites in both fourth and eighth grade.

There is a lot more information available on this; your interested readers can find that in two blog items here:

KSN&C

KSN&C

KSN&C is intended to be a place for well-reasoned civil discourse...not to suggest that we don’t appreciate the witty retort or pithy observation. Have at it. But we do not invite the anonymous flaming too often found in social media these days. This is a destination for folks to state your name and speak your piece.

It is important to note that, while the Moderator serves as Faculty Regent for Eastern Kentucky University, all comments offered by the Moderator on KSN&C are his own opinions and do not necessarily represent the views of the Board of Regents, the university administration, faculty, or any members of the university community.

On KSN&C, all authors are responsible for their own comments. See full disclaimer at the bottom of the page.

Why This Blog?

So far as we know, we only get one lifetime. So, when I "retired" in 2004, after 31-years in public education I wanted to do something different. I wanted to teach, write and become a student again. I have since spent a decade in higher ed.

I have listened to so many commentaries over the years about what should be done to improve Kentucky's schools - written largely by folks who have never tried to manage a classroom, run a school, or close an achievement gap. I came to believe that I might have something to offer.

I moved, in 1985, from suburban northern Kentucky to what was then the state’s flagship district - Fayette County. I have had a unique set of experiences to accompany my journey through KERA’s implementation. I have seen children grow to graduate and lead successful lives. I have seen them go to jail and I have seen them die. I have been amazed by brilliant teachers, dismayed by impassive bureaucrats, disappointed by politicians and uplifted by some of Kentucky’s finest school children. When I am not complaining about it, I will attest that public school administration is critically important work.

Democracy is run by those who show up. In our system of government every citizen has a voice, but only if they choose to use it.

This blog is totally independent; not supported or sponsored by any institution or political organization. I will make every effort to fully cite (or link to) my sources. Please address any concerns to the author.

On the campaign trail...with my wife Rita

An action shot: The Principal...as a much younger man.

Faculty Senate Chair

Serving as Mace Bearer during the Inauguration of Michael T. Benson as EKU's 12th president.

Teaching

EDF 203 in EKU's one-room schoolhouse.

Professin'

Lecturing on the history of Berea College to Berea faculty and staff, 2014.

Faculty Regent

One in a long series of meetings. 2016

KSN&C StatCounter

Disclaimer:

By accessing this website (http://theprincipal.blogspot.com) Kentucky School News and Commentary (hereafter KSN&C), a web browser (hereafter user) consents that she or he is familiar with, understands and absolutely accepts the following weblog disclaimer:

The views expressed by the authors and contributors on this website do not necessarily reflect the views of Kentucky School News and Commentary, those who link to this website, the author’s employers, mothers, fathers, sisters, brothers, other ancestors, blood-relatives, progeny, this website’s web host, moderator, designer, or any other organization in any way connected with this website.

While I presently serve as Chair of the Eastern Kentucky University Faculty Senate (August 2014-May 2016), none of the Moderator's comments are official statements attributable to EKU, its Faculty Senate or any of the institution's entities.

In all cases, comments are the personal views of the author. No individual contributor, author or commenter is paid for their opinion or beholden to a particular point of view. All contributors write in the English language and cannot be held responsible for unfortunate translations that may occur in other languages. KSN&C is not responsible for human errors involving grammar and punctuation.

Comments on this website are the sole responsibility of the author. The author assumes full responsibility, liability, and blame for any libel or litigation that may result from something written in or as a direct result of something written in a comment. The accuracy, completeness, veracity, honesty, exactitude, factuality and politeness of comments are not guaranteed. The content on the blog is not intended to malign any religious, ethnic group, club, organization, company or individual. Readers are advised to employ a healthy dose of rationality. Furthermore, information is always in transition. Web links change, and content published today may be out-of-date next week.

Readers are advised that some images used on the site are not the property of KSN&C but are reduced in size and used under fair-use. The same is true of certain copyrighted material. Any concerns should be addressed to the moderator. Due to the episodic nature of the blog, errors, when pointed out, may not be immediately corrected.

All trademarks, service marks, copyrights, registered names, mottos, logos, insignias and marks used or cited by this website are the property of their respective owners and this website in no way accepts any responsibility for an infringement on any of the above.

Despite any claims to the contrary, nothing on this website should be construed as professional advice. The information provided on this website is of a general, wide-ranging nature and cannot substitute for the advice of a licensed legal professional, physician, psychiatrist or member of the clergy. A competent authority with specialized knowledge operating within the Kentucky Department of Education, local public school district, church school, independent private school, home school, or in the journalistic, law enforcement or legal community is the only one who can address or comment on the specific circumstances covered in the news and commented upon herein. For personal advice, please contact your mother, father, BFF, local bar association, local bar tender, law society, medical board, county hospital, pastor, teacher, phone book, online directory, local emergency number in your jurisdiction, or Google to find a or obtain a referral to a competent professional.

This website has no control over the information you access via outbound link(s) in the post text, sidebar, header, footer or comment sections. This website does not endorse linked websites and cannot guarantee the accuracy of any information found by following said links or the correctness of any analysis found therein and should not be held responsible for it or the consequences of a user’s use of that information. In fact, we’re pretty sure we link to falsehoods perpetrated by others with some frequency. Be warned. Twistifications of supposed facts, biased reporting, and bad analysis is de rigueur for some of the sites we link.

This website may inadvertently link to content that is vacuous, obscene, venomous, frivolous, rotten, antagonistic, harsh, rancorous, acrimonious or repetitive. This website in no way condones, endorses or takes responsibility for such content. Please report anything really ugly to KSN&C’s Moderator.

This website publishes content regularly and said content is maintained in reference to the protections afforded it under local, state, martial, federal, international and school yard law. Publication of information found on this website may be in violation of the laws of the city, county, state, country or other jurisdiction from where you are viewing this website’s content and laws in your jurisdiction may not protect or allow the same kinds of speech or distribution. In the case that the laws of the jurisdiction where this website's content is maintained and those of yours conflict, this website does not encourage, condone, facilitate, recommend or protect the violation of any laws and cannot be responsible for any violations of such laws. We do condone lawful efforts to extend free speech protections to all parts of the world.

Because the World Wide Web is an integrated net of communication, discussion and litigation, this website encourages the distribution of its content. Cross, reciprocal or just plain friendly hyper-linking is consistent with this information sharing and this disclaimer should not be construed as a condemnation of any linking practices. That said, any reproduction of this website’s content must credit the website by name and Uniform Resource Locator (URL). Should you link to this domain or use, reproduce, republish, reiterate, imitate, or duplicate the information contained on this website, you alone are responsible for that action and should, under threat of litigation, credit this website by name and URL. In addition, any user who learns of information from this site, but traces back to our attributed sources in an effort to forego proper mention of KSN&C should seek therapy.

This website is not recommended for inmates, ingrates, illiterates, or anyone professing an irrational fear of CATS or any other mammal, or those who have a penchant for bullying or self- aggrandizement. Women who are pregnant or may become pregnant or are nursing are advised to consult their physician before reading this website. Eating before reading may result in indigestion. This website contains small pieces and is not recommended for children under the age of 4.