Saturday, September 07, 2013

International tests show achievement gaps in all countries, with big gains for U.S. disadvantaged students

In a [recent] EPI report, What do international tests really show about U.S. student
performance?, we disaggregate international student test scores
by social class and show that the commonplace condemnation of U.S. student
performance on such tests is misleading, exaggerated, and in many cases, based
on misinterpretation of the facts. Ours is the first study of which we are
aware to compare the performance of socioeconomically similar students across
nations.

Some critics, disturbed by the
unsophisticated way in which policymakers and pundits use international tests
to condemn American student performance, have commented that American students
in relatively affluent states, like Massachusetts or Minnesota, or students in
schools where few students are from low-income families, perform as well or
better than average students in the highest scoring countries. But while such
comparisons are well-intended, they can’t tell us much because a proper
comparison would be between affluent states in the U.S., and affluent provinces
or prefectures in other countries, or between schools with little poverty in
the U.S. and schools with little poverty in other countries. Critics have not
previously had data by which such comparisons can properly be made.

Yet both of the major international
tests—the Trends in International Mathematics and Science Study (TIMSS)
and the Program on International Student Assessment (PISA)—eventually publish
not only average national scores but a rich database from which analysts can
disaggregate scores by students’ socioeconomic characteristics, school
composition, and other informative criteria. Examining these can lead to more
nuanced conclusions than those suggested from average national scores
alone.

Although TIMSS published average
national results in December, it only plans to release its underlying database
this week. This puzzling procedure ensures that commentators draw quick but
ill-informed interpretations and that policy makers can offer inappropriate
interpretations of the results without fear of contradiction. Analysis of the
database takes time, and headlines from the initial release are sealed in conventional
wisdom before scholars can complete more careful study.

For example, two years ago when PISA
released its latest scores, U.S. Education Secretary Arne Duncan said they
showed American students “are poorly prepared to compete in today’s knowledge
economy. … Americans need to wake up to this educational reality—instead of
napping at the wheel while emerging competitors prepare their students for
economic leadership.” In particular, Duncan stressed results for disadvantaged
U.S. students: “As disturbing as these national trends are for America,
enormous achievement gaps among black and Hispanic students portend even more
trouble for the U.S. in the years ahead.”

Yet a careful analysis of the PISA
database shows that the achievement gap between disadvantaged and advantaged
children is actually smaller in the United States than it is in similar
countries. The achievement gap in the United States is larger than it is in the
very highest scoring countries, but even then, many of the differences are
small.

What’s more, an examination of
trends over the last decade, on multiple administrations of both TIMSS and
PISA, shows that the achievement of the most disadvantaged U.S. adolescents has
been increasing rapidly, while the achievement of similarly disadvantaged
adolescents in some countries that are typically held up as examples for the
U.S.—Finland for example—has been falling just as rapidly. Thus, while the
reading achievement on PISA of the lowest social class students in the U.S.
grew by more than 0.2 standard deviations from 2000 to 2009, it fell by an even
larger amount in Finland. In math, the lowest social class U.S. students also
posted substantial gains, while scores of comparable Finnish students declined.
This is surprising because the proportion of disadvantaged students in Finland
also fell, and we might expect this to make the task of devoting resources to
them easier. Certainly, even for the lowest social class students, Finland’s
scores remain higher than ours, but examination of trends as well as levels
challenges the easy assumption that simply imitating Finnish education is a
recipe for U.S. success.

Once aware of these data trends, it
would be perverse for policy makers to conclude from international test
comparisons that we should upend how we educate disadvantaged youngsters. While
American schools can certainly do better with disadvantaged children, it seems
that our educational system may have more serious problems with the more
advantaged students, relative to other nations.

Since the last PISA release in 2010,
we have been digging deeper into its database, as well as into the older
databases for TIMSS and for both versions of our domestic National Assessment
of Educational Progress (NAEP). We concentrated on scores of adolescents (8th graders
on TIMSS and NAEP, 15 year-olds on PISA) in the U.S., in three top-scoring
countries (Canada, Finland, and Korea), in three similar post-industrial
countries (France, Germany and the U.K.), and in seven American states and
three Canadian provinces for which trends are available because they
voluntarily participated in TIMSS more than once.

The share of disadvantaged students
in the U.S. sample was larger than their share in any of the other countries we
studied. Because test scores in every country are characterized by a social
class gradient—students higher in the social class scale have better average
achievement than students in the next lower class—U.S. student scores are lower
on average simply because of our relatively disadvantaged social class composition.

This social class driven distortion
has been compounded because in 2009, PISA over-sampled low-income U.S. students
who attended schools with very high proportions of similarly disadvantaged
students, artificially lowering the apparent U.S. score. While 40 percent of
the PISA sample was drawn from schools where half or more of students were
eligible for free and reduced-price lunch, only 32 percent of students
nationwide attend such schools.

Our report shows that if we make two
reasonable adjustments to the reported U.S. average, our international ranking
improves. The first adjustment re-weights the social class composition of U.S.
test takers to the average composition of top-scoring countries. The other
re-weights the distribution of lunch-eligible students by the actual intensity
of such students in schools. These adjustments raise the U.S. international
ranking on the 2009 PISA test from 14th to sixth in reading, and from
25th to 13th in mathematics. While there is still room for
improvement, these are quite respectable showings.

A unique aspect of our report is our
consolidation of trend data from all four assessments—PISA, TIMSS and the two
forms of NAEP. From 2000 to 2006, U.S. math scores on PISA fell substantially,
causing great alarm among U.S. policymakers and pundits. But few noticed that
during roughly the same period, U.S. math scores on TIMSS were rising, as they
did on the Main NAEP. We cannot attribute this to an alleged superior alignment
of TIMSS to the U.S. curriculum because in the next period, the pattern was
reversed: from 2006 to 2009, PISA and NAEP scores both rose, while TIMSS scores
were flatter. We are aware of no reasonable explanation for these erratic
patterns, but they suggest that caution is called for before drawing
conclusions from any single test about international comparisons—or about
anything else, for that matter.

Extensive educational research in
the United States has demonstrated that students’ family and community
characteristics powerfully influence their school performance. Children whose
parents read to them at home, whose health is good and can attend school
regularly, who do not live in fear of crime and violence, who enjoy stable
housing and continuous school attendance, whose parents’ regular employment
creates security, who are exposed to museums, libraries, music and art lessons,
who travel outside their immediate neighborhoods, and who are surrounded by
adults who model high educational achievement and attainment will, on average,
achieve at higher levels than children without these educationally relevant
advantages.

Aware of these relationships,
education analysts in the United States pay close attention to the level and
trends of test scores disaggregated by socioeconomic groupings. Indeed, a
central element of U.S. domestic education policy is the requirement that
average scores be reported separately for racial and ethnic groups and for
children who are from families whose incomes are low enough to qualify for the
subsidized lunch program. We understand that a school with high proportions of
disadvantaged children may be able to produce great “value-added” for its
pupils, although its average test score levels may be low. We know much less
about the extent to which similar factors affect achievement in other
countries, but we should assume, in the absence of evidence to the contrary,
that they do. It would be foolish to fail to apply this understanding to
comparisons of international test scores.

The database for TIMSS 2011 is
scheduled for release soon, and next December, PISA will announce results and
make data available from its 2012 test administration. Scholars will then be
able to dig into TIMSS 2011 and PISA 2012 databases and place the publicly
promoted average national results in proper context. We urge policymakers and
pundits to await understanding of this context before drawing conclusions about
lessons from future TIMSS or PISA assessments. We plan to conduct our own
analyses of these data when they become available, and publish supplements to
our report as soon as it is practical to do so, given the care that should
be taken with these complex databases.

1 comment:

Anonymous
said...

I have an idea, all the folks who have been riding teachers about "closing the gap" should go to these other places to help them work on the issue there and let us do our work of teaching ALL students.

KSN&C

KSN&C

KSN&C is intended to be a place for well-reasoned civil discourse...not to suggest that we don’t appreciate the witty retort or pithy observation. Have at it. But we do not invite the anonymous flaming too often found in social media these days. This is a destination for folks to state your name and speak your piece.

It is important to note that, while the Moderator serves as Faculty Regent for Eastern Kentucky University, all comments offered by the Moderator on KSN&C are his own opinions and do not necessarily represent the views of the Board of Regents, the university administration, faculty, or any members of the university community.

On KSN&C, all authors are responsible for their own comments. See full disclaimer at the bottom of the page.

Why This Blog?

So far as we know, we only get one lifetime. So, when I "retired" in 2004, after 31-years in public education I wanted to do something different. I wanted to teach, write and become a student again. I have since spent a decade in higher ed.

I have listened to so many commentaries over the years about what should be done to improve Kentucky's schools - written largely by folks who have never tried to manage a classroom, run a school, or close an achievement gap. I came to believe that I might have something to offer.

I moved, in 1985, from suburban northern Kentucky to what was then the state’s flagship district - Fayette County. I have had a unique set of experiences to accompany my journey through KERA’s implementation. I have seen children grow to graduate and lead successful lives. I have seen them go to jail and I have seen them die. I have been amazed by brilliant teachers, dismayed by impassive bureaucrats, disappointed by politicians and uplifted by some of Kentucky’s finest school children. When I am not complaining about it, I will attest that public school administration is critically important work.

Democracy is run by those who show up. In our system of government every citizen has a voice, but only if they choose to use it.

This blog is totally independent; not supported or sponsored by any institution or political organization. I will make every effort to fully cite (or link to) my sources. Please address any concerns to the author.

On the campaign trail...with my wife Rita

An action shot: The Principal...as a much younger man.

Faculty Senate Chair

Serving as Mace Bearer during the Inauguration of Michael T. Benson as EKU's 12th president.

Teaching

EDF 203 in EKU's one-room schoolhouse.

Professin'

Lecturing on the history of Berea College to Berea faculty and staff, 2014.

Faculty Regent

One in a long series of meetings. 2016

KSN&C StatCounter

Disclaimer:

By accessing this website (http://theprincipal.blogspot.com) Kentucky School News and Commentary (hereafter KSN&C), a web browser (hereafter user) consents that she or he is familiar with, understands and absolutely accepts the following weblog disclaimer:

The views expressed by the authors and contributors on this website do not necessarily reflect the views of Kentucky School News and Commentary, those who link to this website, the author’s employers, mothers, fathers, sisters, brothers, other ancestors, blood-relatives, progeny, this website’s web host, moderator, designer, or any other organization in any way connected with this website.

While I presently serve as Chair of the Eastern Kentucky University Faculty Senate (August 2014-May 2016), none of the Moderator's comments are official statements attributable to EKU, its Faculty Senate or any of the institution's entities.

In all cases, comments are the personal views of the author. No individual contributor, author or commenter is paid for their opinion or beholden to a particular point of view. All contributors write in the English language and cannot be held responsible for unfortunate translations that may occur in other languages. KSN&C is not responsible for human errors involving grammar and punctuation.

Comments on this website are the sole responsibility of the author. The author assumes full responsibility, liability, and blame for any libel or litigation that may result from something written in or as a direct result of something written in a comment. The accuracy, completeness, veracity, honesty, exactitude, factuality and politeness of comments are not guaranteed. The content on the blog is not intended to malign any religious, ethnic group, club, organization, company or individual. Readers are advised to employ a healthy dose of rationality. Furthermore, information is always in transition. Web links change, and content published today may be out-of-date next week.

Readers are advised that some images used on the site are not the property of KSN&C but are reduced in size and used under fair-use. The same is true of certain copyrighted material. Any concerns should be addressed to the moderator. Due to the episodic nature of the blog, errors, when pointed out, may not be immediately corrected.

All trademarks, service marks, copyrights, registered names, mottos, logos, insignias and marks used or cited by this website are the property of their respective owners and this website in no way accepts any responsibility for an infringement on any of the above.

Despite any claims to the contrary, nothing on this website should be construed as professional advice. The information provided on this website is of a general, wide-ranging nature and cannot substitute for the advice of a licensed legal professional, physician, psychiatrist or member of the clergy. A competent authority with specialized knowledge operating within the Kentucky Department of Education, local public school district, church school, independent private school, home school, or in the journalistic, law enforcement or legal community is the only one who can address or comment on the specific circumstances covered in the news and commented upon herein. For personal advice, please contact your mother, father, BFF, local bar association, local bar tender, law society, medical board, county hospital, pastor, teacher, phone book, online directory, local emergency number in your jurisdiction, or Google to find a or obtain a referral to a competent professional.

This website has no control over the information you access via outbound link(s) in the post text, sidebar, header, footer or comment sections. This website does not endorse linked websites and cannot guarantee the accuracy of any information found by following said links or the correctness of any analysis found therein and should not be held responsible for it or the consequences of a user’s use of that information. In fact, we’re pretty sure we link to falsehoods perpetrated by others with some frequency. Be warned. Twistifications of supposed facts, biased reporting, and bad analysis is de rigueur for some of the sites we link.

This website may inadvertently link to content that is vacuous, obscene, venomous, frivolous, rotten, antagonistic, harsh, rancorous, acrimonious or repetitive. This website in no way condones, endorses or takes responsibility for such content. Please report anything really ugly to KSN&C’s Moderator.

This website publishes content regularly and said content is maintained in reference to the protections afforded it under local, state, martial, federal, international and school yard law. Publication of information found on this website may be in violation of the laws of the city, county, state, country or other jurisdiction from where you are viewing this website’s content and laws in your jurisdiction may not protect or allow the same kinds of speech or distribution. In the case that the laws of the jurisdiction where this website's content is maintained and those of yours conflict, this website does not encourage, condone, facilitate, recommend or protect the violation of any laws and cannot be responsible for any violations of such laws. We do condone lawful efforts to extend free speech protections to all parts of the world.

Because the World Wide Web is an integrated net of communication, discussion and litigation, this website encourages the distribution of its content. Cross, reciprocal or just plain friendly hyper-linking is consistent with this information sharing and this disclaimer should not be construed as a condemnation of any linking practices. That said, any reproduction of this website’s content must credit the website by name and Uniform Resource Locator (URL). Should you link to this domain or use, reproduce, republish, reiterate, imitate, or duplicate the information contained on this website, you alone are responsible for that action and should, under threat of litigation, credit this website by name and URL. In addition, any user who learns of information from this site, but traces back to our attributed sources in an effort to forego proper mention of KSN&C should seek therapy.

This website is not recommended for inmates, ingrates, illiterates, or anyone professing an irrational fear of CATS or any other mammal, or those who have a penchant for bullying or self- aggrandizement. Women who are pregnant or may become pregnant or are nursing are advised to consult their physician before reading this website. Eating before reading may result in indigestion. This website contains small pieces and is not recommended for children under the age of 4.