Since the release of the TIMSS report on mathematics and scienceachievement in the final year of secondary school and thecorresponding third volume of NCES' Pursuing Excellence, severalquestions have arisen about the validity of the comparisons and therelevance of the results. Some observers have argued that themethodology was biased against students in the United States inseveral ways, including the fact that the average age of U.S.students was lower than in other countries, that the populationstested in other countries represented a more selective group of theage cohort, and that not all our students who took the advancedmathematics examination had been exposed to the material on theassessment.

Several arguments have also been advanced about why the results ofthe assessments do not matter. Some say we should not be so worriedabout achievement at the end of high school because our students"catch up" in college. Critics cite the fact that our economy is thestrongest in the world as evidence that people do not need the typeof mathematics and science knowledge and skills measured on theTIMSS assessment. Howard Gardner writes that TIMSS measures only "akind of lowest common denominator of facts and skills." Morecrucial, he argues, is the ability to "apply scientific andmathematical concepts to the world around them." Others mentionabilities and qualities such as creativity, flexibility, andinnovation as more crucial for economic success.

Following are responses to the most common questions and criticismsregarding the validity and relevance of the TIMSS assessment ofstudents at the end of secondary school.

1.Differences in age and grade levels

Q. The students in other countries, on average, were older than theU.S. students. Is this because secondary schools in other countriesgo beyond grade twelve? If so, how fair or useful is it to compareU.S. students with those in other countries who are one to three years older and have more years of instruction?

A. The purpose of this component of TIMSS was not to comparestudents of the same age or years of schooling, but rather tocompare students at a similar point in the education system: the endof secondary school. In many societies, including the United States,students completing secondary school are deemed "ready" to enteradult society, part of which is the workforce. Thus, a comparison ofstudents at this point sheds light on the level of knowledgeexpected of and attained by the general population.

While there was a range of ages among students taking theassessment, and NCES found that age was related to achievement, thegap between the average ages of U.S. students and those in othercountries, and consequently, the impact it has on the results, hasbeen overstated. At the extreme, the students participating in TIMSSin Iceland had an average age of 21.2 years, 3.1 years higher thanthe average age of the U.S. students, 18.1. However, theinternational average was far lower: 18.7 years, much closer to theU.S. average. This gap was even smaller on the physics and advancedmathematics assessments, where the average age of our students was18.0 years for both assessments. The international averages on theseassessments were just a few months higher, 18.4 years in physics and18.3 years in advanced mathematics.

Regarding the years of schooling, in other countries, both collegepreparatory and vocational programs may go through grade thirteen orfourteen, but most of those countries also have other secondaryprograms that end earlier, often before grade 12. While eightcountries (out of 21 in the general knowledge assessments) includedsome students in grades above twelfth grade, all of those countriesalso included students in twelfth grade and five of them alsoincluded students below grade 12. It is also important to note thatin some countries, the older average age of students reflects alater school starting age. In Denmark, Slovenia, Norway, Sweden, andparts of the Russian Federation and Switzerland, students start the first grade at the age of seven, compared to our typical startingage of six.

The differences in average ages and years of instruction also needto be considered in light of the content of the general knowledgeassessments. If the content were based on high-level curriculumtopics, then younger students and students with fewer years ofschooling might be at a disadvantage, since it is reasonable tothink that they would be less likely to have been exposed to thesetopics than older students or students with more years of schooling.However, the TIMSS general knowledge assessments did not representadvanced-level content. While the items on these assessments werenot based on any one curriculum, TIMSS analysts have found thetopics on the mathematics general knowledge assessment to be mostsimilar to topics covered by the seventh grade in most countries,and the topics on the science general knowledge assessment similarto topics covered by the ninth grade. When they looked at the itemsin terms of U.S. curricula, they found these topics to be introducedat later grade levels, by the ninth grade for mathematics and by theeleventh grade for science. Thus, it appears that students in theUnited States and in other countries were being tested on topicsthey should have already covered, several years earlier in mostcases.

In the end, it is difficult to use the higher ages of students inother countries to explain our relatively poor performance. Whilemany of the countries which outperformed the U.S. included studentswhose average age was higher than ours, but we were alsooutperformed by countries in which the average age of students waslower than ours. For example, students in New Zealand and Australiawere, on average, younger than U.S. students but still scoredsignificantly higher than our students on both general knowledgeassessments. It is also interesting to note that students in theRussian Federation performed similarly to U.S. students on bothgeneral knowledge assessments, but were more than a year younger andwere all in eleventh grade.

2.Differences in enrollment rates

Q. In other counties, only the best students are still enrolled insecondary school in the late teenage years. Isn't it unfair tocompare our general teenage population-the vast majority of whom arestill in school-with elite populations in other countries?

A. There are several ways to look at secondary school enrollment.One way is to divide the number of young people enrolled insecondary school by the total number in the population within thecorresponding age cohort. Recognizing that this can often overstateenrollment if young people outside the age cohort are enrolled, OECDdata indicate that the majority of countries participating in thiscomponent of TIMSS, including the U.S., have over 85 percent of theage cohort enrolled in secondary schools. Thus, while variation inenrollment rates does exist, the countries are roughly comparable,and more so than in previous years.

Looking at enrollment another way, among seventeen-year-olds, theU.S. actually has a smaller proportion enrolled in school than theaverage for the other TIMSS countries for which this information isavailable (75 percent vs. 82 percent). If higher school enrollmentrates were associated with lower average scores, the bias would bein favor of U.S. students rather than against them. However, inTIMSS, using either of the methods of measuring enrollment, studentsin countries with higher enrollment rates than the U.S. tended toscore significantly higher than the U.S. on both the mathematics andscience general knowledge assessments. Furthermore, in TIMSS, thepattern generally appears to be that higher secondary enrollmentrates are associated with higher levels of performance, rather thanthe reverse.

3.The "unfair" definition of our advanced mathematics population

Q. The advanced mathematics test had calculus questions, but U.S.students whose highest mathematics course was precalculus wereincluded in the test sample. Wouldn't it have been a fairercomparison to look only at those students who had actually takencalculus?

A. The advanced mathematics assessment was not primarily a calculustest. Calculus items comprised only about 25 percent of thequestions. Interestingly, even with precalculus students included inour sample of advanced mathematics students, our weakest contentarea was not calculus, but geometry. This is even more surprisingwhen we take into consideration that advanced mathematics studentsin the United States will typically have studied geometry for a fullyear by the twelfth grade.

The purpose of the advanced mathematics assessment was to compareadvanced mathematics students across countries on content above thelevel of "general knowledge." For this assessment, countries wereasked to create course- or program-based definitions of advancedstudents such that the resulting population represented from between10 to 20 percent of the age cohort. To meet this specification, theUnited States needed to include students in precalculus classes, asonly seven percent of the age cohort were taking or had taken acalculus class. (Would it be fair to compare seven percent in theU.S. with over 16 percent in Canada, 20 percent in France, or 33percent in Austria?) The resulting population in the United Statesrepresented 14 percent of the age cohort, compared to theinternational average of 19 percent. In this respect, the group ofU.S advanced mathematics students was somewhat more selective thanin some other countries.

Even with precalculus students included in our advanced mathematicspopulation, one cannot assume that U.S. students were at any more ofa disadvantage than students in other countries. In none of thecountries were students chosen on the basis of whether they hadtaken calculus. An in-depth curriculum analysis might reveal thatstudents in other countries had more exposure to calculus (or other topics) than U.S. students, but it might also show that U.S.students had relatively more exposure to some topics than studentsin other countries. If U.S. advanced mathematics students do face acurriculum that includes fewer advanced topics than those faced bytheir international peers, that would be an important finding.

End of Part 1 of 2********************************************************