There’s a tweet circulating widely that says that Wisconsin ranks second in the nation in its students’ SAT/ACT scores, while five states that bar teachers from participating in collective bargaining all rank near the bottom of the pile. Here are a few examples:

@WisDems Only 5 states don’t have collective bargaining for educators. Their ACT/SAT rankings: SC-50th/NC-49th/GA-48th/TX-47th/VA-44th.

The information presented here on which states bar collective bargaining in education is correct (although it only reflects the five states where teachers’ unions are illegal, leaving out the other five states where they’re legal but essentially don’t exist). What it doesn’t include, however, is any supporting documentation on the SAT and ACT rankings themselves.

So I’ve taken a look at the data.

State scores on the SAT and ACT are hard to compare directly, because there’s so much variation in how many students take the tests. In addition, I haven’t yet found a source that combines SAT and ACT scores into a composite ranking like the one in the tweet. Looking at charts for SAT and ACT results separately, however, here’s what I found:

Wisconsin ranks 3rd in the nation in SAT scores, but with a participation rate of just 4%. On the ACT, with a much more representative partcipation rate of 69%, it was tied for 17th. In comparison…

Virginia was 34th on the SAT with 67% participation, 13th on the ACT with 22% participation.

Texas was 45th on the SAT with 53% participation, 33rd on the ACT with 33% participation.

Georgia was 48th on the SAT with 74% participation, 34th on the ACT with 44% participation.

North Carolina was 38th on the SAT with 63% participation, 20th on the ACT with 16% participation.

South Carolina was 49th on the SAT with 66% participation, 44th on the ACT with 52% participation.

Wisconsin is clearly above the other five states in both SAT and ACT scores, but the gap isn’t anywhere near as big as the pro-union tweets suggest. Among high ACT participation states, Wisconsin ranks something like 4th in the nation. But among high SAT participation states, Virginia ranks about 5th in the nation — almost all the states with better SAT scores than Virginia have far smaller participation rates, drawing on a far more elite test-taking group.

I’m open to seeing new data on this, but for now I’m going to mark this claim down as highly exaggerated.

Update | Several readers have posted links to the original source for the tweeted claim, which can be found here. Thanks!

Now, about that source. First, it’s from an analysis conducted in 1999, apparently by a University of Missouri law professor named Douglas O. Linder. Linder doesn’t say specifically what year the SAT/ACT numbers come from, but they’re obviously more than a decade old.

Second, the ranking methodology is really weird, and completely unreliable. What the author did was take each state’s ranking on the SAT, add it to each state’s ranking on the ACT, add those two numbers together, and then put them in order. In other words, Wisconsin scored 5th in the country on the SATs and 4th in the country on the ACTs, giving it a total of 9, and only Iowa had a lower total, so Wisconsin was 2nd in the country overall.

This is just silly. As I noted above, almost nobody in Wisconsin takes the SATs — the state has only a 4% participation rate on that test, with the only students taking it being those who are applying to competitive out-of-state colleges that don’t accept the ACTs. When you compare Wisconsin’s SAT average to Georgia’s, you’re comparing the performance of a tiny elite in one state with that of 74% of the graduating class of the other. And on top of that, this chart gives Wisconsin’s SAT score equal weight with its ACT score in determining which state is “better.”

It’s nonsense. It’s meaningless.

Second Update | The scores don’t seem to match the data, either. Linder gives Wisconsin’s average combined Math and Verbal SAT score as 1073, but Wisconsin’s average SAT scores in the late 1990s ranged from 1169 in 1996-97 to 1181 in 1999-2000. Something’s screwy with Linder’s numbers.

Third Update | As I said at the top of the post, I’ve put up a new essay this morning discussing exactly how teachers’ unions do improve student performance, and why. If you’ve read this far, you really should keep reading.

Its a difficult thing to determine from simple stats. States with low participation in a particular test are likely to suffer from a selection bias. In WI you take the SAT if you are going out of state. You go out of State if you are smarter, richer, etc. Which is why our SAT scores are high. You need to combine ACT, and SAT (convert) then adjust the states data for race, income, education, and percent going to college, along with union penetration. that will give you a better handle on the union’s effect. Unions definitely have very bad aspects too them, but I would think their ability to get higher pay for union members, etc greater stability, makes for on average better teachers. Thats not to say you couldn’t create the same environment without unions. When I grad from Madison I knew teachers who went to TX, because of the lower standards and the easy of getting in. The good ones eventually moved back to WI, the bad ones left teaching.

What about this more recent (2009) study on state SAT rankings? It seems much in keeping with the original tweet (WI is at number 2; Georgia, NC and SC at the bottom, along with DC which has notoriously troubled public schools).

Taggert/Angus: Can you get your hands on this Harvard Educational Review article? Alas, I can’t get access to that particular article, but I’m curious about it. Even though the material would be 10 years old, the conclusions would still seem relevant.

Miriam — that chart uses pretty much the same data as I used in my post. As I noted, though, and as the chart you linked to shows, very few students in Wisconsin take the SATs at all. As you can see, no state with higher than 50% participation in SAT testing ranks higher than 24th on the list, and the top scorers — including Wisconsin — are all in the single digits. It’s just not a meaningful measure of a state’s academic achievement.

Oh, and on Miriam’s PS: I’m a big fan of that Gladwell article (surprising as that may be to long-time readers of this site), but what he’s talking about is something a little different. There he’s saying that what you measure determines how your rankings will come out. What I’m saying is that Professor Linder’s rankings are statistically incoherent. They’re literally meaningless, because they don’t measure anything worth measuring in any sort of valuable way.

There are some 10 states in which there are virtually no legally binding K-12 teacher contracts at all (there are none in AL, AZ, GA, MS, NC, SC, TX, and VA; there is only one district with a contract in LA, and two in AR).

Out of these 10 states, only one (Virginia) has an average rank above the median, while four are in the bottom 10, and seven are in the bottom 15. Nine of the 10 states with the highest average ranks are high coverage states, including Massachusetts, which has the highest average score on all four tests. She points out that unions alone aren’t the only contributing factors to low test scores in non union states, but it does make a good argument against those who say unions are to blame for falling schools.

[…] an extensive post about how wrong, misleading and just false the information is when I came across this post that does basically exactly what I was about to do, so I’m just going to encourage you to […]

It can’t all be total nonsense. Only 4% of Wisconsin students take the SAT, but that does not make them an elite group as this article indicates. It just means that 4% of students want to attend school outside of Wisconsin and the SATs are required. It’s unfair to assume they are an elite group and thus are going to have higher scores than students from, say, Georgia. The point is that they are students just like the students from other non-collective-bargaining states. Attending out of state is not reserved for the richer or smarter. Taking the SAT is not reserved for those same groups either. There must be some correlation between the scores and the fact that those states don’t have a collective bargaining agreement. Has anyone looked at graduation rates in these same states and made a correlation that way?

As a public school teacher, I find it particularly disappointing how many of my colleagues are spreading this tweet without looking into it. They are guilty of doing the very thing they criticize students for: accepting what you read on the internet as fact. Pathetic.

Thank you for your efforts to uncover the truth. Obviously, it’s hard to compare based on sample size, etc, but anyone with some sense should find the original tweet hard to accept at face value.

This post was nothing more than propaganda, it is not filled with FACT, instead, it is lies. Check out the official ACT website, to verify the inaccurate information. Remember, just because someone makes a post on the web, doesn’t mean it is FACT.

Good job catcthing the phony data. Some of the results that I’ve seen quoted are actually from 1997. That is “hand picking of data” at its finest.

As for the SAT data it is invalid. Because, as you also caught on, the participation rates are all over the map; with the highest scores being clearly grouped with lowest participation rating states. If one was to do a regression the conclusion might be that state should reduce their participation rates in order to get better scores. Also, a bad conclusion.

There is a reason why Mark Twain was quoted saying, “There are three kinds of lies: lies, damned lies and statistics.”

There seems to be a definite relationship between the number of participants, and the ranking of SAT scores. Much more than if a state has a union or not. How can you extrapolate a reasonable figure for a state that has a 4% participation ratelike Wisconsin to Maine with a 90% rate. No offense, but it seems a little “voodooish” to me.

I came to the same general conclusion when looking at this myself. I would also add in that you need to look at the minority population and income level in a state. Case in point, Georgia has a minority population of ~35%, while Wisconsin is closer to ~6%. Minority groups have long complained about the SAT/ACT and biased quesitons. The NYtimes has also reported on income levels Vs test results.

[…] arguments made the rounds over the past week. They were addressed thoughtfully on one blog, which examined the validity of the data, and pointed out (correctly) that it remained unclear whether differences were attributable to […]

[…] works Indiana, Florida, and Ohio, all by Republican leaders. When you factor in that union teachers do significantly better work than non-union ones on average (yes, there are 3 separate links there), and that when properly […]

OMG! A quick look the map available at http://www.censusscope.org/us/map_nhwhite.html
is revealing Please take time to do this.
Any portrail of teachers in non-collective bargaining states as inferior is a smear showing how low some go to protect an unfair system that fleeces taxpayers. Taxpayers who, nevertheless, truly want to pay teachers fairly. The states in question have overwhelmingly larger African American and Hispanic populations. Children in these classrooms are many times more likley to come from non-english speaking or extemely disadvanted backgrounds. Just look at the map and compare the states being so unfairly disparaged with the state of Wisconsin. It does not take a MIT degree to get the point. Using these unfortunate disparities to bestow pre-eminence on teachers who collectively bargain is shameful. Keep this in mind the next time you considering the logical inference of Angus Johnston. If this is the level reasoning (or the level of trust) we can expect from our educators as they persue their fincanial goals, I fear for the general public’s faith in our schools as we go forward. I am certain that there will be more self serving studies showing how, compensating of multilplicy of factors, teachers collective bargaining makes a significant difference in the education of our children. You can count on it.

“It just means that 4% of students want to attend school outside of Wisconsin and the SATs are required. ”

Tim, the SAT consistently draws higher achieving students. Elite colleges are more likely to ask students to take it and there is a definite correlation between the selectiveness of a school and the quality of those who apply. Anyone can apply to MIT but the truth is that by and large only very good students do.

[…] same level of accuracy as Obama-is-a-Manchurian-Muslim emails. Josh Moser at Ashbrook Center and Angus Johnson at Student Activism critique the original rankings from the right and left wings, respectively. In short, the data are […]

Well, this was somewhat confusing. First you state that Wisconsin’s ACT ranking was 17th, with 69% participation.Later you state that Wisconsin was 4th, combined with 5th on SAT, for a total of 9, which put it in 2nd place overall. Which is accurate: 17th or 4th? That’s a big difference, and I’d like to use the correct number here in Wisconsin.

[…] Angus Johnston of StudentActivism.net makes a great case as to why this is so. First of all, when corrected for participation, the data still supports the fact that Wisconsin consistently outranks states that don’t […]

collective bargaining and unions fleece taxpayers-must be one of those HB gary sock puppets writing there.

But arent teachers and union members taxPAYERS, os are they fleecing themselves too? Or is it really fascist elite ruling class allowing corporations to dodge taxes and create loopholes for them to fit there very large bank accounts to be hidden. I think the original numbers were posted more as a “kick” than anything else. The author and the sockpuppets then were meant to make it into something else. As much as some of you all would like the protest to go away and the rest of the middle class as well (everyone KNOWS teachers are UNDER PAID-well except for college level DIV I COACHES). Really the point is the income and tax disparity…not so much SAT/ACT scores. nice deflection but no deal.

As a retired institutional researcher, I can tell you that some districts (forget about states for a minute) openly recruit “better” performers for SAT – even to the point of providing “scholarships” to pay for testing fees – while they discourage poorer performers from considering sitting for the SAT. Some facilitate these poorer performers to take the ACT, knowing that in their state only the SAT scores are considered as a prestigious indicator of district success. The author is correct in noting that when the overall proportion for students taking the SAT is **so** disparate that direct comparisons of resulting performance are impossible – without further disaggregation. If someone could access the HS transcripts for SAT takers, common demographics including grade level and age at testing, and **all** SAT scores for each student tested, then a more realistic comparison is possible. [I mention “all” SAT scores because it is not uncommon for a student to take the SAT more than once in an attempt to improve their scores. In fact, research has shown that it is not uncommon for younger students to improve overall point count by 200-300 points by the second testing. Also, it is up to each district to decide **which** of these multiple SAT scores to report for each student – the 1st one? The best one? The most recent one? The best for each subtest regardless of date taken?] Also, some districts (and States, for that matter) embrace the goal to include as many students as possible in SAT testing – a tactic opposite to testing only the “elite”. Are these States and districts to be penalized by being considered as “instructionally impotent” because they open the door to college admissions to a larger proportion of youth? Obviously the average SAT score for a larger and more representative sample of the population would be lower than that for the top quartile or decile. It all boils down to smoke and mirrors with numbers. All accountability and “one number tells the whole story” systems with which I am familiar dilute and distill accurate and factual information for the sake of throwing one index per State up on a bar chart. That, my friends, is no information at all – and only the ingenuous, the uninformed, and the manipulative attach meaning to it and promote it in discussions of action.

Certainly the participation percentage matters – if only 4% of your students take the SAT, as in Wisconsin, then you are only getting a small and probably elite subset of all students. If 50 or 60% of students take a test, then you are getting closer to average ability. It would be very helpful if all states required all students to take either or both of these tests – but then, they do not really want people to be able to compare education results, do they? Perhaps someone should survey universities and colleges on how they use these scores – I am sure they have figured out what these scores really mean, as they use them to determine which students to pursue. My two homeschooled sons both scored over 28 on the ACT in Wisconsin, which is about the average score for all of the homeschooled students I know. Makes you wonder how much the homeschool scores are bolstering the public school scores. Why are these factors not tracked?