The authors of this study of 74 children, half homeschooled, half institutionally schooled, conclude that structured homeschooling is best, public schooling next, and unstructured homeschooling worst at producing high levels of academic achievement. They begin with a nice summary of the limits of previous studies of academic achievement, especially of the famous Rudner study of 1999, which we have had several occasions to mention on this blog. They wanted to overcome the sampling biases of this and many other HSLDA-funded studies of academic achievement.

To do this they sent out ads in every medium possible to recruit public and homeschooled students between ages 5 and 10 in Canada. They chose 37 public schooled and 37 homeschooled children from the recruits. These kids were then matched up so as to create pairs of demographically similar children. Later, the homeschoolers were further subdivided into “structured” and “unstructured” once researchers figured out there was a real distinction here in how homeschoolers went about their daily education. There were 25 “structured” homeschoolers and 12 “unstructured.”

Parents filled out a demographic questionnaire and children took the Woodcock-Johnson Test of Achievement. Both homeschoolers and conventionally schooled children were given the test in the same way by the same people, thus avoiding one of the most common problems with most academic achievement studies.

The results were that the structured homeschoolers scored higher than the public schoolers on every part of the test, even though their families reported slightly lower incomes and their mothers had a bit less formal education.

Unstructured homeschoolers, on the other hand, consistently scored lower than institutionally schooled students, and in four of the seven categories they performed below grade level. The authors are quick to note that their sample of 12 is inadequate to make compelling generalizations, but they find this “exploratory analysis” to be intriguing.

The authors end with a discussion of some of the limitations of their study. They note that the study’s reliance on volunteers likely contributed to an inflation of homeschooler scores, but in their case (unlike all other studies) the institutionally-schooled students were ALSO volunteers. They too scored above grade level, just not as much as the structured homeschoolers. The researchers also note that the small sample size did not allow for more nuance about level of structure homeschoolers employ, and they wonder if over time maybe the unstructured kids would catch up.

I have three points to make about this article:

1. Part of me is skeptical about the whole thing. Here’s why. The authors say that at the outset of their study they were only planning on comparing the results of homeschoolers to those of traditionally schooled children. But that’s NOT what they ended up doing. They say that they later came to figure out that there was a distinction between structured and unstructured homeschooling, but it is very convenient for their findings that this division has allowed them to pull out the low scoring homeschoolers from their sample. The authors never end up giving us the actual comparison between their 37 public schoolers and their 37 homeschoolers. Instead, we get a comparison between the 37 public and the 25 best scoring homeschoolers, and then a separate comparison between the 37 public and the 12 worst scoring homeschoolers.

It is impossible to tell from the data provided in the article itself what the results of a straight-ahead comparison between the 37 and 37 would have given, but it looks to me like if you combined the two groups of homeschoolers you’d end up with scores that were pretty close to the average obtained by the public schooled students.

The reason this bothers me is that I’ve recently been reading Ben Goldacre’s very funny Bad Science: Quacks, Hacks, and Big Pharma Flacks, which chronicles several ruses pharmaceutical companies use to cook the books to make their drugs look good. One of their best tricks is to start a study looking for a result in the general population, but when the data doesn’t given them a good result for everyone, they look for any sub-population on which it seemed to work. If they find that the drug works on, say, women of Chinese ancestry, they rework the entire study into a trial about the results of their drug on Chinese women.

Now, Martin-Chang, Gould, and Meuse do not seem to have any political agenda driving their disaggregation of the homeschoolers into a high-performing group and a low-performing group, so I don’t want to accuse them of intentionally cooking the books, but it does seem very strange to me that they don’t even report at all their findings on the original question that inspired their study.

2. I found the distinction between “structured” and “unstructured” homeschoolers a bit unsatisfying, especially given how significant it is in this analysis. Here’s why. As any veteran homeschooler will tell you, there’s no hard-and-fast distinction here. There’s a continuum. Furthermore, researchers have consistently found that over time families tend to move along that continuum from structure to unstructure. This paper’s facile distinction misses both of those points. The paper also fails to consider length of time homeschooling among its subjects. Had the “structured homeschoolers” been homeschooling their entire lives? For just a few months? We don’t know, though the researchers themselves pointed out the significance of this question in their opening discussion of Rudner.

3. Finally, despite these significant misgivings, I want to applaud the study for trying to right some of the flagrant wrongs of nearly all previous studies of academic achievement. They’re not comparing a select group of homeschoolers to national averages like Rudner did and like Ray has done and continues to do (for a long treatment of Ray’s oevre see here and here). They’re not comparing public school kids who take their tests under stressful conditions in school, proctored by outsiders, to homeschoolers who take them in their living rooms with mom looking over their shoulder. They control for demographics. All of this is wonderful. If they had only given us the generic data comparing all homeschoolers to all public schoolers in their sample and given us data comparing the results of longtime homeschoolers with those who have only just begun, the results would be much more valuable. Of course larger sample sizes would be nice too, as would (pipe dream) a longitudinal follow-up in 10 years to see if the results from these young children remained consistent over time. But for all its failings, this study provides a great model future studies could build upon to get us some real data on homeschool/public school comparisons.

17 Responses

Although I understand the concern about the validity of the results, I was pleased to see the distinction made between structured and unstructured home schooled students. Our home school operates as a satellite school out of Christian Liberty Academy, so we are fairly structured. I don’t like being lumped together in studies with very relaxed home school families. Our philosophies may be akin, but the similarity ends there.

Thanks for the time you take to review and collect these things. I appreciate your perspective. Yes, it certainly would have been nice to get the overall numbers AND the breakdowns.

I’m curious about what you mentioned about homeschoolers getting less structured over time. Do you have any links to articles that discuss that? I’d also be interested to know how they define structure. There is a way of approaching those early years that seems less structured in that it encourages much exploration and response, that is still quite structured underneath it all. (Such as a Montessori approach, or Charlotte Mason). I think of that as “unstructured” but it probably is not. I think of ourselves getting more structured overtime as we demand more at 3rd grade, then 5th, then a huge bump at 7th for instance – perhaps we are actually defined as structured all the time according to their definitions.

The oldest (of ten) has done great academically and is entering college a year early with high ACT scores – I guess I’ll have to be on my guard against slipping in the future with the younger kids! It certainly would be nice to take a nap (that’s a joke).

Several studies have found a consistent pattern in new homeschooling moms. The first year they often try to reproduce exactly what the schools they left behind did–separate subjects, each with its own textbook, distinct periods and so on. Then as mothers get more comfortable with what they’re doing they feel freer to be spontaneous, or “eclectic” as many call it, and not as bound to the schedule. Here are some references, in chronological order of publication:

What you say about increasing structure when the children reach secondary education is also right on. It’s a big reason many homeschooled kids go back to institutional schools during those years or do a lot more “school-like” stuff like co-ops, community college classes, online courses, and so forth. What I was saying only really applies to the first three years of homeschooling for newbies.

Oh, I see what you mean then. I suppose it’s true that many homeschooling families move from school-imitation to eclectic over a few years (I started, and remain, a classical homeschooler).

I appreciate being informed about research like this; I don’t know that I would have heard about it otherwise, and it’s nice to have a study that isn’t from HSLDA, even if it’s kind of incomplete in ways. I hope this will be the start of many similar research projects.

It doesn’t say in the abstract how old the children in the study were. As far as I’m concerned, it goes without saying that unschoolers (and children who attend Sudbury schools and the like) would be academically behind more structured homeschoolers and even average schoolchildren, because that’s not the priority in their lives at that point. If you take this kind of snapshot at a time when they are focused on other things and academic learning isn’t a priority for them, all it tells is that they’re behind in that point in time. It’s like comparing apples with oranges. The really pressing question, however, would be what happens to them when they get to the age where their peers are going into into tertiary education or into the job market.

I agree with what you say about secondary education. I’ve seen this happen with several children, from both more structured and unschooling families, who go to high school, often for academic reasons in both cases. My son, who’s nearly 18 and has been unschooling ever since he was about 9, with very little “academics” is now studying math, physics, chemistry etc. so that he can apply to university to study music. Other unschoolers we know who have decided that to catch up with academics after years of unschooling have managed to do this in a relatively short time. There might be differences of opinion as far as my latter statement is concerned, but I think that studies like this which simply note the academic difference between structured and unstructured homeschoolers without looking at looking at the whole picture are a step beyond the likes of the old Ray and Rudner studies, but there’s still room for improvement.

By the way, Milton, if you liked Goldacre’s book, you should look at the blog Science-Based Medicine. It’s a goldmine of information on the skewing of science (and on how to do good science).

Thanks for the blog suggestion Rina. The kids in the study were all age 5-10. The researchers didn’t give a further breakdown by type of homeschooling, so it’s unclear if the unschoolers were all 5, all 10, or what.

Yes, I’d like to also add that unstructured homeschoolers frequently have high self motivation to become “structured” if their goals include university. I’ve known so many kids who weren’t really fluent readers at 10 or 11 as unstructured homeschoolers (so would seem to be academically not be doing well at that particular time), but by 17 were working their way through college reading lists and taking community college lab courses (and at that time would seem to be academically surging ahead of their public schooled and structured homeschool peers, who sometimes are dealing with burnout or unable to internalize their PARENTS’ motivations for them rather than working with self motivation).

At our co-op, the families are about evenly divided between structured and unstructured – but the unstructured are also highly functional families. They may allow their children to follow their interests, but these are parents who are actively facilitating. I must admit I find Vicki’s comment quite antagonistic. Being “lumped in” with “very relaxed homeschoolers”? Please! I think the distinction I see is more about whether the family values education and is functional. When they do – the results are great, regardless of amount of structure. If by relaxed, Vicki means families where there is not much going on – well I have seen that problem in “curriculum” families who are just going through the motions as clearly as I have seen it in “very relaxed homeschoolers.” If Mom and Dad are reading, taking kids to library, providing hands-on opportunities, providing critical thinking development, etc., then in the end, it’s all going to work out.

My three younger children didn’t start to read until they were at least 8. My daughter was 8 and a half before she could read more than a three letter word. Now, at 13, she takes 5 teen fantasy books out of the library and has them read in 3 days. If she, or her two younger brothers, who are also now reading fluidly although not as avidly as she is, had been involved in that study at the age of 8 (what I am assuming is the median age of children involved in it) they would have looked really behind.

I have four boys. They were all late readers. We did virtually no academic work until they were about 10/11. They began to read somewhere between the ages of 7-10. They would definitely have been in the “low” group had they been in this study.

Oldest son had an Ivy level SAT score, and just graduated with A average from a Canadian university, as a Theatre Directing specialist. His highschool math test scores were good enough to get him into sciences. He also took music at a university level. Son two was a PSAT commendation student, had a perfect Verbal SAT, and very high overall SAT. He also has an A average at uni, and will graduate in Dramatic Writing this December, a year early.

We moved from unstructured to more structured study because that was what suited our sons and their needs. While I understand your limited commendation of this study, again, I feel that it completely misses the point of what homeschooling does/can do for children–allow them to develop in a way that is conducive to ultimately healthy human development and adult success. Measuring the “achievement” levels of 5-10 yr. old homelearners is, in my opinion, a ridiculously futile and pointless exercise.

I was discussing this with a fellow unschooling mother and we were wondering what the four areas were where the unstructured homeschoolers were below grade level. I’m guessing that one of these is likely written expression. Can you tell us what they are?

Oh yes – let’s not even talk about writing. My college age sons did little significant writing until the high school years and never formally studied grammar or spelling. They each received one of a few precious A’s awarded in their classes in freshman comp at university – while admittedly having to work hard to learn some of the conventions.

However – what they had done was read, think, debate, draw, spend hours thinking through building things, organize tool and material lists, analyze music and poetry and advertising, etc.

One of the boys’ college profs commented on the relief his writing provided since it was not a “five paragraph formula essay.”

But if you’d asked them to write something when they were 10, their efforts would not have been at all comparable to their schooled peers.

This is, as they say, not a bug, but a feature.

This, Dr. Gaither, is why we don’t succumb to regulation that you keep suggesting we should not be afraid of. Our whole approach is simply totally un-grasped. To keep applying schooled notions of achievement to homeschoolers at specified ages is like trying to measure rain with yellow.

Perhaps schools have to do it because they are spending taxpayer money, and perhaps they should have to justify that such a non-organic, unnatural, one-size-fits-all institutionalize approach is not harming any more than the 1/3 of children we know it fails.

But we homeschoolers aren’t using tax payer money. We are taking responsibility for our kids. We know full well that having children in a rich environment with loving involved parents who work hard, take an interest in the world around them, demonstrate lifelong learning, and model literacy and numeracy, will provide the tools necessary for kids to meet their goals, without the pathology of stigmatizing them along the way for not reaching arbitrary benchmarks.

When the school system can say the same thing about its customers, I’ll consider their evaluation methods to be relevant.

I should be clear – my kids did do things like draw comic strips with dialogue, tell stories, invent elaborate games with story lines, etc. My strong sense is that all these things and others like them (including the things in my last comment) contributed far more to their being good writers as young adults than having been able to churn out sample sentences and paragraphs as second graders — because what they are doing is COMPOSING, you see. A lifetime of COMPOSING, and imagining and building confidence and content (you have to have something to SAY!) was so effective! The writing-things-out seems mostly to be a code you can get by reading and WANTING to generate the code when you are motivated to do so. And it turns out that allowing self-motivation to develop is also effective! Except if you jump in and try to take quantitative snapshots and force production of “work” that kids aren’t ready for.

Regarding Rina’s question above about what subjects the unschoolers were below grade level. The seven categories being assessed were letter-word recognition, reading comprehension, vocabulary, science, social science, humanities, and calculation. Unstructured homeschoolers scored below grade level on reading comprehension, social science, humanities, and calculation. Written expression was not assessed in this test battery.

Thanks, Milton. I didn’t see your last response there till now. It figures that reading comprehension would be later in unschoolers who are later readers and calculation too, particularly with respect to multiplication and division. I don’t know what the calculation tests look like, whether they’re word problems or mathematical statements, but just out of curiosity, I gave my youngest (who’s about to turn 9 in a week) the problem 40 (the symbol divided by, which I can’t write on the keyboard) 5. He asked, “What’s that?” and I said, “Divided by – it means how many times does 5 go into 40.” So he said “Two times is 10, 4 is 20, 6 is 30, so it’s 8,” and wrote down 8. Then I asked him what half of a hundred was and he said “50” and in response to the question of what a quarter of a hundred was, he said, “It must be half of 50, so it’s 25.” So the ability to do mathematical calculations at (or above) grade level can translate into an unschooled child producing results below grade level in a standardised test if they haven’t yet been taught the notation. BTW, when I asked him what 4/5 of 100 was, he reached the limits of his patience with me, but I know I could teach it to him in 10 minutes at this stage if I (and he) wanted to. We just don’t because neither of us feels it’s a priority right now ;-)

I would be much more interested in leaving school exam results such as the end of age 18. That would provide a truer picture of competancy. In my little lab of 5 unschooled children (which is 13.5 percent of their studies of 37 homeschoolers!) I asked my children to write the provinces standardized tests. They all scored in the acceptable range (50-85%) and some of them scored in the excellent range (85-100%) in some of the grade 6, and 9 subject tests.
In the grade 12 government exams, two children wrote and scored an average of 78% in seven exams. And they write them in public places and have them marked and supervised by government appointed people – not MOM!
If anyone else has had their unschooled children write standardized tests, please contact me at jarnall@shaw.ca as I am writing a book about this.
Judy Arnall