March 5, 2008

Theory III: Improve Household Incomes

Hot on the heels of the Minneapolis busing failure story, let's take a look at what the data shows with respect to the performance of low-SES attending high-SES school districts.

I ran three regressions for household income versus low-SES student performance. For trial one, I used the percentage of households with incomes greater than $75,000. For trial two, I used the percentage of households with incomes greater than $100,000. And, for trial three, I used the percentage of households with incomes greater than $150,000. All the regressions showed nearly identical results, so I'm only going to discuss the middle trial, trial two.

The theory is that low-SES students will perform better in high-SES school districts. Apparently, these high-SES districts attract better teachers, have better students with better motivation, have more resources, can pay more attention to the small number of low-SES students, and the like.

For this example I'm using community household income as a proxy for high-SES. Specifically, I used the percentage of households having incomes greater than $100,0000 as my measure of high-SES. The greater the percentage of higher income households, the higher the SES of the school district. Let's compare the performance of low-SES students to the percentage of high income households.

Here's the regression results:

R = 0.13 (there is a weak association between high household income school districts and the performance of low-SES students)

R2 = 0.016 (there is a poor fit between the data)

P = 0.007 (the results are statistically significant)

I added a trend line (black line) to the graph, but with a correlation this low, the trend is not reliable. I also added a horizontal line showing the mean of the performance for all school districts in Pennsylvania (mean = 60.7). As you can see, few school districts had low-SES student performance that exceeds the mean score. Low-SES students tend to be underperformers. RWP found that this underperformance was statistically significant.

Interpretation

The mean school district contains about 11% high-income households. This means that if we were to bus all the low-SES students from school districts with less than 11% high-income households to school districts with greater than 11% high-income households, most students would be sent to high-SES school districts having between 11% to about 25% high-income households. That's not much of an improvement and the regression suggests that low-SES student won't improve much anyway. Even if you were to believe the slightly positively sloping trend line, the expected gains in student performance would be minimal.

But there is good reason to question the trend line. Take a look at this graph:

This is a graph comparing high-income households to the fraction of moderately poor households in the school district. I've defined moderately poor households as households with between $15k and $29k of household income. The graph gives the percentage of moderately poor households relative to all poor households (household income less than $29k). As you can see from the graph as the percentage of wealthy households increases so does the fraction of moderately poor households. In other words, the higher-SES school districts have more low-SES students falling in the lower middle class as opposed to the lower or under class.

This is an important finding. The disaggregated data for low-SES (or "economically disadvantaged" under NCLB) students includes students receiving free or reduced lunches. A student can be from a family having a household income approaching $50k and still be included in the low-SES group. But, as the graph above indicates, not all low-SES kids are created equally. The low-SES kids in wealthier school districts tend to have higher household incomes. And, we already know that SES is positively correlated with student achievement.

Many books have been sold by the Kozol/Rothstein clique of SES hustlers who claim that our education woes can be cured if only we could send these kids to high-SES school districts. These claims are founded on the error of ascribing causation to a weak correlation (the top graph) and failing to account for the substantial intra-group differences (the bottom graph) of students defined as low-SES or economically disadvantaged.

13 comments:

I think it's worth remarking that nobody who does not already agree with you will be convinced by this highly localized data.

There are numerous reasons to think of Pennsylvania as a special case in education generally - everything from Edison Schools on forward - so much so that I actually have a special category for it in my database.

And even were this not the case, it is highly unlikely that you can generalize to the entire world from the data in Pennsylvania.

Stephen, my goal is not to convert the zealots and true believers, they are beyond my reach. Mere data does not suade them. My goal is more humble-- to prevent them in the first place.

I never claimed that this data is generalizable to the rest of the world. No doubt this analysis doesn't apply to the Congo. But, I would bet that it generalizes to most other US states and many other western countries. All you need is a spreadsheet and a data file (I know where 49 other files are located) to prove me wrong.

I believe that Edison only has schools in Philly and Chester. And, they perform like public schools, i.e., not very well. If anything, PA would be a state would be the perfect counterexample-- well funded schools, strong teachers' union, big salaries, large state funding of low-SES schools. Am I missing something?

But, I'll take you up on the challenge. Pick another state that you think is less the exception and I'll run the analysis as long as there is demographic data and test scores.

None of the studies or reports I've seen have shown any significant overall academic improvement for low SES students who are allowed to attend wealthier schools and/or private schools.

This has always been one of my arguments against busing, which tends to be seen as a nice, liberal panacea to fix the awful educational outcomes of low income students.

As a law student, I found many of my colleages (in an education law class) firmly resistant to the idea that busing doesn't improve the academic performance for low income kids. They just couldn't believe it, and the evidence couldn't convince them.

As a former teacher (now lawyer), in my opinion most of the low academic performance of low SES kids has to do w/ a combination of poor parental support, community values that place little value on education, media/rap messages, peer pressure, lack of effort and disruptive behavior. It's not just a simple fix.

Better teachers, more money for the schools, etc. will help the problem, but not eradicate it. Any suggestions to repair these root problems? That would be a great study.

in my opinion most of the low academic performance of low SES kids has to do w/ a combination of poor parental support, community values that place little value on education, media/rap messages, peer pressure, lack of effort and disruptive behavior. It's not just a simple fix.

I think that many of these are effects rather than causes-- effects of historically and individually performing poorly in school.

I think that much of these effects can be reduced by adopting better teaching practices and reducing the incidence of early academic failure in the first place.

See the improvement of affective measures for the DI model in Project Follow Through. In a recent article in the DI news on motivation, Engelmann claims that motivation problems in school should be viewed first as instructional problems.

In PFT, All the students were tested with the Coopersmith Self-Esteem Inventory and the Intellectual Achievement Responsibility Scale (IARS). The Coopersmith measures children's feelings about themselves and school; the IARS measures the degree to which children take responsibility for their successes and failures.

The DI students scored significantly higher in these affective measures than the other students.

Here's Engelmann's interpretation of those results. "[w]e assume that children are fundamentally logical. If we do our job of providing them with experiences that show they are smart, they will conclude that they are smart. If they experience success in school that can also be measured in the neighborhood, those experiences serve as fuel for the conclusion that students are competent. At the time of the evaluation, I had heard more than 100 stories of our children helping older siblings learn to read or do homework. The children knew that they could do things the average kid on the street could not do."

Kderosa: Thanks for your response to my request for information. I see from the report that certain teaching strategies (DI) appear to have positive results in helping elementary students at risk for failure. My experience was in HS and junior high, so I was not previously familiar with DI techniques.

However, this doesn't change my main point that SES level and other personal (rather than school) characteristics have a huge influence on students' academic achievement.

I think you and I agree that such at-risk students CAN be helped to a greater or lesser degree through intensive educational intervention. However, my point is that these strategies are largely unnecessary for students with better SES backgrounds. That is, it takes a huge amount of educator effort to make up for these SES disadvantages.

Also, such interventions may not hold up over the years, as students succumb to high school peer pressure and other influences. It's a battle.

As a former teacher of low-SES students, I get angry when it appears that the public blames the teachers for not being able to quickly fix all of the multiple problems that come along with many of our low-income students. It takes strong teaching, strong administrative support, etc. to make headway with these kids, and to keep up the gains. The same teaching effort isn't required to get good test scores from high SES kids (though they can have their own set of problems!).

However, this doesn't change my main point that SES level and other personal (rather than school) characteristics have a huge influence on students' academic achievement.

There is a correlation, but that does not imply a causation. The question is "can these influence be overcome, at least partially, by improved teaching" I think the answer is yes if the teaching is improved from early on.

Also, such interventions may not hold up over the years, as students succumb to high school peer pressure and other influences. It's a battle.

The DI intervention, which only lasted until third grade, did have a lasting effect on many students.

It takes strong teaching, strong administrative support, etc. to make headway with these kids, and to keep up the gains. The same teaching effort isn't required to get good test scores from high SES kids (though they can have their own set of problems!).

I agree with this. Teaching low-SES kids is a job of first magnitude and requires excellence in the entire school structure, including the management. I do not think that many schools, expecially those run by political entities, are up to the task.

Indeed, I would go so far as to suggest that the use of statistics in the last few posts would offer support *only* to zealots and true believers.

Backing up this statement might be more persuasive.

This may be so. But the fact remains, you haven't proven it.

Nor have you proven your position. All you need to do is find one counterexample.

Give me some time, I'll ad to the list of analyzed states.

I pick... Finland.

Cute. I read that article too. More opinion than hard data though. Point me to the dataset and I'll run the analysis. I'd expect to see Finnish performance similar to, say, the performance of white Minnesotians. But, I wouldn't expect Finnish performance to gernalize to a more diverse group such as found in most of the rest of the US.

I have already observed that the data is for one year for a single state. I need no more data in order to back up my statement. The narrowness of the survey you cite is sufficient to invalidate it.

>>This may be so. But the fact remains, you haven't proven it.

>Nor have you proven your position. All you need to do is find one counterexample.

The only position I am advancing here is that your data are insufficient to support your conclusion.

You can't draw me into a scenario where I do your research for you.

If you don't have the data, you don't have the conclusion. period. There's no onus on me to 'disprove' what you've asserted.

> Point me to the dataset and I'll run the analysis.

Again, there's no reason why I should do your research for you.

--

The fact is, this entire series of posts generated form the Pennsylvania 'data' are complete fabrications.

You have picked some data you say (without proof) is generalizable, and you are drawing an entire series of spurious conclusions from it.

Nobody with any degree of critical reasoning ability will be convinced by such data. The arguments you are offering are non-sequeters: the conclusions simply do not follow from the evidence. This is a point of logic, a point of reason.

Which means, the only people that would be convinced by the arguments you are offering are those to whome reason holds no sway, one way or another. In other words - the people who have already bought into the snake oil you are selling, the zealots and true believers.

Oddly enough PA has twice the population as Finland and your PISA data is only for two years, yet that hasn't stopped you from using Finland as your poster child for your theories.

Actually, I performed a similar analysis for 2002 PA data (posted on this blog). Not unsuprisingly, it showed the same thing. A trend is born.

And just so we're clear, this data doesn't "prove anything." these are just correlations. But the data can be used as a counterexample to refute the theories based solely on correlation data, i.e., all of them.

And, I do expect every other state to show the same corelations because that's what was found in the Coleman report.

I continue to find your dual standards of proof amusing. For theories you agree with, you are willing to accept almost any data regardless of the infirmities. For theories you disagree with, almost no data is sufficient or acceptable. I'm sure you understand why this is a problem, a common problem among the true believers.

About D-Ed Reckoning

The primary problem with K-12 education today is the problem of dead reckoning--an estimate based on little or no information. We don't know what a good K-12 education system is because we've never seen one operating. A good education system is one that is capable of educating almost every child.