CHAPTER 9 Nonexperimental Social Science - Uncontrolled jim manzi

Imagine that a hypothetical US president is considering his options vis-à-vis Iran’s rapidly developing nuclear weapons program. First a science adviser enters the room and predicts that if the Iranians take a certain quantity of fissile material and compress it into a sphere of a particular size under specific conditions, then it will cause an explosion large enough to destroy a major city. Next a historian enters the room and predicts that if external attempts are made to thwart Iranian nuclear ambitions, then a popular uprising will ensue sooner or later, and force changes in governments until Iran has achieved nuclear capability. The president would be incredibly irresponsible to begin debating nuclear physics with his science adviser, even if the president happened to have trained as a physicist. Conversely, the president would be incredibly irresponsible not to begin a debate with the historian. This likely would include having several historians present different perspectives,Read more at location 1577

(1) a priori beliefs about human nature, and conclusions that are believed to be logically derivable from them, (2) analysis of historical data, which is to say, data-driven theory-building, and (3) a review of the track record of prior predictions made using the predictive rule in question.Read more at location 1595

Please view the following film taken from a long series of huge explosions that result when independent evaluators combine the materials I described in the manner I described.Read more at location 1598

Mankiw summarized fourteen findings that have achieved widespread acceptance among economists. Among them are: Fiscal policy (e.g., tax cut and/or government expenditure increase) has a significant stimulative impact on a less than fully employed economy. A large federal budget deficit has an adverse effect on the economy. A minimum wage increases unemployment among young and unskilled workers. In fact, 10 to 20 percent of practicing economists disagree with each of these assertions; but more fundamentally, even if we assume them to be correct, they are too vague to really help settle policy arguments.Read more at location 1617

although experiments can help and should be aggressively pursued, our scientific knowledge of any human social organization will remain extremely limited even when these experiments are deployed extensively.Read more at location 1644

Prior to the creation of modern social science, we simply had history, with its tradition of recording facts and making assertions based on these facts plus narrative appeals to commonly held understandings of human motivations and experiences. This was nonscientific, in that it did not make claims for the kinds of reliable, nonobvious, and useful predictive rules that characterize science. In the terms of this book, history is informed common sense.Read more at location 1663

numerous thinkers attempted to apply scientific methods to the study of human social behavior. The French Enlightenment, in particular, was central to the creation of the modern social science ideology. Auguste Comte and Henri de Saint-Simon were explicit in arguing that the methods of natural science provided the model for developing predictive laws for human society. Comte argued that human understanding in various fields proceeded in three stages: theological, metaphysical, and finally, positive.Read more at location 1667

Comte believed that humanity had achieved “positive” (i.e., scientific) understanding in various fields in the order of their complexity: mathematics, astronomy, physics, chemistry, biology, and finally sociology.Read more at location 1672

It was clear to the earliest social scientists that the natural sciences of their era (astronomy, chemistry, and physics) achieved spectacular success by discovering and stating physical laws as equationsRead more at location 1673

In an argument somewhat akin to Sir Karl Popper’s doctrine of falsification, Mill argued that one role of empirical observation is to “verify” a given causal rule by “by comparing,Read more at location 1681

For example, a social scientist might promulgate a predictive rule that a US president will fail to win reelection if the unemployment rate exceeds 10 percent. If, in a specific future election, a president were to win reelection with 11 percent unemployment, the social scientist might observe that there was a disturbing cause created by the fact that the nation was at war and the president was viewed as an indispensable leader.Read more at location 1685

The problem of how we can develop nonobvious, reliable predictive rules without controlled experiments has so far been deadly to Comte and Mill’s dream of rational social science.Read more at location 1696

The most widely discussed finding in the book was a regression analysis, based in part on an updated version of analysis from his 2004 academic paper, “Partisan Politics and the US Income Distribution,” which reviews the changes in incomes for the rich versus the poor under Democratic versus Republican presidents from 1948 to 2005. Bartels asserts that the differences in the behavior of Republican versus Democratic presidents have been a leading cause of the rich gaining relative income versus the poor, saying these presidential differences were “the most important single influence on the changing US income distribution over the past half-century.”Read more at location 1715

decisions by the Congress, Supreme Court, and Federal Reserve; changes in international economic competition; technological developments that enhance some people over others; changes in immigration rates and sources; changes in social mores and beliefs; and being at war or peace.Read more at location 1721

“Fortunately, from the standpoint of political analysis, the very fact that these social and economic trends have been gradual and fairly steady implies that their effects are unlikely to be confounded with the effects of alterations in control of the White House.”Read more at location 1729

he adds two variables to his equations to fit his curves to the historical trend of the data, rather than to explain this historical trend as a function of underlying causes. One of these trend variables is the number of years since 1948 for each year, and the final variable in the model is the square of the number of years since 1948 for each year. And that’s it.Read more at location 1742

assumption that out of all the potential confounding causes for inequality, only oil prices and female labor participation should be included in the model as specific causes, and that the model has captured all of the other possible causal factors through his “linear and quadratic trend terms.”Read more at location 1751

Using the raw Census data tables, I observed that income inequality does tend to rise under Republican presidents (lagged one year—e.g., Jimmy Carter gets credit for 1981) and fall under Democratic presidents (lagged one year). But when I did the simple test of changing the lag to two years, the entire apparent effect disappears:Read more at location 1760

he cites two academic papers that he believes show his assumption is “consistent with macroeconomic evidence regarding the timing of economic responses to monetary and fiscal policy changes.” But first note that a president can affect a far broader range of policies than monetary and fiscal policy—for example, regulatory decisions, Supreme Court and Federal Reserve appointments, negotiating trade treaties, antitrust enforcement, seeking out or settling wars,Read more at location 1766

And these papers don’t appear to claim a one-year rather than a two-year window for the impacts they do analyze. One paper estimates that (1) the peak impact of a tax shock on GDP should be reached by one to two years after the taxes change, and thereafter continue indefinitely; and (2) the peak impact of a spending shock should not be reached until two to four years after the spending change, and then continue indefinitely. The other paper estimates that numerous effects of monetary shocks extend for two years or more.Read more at location 1772

For this kind of a social reality, such model-tuning (for example, the one-year lag versus a two-year lag; including oil prices and female labor force participation versus the myriad other potential control variables; using a linear plus quadratic trend terms versus searching for additional explicit control variables, etc.) is inevitable, because the complexity of the real world overwhelms the tool of regression analysis.Read more at location 1780

Among the most widely discussed passages in Freakonomics was Levitt’s assertion that a significant fraction of US crime reduction in the 1990s can be linked to changes set in motion by Roe v. Wade in 1973. The basic asserted causal mechanism is that the increase in abortions disproportionately eliminated potential future criminals.Read more at location 1789

several fertility control technologies—most importantly the birth control pill—plus a huge variety of social trends that plausibly affect abortion rates and/or crime emerged in the same era as legalized abortion. The argument Levitt makes in his professional publications is that we can control for these other effects. But this is extremely difficult if these other effects became evident at the times, in the places, and for the population subgroups where abortion legalization had its first effects.Read more at location 1798

Freakonomics presents the results of a natural experiment: the five states that liberalized abortion laws prior to Roe (Levitt terms these “early legalizers”) experienced a crime reduction prior to the nonrepeal states.Read more at location 1803

The logic of the various regression analyses in the paper was (as with the Bartels analysis) to “hold all other factors equal” and isolate the causal change created by the change in the abortion rate.Read more at location 1812

other academics published alternative versions of the same analysis, using slightly different assumptions, that did not show any such effect. Levitt and Donohue, of course, quickly replied by arguing that one should use their preferred specifications.Read more at location 1815

two Federal Reserve economists published a crucial criticism in which they showed that the software implementation of the equations presented in DL 2001 had an important error and that once this was corrected and some other technical changes were made, the asserted effect of abortion on crime was no longer evident.Read more at location 1817

Other academics then attempted to replicate the same analysis for the effect of legalization of abortion in the United Kingdom. They also discovered that depending on the exact specification of data sets and assumptions made in the regression model, the effect on crime would sometimes appear, and sometimes not.Read more at location 1826

The fragility of the results in this paper serve to emphasize the difficulty researchers have in identifying causal effects of social change such as abortion legalization on crime rates some years hence, particularly given the myriad of other social changes occurring over the same time and which may dilute any effect.Read more at location 1830

Once again, regression analysis cannot tell us the effects of abortion on crime, because different reasonable assumptions for the analysis lead to completely different answers.Read more at location 1833

One way to get around all of this confusion would be to run an experiment. A purposeful experiment to force a random sample of states to implement abortion legalization has never happened in American history, and almost certainly never will.Read more at location 1837

New York declines 35 percent, while Alaska increases 50 percent; California is down 14 percent, and Hawaii is up 11 percent; Washington is almost exactly flat. The total rate across the early legalizers goes down versus the rest of the country only because New York and California are so much larger than the other three states. The natural experiment cannot resolve the question of the causal impact of abortion on crime, either.Read more at location 1857

Second, a national society is holistically integrated; therefore, it is hard to get causal impermeability between the test and control groups. In the abortion-crime debate, for example, I indicated that a significant technical issue was how to account for the reality that people move between states.Read more at location 1866

Third is the possibility of systematic, unobserved bias between the individuals or places that are subject to the treatment in the natural experiment as compared to those that are not. Consider the abortion-crime example. All kinds of plausible differences in political culture, social evolution, rational expectation for future challenges, and so on could vary between the early legalization states and the rest of the country.Read more at location 1873

This is the irreducible problem for any such social natural experiment that does not use strict randomization for assignment to the test population, no matter how large the same size.Read more at location 1880

The actual event that inspired this observation was that, one day in 1961, Lorenz entered .506 instead of .506127 for one parameter in a climate-forecasting model and discovered that it produced a wildly different long-term weather forecast.Read more at location 1887

Businesses are notoriously practical and results-oriented, and have sunk vast resources into trying to develop useful, reliable predictions for behavior in the absence of experiments. In doing so, they have run into the same problems and hit the same dead ends. I know, because I spent years doing it.Read more at location 1893