www.elsblog.org - Bringing Data and Methods to Our Legal Madness

03 January 2018

A recent thread on the Stata listserv alerted me to a user-created command, radar, that generates "spider" graphs (see, e.g., below). The radar command "produces a radar plot from at least two variables. The first variable must always contain the label for the axes and the second variable must be numeric."

While I'm not entirely sure how such graphs can be deployed effectively, I was delighted to learn that this graphic potential existed. For those interested, just use the "findit" command to locate, download, and install the "radar" graphics module.

"In this paper, we develop a new method that uses empirical Bayes techniques to more accurately incorporate the individual-specific information. While some non-injurious factors will always be unobserved, it is possible to estimate the percentage of the variance due to these factors. Given this estimate, our method calculates individual-specific damages by using empirical Bayes methods that combine a general estimate, based on a regression coefficient, with individual-specific estimates based on regression residuals. It thus combines many of the best features of previous approaches and results in considerably more accurate payments than existing statistical procedures."

14 December 2017

What began as a "good idea" and complemented other resources has quickly evolved into an invaluable stand-alone resource for novice and experienced Stata users alike. The current collection in Stata's instructional video library (hosted on YouTube) is now both wide and deep; I recommend this resource to all.

08 December 2017

As the "logit" command is quite common in ELS and as many political scientists have embraced marginal effects, I thought this recent and helpful exchange (here), featuring concrete coding suggestions, might interest some readers.

29 November 2017

Interesting discussion (here) about various motivations for and objectives of robustness checks. Lurking between the lines of the discussion is an effort to distinguish "standard" robustness checks from "replication" efforts. As Gelman (Columbia-Statistics) notes: "For example, maybe you have discrete data with many categories, you fit using a continuous regression model which makes your analysis easier to perform, more flexible, and also easier to understand and explain—and then it makes sense to do a robustness check, re-fitting using ordered logit, just to check that nothing changes much." From my way of thinking, robustness checks evidence (to editors or readers) that a paper's main substantive findings are not solely "model- or assumption-specific."

13 November 2017

Increasingly "fed up" the concepts of "'power,' 'type 1 error,' and 'type 2 error,' because all these are defined in terms of statistical significance," Andrew Gelman (Columbia--Statistics) developed and advanced "Type S (Sign) and Type M (Magnitude) Errors." Gelman, conceding that "concepts of Type S and Type M errors are not perfect," nonetheless concludes they represent a helpful "step forward." Gelman (with John Carlin) described his thinking in a 2014 paper, Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. As well, the paper is summarized in a more recent blog post (here).

While few are ready to completely jettison more traditional power testing, Gelman and Carlin's alternative "Type S and Type M" struck many as a useful complement. Additional evidence of the possible promise of "Type S" and "Type M" tests is found in the recent release of Daniel Klein's user-written Stata command add-on, entitled rdesigni. (For some background on the new rdesigni command click here for a recent list-serv discussion.)

Those already equipped with Stata and wanting to explore this new command need only execute to following commend (with Stata launched):

11 October 2017

An asymmetric relation between CIs (confidence intervals) and statistical significance constantly trips up students (and others). A quick-and-dirty summary follows (a fuller discussion, with helpful examples, is found on the Stata Blog here).

In short: if the individual CIs do not overlap, then the difference is statistically significant. However, the reverse is not necessarily true. That is, CIs for individual parameters can overlap even if their difference is statistically significant. What is also true is that if the difference is statistically significant, the CI for the difference will exclude zero.

07 September 2017

In a scenario described in a recent exchange on the Stata List, a researcher seeks to accomplish a discrete task: specifically, to "estimate some marginal effects for different income levels" (and in the coding the "group" var identifies the various income groups ["group==1", group==2", etc.]). The researcher correctly identify three separate--though similar--coding approaches toward the desired task. The three separate coding approaches yield slightly distinct sets of results (in the scenario, two). Consequently, as a commentator notes, the correct coding approach for any given task "depend[s] on your research goals, [and] you have to choose the one that fulfills them." One important take-away is that is often critical to understand how subtly different coding approaches--each pursuing a common objective--can yield results that differ in critical, yet subtle, ways. As a commentator observes: "We are all guilty, much of the time, of throwing around the term 'marginal effects' without specifying which of the infinitely many different marginal effects are associated with any model" (emphasis added).

22 July 2017

To quickly summarize a discussion on the Stata Blog (here), for those seeking to test whether coefficients for two independent variables in the same regression estimation systematically differ from one another, the basic coding structure follows:

reg a b c

test b = c

A resulting F-test statistic that falls below your significance threshold implies that the difference between the coefficients for independent variables b and c is significant. Obviously, a suite of other post-estimation commands also exists for a host of other issues.

14 July 2017

While I haven't yet had a chance to use it myself, the recently-released Stata v.15 includes the npregress command. Nonparametric regression, similar to the more traditional parametric regression, predicts a mean of an outcome for a given set of covariates. What separates parametric and nonparametric models, however, includes assumptions about the functional form of the mean conditional on the covariates. In particular, unlike parametric models, "nonparametric regression makes no assumptions about the functional form." Thus, output from nonparametric models supports valid inferences "regardless of the true functional form."

A recent Stata Blog post (here) describes the new command in much greater detail and provides helpful comparisons illustrating output differences between parametric and nonparametric models.

20 June 2017

While I've always believed that some qualitative techniques can complement quantitative work in many important ways, conventional wisdom too often positions qualitative and quantitative scholarship as somewhat at odds with one another. Moreover, over the years we've witnessed a significant migration of empirical legal scholarship from student-edited law reviews to faculty-edited peer-reviewed journals. Positioning itself at the intersection of these two points, Qualitative Methods for Law Review Writing outlines how legal scholars writing in law reviews can use qualitative methodological techniques from political science, sociology, and history to help support more generalizable, causal claims. In particular, the authors, Katerina Linos (Berkeley) and Melissa Carlson (grad student--political science), emphasize how legal scholars can use sampling and case selection techniques, as well as "process tracing," among other methods, to more effectively assess their claims' validity. While this paper is aimed at law review writing, its general point warrants broader attention. The paper's abstract follows.

"Typical law review articles not only clarify what the law is, but also examine the history of the current rules, assess the status quo, and present reform proposals. To make theoretical arguments more plausible, legal scholars frequently use examples: they draw on cases, statutes, political debates, and other sources. But legal scholars often pick their examples unsystematically and explore them armed with only the tools for doctrinal analysis. Unsystematically chosen examples can help develop plausible theories, but they rarely suffice to convince readers that these theories are true, especially when plausible alternative explanations exist. This project presents methodological insights from multiple social science disciplines and from history that could strengthen legal scholarship by improving research design, case selection, and case analysis. We describe qualitative techniques rarely found in law review writing, such as process tracing, theoretically informed sampling, and most similar case design, among others. We provide examples of best practice and illustrate how each technique can be adapted for legal sources and arguments."

06 June 2017

As structural equation models have largely overtaken what was once called two- (or three-) stage least squares models, a recent YouTube tutorial (here) could not be more timely or helpful. As Chuck Huber is the presenter, the tutorial is illustrated with Stata commands. Particularly intriguing are the robust graphical possibilities (e.g., path diagrams). While almost two hours long, this tutorial is well worth the time investment.

11 May 2017

As legal scholars begin to digest AI's ramifications for law and legal analysis, a recent paper exploits the Supreme Court Database to illustrate AI's ability topredict case outcomes. In A general approach for predicting the behavior of the Supreme Court of the United States, Daniel Katz (Chicago-Kent) et al. explore almost two centuries of Supreme Court decisions and use an algorithm to successfully predict 70.2% of the case outcomes and 71.9% of individual justice-level votes. The abstract follows.

"Building on developments in machine learning and prior work in the science of judicial prediction, we construct a model designed to predict the behavior of the Supreme Court of the United States in a generalized, out-of-sample context. To do so, we develop a time-evolving random forest classifier that leverages unique feature engineering to predict more than 240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015). Using only data available prior to decision, our model outperforms null (baseline) models at both the justice and case level under both parametric and non-parametric tests. Over nearly two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the justice vote level. More recently, over the past century, we outperform an in-sample optimized null model by nearly 5%. Our performance is consistent with, and improves on the general level of prediction demonstrated by prior work; however, our model is distinctive because it can be applied out-of-sample to the entire past and future of the Court, not a single term. Our results represent an important advance for the science of quantitative legal prediction and portend a range of other potential applications."

12 January 2017

While in the legal domain such research design issues typically involve law and psychology papers, the "conventional wisdom" dwells on standard external validity challenges. As Andrew Gelman's (Columbia--Statistics) post makes clear, however, other challenges also lurk. That is to say, while a randomization strategy is invariably preferred, it cannot alone eliminate all research design concerns. As Gelman helpfully notes, "there’s more to inference than unbiasedness."

03 January 2017

Lagging variables is frequently justified (or even necessary) in empirical legal research. This is particularly so in event-study research that seeks, e.g., to explore the influence of a new law, statute, court decision on various outcome variable(s) of interest. In such contexts researchers need to account for a natural delay in an hypothesized intervention's effect. This topic was raised in a recent Stata blog post (here) that also references relevant Stata coding.

As explained in the comments (slightly edited): "As for the reason behind it [lagging a var], ... this is a scientific, not a statistical issue.... Just generally, it is often the case when longitudinal data is available, that one expects the effects of one variable on another to appear with a delay. That is, this year's value of Y may depend on last year's value of X rather than on the current value. Indeed, it can be more general: sometimes one expects Y to depend on a value of X from several years back, or even jointly on more than one past year. Just when those situations arise depends on the subject matter and the science."