You were saying something about best intentions

Michael Simkovic thinks I don’t understand how grant-funded research works. He also thinks I criticized him for failure to disclose that the S&M million-dollar degree articles were funded by Access Group and LSAC. I didn’t criticize him for failing to disclose this, since I never claimed these articles was funded in that way:

So great is [Access Group’s and LSAC’s] enthusiasm for these findings that they are, as the intrepid scamblogger Dybbuk reveals, generously funding their further propagation. (emphasis added)

As for how grant funding works, Prof. Simkovic is much troubled by the suggestion that the sources of his funding could affect the character of what I believe are referred to in the relevant literature as “deliverables:”

Frank and I are interested in methodological rigor, not in particular results or outcomes, which in any case are unknowable until after we analyze the data. We believe in maximizing the transparency of the methods we use for our research so that it can be replicated or challenged by future empirical researchers. There has never been any effort by LSAC or Access Group to influence or censor our results.

Yet Prof. Simkovic and his co-author have levied exactly the same criticisms regarding how the funding of research might influence the conclusions reached that I suggested might be applicable to their work:

“Two professors at Seton Hall University, Frank Pasquale and Michael Simkovic, have been tracking studies released by Lumina-funded think tanks that criticize federal student lending, which is Sallie Mae’s direct competitor. For example, Lumina has given the New America Foundation, a nonpartisan think tank that focuses on issues ranging from national security to technology, nearly $3 million since 2008… “It’s hard to make sense of a lot of what Lumina is advocating on student loans unless you think of how it would benefit Sallie Mae,” says Michael Simkovic, an associate professor at Seton Hall… There is “no connection at all” between Sallie Mae and The New America Foundation, said Kevin Carey, director of the Education Policy Program at the New America Foundation, who said two of the three studies that criticized federal student loans were funded exclusively by the Gates Foundation.”

“A number of recent Brookings studies have been singled out for criticism by academics and others… One study, released in June, analyzed Federal Reserve Board data that tracks student debt and income levels in young households to conclude that typical student borrowers were no worse off now than they were a decade ago and that reports of a student debt crisis may be overblown. The study contradicted arguments from critics of the for-profit student-loan industry… Michael Simkovic, a visiting associate professor of law at the University of North Carolina at Chapel Hill and an expert on lending issues, said that if Brookings’s reports on student debt were to dictate policy, they would “boost the profits of the student lenders like Sallie Mae.” Critics pointed to a potential tie between Brookings and the lending industry: $1.9 million in donations since 2009 from the Lumina Foundation… Lumina did not underwrite the June study, according to co-authors Matthew M. Chingos and Beth Akers…. Lumina’s director of strategy, Zakiya Smith, said suggestions of meddling were “mind-boggling.”

Prof. Simkovic also says the money from two groups who are, to put it mildly, eager to see certain sorts of results from S&M’s further work is only going toward their salaries, as well as some research costs, so what’s the problem? This is a remarkably naive (to put it charitably) interpretation of what benefit Seton Hall and S&M are getting, respectively, from the grant-funding of this research. Grant-funding of their salaries, or part of their salaries, takes that money off the rest of Seton Hall Law School’s budget, which, given the fact that the school was recently on the verge of laying off its untenured faculty (including, presumably, Prof. Simkovic), adds up to a pretty powerful incentive to come up with the right results at the remains of the day.

But as the quotes from him above illustrate, he understands all this perfectly well, or at least he does when it’s not in his self-interest to forget it. (“It is difficult to get a man to understand something when his salary depends etc.”).

Prof. Simkovic gives some anecdotal examples, off the top of his head, of grant-funded law school research, to illustrate that I supposedly don’t understand how common such research is. My statements in regard to this aren’t made off the top of my head: they are based on studying the budgets of several dozen law schools over the past couple of years. Those budgets reveal that grant-funding, on average, covers a tiny percentage (usually much less than five percent, and often less than one percent) of these schools’ operating costs.

*An amusing side note: Prof. Simkovic doesn’t actually link to LGM, but rather to a PDF of my post. Apparently the landlord who is leasing his cyber-real estate out for Prof. Simkovic’s benefit cannot bear the thought of providing LGM with any link love.

Post navigation

My favorite takedown of ideologically-driven grant funded research came from Judge Bernard Friedman in the Michigan marriage equality case. Speaking of the work of Mark Regnerus to support the notion that male-female couples produce healthier children than same-sex couples, Judge Friedman took Regnerus apart:

The Court finds Regnerus’s testimony entirely unbelievable and not worthy of serious consideration. The evidence adduced at trial demonstrated that his 2012 “study” was hastily concocted at the behest of a third-party funder, which found it “essential that the necessary data be gathered to settle the question in the forum of public debate about what kinds of family arrangement are best for society” and which “was confident that the traditional understanding of marriage will be vindicated by this study.” See Pls.’ Motion in limine to Exclude Testimony of Mark Regnerus, Ex. 9. In the funder’s view, “the future of the institution of marriage at this moment is very uncertain” and “proper research” was needed to counter the many studies showing no differences in child outcomes. Id. The funder also stated that “this is a project where time is of the essence.” Id. Time was of the essence at the time of the funder’s comments in April 2011, and when Dr. Regnerus published the NFSS in 2012, because decisions such as Perry v. Schwarzenegger, 704 F. Supp. 2d 921 (N.D. Cal. 2010), and Windsor v. United States, 833 F. Supp. 2d 394 (S.D.N.Y. 2012), were threatening the funder’s concept of “the institution of marriage.”

While Regnerus maintained that the funding source did not affect his impartiality as a researcher, the Court finds this testimony unbelievable. The funder clearly wanted a certain result, and Regnerus obliged. Additionally, the NFSS is flawed on its face, as it purported to study “a large, random sample of American young adults (ages 18-39) who were raised in different types of family arrangements” (emphasis added), but in fact it did not study this at all, as Regnerus equated being raised by a same-sex couple with having ever lived with a parent who had a “romantic relationship with someone of the same sex” for any length of time. Whatever Regnerus may have found in this “study,” he certainly cannot purport to have undertaken a scholarly research effort to compare the outcomes of children raised by same-sex couples with those of children raised by heterosexual couples. It is no wonder that the NFSS has been widely and severely criticized by other scholars, and that Regnerus’s own sociology department at the University of Texas has distanced itself from the NFSS in particular and Dr. Regnerus’s views in general and reaffirmed the aforementioned APA position statement.

[pp. 13-14]

Note: after listening to Regnerus’ testimony, including cross-examination, and after lining it up with the email record and other documents related to the funding process, Friedman didn’t say Regenerus’ work was “unpersuasive” but that is was “unbelievable and not worthy of serious consideration.”

The same might be said of Simkovic here.

Of course, if Simkovic wants to offer his email trail and grant funding documentation for examination to support his assertions of independence, we could probably find some folks who would take the time to read it and offer a few thoughts on it.

In avoiding providing a link to your article, he violates the copyright on your work. If you wanted to be mean, you could issue a DMCA takedown notice to Typepad.

Peterr

Thus leading to the inevitable “I’m being repressed! I’m being repressed!” chant.

Now a note to the dean at Seton Hall, on the other hand, might bring about a rather amusing conversation (amusing from the LGM point of view, of course). “Is this how your faculty typically engage in academic give-and-take, by avoiding direct conversation?”

Of course, from Simkovic’s POV, this would be seen as being mean, too.

JustRuss

amusing from the LGM point of view, of course)

I’m not sure how one can work ketchup, vodka, or dead horses into that conversation, but I’m not about to tell this crowd it can’t be done.

Peterr

“. . . One might speculate on whose grants have been funding LGMs research into dead horses, or the superiority of certain condiments, or the appropriate alcohol on which to build a martini. But people are not panting over their vodka-ketchup martinis worrying that if there are insufficient numbers of dead horses, it will end up shuttering the blogosphere.

“But there are plenty of folks worrying that law schools are driving a non-trivial number of their graduates into penury, with increasing loan amounts and decreasing numbers of jobs of sufficient quality so as to repay those loans. As the data piles up, one can see where a single outlier study that concludes “there’s nothing to see here — move along, move along” might be looked upon with a bit of a jaundiced eye — especially when the author of that study takes pains to attack his critics in a manner to make meaningful back and forth much more difficult. Is it Seton Hall’s style NOT to engage critics directly? . . .”

Or something like that.

I’ll leave it to others to work the elimination of the Air Force into the discussion.

(Aside: there is not nearly enough grant funding for vodka-ketchup-dead horse research projects.)

Unemployed_Northeastern

In other recent news,

“Years of using a Harvard nameplate to flog his insistence that polar bears are doing fine, and that sunspots might explain planetary warming better than the Industrial Revolution does, may finally have caught up with Wei-Hock Soon.

Mr. Soon, an astrophysicist at the Smithsonian Astrophysical Observatory, has endured a barrage of news reports this week detailing his acceptance of $1.2-million in support from energy companies and others hostile to government limits on fossil-fuel use.”

Prof. Simkovic gives some anecdotal examples, off the top of his head, of grant-funded law school research, to illustrate that I supposedly don’t understand how common such research is. …grant-funding, on average, covers a tiny percentage (usually much less than five percent, and often less than one percent) of these schools’ operating costs.

The old “I know this one guy, so all your data and statistics are meaningless” argument. Sadly, it tends to show up in academia more than one would hope. I can never tell if it’s due to weaselitus or sincere “I really want this to be true” stupidity.

Crusty

I would imagine that there’s some BS ethical reason not to do this, but I’d like all the professors at the crap law schools propagandizing that law school is a wonderful way to spend money you may or may not have to put together a fake resume, where the main credential is that they have just graduated from the law school they teach at. Don’t give him bottom of the barrel grades, but don’t make him or her top 10% or law review. Maybe, top quarter and some experience on a non-law review journal. Undergrad from the large respectable state university in which the law school is located. Apply for jobs, through on campus recruiting (if it exists) and by sending out resumes and cover letters. See how that goes and report back.

Tracy Lightcap

Well, that’s actually a pretty good idea. You couldn’t use the actual profs; they’d never give their permission. You could send out resumes with made-up names, however, and see what kind of a return you get. Making the names coincide with different ethnic groups would be a nice touch too.

You might have a little trouble with getting that by an IRB, but you aren’t requiring a response; it’ll be up to the organizations involved. There’s a nice dissertation in there for somebody who’s enterprising.

Lee Rudolph

There’s a nice dissertation in there for somebody who’s enterprising

and is willing to antagonize a whole slew of already probably pretty unhappy (all things considered) law professors.

Tracy Lightcap

Same song, second verse. Look, S & M did some research that had results some people with a pecuniary interest found compatible and they offered to give them grants. Like I said earlier: Stop the presses!! Get out the 64 point type!! This kind of thing is about a common as pig tracks in the woods, as we say down South. Their original paper was a well constructed piece of research and didn’t say anything unusual: using retrospective data, they found that people with advanced degrees, like an LLB, have done better in terms of income over their life spans then people who don’t have an advanced degree (duh). Their new piece, still obviously in draft, is also fairly innocuous, until you get to the part on discounting. I pretty sure they’ll end up doing something about that as well as some of the other parts of their model.

I think that saying their new results are tendentious because of who helped fund their research is a cop-out. If the results aren’t valid, then tell us why. If it turns out that a close analysis of their new work shows that S & M are cooking their results to suit their funders, then let them have it. You’ll have more then an ad hominem argument to work with.

And, btw, same goes for Simkovic: his “Sallie Mae is the Devil” comments aren’t worth the ink used to print them. This is, btw, the only thing that makes me suspicious of his research so far.

Unemployed_Northeastern

“And, btw, same goes for Simkovic: his “Sallie Mae is the Devil” comments aren’t worth the ink used to print them. This is, btw, the only thing that makes me suspicious of his research so far.”

Really? The fact that his “law school grads’ earnings aren’t affected by graduating into a recession” study directly contradicts more than a half dozen other recent labor economics studies concerning lower lifetime wages for recession-era college, law, and Stanford MBA graduates doesn’t bother you? How about his Million Dollar Degree presumptions that law students earn $24k while in law school (only possible if you get a BigLaw SA position, which go to a small minority of law students) or that the average student debt load is a mere $85k (more like $140k, exclusive of interest on unsubsidized loans, bar loans, and undergrad debt)? Or that the 25th percentile of college graduates is the same cohort as the 25th percentile of law school graduates? I was unaware that people who could barely get through Corinthian could get into and out of an accredited law school. For the period that his study covers, virtually all law schools had competitive admissions. If the cohorts are the same, why does the US have about 50 unaccredited law schools? Who do they serve?

Tracy Lightcap

Thanks for raising some substantive issues, but I’m not going to get into a kerfuffle about a draft paper. No doubt there will be revisions to their new piece and some of them might address your concerns. That’s what happened last time. We’ll just have to see what they say. You might bring these concerns up somewhere besides here and see what happens.

As to the Million paper, if I’m not mistaken S & M addressed these concerns in the last go-round. You can check their responses to critics and see; I’m not worked up enough about what I considered a “duh” set of results to do that myself. I will note, however, that my son earned right at $24K in law school working for a 2 person firm in Atlanta, so it is possible, albeit rare, to do it without grabbing the Gold Ring. (He did graduate with substantial debt, however, which he is now paying off.) No doubt he was the exception that yadda, yadda, yadda.

I wish all the responses here had been like this one. It might very well be that S & M are cooking the books and raising empirical issues is the way to find out.

Unemployed_Northeastern

“You might bring these concerns up somewhere besides here and see what happens.”

I’m not sure what critiques of S&M you’ve come across, but mine is pretty elementary. Take a gander at what Matt Leichter and Steven J. Harper have written in The American Lawyer and on their own websites. Both are lawyers who possess graduate degrees in the social sciences (as opposed to Simkovic, who merely has a BA and a JD); Harper’s master’s degree is in economics and he was a partner at one of the nation’s largest law firms for 30 years, Kirkland & Ellis. They have really done a number on the Million Dollar Degree study in terms of its methodology, outcomes, and prediction that law is not undergoing structural change. S&M’s replies, to the extent they gave them, are almost wholly unsatisfactory to my mind. Deborah Jones Merrit, a law prof at OSU, has also done some good work in this regard.

Did S&M make any attempt to adjust for the prestige of various law schools? (of which USN, for all its faults , is a decent proxy) They didn’t did they? I think they just put this in the “too hard” category. Instead they just sort of implied school prestige doesn’t have much impact on individual outcome. How… convenient, considering Seton Hall is a fairly badly ranked (and grossly overpriced) school.

Tracy Lightcap

Actually, they did do that in the Million Dollar study; they know that law school quality counts. I haven’t looked at the new draft study carefully enough to see if they did it there.

petesh

Well, the company in hell is likely to be stimulating.

OhioDocReviewer

It’s all too fitting that Simkovic and McIntyre can be referred to in shorthand as S & M.

(Yes, I know it’s from the photo, but I’m loving the style of that picture.)

Tracy Lightcap

NE, see my reply above. The site engine shunted me to the wrong place again.

gertrudesays

Also, if you laugh at this you’re going to Hell.

Of course I am, silly!

observer

Whatever happens in this industry, and whatever happens in back-biting academia, Paul Campos should (god willing and the creeks don’t rise) come out the better for it. You’re a good man, Professor, and the world needs more of them.

MacK

Personal history.

I started my private practice legal career doing a mix of international trade (Section 301, Special 301, §337) and Antitrust/Competition (US and EU.) In one early matter we were having a quick status meeting and one partner asked “has anyone called the whores yet?” and “have we picked a whore?” I was a little startled and wondered what he meant; it turned out that he was referring to the economics expert witnesses. Later, at depositions, each side’s economists would attend the deposition of the other – and I expected some tension, because they had both taken such diametrically opposite views, argued with such vehemence, that I thought they would argue. But they were perfectly friendly, none gave a toss what the other guy had said, some were on the same faculty. Later yet, I remember having to ask an economic expert to tone down his report in support of our side of a case, on the basis that it was so “over the top” as to be ludicrous. One expert describes his experiences here: http://www.legalaffairs.org/issues/March-April-2003/review_marapr03_moss.msp

In trade cases it matters a lot to produce articles and studies supporting the outcome your client wants (the Washington Post and BNA at least picks them up, and publishes articles that are often one long quote.) Economists and law professors are regularly hired (or suppoted) to produce these sorts of papers, often published in a way the obfuscate’s the backer. PhARMA (the lobbyists for the US pharmaceutical industry used to do this a lot.) It used to hilariously easy to shoot there reports down – which would argue that Country A, B or C’s drug laws and public procurement rules were damaging to US trade, by simply citing the CFR reg. or US law which was often word-for-word identical (the law profs had not bothered to check), so that if the diatribe was correct and Country A, B or C was found by the US to breach trade rules – well so did the US (this tended to lead to a loss of interest at USTR (they’d have to explain in Brussels.))

It really was quite astonishing how one could find reports and studies that showed diametrically opposite outcomes – from law professors and economists who were in broad agreement in economics and law – some had even co-authored papers. There was a sort of “nod & wink” mindset in which no one took a professor to task for such a study that actually conflicted with much of their other work – it seems that everyone, especially in economics was forgiven a little whoring.

Although this practice is widespread, generally I have found it to be ineffectual (and while people might say “he would say that”) I don’t do it. The reason may be seen in the response to S & M – too many people in legal practice have seen this sort of crap too many times (and that includes judges and antitrust/competition agencies), and have learned enough economics as well as the underlying tricks in these studies to work on professionals – look at how fast S & M’s study got torn apart. Other than the motivatedly credulous, the profoundly intellectually dishonest – and journalists (in a special class of lazy ignorance), and the uneducated (prospective law students), no one has accepted it. Of course, since the target was the lazy journalist and the prospective law student, I suppose the study served its purpose.

weirdnoise

“The dismal science,” indeed…

Prostitution can be an honest, perhaps even honorable occupation. Calling these liars-for-hire “whores” is giving them far too much legitimacy.