Posted
by
kdawson
on Tuesday June 29, 2010 @02:18PM
from the transparency-wins dept.

jamie found a story up on Daily Kos revealing that the polling firm they had contracted with for 18 months, Research 2000 or R2K, apparently made up or at least manually tweaked its polling results. The blog published a preliminary report by a team of statistics gurus (Mark Grebner, Michael Weissman, and Jonathan Weissman), and it is an exemplar of clarity and concision. The team reports, "We do not know exactly how the weekly R2K results were created, but we are confident they could not accurately describe random polls." Daily Kos will be filing a lawsuit against its former pollster. "For the past year and a half, Daily Kos has been featuring weekly poll results from the Research 2000 (R2K) organization. These polls were often praised for their 'transparency,' since they included detailed cross-tabs on sub-populations and a clear description of the random dialing technique. However, on June 6, 2010, FiveThirtyEight.com rated R2K as among the least accurate pollsters in predicting election results. Daily Kos then terminated the relationship. One of us (MG) wondered if odd patterns he had noticed in R2K's reports might be connected with R2K's mediocre track record, prompting our investigation of whether the reports could represent proper random polling. ... This posting is a careful initial report of our findings, not intended to be a full formal analysis but rather to alert people not to rely on R2K's results."

[quote]or anything ever aired on "Air America" before it went bankrupt.[/quote]

I actually liked Rachel Meadow on Air America. Every night she would give the daily death tolls from Iraq and Afghanistan. Something that no other news/talking head program that I have been able to find on my radio dial does. The rest of the line up was pretty 'meh' though.

I wonder if she still does that at the beginning of her program? It seemed that when Bush was in office, the left screamed bloody murder when it came to the war(s). Now that their guy is in office, you can hear chirping from crickets.

I wonder if she still does that at the beginning of her program? It seemed that when Bush was in office, the left screamed bloody murder when it came to the war(s). Now that their guy is in office, you can hear chirping from crickets.

She doesn't do that on her tv show on MSNBC, but she goes after Obama all the time on lots of other stuff, as does Keither Olbermann, Jon Stewart and Stephen Colbert. To think that 'the left' doesn't go after Obama for anything is to completely ignore some of the biggest names

You're using that word, liberal, but it certainly doesn't mean whatever it is you think it means. If Obama were liberal, he would have signed an executive order halting DADT his first week in office, appointed an Attorney General that would be laying waste to Bushco and Wall Street with indictments, pushed for at least a 70% top marginal tax rate, withdrawn from Iraq and Afghanistan, pushed for single payer, pushed for a real green energy bill....

Olbermann continued to close his show with the number of days since "Mission Accomplished" right up until a month or so ago, when he switched to the number of days since the deepwater horizon leak started. Much of the left does not pull punches against Obama for taking his time extracting us from these debacles.

The far left idealists can get quite heated against Obama. Me, all I have to do is imagine how McCain would have responded, then after I've wiped off all the cold sweat and stopped gritting my teeth, I have no regrets about 08.

Should he be down there trying to clean the gulf with his fucking kidneys? Shutting down other drilling was a pretty big step, and anybody who thinks that there is something more that could be done is ignoring the enormous scope of the problem (there are lots of dumb-shit PR things that could be done, but not much that would really do anything about the oil, the biggest problem is that there were not enough resources to deal with it in place before it happened, not t

Should he be down there trying to clean the gulf with his fucking kidneys?

I think there are a lot of people who forget that Obama wasn't facing off in the last election against Aquaman, but for the record John McCain wouldn't be using telekinetic powers to summon a posse of dolphins to plug the leak in the gulf either. It's either that or else we've somehow come to think that knee-jerk, blustery statements/actions that don't address the real problem (like the Patriot act or the invasion of Iraq) are a good thing.

They explain the criteria by which they selected R2K, and it seemed fine. (RTFA)

That they caught R2K at this, and were willing to expose it, while other polls have also exhibited some of these patterns and continued to be used by their clients, says more good things about dkos than bad.

If this was the National Review Online, or Free Republic, or what have you, there would be a huge push to cover this up and blame the "liberal media"(whatever the hell THAT is) for any accusations that they did something wrong.

If this was the National Review Online, or Free Republic, or what have you, there would be a huge push to cover this up and blame the "liberal media"(whatever the hell THAT is) for any accusations that they did something wrong.

I doubt they would have questioned the results to begin with, much less investigated...

He, like many people educated in a day and age where truth is no longer held to the rigorous standards it once was, simply begins his line of thinking with his beliefs, and find the facts that best support those beliefs. Even if that means extracting them from their surrounding context entirely.

But, oh yes, many of the facts he uses are technically true. He's downright wrong on occasion, for sure, the point is that he's not wrong because his facts are wrong, rather, he's wrong because a comprehensive, holistic look at the facts does not influence his opinion at all. They're just a tool to him to propagate his beliefs.

And when you look at it from another perspective - you will probably exclude the nerd section of the population who never answers calls from 800-numbers (or other known junk callers) by using technology to divert the calls into a tarpit or something.

At least the open source telephony switch Asterisk do have a blacklist function where blacklisted numbers can be stored and used to perform a response like "The number you have dialed is not in use".

The headline on this article was stupidly misleading. Months ago, if not over a year, Nate Silver at fivethirtyeight.com [fivethirtyeight.com] called out R2K for just this one thing. You may remember Silver's interesting observation that the least significant digits in the polling results did not follow a normal random distribution. For example there may have been too many.9's in the results (58.9 or 63.9, etc) while there were few instances of other digits.

The pollster was subscribed to by DailyKos, among hundreds of other news organizations, and the results were skewed IN FAVOR OF RIGHT-WING CAUSES, not left-wing, so the assumption that DailyKos was somehow complicit in this is absolutely not true. (And I've rarely, if ever, read DailyKos, so I have no personal interest in defending them.. the headline is just grossly misleading).

(Hey, I'd be much happier if people named products with distinguishable proper names rather than generic sounding word combinations and worse yet, acronyms, so you have my sympathies for getting them mixed up.)

Probably somebody paying attention to Daily Kos' record. You show me the times and places they've been inaccurate. Note I didn't say biased, I said inaccurate. Ann Coulter and Rush Limbaugh make things up, they literally lie on air and in print, so throwing them together with Daily Kos, which at worst selectively covers stories that illustrate its world-view, is a troll-worthy attempt to muddy the waters for the benefit of right-wing hacks at the detriment of honest left-wing news outlets.

Somebody really fucked up the metamoderation. I used to metamoderate all the time when you had to rate moderations as fair, unfair, or neutral. I thought that worked pretty well, but now metamoderation is an up or down vote, and worst of all, by doing so you have tagged the post and it shows up on your damn profile page! I only made that mistake once, and I'm not metamoderating again until that goes away. I don't want my profile page cluttered with random posts from other people.

"Only if you're fond of false equivalencies, or an Obama fanboy. Of which there definitely are some on Dkos..."

Some? Get real. The agenda on DKos is Liberalism, pure and simple. Its says so right on the front page. If you want a politically left view point, then DKos is for you. If want a right, then its Limbaugh. If you want pure news with out a slant... well, I guess you're SOL.

Uh, there is a HUGE fight ongoing at dkos between the Obamabots and the haters, as they refer to each other. It's hilarious. All the real liberals are pissed as fuck at Obama, because he's a conservative in lib clothing, and we all see that now. But on dkos, we see the equivalent of the Bushtards who approved of the dolt until the end. They can not own up to the fact that they were scammed, and so they will defend Obama to the death even as he bends them over the fence for his corporate masters.

Right now it seems to be:
- against big government, but for the Patriot Act
- against a Health Care ID card, because that's big brother - but the Arizona "Show us your papers" law? No problem
- for State's rights, unless it's Bush v. Gore, Bush v. California EPA laws, or SCOTUS vs. state gun laws
- against deficits, unless a Republican's in the White House
- against "Islamofascists", unless their Saudis, in which case nothing to see here
- against abortion

Conservatism is about the protection of individual rights and liberties, freedom. It is about the reduction of government intervention. Let's see what Obama has done and his responses to certain topics:

The lower class needs healthcare: response is to increase government spending with entitlements

The economy is tanking badly: response is to create a slush fund of a trillion of dollars (give or take a billion)

BP screwed up and caused an oil spill: response is to send in the lawyers, SWAT?, setup several m

Wrong. Now that teabaggers know what the term means, they call themselves tea partiers. But back in the day, they carried teabags around and called themselves teabaggers.

Here's an article backing up that fact, but I warn you, it is from that den of liberal iniquity, Billy Buckley's The National Review, so take it with the grain of salt that any reading of The National Review requires.

Nope, that proves nothing other than some people are as ignorant as yourself. That article was written in December, 2009. And the author apparently didn't know anything about the Tea Parties that had been happening for almost three years - he seems under the (mistaken, or intentionally misleading) assumption it had something to do with Obama's election.

Haha. You guys still haven't figured out that "progressive" is a term we've been using for a deviant sexual act. LOL! That's why we guffaw when you call yourself a "progressive". I mean, the only thing we can thing to ourselves is, "well, yea, somebody tried to progressivize me in college, but I just wasn't drunk and horny enough to get into it."

The first big day for this movement was Tax Day, April 15. And organizers had a gimmick. They asked people to send a tea bag to the Oval Office. One of the exhortations was “Tea Bag the Fools in D.C.” A protester was spotted with a sign saying, “Tea Bag the Liberal Dems Before They Tea Bag You.” So, conservatives started it: started with this terminology. But others ran with it and ran with it.

Well, you just stick to bantering about terminology and calling names while we prepare to slaughter and gut the Big Government incumbents in November. You can camp out on the articles on Wikipedia and spend your time that way, I guess. Enjoy your broken-winged Chicago thug, by the way. He's your albatross for awhile longer. Shoulda picked Hillary, I guess.

Like I said, it has no place in polite conversation, regardless of who uses it. Plus, many who are being tarred with this brush had no part in its origins.

And none of that changes my original point, which was that it is used as a pejorative term to attack people rather than engage in debate, and therefore is usually used by people whose ability to engage in an interesting discussion is less developed than their desire to mock those who think differently.

In January of 09, they had a Facebook page that had some back and forth discussion about the 'alternate' meaning of teabag with some surprised disdain when they were informed as to what the term meant. They were apparently unaware at that point.

The fundamental problem with the Tea Party movement is that, the libertarian (economically), interventionist (with regard to foreign policy), and xenophobic (with respect to our relationship with the rest of the civilized) ideas that are underlying their platform are demonstrably awful for any society that adopts them. Sometimes there is a right and a wrong answer, all opinions are not equally valid and worthy of debate - yet we will tolerate all of them. The Tea Party movement is wrong, plain and simple. They do not have a legitimate contribution to make to the debate of how to govern society because their answer to that question is, "Don't."

That's not an ideology, it's an emotional response. It is essentially fear based isolationism taken to the extreme and applying it as far down as it will go; to the individual person. That, mixed with religious zealotry, and a sub-culture that worships guns and violence has the potential to set the US back 50-100 years in terms of social progress, equality, and the expansion of rights (as is understood by the ability to actually make life choices and have the MEANS to carry them out).

We have to tolerate them, as Americans they are within their right to express themselves, but anyone who does not stand up and say, scream, "NO!" to their hateful, backward, intolerant, reactionary rhetoric is the very antithesis of patriotic.

Taking a quick scan of the teaparty.org home page, they seem to be more anti-Democrat than for anything in particular.

If they were really an organization of principle instead of partisanship they would be trying to push both parties to work with their principles, in particular note their stance on Gen. McChrystal's comments. It doesn't matter whether he was right or wrong, that level of public insubordination is unprofessional and behavior unbecoming an officer in the US military.

For the FY2009, the federal budget was approximately 3.1 trillion dollars. Of that, 1.89 (~61%) trillion was mandatory (entitlement) spending - Social Security, Medicare, National Debt service, Unemployment, welfare, and the like. The remaining 1.21 trillion (~39%) was discretionary spending.

Of that 1.21 trillion, about 515 billion (~42%) was spent on the Department of Defense.

Did you know the original Teabaggers were protesting the fact that the wealthy British lowered taxes on their own tea below the taxes on the colonial tea?

That's a pretty inaccurate depiction of the Tea Act and why the colonists opposed it. In essence, the British government was protecting it's own favored company (East India Company), in favor of other traders (and smugglers, because tea carried a hefty tax). So actually the colonists favored free trade instead of crony capitalism (or fascism, if you prefer), and when the British government tried to pass laws that provided monopolies for East India, the colonists rebelled.

I think that's a pretty good analogy with motivations of the modern-day Tea Party protesters.

Those who aren't used to phrases used with "political" centric organizations might mistake the title as saying someone who is on Daily Kos' payroll flubbed the numbers, rather than a company working on contract with them.

For me, the surprising part of this isn't so much that R2K made up poll results, but that the results actually were noticably less accurate than traditional polling, which I like to think of as representing a broad cross-section of people who still have landlines with no caller ID for some reason (or are desperate enough to talk to another human being that they'll answer their landline anyway).

Unlike the many Republican outfits which used partly- or wholly-fabricated polls by Strategic Vision, or the many media outlets which continue to use the horribly flawed Rasmussen polls to create eye-catching headlines, Kos immediately dumped the pollster, did an investigation, owned up to the errors publicly, and is now pursuing legal recourse.

This is exactly how you would expect an honest media organization (if one with a considerable political agenda) to behave. Too bad the mainstream media and those on the other side of the aisle don't seem to want to do the same.

The quote you gave merely says that the results that Rasmussen is getting at this time show a public more greatly favoring Republican positions not that Rasmussen is fudging the numbers to get those results. The rest of the article can be dismissed since the source for the article clearly favors Democratic policy over Republican

You're also lazy, and ill informed. You could have spent a fraction of a second (0.15 seconds) with Google to find about 3,860,000 results for the search term "Rasmussen bias" to discover that, yes, in fact, there is some discussion of this point.

Nate Silver on Possible Biasin Rasmussen Reports [fivethirtyeight.com]
"What Rasmussen has had is a "house effect". So far in the 2010 cycle, their polling has consistently and predictably shown better results for Republican candidates than other polling firms have. But such house effects can emerge from legitimate differences of opinion about how to model the electorate. And ultimately, these differences of opinion will be tested -- based on what happens next November. If Rasmussen's opinion turns out to be wildly inaccurate, that will impeach their credibility, and believe me, we will point that out. Likewise, if they turn out to be right when most other pollsters are wrong, we will point that out too."

You're also lazy, and ill informed. You could have spent a fraction of a second (0.15 seconds) with Google to find about 3,860,000 results for the search term "Rasmussen bias" to discover that, yes, in fact, there is some discussion of this point.

I know there is discussion. Even your quote says differences "can emerge from legitimate differences of opinion about how to model the electorate" and FiveThirtyEight has, in the past, noted Rasmussen's surprising accuracy in predicting election outcomes. Your link would not support the GPP's description of "horribly flawed" to Rasmussen -- merely "hmm that's interesting".

Yes, the NYT fired Jayson Blair once they could no longer hide the fact that he made things up. The NYT had known for years that his "news" articles were fabrications before they fired him. BTW, unlike Bill Kristol, who writes opinion pieces, Jayson Blair was supposedly writing news articles.

I used to work a different major polling company, and I can assure you R2K is not alone in just making up numbers. Easily 80% of surveys that went through my region were completely falsified, and the remaining 20% rarely matched the demographic they were supposed to be answering for. Survey administrators have quotas, and then get paid extra for additional surveys past that, but there is basically nothing done to verify any of the surveys turned in, and everybody in the company knows it. Don't always trust what you read, especially not statistics.

For headline skimmers, this post would produced a completely inaccurate sense of what the article was all about... at length, the D.KOS are the ones who found out about this and are doing something something about it. That's good... but if you just read the headline, you'd come away thinking that D.KOS were the culprits instead.

Kos' sponsors paid R2K for a year and a half because they got the results they wanted to hear, even though everybody knew the results were complete BS. They finally got called out on it so they go into damage control mode....That's good?

There's a difference between being sold defective product and knowingly selling said defective product. Will Daily Kos likely and deservedly lose some credibility from this? Probably. But to say that they're a culprit is to imply that they were knowingly complicit in the fraud that they are alleging.

But my point is that there aren't equal numbers. Slashdot has always leaned one way--left. In this one case where a left-wing site is the topic of discussion, the accusation is being made that Slashdot is a conservative site.

By the way, the Kos supporters with mod points are out in full force abusing the -1 Overrated moderation.

of course this will turn into a "bash the left" and a "bash the right" thread. when ideology isn't the point. polling is the idiocy in question

and the guy who manipulated the numbers is clearly an amateur. the way you do proper poll manipulation is LOAD THE QUESTION. you poll people with a question with the proper turn of phrase to lead them towards the answer you want. then, when you present the answers to the poll, you also cage the results in such a way to lead the audience in the way you want them to interpret the results

polling is fucking joke. all results from the left, or the right, is complete bullshit, and a waste of your time

polling is fucking joke. all results from the left, or the right, is complete bullshit, and a waste of your time

Or, to put it another way, it's absolutely impossible to know what the people want without asking every single one of them.

Genius.

No, wait, sorry, I meant "bullshit". Polling is a tool, and an extremely important one. Can it be done very poorly? Yes, of course, But that needn't necessarily be true. And it's the only option for understanding a population when there's millions and millions of individuals.

Depends on what you mean by "joke". Using polling data, Nate Silver predicted the 2008 election within less than a half a percent. That's pretty accurate, but it could still be a joke for some definitions of joke.

Daily Kos wasn't trying to manipulate anything -- notice that they fired R2K once fivethirtyeight's statistics showed them to be least accurate at predicting election results, long before there was any evidence of fraud?

of course this wouldn't poll very high. even those who oppose obama, the vast majority do so for genuine reasons of policy, not out of some demented partisan bloodsport (although unfortunately, the unthinking zombie partisan hordes seem to be on the rise in this country)

In Neal Stephenson's Cryptonomicon [wikipedia.org], there's a scene early in the book where the Allies are assembling the personnel for Station X (aka Bletchely Park [bletchleypark.org.uk]). Statistician, turned Nazi codebreaker Lawrence Waterhouse, points out that his Nazi counterpart Rudy von Hacklheber, would notice something was amiss with the Allied personnel changes based the statistics of people being transfered to Bletchely Park, and then quickly deduce that the Allies are attempting to break the Enigma code. To camouflage the transfers, Waterhouse suggests creating ficticious personnel and have some of them transfered to Bletchely Park as well. However the military can't just make any random fake person, the fictious people must be statisitically drawn from a distribution that when added to distribution of real Bletchely Park personnel, the combined distribution is statistically insignificant [wikipedia.org] (i.e. fail to reject the null hypothesis) than any other large military base.

If Research 2000 did what is suggested, they failed to taint the polls with the right kind of fake data, just like what the novel warned about.

When you are attempting to do as little work as possible and still get the million dollars a year kos spends on polls, mocking with the data in such a way that nothing amiss can be detected is rather counter productive: You might as well do the polls right anyway.

They did look at it critically. Research 2000 was fired by Daily Kos before anyone noted any impropriety in the figures, simply because the numbers weren't matching up with reality. Shortly after this happened, Grebner, Weissman, and Weissman approached Markos with evidence of deliberate impropriety.

Does Daily Kos have a responsibility to not promote questionable information as truth? Of course, and they've apologized for the situation. But keep in mind that this information is only coming to light because someone with sufficient statistical background took the time to pore over the data. That sort of expertise is hard to come by, which is the reason why smaller media/news outlets contract out to firms like Research 2000 in the first place!

It's only relatively recently that there's been much interest in the science of polling. Before the emergence of aggregation sites like FiveThirtyEight or Pollster.com, it was extremely rare that you'd ever see this kind of statistical analysis of polling data. The traditional method of testing a pollster's reliability was simple trial and error over a period of several elections. Really, that's *still* the primary method. If anything, Research 2000 only got scrutinized in this case because of the issues with their accuracy that led to them being dropped in the first place.

For me, it's not really a partisan issue, despite the highly politicized nature of Daily Kos. It has more to do with the size of the media outlet. I would expect a major news organization with dozens or hundreds of employees, like Fox News or MSNBC, to be able to detect problems like these very quickly. A relatively small blog with maybe a dozen part-time employees like Daily Kos, or Red State, or whatever, I'm more willing to give a pass. At least at first: I'd expect Markos to learn his lesson from this and be more proactive in ensuring that it doesn't happen again.