The CFTC’s war on prediction markets goes beyond Intrade. In 2010, the commission rejected an application by two exchanges to sell options on box-office receipts. Commissioner Bart Chilton said that such contracts would be unduly subject to manipulation by deep-pocketed Hollywood studios. But excluding studios from participating, he added, would mean these markets “would not result in the actual managing and assuming of price risks—something required for our approval.”

…

Yet in its 11 years on the Web, Intrade has proved neither frivolous nor manipulable. Intrade and similar services extend the predictive power of markets to offbeat but important topics from terrorism to flu outbreaks.

In a 2008 paper, a group of 19 economists, including Nobel laureate Kenneth Arrow, argued that prediction markets could potentially prove extremely useful. They are often more accurate than traditional forecasting tools — in part because they aggregate a wide range of informed opinions. ”The range of applications is virtually limitless,” the economists wrote, “from helping businesses make better investment decisions to helping governments make better fiscal and monetary policy decisions.”

The decision to cancel FutureMAP was at the very least premature, if not wrong-headed. The bulk of evidence on prediction markets demonstrate that they are reliable aggregators of disparate and dispersed information and can result in forecasts that are more accurate than those of experts. If so, prediction markets can substantially contribute to US Intelligence Community strategic and tactical intelligence work.

In a recent paper by Seth I. Stephens-Davidowitz Google Search Data is used to assess how much racist-sentiment affected the 2008 vote. The method is interesting, and the outcome nothing short of sensational:

The results imply that, relative to the areas in the United States with the lowest racial animus, racial animus cost Obama between 3.1 percentage points and 5.0 percentage points of the national popular vote. This implies racial animus gave Obama’s opponent roughly the equivalent of a home-state advantage country-wide. The cost of racial animus was not decisive in the 2008 election. But a four percentage point loss by the winning candidate would have changed the popular vote winner in the majority of post-war presidential elections.

Now, read the paper for yourself and determine if you agree with the methodology or not, but it does look interesting. Ultimately it becomes a correlation/causation issue to some extent, but still – interesting.

The notion that we could track prejudice and racist sentiment like this seems to open for new ways to track the dark sides of society and perhaps bring our creativity to bear on the problem of how to counter the fear and ignorance that underpins the memes involved.

In recent years there has been a growing interest in crowdsourcing methodologies to be used in experimental research for NLP tasks. In particular, evaluation of systems and theories about persuasion is difficult to accommodate within existing frameworks. In this paper we present a new cheap and fast methodology that allows fast experiment building and evaluation with fully-automated analysis at a low cost. The central idea is exploiting existing commercial tools for advertising on the web, such as Google AdWords, to measure message impact in an ecological setting. The paper includes a description of the approach, tips for how to use AdWords for scientific research, and results of pilot experiments on the impact of affective text variations which confirm the effectiveness of the approach.

A similar approach is described in the book Supercrunchers and has been used by political consultants to do message testing in a fast and relatively cheap way. As we proceed down the road I think secondary research uses can be found for a lot of the services online. The secondary uses of data allow for a wide variety of creative uses that explore language, social structures and economics. The data shadow of the web is the next frontier for many of these empirical sciences.

Besides exploring the theoretical notion the authors provide a great hands-on description of how to set up your experiment. Very intriguing.

(Full disclosure – I work at Google, but write here in a private capacity. The study was done in cooperation with Google and was funded by our research programme Google Research Awards to a part. I still think it is really, really interesting – but of course you should know those two things when reading this post!)

Peter Thiel is turning out to be a very, very original thinker. In today’s New York Times David Brooks mentions a lecture he gave at Stanford on starting up a company. Thiel’s main point is subtle and incredibly interesting: he says that people are confused when they think capitalism is about competition. In fact, with perfect competition there is no profit and you are forced to reinvest all the money you make to stay in place. Capitalism is about doing something in a large market where there is very little competition, and thus making huge profits. And then he adds that this is a consequence of many people confusing what is hard with what is valuable. There is so much sheer insight in these lecture notes that Blake Masters have put up that there is no excuse not to read them. Favorite quotes below:

The usual narrative is that capitalism and perfect competition are synonyms. No one is a monopoly. Firms compete and profits are competed away. But that’s a curious narrative. A better one frames capitalism and perfect competition as opposites; capitalism is about the accumulation of capital, whereas the world of perfect competition is one in which you can’t make any money.

On value and difficulty as a proxy for value

Intense competition makes things hard because you just beat heads with other people. The intensity of competition becomes a proxy for value. But value is a different question entirely. And to the extent it’s not there, you’re competing just for the sake of competition. Henry Kissinger’s anti-academic line aptly describes the conflation of difficulty and value: in academia at least, the battles are so fierce because the stakes are so small.

That seems true, but it also seems odd. If the stakes are so small, why don’t people stop fighting so hard and do something else instead? We can only speculate. Maybe those people just don’t know how to tell what’s valuable. Maybe all they can understand is the difficulty proxy. Maybe they’ve bought into the romanticization of competition. But it’s important to ask at what point it makes sense to get away from competition and shift your life trajectory towards monopoly.

On why life is not war

The perfect illustration of competition writ large is war. Everyone just kills everyone. There are always rationalizations for war. Often it’s been romanticized, though perhaps not so much anymore. But it makes sense: if life really is war, you should spend all your time either getting ready for it or doing it. That’s the Harvard mindset.

But what if life isn’t just war? Perhaps there’s more to it than that. Maybe you should sometimes run away. Maybe you should sheath the sword and figure out something else to do. Maybe “life is war” is just a strange lie we’re told, and competition isn’t actually as good as we assume it is.

On AI:

Artificial Intelligence is probably an underrated field. People are burned out on it, largely because it has been overrated and overstated for many decades. Few people think AI is or will soon be real at this point. But progress is increasingly relentless. AI performance in chess is increasing. Computers will probably beat humans in Go in 4 or 5 years. AI is probably a good place to look on the tech frontier. The challenge is that no one knows how far it will go.

The fascinating thing with Thiel’s argument is that it contains a tip on how to live your life as well on how to start your business. And maybe the trick is not viewing your life as very different from an investment in a startup. And realizing that life is not war, difficulty is not a good proxy for value and competition is opposed to capitalism.

The European Union has a research policy agenda that varies wildly. In one project, FuturICT, it has set out to examine the following, according to their website:

The ultimate goal of the FuturICT flagship project is to understand and manage complex, global, socially interactive systems, with a focus on sustainability and resilience. Revealing the hidden laws and processes underlying societies probably constitutes the most pressing scientific grand challenge of our century and is equally important for the development of novel robust, trustworthy and adaptive information and communication technologies (ICT), based on socially inspired paradigms.

Oooookay. That is pretty ambitious. Now, here is a question for you. Is this the kind of research we want? I must confess to being very much of two minds here. On one side I do like the broad approach and much of what the project has been doing is interesting. A recent paper outlines in-depth some challenges for complexity sciences that I found interesting. Again, though, the scoping is a bit, hm, exorbitant:

FuturICT foundations are social science, complex systems science, and ICT. The main concerns and challenges in the science of complex systems in the context of FuturICT are laid out in this paper with special emphasis on the Complex Systems route to Social Sciences. This include complex systems having: many heterogeneous interacting parts; multiple scales; complicated transition laws; unexpected or unpredicted emergence; sensitive dependence on initial conditions; path-dependent dynamics; networked hierarchical connectivities; interaction of autonomous agents; self-organisation; non-equilibrium dynamics; combinatorial explosion; adaptivity to changing environments; co-evolving subsystems; ill-defined boundaries; and multilevel dynamics. In this context, science is seen as the process of abstracting the dynamics of systems from data. This presents many challenges including: data gathering by large-scale experiment, participatory sensing and social computation, managing huge distributed dynamic and heterogeneous databases; moving from data to dynamical models, going beyond correlations to cause-effect relationships, understanding the relationship between simple and comprehensive models with appropriate choices of variables, ensemble modeling and data assimilation, modeling systems of systems of systems with many levels between micro and macro; and formulating new approaches to prediction, forecasting, and risk, especially in systems that can reflect on and change their behaviour in response to predictions, and systems whose apparently predictable behaviour is disrupted by apparently unpredictable rare or extreme events. These challenges are part of the FuturICT agenda.

Oh, just that? Where is your ambition, project members? Joking aside, it is exhilarating to see someone aim for the stars like this. But will it succeed? One problem I have is that I do not know what it would look like for the project to succeed. Accomplishing the singularity (finally!) or producing a god-like AI? Or just cataloguing a series of really interesting problems?

So I hesitate. On one side: good for EU that it dares address these challenges head on! On the other side: what exactly are you doing? Then I remember the millions that the EU plowed down into Electronic Copyright Management Systems like Imprimatur. Maybe we are better off with a project that states the following:

The FuturICT flagship proposal intends to unify hundreds of the best scientists in Europe in a 10 year 1 billion EUR program to explore social life on earth and everything it relates to. The FuturICT flagship proposal will produce historic breakthroughs and provide powerful new ways to manage challenges that make the modern world so difficult to predict, including the financial crisis.

Oh, good. What is all the fuss on the stock markets about, then? So, what do you think. Flip or flop? My jury was caught in a combinatorial participatory sensing explosion.

Tomorrow I will participate in Stockholm Internet Forum. I am looking forward to the panels and discussions, but also to meeting some old friends. In my own thinking about the conference I have ended up sketching out three themes that I want to examine.

The first is the notion that we cannot divide up responsibility in different sectors of a society. Saying that government has the responsibility to do x, and corporations have the responsibility to do y simply muddles the question. We have an overall responsibility that is dependent on everyone doing their part, and doing it well. That needs to be the basic insight here.

The second is that transparency is paramount. That we need transparency to ensure that the powers that we grant our governments over ourselves are exercised responsibly and with great care. Power observed and accounted for is power measured and careful exercised. More about this tomorrow.

The third is that I think that we have been under-theorizing the role of free flow of information generally. We need to understand information flows as fundamental to economic growth, intellectual achievement and cultural production. We need to build models that go beyond industrial society measures, and really understand the “space of flows” to craft a good internet society.

The good thing is that I will know more about all these things by the end of day tomorrow. The speakers and panels are top-notch and I am looking forward to learning more about this, as I happen to believe, important subject.

Update: Swedish government put out a press-release in which they were kind enough to count me among the guests the named by name. And I can die now, since they got my title right:

Prominent participants include Frank La Rue, UN Special Rapporteur on freedom of expression; Alec J Ross, Senior Advisor for Innovation to Secretary of State Hillary Clinton; Rebecca MacKinnon, one of the founders of Global Voices Online; Nicklas Lundblad, policy geek at Google; Suneet Singh, CEO of DataWind, which has launched the budget tablet Aakash; and Måns Adler, founder of the company and Internet service Bambuser for live broadcast of mobile phone videos.

Will knowledge about what our world is like help us craft a philosophy to deal better with our lives? It is not a trivial question. Knowing what the world is like could be both a blessing and a curse. There are those who think that the universe, such as it is, must have meaning for it to be possible for us to sustain meaning — this is in part the sense in which everything is allowed if God is dead in Dostoyevsky — but a perhaps more interesting position is to build meaning of sorts from established meaninglessness. Lawrence M Krauss recently wrote an article for the LA Times arguing exactly this, exploring what modern physics knows about the universe. His final words are interesting:

Imagining living in a universe without purpose may prepare us to better face reality head on. I cannot see that this is such a bad thing. Living in a strange and remarkable universe that is the way it is, independent of our desires and hopes, is far more satisfying for me than living in a fairy-tale universe invented to justify our existence.

There is truth to this. This is an almost Spinozan contemplation of the universe from the point of eternity, and it is in a sense liberating. If the universe is unlikely, without purpose, indeed, absurd, our choices seem less heavy. If we are the stuff of stars eating popcorn and having beer, if we are the unique and unlikely consequences of colliding particles and the children of collapsed wave forms, well, then maybe it is ok to be a bit late for work occasionally, or to just mess up.

Severalstudies have shown that people who contemplate death for a few minutes each day are happier. I think the same holds for those that look into the night sky and see the stars for what they are: our relatives, improbable and fantastic and uncaring. In fact, I would bet that if we could compare three groups: those that think about death, those that stare into the dark night sky and those that do neither, the second would be the happiest. That is an experiment I would like to try sometime.

Camus was right. We have to imagine that Sisyphus is happy – and so we could be too in a Sisyphean universe that cares little about our hopes and wants. In fact, there is something distinctly liberating about that isn’t there?

I just finished an excellent little paper by Henry Cohn and Douglas N Arnold. It is called “Mathematicians take a stand” and encourages all mathematicians to join a boycott against publisher Elsevier. The boycott — over at Cost of Knowledge — has its root in the growing prices of academic journals and how they slow down the dissemination of research and hamper the human project of building our knowledge together. Today more than 8000 researchers – not only mathematicians – has agreed to boycott the publisher if they do not allow for other easier ways for researchers to distribute their work and help build the future of knowledge.

The authors note in their paper:

While the mathematical literature itself is a treasure, the current system of scholarly publishing is badly broken. Elsevier is the largest and, in our view, the most egregious example of what is wrong. We hope many readers will agree with us that by choosing to withdraw our cooperation from Elsevier, we are sending a valuable message to them and to the scholarly publishing industry more broadly. Please consider joining the movement at http://thecostofknowledge.com.

What is our vision for the future? The mathematical community needs a period of experimentation and healthy competition, in which a variety of publishing models can flourish and develop. Possibilities include various approaches to open access publishing, refereed journals tightly integrated with the arXiv or similar servers, increased reliance on non-prot publishers, hybrid models in which community-owned journals subcontract their operations to commercial publishers, commercially-owned journals with reasonable prices and policies, etc. It is too early to predict the mix of models that will emerge as the most successful. However, any publisher that wants to be part of this mix must convince the community that they oversee peer review with integrity, that they aid dissemination rather than hinder it, and that they work to make high-quality mathematical literature widely available at a reasonable price.

Let’s work together to foster good practices and build better models. The future of mathematics publishing is in our hands.

And with the future of publishing the future of the subject also hangs in the balance. The way knowledge is disseminated, and, that it at a very minimum is not actively hindered to reach people everywhere is essential to the future of the Information Society.