KATE CRAWFORD. There are these hard limits that are reached in the epistemology of “Collect it all” where we reach a breakdown of meaning, a profusion and granularization of information to the point of being incomprehensible, of being in an ocean of potential interpretations and predictions. Once correlations become infinite, it’s difficult for them to remain moored in any kind of sense of the real. And it’s interesting how, for both of us, that presents a counter-narrative to the current discourse of the all-seeing, all-knowing state apparatus. That apparatus is actually struggling with its own profusion of data and prediction. We know that there are these black holes, these sort of moments of irrationality, and moments of information collapse.

KATE CRAWFORD. (…) the thing that got me through were these moments of humor. It’s very dark humor, but in the archive there are so many moments of this type. Some of the slides in particular are written in this kind of hyper-masculinist, hyper-competitive tone that I began to personalize as “the SIGINT Bro.”

KATE CRAWFORD. The other thing that I would love to talk to you about–and this is switching from the state to corporate uses of data, because I know both you and I are interested in how those two are really merging in particular ways–is IBM’s terrorism scoring project (…). I know we are both interested in how this type of prediction is a microcosm of a much wider propensity to score humans as part of a super-pattern.

HITO STEYERL. I’m really fascinated by quantifying social interaction and this idea of abstracting every kind of social interaction by citizens or human beings into just a single number; this could be a threat score, it could be a credit score, it could be an artist ranking score, which is something I’m subjected to all the time. For example, there was an amazing text about ranking participation in jihadi forums, but the most interesting example I found recently was the Chinese sincerity social score. I’m sure you heard about it, right? This is a sort of citizen “super score,” which cross-references credit data and financial interactions, not only in terms of quantity or turnover, but also in terms of quality, meaning that the exact purchases are looked into. In the words of the developer, someone who buys diapers will get more credit points than someone who spends money on video games because the first person is supposed to be socially “more reliable.” Then, health data goes into the score–along with your driving record, and also your online interactions. Basically it takes a quite substantial picture of your social interactions and abstracts it into just one number. This is the number of your “social sincerity.” It’s not implemented yet–there are some precursors in the form of extended credit scores which are already being rolled out–but it is supposed to be implemented in 2020, which is not that long from now. I’m completely fascinated by that.

KATE CRAWFORD. When I think about the Chinese citizen credit score is that here, in the West, it gets vilified as a sort of extremist position, like, “Who would possibly create something so clearly prone to error? And so clearly fascist in its construction?”[DE TODOS MODO, DEJAMOS LA POLÍTICA EN MANOS DE ESTE TIPO DE SISTEMAS MATEMATIZABLES] Yet, having said that, only last week we saw that an insurance company in the UK, the Admiral Group, was trying to market an app that would offer people either a discount on their car insurance or an increase in their premium based on the type of things they write on Facebook.

As for the IBM terrorist credit score, it’s being tested and deployed on a very vulnerable population that has absolutely no awareness that it is actually being used against them; also, it’s drawing upon these terribly weak correlations from sources like Twitter (…), it’s critically important that we question these knowledge claims at every level.

HITO STEYERL. (…) we are kind of back in the era of crude psychologisms, trying to attribute social, mental, or social-slash-mental illnesses or deficiencies with frankly absurd and unscientific markers.

KATE CRAWFORD. (…) what we now have is a new system called Faception that has been trained on millions of images. It says it can predict somebody’s intelligence and also the likelihood that they will be a criminal based on their face shape. Similarly, a deeply suspect paper was just released that claims to do automated inferences of criminality based on photographs of people’s faces. (…) Phrenology and physiognomy are being resuscitated, but encoded in facial recognition and machine learning.

(…) we’re seeing these historical returns to forms of knowledge that we’ve previously thought were, at the very least, unscientific, and, at the worst, genuinely dangerous.

HITO STEYERL. I think that maybe the source of this is a paradigm shift in the methodology. As far as I understand it, statistics have moved from constructing models and trying to test them using empirical data to just using the data and letting the patterns emerge somehow from the data. This is a methodology based on correlation. They keep repeating that correlation replaces causation. But correlation is entirely based on identifying surface patterns, right? The questions–why are they arising? why do they look the way they look?–are secondary now. If something just looks like something else, then it is with a certain probability identified as this “something else,” regardless of whether it is really the “something else” or not. Looking like something has become a sort of identity relation, and this is precisely how racism works. It doesn’t ask about the people in any other way than the way they look. It is a surface identification, and I’m really surprised how no one questions these correlationist models of extracting patterns on that basis. [The] IBM’s Hollerith machines, (…) were used in facilitating deportations during the Holocaust. This is why I’m always extremely suspicious of any kind of precise ethnic identification.

HITO STEYERL. There is a danger that if one tries to argue for more precise recognition or for more realistic training sets, the positive identification rate will actually increase, and I don’t really think that’s a good idea.

KATE CRAWFORD. Google has so much information (…) but that connection between its enormous seas of data and actually connecting that to instrumentalize the knowledge is still very weak.

[If] you are currently misrecognized by a system, it can mean that you don’t get access to housing, you don’t get access to credit, you don’t get released from jail. So you want this recognition, but, at the same time, the more the systems have accurate training data and the more they have deeper historical knowledge of you, the more you are profoundly captured within these systems.

We are being seen with ever greater resolution, but the systems around us are increasingly disappearing into the background.

KATE CRAWFORD. The narrative that’s being driven by Silicon Valley is that the biggest threat from AI is going to be the creation of a superintelligence that will dominate and subjugate humanity. (…) But to everybody else, those threats are already here. We are already living with systems that are subjugating human labor and particular subsets of the human population in ways that are harsher than others.

[One] of the things that is going to happen in the US is the complete automation of trucking. Now, trucking is one of the top employers in the entire country, so we’re looking at the decimation of a dominant job market.

HITO STEYERL. As people get replaced by systems, one of the few human jobs that seems to remain is security.

KATE CRAWFORD. I often think about this concept of solidarity in a world where so many of these stacks that overlay everyday interactions are trying to individualize and hyper-monetize and atomize not just individuals, but every sort of interaction. Every swipe, every input that we make, is being categorized and tracked. The idea, then, of solidarity across sectors, across difference, feels so powerful because it feels so unattainable.

HITO STEYERL. Have you seen any example of an AI that was focused on empathy or solidarity? Do you see the idea of comradeship anywhere in there? KATE CRAWFORD. ELIZA is the most simple system there is. She is by no means a real AI and she’s not even adapting in those conversations, but there’s something so simple about having an entity ‘listen’ and just pose your statements back to you as questions. (…) ELIZA as an empathy-producing machine because she was a simple listener. She wasn’t trying to be more intelligent than her interlocutors, she was just trying to listen, and that was actually very powerful.