Daniel Suarez: Bot-Mediated Realities

I enjoy exposure to new world views, the feeling of one’s brain being stretched to fit a new frame. For that reason, I enjoyed Daniel Suarez’s talk to the Long Now Foundation, entitled “Daemon: Bot-Mediated Realities”. You can listen to the talk as I did, or read Paul Saffo’s summary.

Suarez sees a world in which bots run everything from airplanes and cars to economies and financial systems, and we have decreasing control over them. The automated systems are good for us, enabling us to do more with fewer people, but Suarez reminds us of the downsides: the specialization of knowledge combined with exploitability of software and the easily-imagined situation of still-running code the workings of which nobody understands. It’s certainly changed the way I look at the things around me: inside every intelligent object I wonder, “who knows how these algorithms work? How long will it live?”. Not in a paranoid tinfoil helmet conspiracy way, just becoming aware of the fragility of the software I took for granted.

It was at this point that Suarez’s talk took a turn for the wishful. His solution to the possible nightmarish future of mankind at the mercy of bots that can’t be repaired or replaced was “let’s recreate the Internet, only with strong crypto and human-vouched IDs, and we’ll only permit bots that a quorum of humans have read and validated the source code to, and …” and I had to ask, “dude, have you ever worked with security people?”. The 9/11 terrorists had government IDs, and it’s easy to imagine malicious code doing so in the future. Reading the source code is time-consuming, therefore expensive, and no panacea–bugs can still exist in code that has been audited. The solution to fragile technology isn’t more fragile technology unless you can failover in a redundant array of inexpensive Earths.

I think the greatest value of his talk was from the long view of software: we’re creating actors that live beyond us, and we (software developers and society, the users of the bots) aren’t planning for succession or failure. Come for the world view, but leave early to avoid the questionable solutions.

In a more technical direction, Richard P. Gabriel’s paper ‘Design Beyond Human Abilities’ explores some of the issues involved in implementing such large automated systems, and poses a radical challenge to traditional thinking about software design.

While it doesn’t really address the hubris of trying to fix technology with more technology, it does provide some insights into how to make the technology less fragile.

These kind of concepts look like Terminator movie series. But, I doubt, in future we would be in some of those kinds of messed up situations where we have to obey what a machine says.

http://www.meemz.com Alex Tolley

Any highly specialized and constrained process will ultimately be fairly brittle and prone to failure. If S/W is to be more robust, it will have to be more like life. But that comes with the price that it will not be under our control any more.

http://www.nectardesign.com/ DennisSC

The sad fact is that any sort of system, supervised by humans or bots, is subject to mischief and, far more often, plain incompetence. I live in Southern California and we just had a truly horrific train crash out here which, if I’m to believe the radio, was caused by an engineer failing to stop at a red light. Still, there’s needs to be a way, I think, to somewhat “democratize” this, for lack of a better word, so that the human factor, flawed as it obviously is, is always kept coherently in the loop.

http://www.stapleton-gray.com Ross Stapleton-Gray

The train crash is being reported as possibly because the engineer was busy texting and missed the signal, which is truly horrible. Hope s/he at least died knowing what was up with Jamie Lynn Spears.

In William Gibson’s “Virtual Light,” people were accustomed to technical failures, e.g., robotically-guided cars whacking into things, as “SITS” (“something in the silicon”).

http://www.radar.com/nat gnat

@Manoj: anyone who follows a GPS around town is doing what a computer says to do. You even get chastized if you deviate from the algorithm’s chosen route! This isn’t abandoned technology, unless you happen to own a GPS unit for which no new maps are being released ….

@Alex Tolley: I’d never heard it put that way before, but yes–life is the most failure-tolerant thing we know. Great point, thanks!

Falafulu Fisi

There was a similar view expressed by Sun MicroSystem’s former chief scientist, Bill Joy in his Wired Magazine article published in 2001.

Why the future doesn’t need us (Our most powerful 21st-century technologies – robotics, genetic engineering, and nanotech – are threatening to make humans an endangered species).

His view is that the next revolution in IT comes from Physicists and not from computer scientists, electrical engineers, software architects, etc… This is true as physicists are starting to make breakthroughs in the domain of Quantum Computing which will make classical computing redundant at some stage. Programmers will need to be familiar with Quantum Physics (which is something very difficult) in order to write software, when the age of Quantum computing arrives.

http://code-itch.wordpress.com Hari Jayaram

This is a great talk from the Long now . I agree with your analysis saying that David Suarez posed the problem very well.
Nobody thinks anymore . Everything from whether my insurance claim gets accepted to whether anyone gets screened at the Airport is relegated to a “Bot” and thousands of call centers exist to take orders from these Bots and keep the system “functioning”.
Its a scary Future indeed.
My personal solution is force everyone to use their head. And patronize business that has not optimized at the cost of customer satisfaction even if it means spending 2 hours arguing with a person.

Jim Stogdill

Norbert Wiener, began talking about these kinds of issues in the 1950’s and ’60’s. You may find his classic piece God and Golem, Inc interesting: http://tinyurl.com/6ka3cu

Falafulu Fisi

Manjo said…These kind of concepts look like Terminator movie series.

Yes, correct. The embedded algorithm in the chip depicted in Terminator movie is called Artificial Neural Network (ANN) and it is model based on how human neuron functions. ANN is a self-learning algorithm ie, improving its current output by adapting to both changes in its past outputs & inputs, ie, its output at time t depends on its input/output at time t-1, t-2, t-3, … t-n. This is how exactly how human learns its external world via memory adaptation (ie, processing of current/past stored information – input/output). There are already ANN-based electronic chip being developed and made available for commercial use over the last 2 decades or so, but these are far away from the capability shown in Terminator. Technology is still evolving, and autonomy shown in Terminator, has just been started to be developed over the last decade or so, ie, to combine learning capability with autonomous decision making to make bots more intelligent. These 2 fields of computing are different domains,ie, Machine Learning & Autonomous Multi-agent software system. Combining these 2 paradigms of computing (ie, hybridizing) makes it superior than its corresponding 2 stand-alone counterparts.

Manjo said…But, I doubt, in future we would be in some of those kinds of messed up situations where we have to obey what a machine says.

Umm! It is already happening, but it depends on what your interpretation of the scenario. If you mean a dictatorial machine such as Terminator, then the answer is No. If you mean if it happens in everyday business world, then the answer is YES. It is already happening today in the domain of finance, medicine, defense, network security, and more…

http://www.echomusic.com.sg/ James

The idea of the world taken over by bots is indeed scary. While it may takes times for ANN to develop to similar capacity as terminator, it is still a possibility. Who will ever imagine flying in a aircraft and travelling at the speed of sound in the 18 century?

Lets hope there will be still control by humans at the end times.

http://www.blackshotsmp.blogspot.com/ Blackshot

These kind of concepts look like iRobot movie. It will be rather scary when we are controlled by robots, and watching them rise in numbers and in power.

http://www.blackshotsmp.blogspot.com/ Blackshot

These kind of concepts look like iRobot movie. It will be rather scary when we are controlled by robots, and watching them rise in numbers and in power.

Featured Video

The growing role of software architects: “Architecture has become much more interesting now because it’s become more encompassing," says Neal Ford, software architect and meme wrangler at ThoughtWorks.