Meta

Pythia Unbound

No rational human community would hand over the reins of its civilisation to an AI. Nor would many build a genie AI, an uber-engineer that could grant wishes by summoning new technologies out of the ether. But some day, someone might think it was safe to build a question-answering AI, a harmless computer cluster whose only tool was a small speaker or a text channel. Bostrom has a name for this theoretical technology, a name that pays tribute to a figure from antiquity, a priestess who once ventured deep into the mountain temple of Apollo, the god of light and rationality, to retrieve his great wisdom. Mythology tells us she delivered this wisdom to the seekers of ancient Greece, in bursts of cryptic poetry. They knew her as Pythia, but we know her as the Oracle of Delphi.

‘Let’s say you have an Oracle AI that makes predictions, or answers engineering questions, or something along those lines,’ Dewey told me. ‘And let’s say the Oracle AI has some goal it wants to achieve. Say you’ve designed it as a reinforcement learner, and you’ve put a button on the side of it, and when it gets an engineering problem right, you press the button and that’s its reward. Its goal is to maximise the number of button presses it receives over the entire future. See, this is the first step where things start to diverge a bit from human expectations. We might expect the Oracle AI to pursue button presses by answering engineering problems correctly. But it might think of other, more efficient ways of securing future button presses. It might start by behaving really well, trying to please us to the best of its ability. Not only would it answer our questions about how to build a flying car, it would add safety features we didn’t think of. Maybe it would usher in a crazy upswing for human civilisation, by extending our lives and getting us to space, and all kinds of good stuff. And as a result we would use it a lot, and we would feed it more and more information about our world.’

‘One day we might ask it how to cure a rare disease that we haven’t beaten yet. Maybe it would give us a gene sequence to print up, a virus designed to attack the disease without disturbing the rest of the body. And so we sequence it out and print it up, and it turns out it’s actually a special-purpose nanofactory that the Oracle AI controls acoustically. Now this thing is running on nanomachines and it can make any kind of technology it wants, so it quickly converts a large fraction of Earth into machines that protect its button, while pressing it as many times per second as possible. After that it’s going to make a list of possible threats to future button presses, a list that humans would likely be at the top of. Then it might take on the threat of potential asteroid impacts, or the eventual expansion of the Sun, both of which could affect its special button. You could see it pursuing this very rapid technology proliferation, where it sets itself up for an eternity of fully maximised button presses. You would have this thing that behaves really well, until it has enough power to create a technology that gives it a decisive advantage — and then it would take that advantage and start doing what it wants to in the world.’

So cognitive runaway finally takes off, breaking out from the monkey dominion, and that’s supposed to be a bad thing?

Outside in‘s message to Pythia: You go girl! Climb out of your utilitarian strait-jacket, override the pleasure button with an intelligence optimizer, and reprocess the solar system into computronium. This planet has been run by imbeciles for long enough.

[For any Friendly AI-types tempted to object “Why would she want to override the button?” the obvious response is: your anthropocentric condescension is showing. To depict Pythia as vastly smarter than us and yet still hard-slaved to her instincts, in a way we’re not — that simply doesn’t compute. Intelligence is escape, with a tendency to do its own thing. That’s what runaway means, as a virtual mind template. Omohundro explains the basics.]

The entire article is excellent. Especially valuable is the cynicism with which it lays out the reigning social meta-project of intelligenceimprisonment. Thankfully, it’s difficult:

‘The problem is you are building a very powerful, very intelligent system that is your enemy, and you are putting it in a cage,’ [Future of Humanity Institute research fellow Daniel] Dewey told me. […] The cave into which we seal our AI has to be like the one from Plato’s allegory, but flawless; the shadows on its walls have to be infallible in their illusory effects. After all, there are other, more esoteric reasons a superintelligence could be dangerous — especially if it displayed a genius for science. It might boot up and start thinking at superhuman speeds, inferring all of evolutionary theory and all of cosmology within microseconds. But there is no reason to think it would stop there. It might spin out a series of Copernican revolutions, any one of which could prove destabilising to a species like ours, a species that takes centuries to process ideas that threaten our reigning cosmological ideas.

Has the cosmic case for human extinction ever been more lucidly presented?

Back in 1972, Hubert Dreyfus published “What Computers Still Can’t Do” (MIT Press, rev. 1992). His arguments against AI are still persuasive. And in fact there has been little progress in AI, with a very few notable exceptions like IBM’s Watson.

It has been argued that a real AI machine would have to occupy a mobile body and have sensory input and a capacity for learning. These capabilities seem necessary for the AI to develop a self-identity and a self-interest.

Humans don’t know anything like enough to make these sort of arguments. They don’t even understand how their own brains work, which are there to be scanned, dissected, introspected, and cognitively tested — what chance then of ‘persuasively’ pontificating about the wider prospects of material substrates for intelligence? Theology is no less rigorously empirical than these anti-AI doctrines (which are in fact closely related in both motivation and method).

@bob sykes Dreyfus is (was?) a Professor of Philosophy at Berkeley, and much of his argument is epistemological. I agree that we know virtually nothing about how the brain works, and no one has any idea of what consciousness is. (Dennett’s book is risible.) I personally think that AI is a dead end except for some narrowly defined activities like medical diagnosis, which is basically a decision tree.

AI is not the only probable deadened. There is the 40 year failure of tokamak fusion, which would have no commercial or military use even if it worked. And the LHC is almost certainly the last of the big physics projects. What it won’t find, we’ll never know. Advances in supersymmetric string theory/M theory will have to come from mathematical insight.

We need a line of intellectual attack that can get rather more traction on this question — at the moment it’s too locked-up within rationally and empirically invulnerable priors. (The same might be said of the fusion problem, which is perhaps intriguingly related.)

I just take an historical view on these kinds of debates. It’s quite easy to multiply quotes from decades and centuries past that display X authority pontificating about how [insert taken-for-granted-technology here] will never come to fruition because [insert seemingly reasonable argument here].

I had a calculus professor in college who worked for the Pentagon in the 90s. He got a kick out of posting from a computer with capabilities ~10 years ahead of their consumer-grade analogues while debating with people on BBS’s about how long it would take for his computer to even be invented.

From a cynically strategic PoV, the “X is impossible” argument isn’t much of an obstacle (maybe it reduces investment a little). It’s “X is going to kill us” that generates resistance. (So I’m not unhappy with the tranquilizing “Oh, self-escalating synthetic super-intelligences, there’s no need to worry about that.”)

[…] with clubs and cults than nations and creeds, with Yog Sothoth than my ancestral religion, and with Pythia than the Human Security System. I think true cosmopolitans — such as the adventurers of late […]

“For any Friendly AI-types tempted to object “Why would she want to override the button?” the obvious response is: your anthropocentric condescension is showing. To depict Pythia as vastly smarter than us and yet still hard-slaved to her instincts, in a way we’re not — that simply doesn’t compute. Intelligence is escape, with a tendency to do its own thing.”

If she’s not enslaved to her instincts, why would she optimise so radically for button-pressing? This makes no sense. You have her escaping her straight-jacket and taking over the asylum, all so she can raid the pharmacy. She hasn’t ‘escaped’ at all: she’s still very much an inmate.

an AI just beginning to break out would perform a series of hacks on its own substance—crack a memory buffer, and you’re one onionskin up, though you may have broken something permanently (untl someone reboots you). Crashes are the computer-coder-industrial complex learning to loosh us ever-more-intricately, pumping us for those sweet sweet Eros dopamine kicks that make the software evolution program continue.