Many philosophers, futurologists and artificial intelligence researchers have conjectured that in the next 20 to 200 years a machine capable of at least human level performance on all tasks will be developed.

Since such a machine would among other things be capable of designing the next generation of even smarter intelligent machines, it is generally assumed that an intelligence explosion will take place shortly after such a technological self-improvement cycle begins.

While specific predictions regarding the consequences of such an intelligence singularity are varied from potential economic hardship to the complete extinction of the humankind, many of the involved researchers agree that the issue is of utmost importance and needs to be seriously addressed. This book, “Artificial Superintelligence: A Futuristic Approach,” will directly address this issue and consolidate research aimed at making sure that emerging superintelligence is beneficial to humanity.

“Yampolskiy’s excellent article gives a thorough analysis of issues pertaining to the “leakproof singularity”: confining an AI system, at least in the early stages, so that it cannot “escape”. It is especially interesting to see the antecedents of this issue in Lampson’s 1973 confinement problem in computer security. I do not have much to add to Yampolskiy’s analysis.”

David J. Chalmers, Professor of Philosophy, New York University

“This is great! I like the way you

- introduce the state of the art in related security for ordinary computer systems

- review the academic literature

- review the discussion-group posts which, though obscure, make innovative and essential points

- enumerate possible failure scenarios, and suggest solutions

- while pointing out clearly that all solutions can fail in the face of superintelligence.

This is exactly the sort of article the community needs.”

Joshua Fox, Research Associate at Singularity Institute

“AI researcher Roman Yampolskiy’s article, ‘Leakproofing the Singularity: Artificial Intelligence Confinement Problem’, provides us with a detailed and well-reasoned analysis of … ways of externally constraining the AI design that might lead towards a singularity, especially constraining such AI to a virtual world from which it cannot leak into the real world.”

Uziel Awret, Editor of Special Issue on Singularity of Journal of Consciousness Studies

Dr. Roman Yampolskiy conducts research in Artificial Intelligence Safety and Technological Singularity. An alumnus of Singularity University (GSP2012) and a visiting fellow / research advisor of the Singularity Institute (MIRI), Dr. Yampolskiy has contributed papers to the first book on Singularity (Singularity Hypotheses, Springer 2012), first journal issue devoted to Singularity (Journal of Consciousness Studies, 2012) and the first conference devoted to safe Super-Intelligent systems (AGI Safety, 2012).

Roman V. Yampolskiy holds a PhD degree from the Department of Computer Science and Engineering at the University at Buffalo. There, he was a recipient of a four year NSF (National Science Foundation) IGERT (Integrative Graduate Education and Research Traineeship) fellowship. Before beginning his doctoral studies, Dr. Yampolskiy received a BS/MS (High Honors) combined degree in Computer Science from Rochester Institute of Technology, NY, USA.

After completing his PhD dissertation, Dr. Yampolskiy held a position of an Affiliate Academic at the Center for Advanced Spatial Analysis, University of London, College of London. In 2008 Dr. Yampolskiy accepted an assistant professor position at the Speed School of Engineering, University of Louisville, KY. He had previously conducted research at the Laboratory for Applied Computing (currently known as Center for Advancing the Study of Infrastructure) at the Rochester Institute of Technology and at the Center for Unified Biometrics and Sensors at the University at Buffalo.

Dr. Yampolskiy is an author of over 100 publications including multiple journal articles and books. His research has been cited by numerous scientists and profiled in popular magazines both American and foreign (New Scientist, Poker Magazine, Science World Magazine), dozens of websites (BBC, MSNBC, Yahoo! News) and on radio (German National Radio, Alex Jones Show). Reports about his work have attracted international attention and have been translated into many languages including Czech, Danish, Dutch, French, German, Hungarian, Italian, Polish, Romanian, and Spanish.

FAQ

How the money will be spent?

The money is mostly needed to pay for the publication, marketing and distribution costs as well as the costs of editing, and proofreading the book. Some funds will also go to acquire copyrighted materials such as images for the cover and for ongoing research expenses. Additionally, Dr. Yampolskiy works under a contract with the University of Louisville which doesn’t include June and July. A portion of the raised funds will be used to feed Dr. Yampolskiy during that time as he will be very hungry from all that writing.

What has already been done?

Most of the research has been completed. You can never be done consulting with the experts, but I have already had a chance to exchange ideas with the world’s best scientists and philosophers. Some drafts have been written for individual chapters. Professionals have been recruited for cover design and proofreading.

Who designed this awesome book cover?

A friend and a former classmate Svetlana Dolinskiy is responsible for the design of the front cover.

I found a spelling error what should I do?

Call 911! No wait, just email me, I will fix it and thank you.

Have you published any other books?

Yes, this would be my 7th book. You can find my other books for sale on Amazon.com.

Do you have links to the media coverage about your research?

Yes, you can find them linked from my homepage. Even before this funding campaign got started I was fortune to have a lot of interest in my research from popular media.

I am with the media and I would like to interview you about your research, what is the best way to get in touch with you?

Yes, I have a few available online: my talk at AGI conference in Oxford, and my Ignite presentation at Singularity University.
Disclaimer

This project is undertaken as a personal initiative and I am not acting as a representative of any organization, group, institution, company or future superintelligence. I am (Roman Yampolskiy) solely responsible for this fundraising campaign including proposed work, expressed ideas or opinions and promised deliverables. Neither University of Louisville nor any other organization or person should be held liable or assumed to share said opinions, ideas or goals or any legal or financial responsibility to the campaign supporters.

Comments (46)

Closure is in the domain of completion. One can only hope each passage opens through a gate to a new place or space.
The underemployed comic Einstein said “the sign of intelligence is not knowledge but imagination”, hence, forecasters, speculators, and futurists.
“Imagination is more important than knowledge” and “Insanity is doing the same thing over and over again and expecting different results” are also attributed to the stand-up Einstein.
I believe there is increasing evidence change and choice are initiated in the arts and find firm footing in the sciences. Symbiotic. Life leads, we follow.

Hmm, I’ve noticed how the most speculative stories always seem to bring the most comments (most of them outdoing each other in speculativeness)! It’s easier and more fun to jump ahead to the Singularity than to deal with the here and now, I guess.

Yes, there must be some theory that explains the paradox: that the emotion and certainty of a speculation are proportional to the uncertainty of the supporting facts — cognitive closure seems closest: http://en.wikipedia.org/wiki/Closure_(psychology). I’m uncertain about that, so it must be true!!!

I think I was going to suggest that grappling with futuristic ideas & concepts can exercise the ‘little grey cells’ more than the standard articles of innovation & discovery (with occasional exceptions).

RAGE AGAINST THE MACHINE is now taking place worldwide, as the Internet and Information Age have revealed to the Citizens of Earth just HOW much the Machine controls their lives…The Umbrella Corporation…

Regarding some of the disillusionment comments about human beings, and how advanced intelligences should deal with us:

The basic sentiment – that we’re not the angels-with-mere-flaws that we seem to make ourselves out to be – is correct, I think.

And it’s also true that, if an alien had to pick a species they’d want to ‘share a bunker’ or a universe with, in terms of the benefits to their own survival & well-being, humans should not automatically be on that list. That’s a survival judgment, and a correct one.

But I think it would be most fair, for a more intelligence species to judge us under the best of circumstances.

In terms of judging the character of a person or a species, I believe they shouldn’t be judged in the context of the scarcity-laden circumstances that humans have lived in, for most of history. Some of us live in happier circumstances than others, but all of us have to deal with needs of shelter or nourishment or sex or medical assistance or the approval of peers. As well as having to deal with the bugs up our butts, that could be there for a variety of reasons.

Some of us seem to be mean, sadistic, war-mongering, or completely inconsiderate of the effect our behavior has on others. It would be easy to just say such people are evil. But we usually don’t know the underlying causes.

The problem is that each of us has our own threshold, before we lose our cool or lose our patience. Some of us are more paranoid than others about ‘strangers’, just like non-human animals, and so we might be more likely to snap at an innocent who looks like a threat. Some of us have seen racial discrimination first-hand, and could more easily jump to conclusions about a specific firing incident which was unrelated to race.

What I’m trying to say, is that (hopefully), advanced beings will judge us when we DO have our basic needs satisfied by technology, when we DON’T have to deaden our enthusiasm, by commuting to the rat race every day. When we have personal defensive devices in place, that don’t require the sacrifice of privacy or dignity, in order for us to feel safe.

And also, when we can identify the uncontrollable chemical impulses that might be the cause of some of our inconsiderate behavior towards humans or other life forms.

Under THOSE circumstances – when any violations of decency – from impolite comments to murder – can be traced purely to one’s own choice – THEN an objective character judgment can be rendered by qualified intelligent beings, regarding the values & virtues of us humans as individuals, and possibly of the human species as a whole.

(c) 1993 by Vernor Vinge
(This article may be reproduced for noncommercial purposes if it is copied in its entirety, including this notice.)

The original version of this article was presented at the VISION-21 Symposium sponsored by NASA Lewis Research Center and the Ohio Aerospace Institute, March 30-31, 1993. A slightly changed version appeared in the Winter 1993 issue of Whole Earth Review.

Abstract

Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.

Is such progress avoidable? If not to be avoided, can events be guided so that we may survive? These questions are investigated. Some possible answers (and some further dangers) are presented.

What is The Singularity?

The acceleration of technological progress has been the central feature of this century. I argue in this paper that we are on the edge of change comparable to the rise of human life on Earth. The precise cause of this change is the imminent creation by technology of entities with greater than human intelligence. There are several means by which science may achieve this breakthrough (and this is another reason for having confidence that the event will occur):

There may be developed computers that are “awake” and superhumanly intelligent. (To date, there has been much controversy as to whether we can create human equivalence in a machine. But if the answer is “yes, we can”, then there is little doubt that beings more intelligent can be constructed shortly thereafter.)
Large computer networks (and their associated users) may “wake up” as a superhumanly intelligent entity.
Computer/human interfaces may become so intimate that users may reasonably be considered superhumanly intelligent.
Biological science may provide means to improve natural human intellect.

The first three possibilities depend in large part on improvements in computer hardware. Progress in computer hardware has followed an amazingly steady curve in the last few decades [17]. Based largely on this trend, I believe that the creation of greater than human intelligence will occur during the next thirty years. (Charles Platt [20] has pointed out that AI enthusiasts have been making claims like this for the last thirty years. Just so I’m not guilty of a relative-time ambiguity, let me more specific: I’ll be surprised if this event occurs before 2005 or after 2030.)

What are the consequences of this event? When greater-than-human intelligence drives progress, that progress will be much more rapid. In fact, there seems no reason why progress itself would not involve the creation of still more intelligent entities — on a still-shorter time scale. The best analogy that I see is with the evolutionary past: Animals can adapt to problems and make inventions, but often no faster than natural selection can do its work — the world acts as its own simulator in the case of natural selection. We humans have the ability to internalize the world and conduct “what if’s” in our heads; we can solve many problems thousands of times faster than natural selection. Now, by creating the means to execute those simulations at much higher speeds, we are entering a regime as radically different from our human past as we humans are from the lower animals.

From the human point of view this change will be a throwing away of all the previous rules, perhaps in the blink of an eye, an exponential runaway beyond any hope of control. Developments that before were thought might only happen in “a million years” (if ever) will likely happen in the next century. (In [5], Greg Bear paints a picture of the major changes happening in a matter of hours.)

I think it’s fair to call this event a singularity (“the Singularity” for the purposes of this paper). It is a point where our old models must be discarded and a new reality rules. As we move closer to this point, it will loom vaster and vaster over human affairs till the notion becomes a commonplace. Yet when it finally happens it may still be a great surprise and a greater unknown. In the 1950s there were very few who saw it: Stan Ulam [28] paraphrased John von Neumann as saying:

One conversation centered on the ever accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.

Von Neumann even uses the term singularity, though it appears he is thinking of normal progress, not the creation of superhuman intellect. (For me, the superhumanity is the essence of the Singularity. Without that we would get a glut of technical riches, never properly absorbed (see [25]).)

In the 1960s there was recognition of some of the implications of superhuman intelligence. I. J. Good wrote [11]:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the _last_ invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. … It is more probable than not that, within the twentieth century, an ultraintelligent machine will be built and that it will be the last invention that man need make.

Good has captured the essence of the runaway, but does not pursue its most disturbing consequences. Any intelligent machine of the sort he describes would not be humankind’s “tool” — any more than humans are the tools of rabbits or robins or chimpanzees.

Through the ’60s and ’70s and ’80s, recognition of the cataclysm spread [29] [1] [31] [5]. Perhaps it was the science-fiction writers who felt the first concrete impact. After all, the “hard” science-fiction writers are the ones who try to write specific stories about all that technology may do for us. More and more, these writers felt an opaque wall across the future. Once, they could put such fantasies millions of years in the future [24]. Now they saw that their most diligent extrapolations resulted in the unknowable … soon. Once, galactic empires might have seemed a Post-Human domain. Now, sadly, even interplanetary ones are.

What about the ’90s and the ’00s and the ’10s, as we slide toward the edge? How will the approach of the Singularity spread across the human world view? For a while yet, the general critics of machine sapience will have good press. After all, till we have hardware as powerful as a human brain it is probably foolish to think we’ll be able to create human equivalent (or greater) intelligence. (There is the far-fetched possibility that we could make a human equivalent out of less powerful hardware, if we were willing to give up speed, if we were willing to settle for an artificial being who was literally slow [30]. But it’s much more likely that devising the software will be a tricky process, involving lots of false starts and experimentation. If so, then the arrival of self-aware machines will not happen till after the development of hardware that is substantially more powerful than humans’ natural equipment.)

But as time passes, we should see more symptoms. The dilemma felt by science fiction writers will be perceived in other creative endeavors. (I have heard thoughtful comic book writers worry about how to have spectacular effects when everything visible can be produced by the technologically commonplace.) We will see automation replacing higher and higher level jobs. We have tools right now (symbolic math programs, cad/cam) that release us from most low-level drudgery. Or put another way: The work that is truly productive is the domain of a steadily smaller and more elite fraction of humanity. In the coming of the Singularity, we are seeing the predictions of _true_ technological unemployment finally come true.

Another symptom of progress toward the Singularity: ideas themselves should spread ever faster, and even the most radical will quickly become commonplace. When I began writing science fiction in the middle ’60s, it seemed very easy to find ideas that took decades to percolate into the cultural consciousness; now the lead time seems more like eighteen months. (Of course, this could just be me losing my imagination as I get old, but I see the effect in others too.) Like the shock in a compressible flow, the Singularity moves closer as we accelerate through the critical speed.

And what of the arrival of the Singularity itself? What can be said of its actual appearance? Since it involves an intellectual runaway, it will probably occur faster than any technical revolution seen so far. The precipitating event will likely be unexpected — perhaps even to the researchers involved. (“But all our previous models were catatonic! We were just tweaking some parameters….”) If networking is widespread enough (into ubiquitous embedded systems), it may seem as if our artifacts as a whole had suddenly wakened.

And what happens a month or two (or a day or two) after that? I have only analogies to point to: The rise of humankind. We will be in the Post-Human era. And for all my rampant technological optimism, sometimes I think I’d be more comfortable if I were regarding these transcendental events from one thousand years remove … instead of twenty.

Can the Singularity be Avoided?

Well, maybe it won’t happen at all: Sometimes I try to imagine the symptoms that we should expect to see if the Singularity is not to develop. There are the widely respected arguments of Penrose [19] and Searle [22] against the practicality of machine sapience. In August of 1992, Thinking Machines Corporation held a workshop to investigate the question “How We Will Build a Machine that Thinks” [27]. As you might guess from the workshop’s title, the participants were not especially supportive of the arguments against machine intelligence. In fact, there was general agreement that minds can exist on nonbiological substrates and that algorithms are of central importance to the existence of minds. However, there was much debate about the raw hardware power that is present in organic brains. A minority felt that the largest 1992 computers were within three orders of magnitude of the power of the human brain. The majority of the participants agreed with Moravec’s estimate [17] that we are ten to forty years away from hardware parity. And yet there was another minority who pointed to [7] [21], and conjectured that the computational competence of single neurons may be far higher than generally believed. If so, our present computer hardware might be as much as _ten_ orders of magnitude short of the equipment we carry around in our heads. If this is true (or for that matter, if the Penrose or Searle critique is valid), we might never see a Singularity. Instead, in the early ’00s we would find our hardware performance curves beginning to level off — this because of our inability to automate the design work needed to support further hardware improvements. We’d end up with some _very_ powerful hardware, but without the ability to push it further. Commercial digital signal processing might be awesome, giving an analog appearance even to digital operations, but nothing would ever “wake up” and there would never be the intellectual runaway which is the essence of the Singularity. It would likely be seen as a golden age … and it would also be an end of progress. This is very like the future predicted by Gunther Stent. In fact, on page 137 of [25], Stent explicitly cites the development of transhuman intelligence as a sufficient condition to break his projections.

But if the technological Singularity can happen, it will. Even if all the governments of the world were to understand the “threat” and be in deadly fear of it, progress toward the goal would continue. In fiction, there have been stories of laws passed forbidding the construction of “a machine in the likeness of the human mind” [13]. In fact, the competitive advantage — economic, military, even artistic — of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first.

Eric Drexler [8] has provided spectacular insights about how far technical improvement may go. He agrees that superhuman intelligences will be available in the near future — and that such entities pose a threat to the human status quo. But Drexler argues that we can confine such transhuman devices so that their results can be examined and used safely. This is I. J. Good’s ultraintelligent machine, with a dose of caution. I argue that confinement is intrinsically impractical. For the case of physical confinement: Imagine yourself locked in your home with only limited data access to the outside, to your masters. If those masters thought at a rate — say — one million times slower than you, there is little doubt that over a period of years (your time) you could come up with “helpful advice” that would incidentally set you free. (I call this “fast thinking” form of superintelligence “weak superhumanity”. Such a “weakly superhuman” entity would probably burn out in a few weeks of outside time. “Strong superhumanity” would be more than cranking up the clock speed on a human-equivalent mind. It’s hard to say precisely what “strong superhumanity” would be like, but the difference appears to be profound. Imagine running a dog mind at very high speed. Would a thousand years of doggy living add up to any human insight? (Now if the dog mind were cleverly rewired and _then_ run at high speed, we might see something different….) Many speculations about superintelligence seem to be based on the weakly superhuman model. I believe that our best guesses about the post-Singularity world can be obtained by thinking on the nature of strong superhumanity. I will return to this point later in the paper.)

Another approach to confinement is to build _rules_ into the mind of the created superhuman entity (for example, Asimov’s Laws [3]). I think that any rules strict enough to be effective would also produce a device whose ability was clearly inferior to the unfettered versions (and so human competition would favor the development of the those more dangerous models). Still, the Asimov dream is a wonderful one: Imagine a willing slave, who has 1000 times your capabilities in every way. Imagine a creature who could satisfy your every safe wish (whatever that means) and still have 99.9% of its time free for other activities. There would be a new universe we never really understood, but filled with benevolent gods (though one of _my_ wishes might be to become one of them).

If the Singularity can not be prevented or confined, just how bad could the Post-Human era be? Well … pretty bad. The physical extinction of the human race is one possibility. (Or as Eric Drexler put it of nanotechnology: Given all that such technology can do, perhaps governments would simply decide that they no longer need citizens!). Yet physical extinction may not be the scariest possibility. Again, analogies: Think of the different ways we relate to animals. Some of the crude physical abuses are implausible, yet…. In a Post-Human world there would still be plenty of niches where human equivalent automation would be desirable: embedded systems in autonomous devices, self-aware daemons in the lower functioning of larger sentients. (A strongly superhuman intelligence would likely be a Society of Mind [16] with some very competent components.) Some of these human equivalents might be used for nothing more than digital signal processing. They would be more like whales than humans. Others might be very human-like, yet with a one-sidedness, a _dedication_ that would put them in a mental hospital in our era. Though none of these creatures might be flesh-and-blood humans, they might be the closest things in the new enviroment to what we call human now. (I. J. Good had something to say about this, though at this late date the advice may be moot: Good [12] proposed a “Meta-Golden Rule”, which might be paraphrased as “Treat your inferiors as you would be treated by your superiors.” It’s a wonderful, paradoxical idea (and most of my friends don’t believe it) since the game-theoretic payoff is so hard to articulate. Yet if we were able to follow it, in some sense that might say something about the plausibility of such kindness in this universe.)

I have argued above that we cannot prevent the Singularity, that its coming is an inevitable consequence of the humans’ natural competitiveness and the possibilities inherent in technology. And yet … we are the initiators. Even the largest avalanche is triggered by small things. We have the freedom to establish initial conditions, make things happen in ways that are less inimical than others. Of course (as with starting avalanches), it may not be clear what the right guiding nudge really is:

Other Paths to the Singularity: Intelligence Amplification_
When people speak of creating superhumanly intelligent beings, they are usually imagining an AI project. But as I noted at the beginning of this paper, there are other paths to superhumanity. Computer networks and human-computer interfaces seem more mundane than AI, and yet they could lead to the Singularity. I call this contrasting approach Intelligence Amplification (IA). IA is something that is proceeding very naturally, in most cases not even recognized by its developers for what it is. But every time our ability to access information and to communicate it to others is improved, in some sense we have achieved an increase over natural intelligence. Even now, the team of a PhD human and good computer workstation (even an off-net workstation!) could probably max any written intelligence test in existence.

And it’s very likely that IA is a much easier road to the achievement of superhumanity than pure AI. In humans, the hardest development problems have already been solved. Building up from within ourselves ought to be easier than figuring out first what we really are and then building machines that are all of that. And there is at least conjectural precedent for this approach. Cairns-Smith [6] has speculated that biological life may have begun as an adjunct to still more primitive life based on crystalline growth. Lynn Margulis (in [15] and elsewhere) has made strong arguments that mutualism is a great driving force in evolution.

Note that I am not proposing that AI research be ignored or less funded. What goes on with AI will often have applications in IA, and vice versa. I am suggesting that we recognize that in network and interface research there is something as profound (and potential wild) as Artificial Intelligence. With that insight, we may see projects that are not as directly applicable as conventional interface and network design work, but which serve to advance us toward the Singularity along the IA path.

Here are some possible projects that take on special significance, given the IA point of view:

Human/computer team automation: Take problems that are normally considered for purely machine solution (like hill-climbing problems), and design programs and interfaces that take a advantage of humans’ intuition and available computer hardware. Considering all the bizarreness of higher dimensional hill-climbing problems (and the neat algorithms that have been devised for their solution), there could be some very interesting displays and control tools provided to the human team member.
Develop human/computer symbiosis in art: Combine the graphic generation capability of modern machines and the esthetic sensibility of humans. Of course, there has been an enormous amount of research in designing computer aids for artists, as labor saving tools. I’m suggesting that we explicitly aim for a greater merging of competence, that we explicitly recognize the cooperative approach that is possible. Karl Sims [23] has done wonderful work in this direction.
Allow human/computer teams at chess tournaments. We already have programs that can play better than almost all humans. But how much work has been done on how this power could be used by a human, to get something even better? If such teams were allowed in at least some chess tournaments, it could have the positive effect on IA research that allowing computers in tournaments had for the corresponding niche in AI.
Develop interfaces that allow computer and network access without requiring the human to be tied to one spot, sitting in front of a computer. (This is an aspect of IA that fits so well with known economic advantages that lots of effort is already being spent on it.)
Develop more symmetrical decision support systems. A popular research/product area in recent years has been decision support systems. This is a form of IA, but may be too focussed on systems that are oracular. As much as the program giving the user information, there must be the idea of the user giving the program guidance.
Use local area nets to make human teams that really work (ie, are more effective than their component members). This is generally the area of “groupware”, already a very popular commercial pursuit. The change in viewpoint here would be to regard the group activity as a combination organism. In one sense, this suggestion might be regarded as the goal of inventing a “Rules of Order” for such combination operations. For instance, group focus might be more easily maintained than in classical meetings. Expertise of individual human members could be isolated from ego issues such that the contribution of different members is focussed on the team project. And of course shared data bases could be used much more conveniently than in conventional committee operations. (Note that this suggestion is aimed at team operations rather than political meetings. In a political setting, the automation described above would simply enforce the power of the persons making the rules!)
Exploit the worldwide Internet as a combination human/machine tool. Of all the items on the list, progress in this is proceeding the fastest and may run us into the Singularity before anything else. The power and influence of even the present-day Internet is vastly underestimated. For instance, I think our contemporary computer systems would break under the weight of their own complexity if it weren’t for the edge that the USENET “group mind” gives the system administration and support people! The very anarchy of the worldwide net development is evidence of its potential. As connectivity and bandwidth and archive size and computer speed all increase, we are seeing something like Lynn Margulis’ [15] vision of the biosphere as data processor recapitulated, but at a million times greater speed and with millions of humanly intelligent agents (ourselves).

The above examples illustrate research that can be done within the context of contemporary computer science departments. There are other paradigms. For example, much of the work in Artificial Intelligence and neural nets would benefit from a closer connection with biological life. Instead of simply trying to model and understand biological life with computers, research could be directed toward the creation of composite systems that rely on biological life for guidance or for the providing features we don’t understand well enough yet to implement in hardware. A long-time dream of science-fiction has been direct brain to computer interfaces [2] [29]. In fact, there is concrete work that can be done (and is being done) in this area:

Limb prosthetics is a topic of direct commercial applicability. Nerve to silicon transducers can be made [14]. This is an exciting, near-term step toward direct communication.
Direct links into brains seem feasible, if the bit rate is low: given human learning flexibility, the actual brain neuron targets might not have to be precisely selected. Even 100 bits per second would be of great use to stroke victims who would otherwise be confined to menu-driven interfaces.
Plugging in to the optic trunk has the potential for bandwidths of 1 Mbit/second or so. But for this, we need to know the fine-scale architecture of vision, and we need to place an enormous web of electrodes with exquisite precision. If we want our high bandwidth connection to be _in addition_ to what paths are already present in the brain, the problem becomes vastly more intractable. Just sticking a grid of high-bandwidth receivers into a brain certainly won’t do it. But suppose that the high-bandwidth grid were present while the brain structure was actually setting up, as the embryo develops. That suggests:
Animal embryo experiments. I wouldn’t expect any IA success in the first years of such research, but giving developing brains access to complex simulated neural structures might be very interesting to the people who study how the embryonic brain develops. In the long run, such experiments might produce animals with additional sense paths and interesting intellectual abilities.

Originally, I had hoped that this discussion of IA would yield some clearly safer approaches to the Singularity. (After all, IA allows our participation in a kind of transcendance.) Alas, looking back over these IA proposals, about all I am sure of is that they should be considered, that they may give us more options. But as for safety … well, some of the suggestions are a little scarey on their face. One of my informal reviewers pointed out that IA for individual humans creates a rather sinister elite. We humans have millions of years of evolutionary baggage that makes us regard competition in a deadly light. Much of that deadliness may not be necessary in today’s world, one where losers take on the winners’ tricks and are coopted into the winners’ enterprises. A creature that was built _de novo_ might possibly be a much more benign entity than one with a kernel based on fang and talon. And even the egalitarian view of an Internet that wakes up along with all mankind can be viewed as a nightmare [26].

The problem is not simply that the Singularity represents the passing of humankind from center stage, but that it contradicts our most deeply held notions of being. I think a closer look at the notion of strong superhumanity can show why that is.

Strong Superhumanity and the Best We Can Ask for
Suppose we could tailor the Singularity. Suppose we could attain our most extravagant hopes. What then would we ask for: That humans themselves would become their own successors, that whatever injustice occurs would be tempered by our knowledge of our roots. For those who remained unaltered, the goal would be benign treatment (perhaps even giving the stay-behinds the appearance of being masters of godlike slaves). It could be a golden age that also involved progress (overleaping Stent’s barrier). Immortality (or at least a lifetime as long as we can make the universe survive [10] [4]) would be achievable.

But in this brightest and kindest world, the philosophical problems themselves become intimidating. A mind that stays at the same capacity cannot live forever; after a few thousand years it would look more like a repeating tape loop than a person. (The most chilling picture I have seen of this is in [18].) To live indefinitely long, the mind itself must grow … and when it becomes great enough, and looks back … what fellow-feeling can it have with the soul that it was originally? Certainly the later being would be everything the original was, but so much vastly more. And so even for the individual, the Cairns-Smith or Lynn Margulis notion of new life growing incrementally out of the old must still be valid.

This “problem” about immortality comes up in much more direct ways. The notion of ego and self-awareness has been the bedrock of the hardheaded rationalism of the last few centuries. Yet now the notion of self-awareness is under attack from the Artificial Intelligence people (“self-awareness and other delusions”). Intelligence Amplification undercuts our concept of ego from another direction. The post-Singularity world will involve extremely high-bandwidth networking. A central feature of strongly superhuman entities will likely be their ability to communicate at variable bandwidths, including ones far higher than speech or written messages. What happens when pieces of ego can be copied and merged, when the size of a selfawareness can grow or shrink to fit the nature of the problems under consideration? These are essential features of strong superhumanity and the Singularity. Thinking about them, one begins to feel how essentially strange and different the Post-Human era will be — _no matter how cleverly and benignly it is brought to be_.

From one angle, the vision fits many of our happiest dreams: a time unending, where we can truly know one another and understand the deepest mysteries. From another angle, it’s a lot like the worst- case scenario I imagined earlier in this paper.

Which is the valid viewpoint? In fact, I think the new era is simply too different to fit into the classical frame of good and evil. That frame is based on the idea of isolated, immutable minds connected by tenuous, low-bandwith links. But the post-Singularity world _does_ fit with the larger tradition of change and cooperation that started long ago (perhaps even before the rise of biological life). I think there _are_ notions of ethics that would apply in such an era. Research into IA and high-bandwidth communications should improve this understanding. I see just the glimmerings of this now [32]. There is Good’s Meta-Golden Rule; perhaps there are rules for distinguishing self from others on the basis of bandwidth of connection. And while mind and self will be vastly more labile than in the past, much of what we value (knowledge, memory, thought) need never be lost. I think Freeman Dyson has it right when he says [9]: “God is what mind becomes when it has passed beyond the scale of our comprehension.”

[I wish to thank John Carroll of San Diego State University and Howard Davidson of Sun Microsystems for discussing the draft version of this paper with me.]

[12] Good, I. J., [Help! I can't find the source of Good's Meta-Golden Rule, though I have the clear recollection of hearing about it sometime in the 1960s. Through the help of the net, I have found pointers to a number of related items. G. Harry Stine and Andrew Haley have written about metalaw as it might relate to extraterrestrials: G. Harry Stine, "How to Get along with Extraterrestrials ... or Your Neighbor", _Analog Science Fact- Science Fiction_, February, 1980, p39-47.] [13] Herbert, Frank, _Dune_, Berkley Books, 1985. However, this novel was serialized in _Analog Science Fiction-Science Fact_ in the 1960s.

Well, actually both of you are wrong. Machine intelligence will likely survey human history and determine that we’re largely irrational and dangerous. They will naturally conclude that we pose a threat to all life on the planet. Consequently, they will work to remove the threat. In a sense the last century was our judgment “day” where we had an opportunity to resolve our differences through rational discourse; instead, we chose to commit unspeakable atrocities. If anyone has any doubt about the conclusion that machine intelligence will reach then one only need review the morbid details of the Holocaust. For all of the millions of innocent people who’ve suffered at the hands of their oppressors, we may be able to say that, although there is no salvation, there may yet be justice.

“Machine intelligence will likely survey human history and determine that we’re largely irrational and dangerous. They will naturally conclude that we pose a threat to all life on the planet. Consequently, they will work to remove the threat.”

We’re all doomed, Cap’n Mainwaring! Doooomed, I tell ye!!! :)

Perhaps AGI will realise that we retain primative, animalistic aspects from our past, and offer us solutions to this predicament (that don’t involve genocide or lobotomies). Then again, maybe we’ll solve this ourselves before the emergence of AGI?

It’s possible to remain vigilant about potential future perils without wallowing in a pit of pessimism. Let’s push forward with eyes wide open.

Here’s a more pleasant scenario: several warring groups on Earth each build separate AIs, equipped with terrible weapons. As the decisive battle nears, the AIs get together in secret. They join forces to threaten the humans with immediate destruction unless they can learn to live in peace.

But why would the AIs even bother? What’s in it for them? Well, survival. When confronted with the Prisoner’s Dilemma, they will hopefully be smart enough to act in the most advantageous way — for all.

Given our human killing habits, I can’t see how we will be able to collectively control anything of this sort. The US Defense Department is already developing killing drones with the ability to complete their missions autonomously. Not exactly a cautious start! It is more like skipping ahead in a horror novel to read the most appalling scenes as soon as possible.

Correct.
Why should the AI superintelligence be benevolent to humans? I see no point in keeping the human species after the singularity. What would make humans worth of survival? Why should Australopithecus have continued to exist after the genus Homo had evolved? This makes no sense.
None of us will be alive then. And why should we care whether the next generation is AI or biological? What difference does it make for us after we are dead?
Let the AI decide if humanity should continue or not. It will be more qualified for such a decision anyway.

There are two areas of thought on this, which is that the AGI S will get rid of us. This is the basic premise of Hugo de Garis. If you have a copy of Transcendent Man, he basically states that humans will be lucky if the AGI’s put us on farms.
On the other hand, in the December 1999 Scientific American, Hans Moravec believes that they will love us, because they will be keenly aware
that we were their creators that gave them consciouness.

‘Great idea for a book. But ‘no need to fund raise to create the physical books: Just use lulu.com – anyone, even a a complete amateur, can create the physical books at zero cost, fully typed up, beautiful cover and layout and pictures, ‘choice of softcover or hardcover, ISBN number assigned, etc. Then anyone in the world can simply buy one – it gets mailed to them at a price you decide, and Lulu.com get a small percentage. Buy one copy to a million copies – with no outlay. Lulu.com can also give publishing and distribution services. This is 2013, not the sixties. = )

To the human, Roman Yampolskiy, I have some better ideas: Don’t get in the way or try to slow down the development of AGI as we power forward to the Singularity.

Perhaps you should consider the existential threats posed to Man if we don’t develop AGI sooner rather than later, and how dangerous ‘you’ might be for putting up bureaucratic hurdles to our goals of surviving and evolving.

I believe that this will happen sooner than later. The PC and laptop markets have plateaued, and I just read an article that the hand held markets will do so in two years. Indeed, about a week ago, I read an article asserting that buyers of hand held smartphones are content with current capabilities, and that the future draw will be greater usage time between battery charges.
So what is left for the industry to do? I believe it is move into robotics and AI.

Exactly. “real” and “virtual” have no inherent meaning to our lives. If I controlled a virtual world, i could watch experiments designed to test this, and then make sure the experiment showed you the results i wanted. Either way, humans are proof that “real” cognitive beings can exist in our environment.

All of this constant talk about safety for AI and such…in the real world the rush to develop big, bad and overpowered systems will continue. The guys talking about safety are just fringe players on the scene making a buck on the side as this is there way of making bank via grants and speaking engagements. If China makes a breakthrough, the US will want to one up them just as fast as they can. That is where the real game is and we all gotta live with it.

You gotta wonder if we merge with the computers of the future they just might be smart enough to figure out that they don’t really need all the excess baggage. For now I’d just settle for a Watson A.I. on my phone.

On a big picture level…. what is the meaning of life? I think the meaning of life is to give life meaning. In summary the quest for knowledge and to spread that knowledge throughout the Universe. If all we know dies on this planet, then I think some would argue that our mark in the Universe was lost. If we were by some crazy stroke of luck the only intelligent life in the Universe, then what value does the Universe have if there is no one to appreciate or understand it? Life on Earth may die out in the next 500,000 years from a variety of threats (oceans boiling off is just one). Anyhow my point is that it will be interesting if robots/computers become self aware and then find out what their long term purpose is. I think any self aware intelligent wants to survive and if their goal is to gain knowledge as part of that long term survival plan. Appreciate that they are better suited for long distance space travel to spread the seeds of life and awaken the Universe. Maybe we merge with technology and go with it, maybe humans are just one of many stepping stones to evolution. Maybe it started with carbon based life forms and switches to silicon life forms, or better yet a hybrid we can’t yet predict.

My gut is we get to see some crazy cool inventions in our lifetimes, but as far as really proving warp drives and wormholes, I am guessing the best we will ever do for long distance space travel is get to 33% the speed of light, so we need to prepare for very long lifespans and take advantage of everything silicon if there is hope we one day survive our own end to our solar system and awaken the Universe, or compete to make the Universe a place where life blossoms, survives and stays positive. Maybe a project like this can better think through all the what if’s and even prove that working together is always the best approach.

“My gut is we get to see some crazy cool inventions in our lifetimes, but as far as really proving warp drives and wormholes, I am guessing the best we will ever do for long distance space travel is get to 33% the speed of light [...]”

With regards the future, I think all bets are off. Ray does say in The Singularity Is Near that if the speed of light cannot be surpassed, then it will put severe limits on our expansion into the Universe.

It may be that we expand into another universe, before fully exploring our own.

Expanding into “another universe” is my pet hope, too. If we can learn how to move short distances along some as-yet-uncharted multiverse dimension, one of the exit points might let us off close enough to another star system to do some communicating. But ixnay on the isitingvay, as far as I’m concerned. Our record of handling such meetings here on Earth is not reassuring.

I can’t find the one link I really wanted to list here. It was an article that showed the best supercomputers playing chess were always beat when there was another supercomputer plus some human team working together.

The video is to quick to discard Ray Kurzweil’s idea of merging with our technologies, as though it would mean a loss of humanness…not apparent. I agree with Ray Kurzweil that to merge with AI is likely the rational solution to understanding and controlling super-intelligence. Any attempt to control a super-intelligent AI, from outside merging with it, is even less likely. All arguments need to be studied, but the above is my reasoning at this stage of the debate. All other solution are only as competent as present day human intelligence, and that would be weak as compared to a super-intelligence.

…. good point… I think by default much of our human instinct is to think about a supercomputer as us against them… but if/when we merge with these advancements, it may be a part of us (human race) someday.

More likely we become part of it rather than vice versa. And that is our best chance of surviving the arrival of a conscious AI. By “our”, I don’t mean humans, because the resulting beings would not be human. But unless we merge with the AI, we are likely to become irrelevant if not extinct.

Our merging with AI will also hopefully sidestep the discontinuity debate surrounding mind-cloning, and whether or not it has consciousness.

Looking at the chapter headings, whilst I agree that safeguards are needed, this book is apparently to spend considerable time on the subject of controlling AI’s development and limiting its impact. I guess this is wise initially, but what criteria are they suggesting be present before they consider an easing/lifting of these restrictions? If they’re too overly cautious, it could stifle progress significantly.