From: DNA : the story of the genetic revolution by James D. Watson (2017)

The unexpectedly low human gene count provoked several op-ed page ruminations on its significance. These tended toward a common theme. Stephen Jay Gould (whose premature death tragically silenced an impassioned voice), writing in the New York Times, hailed the low count as the death knell of reductionism, the reigning doctrine of virtually all biological inquiry. This doctrine holds that complex systems are built from the bottom up. Put another way: To understand events at complex levels of organization, we must first understand them at simpler levels and piece together these simpler dynamics. And so it follows that by understanding the workings of the genome, we will ultimately understand how organisms are assembled. Gould and others took the surprisingly small human gene count as evidence that such a bottom-up approach is not only unworkable but also invalid. In light of its unexpected genetic simplicity, the human organism, argued the antireductionists, was living proof that we cannot begin to understand ourselves in relation to a sum of smaller processes. To them, our low gene number implied that nurture, not nature, must be the primary determinant of who each one of us is. It was, in short, a declaration of independence from the tyranny supposedly exercised by our genes.

Like Gould, I well appreciate that nurture plays an important part in shaping each of us. His evaluation of nature’s role, however, is utterly wrong: our low gene count by no means invalidates a reductionist approach to biological systems, nor does it justify any logical inference that we are not determined by our genes. A fertilized egg containing a chimp genome still inevitably produces a chimp, while a fertilized egg containing a human genome produces a human. No amount of exposure to classical music or violence on TV could make it otherwise. Yes, we have a long way to go in developing our understanding of just how the information in those two remarkably similar genomes is applied to the task of producing two apparently very different organisms, but the fact remains that the greatest part of what each individual organism will be is programmed ineluctably into its every cell, in the genome. In fact, I see our discovery of a low human gene count as good news for standard reductionist approaches to biology: it’s much easier to sort through the effects of 21,000 genes than 100,000.

Elon Musk, Stuart Russell, Ray Kurzweil, Demis Hassabis, Sam Harris, Nick Bostrom, David Chalmers, Bart Selman, and Jaan Tallinn discuss with Max Tegmark (moderator) what likely outcomes might be if we succeed in building human-level AGI, and also what we would like to happen.

The Beneficial AI 2017 Conference: In our sequel to the 2015 Puerto Rico AI conference, we brought together an amazing group of AI researchers from academia and industry, and thought leaders in economics, law, ethics, and philosophy for five days dedicated to beneficial AI. We hosted a two-day workshop for our grant recipients and followed that with a 2.5-day conference, in which people from various AI-related fields hashed out opportunities and challenges related to the future of AI and steps we can take to ensure that the technology is beneficial.

A superintelligent AI is created with the goal of interfering as little as necessary to prevent the creation of another superintelligence. As a result, helper robots with slightly subhuman intelligence abound, and human-machine cyborgs exist, but technological progress is forever stymied.

Protector god

Essentially omniscient and omnipotent AI maximizes human happiness by intervening only in ways that preserve our feeling of control of our own destiny and hides well enough that many humans even doubt the AI’s existence.

Enslaved god

A superintelligent AI is confined by humans, who use it to produce unimaginable technology and wealth that can be used for good or bad depending on the human controllers.

Conquerors

AI takes control, decides that humans are a threat/nuisance/waste of resources, and gets rid of us by a method that we don’t even understand.

Descendants

AIs replace humans, but give us a graceful exit, making us view them as our worthy descendants, much as parents feel happy and proud to have a child who’s smarter than them, who learns from them and then accomplishes what they could only dream of—even if they can’t live to see it all.

Zookeeper

An omnipotent AI keeps some humans around, who feel treated like zoo animals and lament their fate.

1984

Technological progress toward superintelligence is permanently curtailed not by an AI but by a humanled Orwellian surveillance state where certain kinds of AI research are banned.

Reversion

Technological progress toward superintelligence is prevented by reverting to a pre-technological society in the style of the Amish.

Selfdestruction

Superintelligence is never created because humanity drives itself extinct by other means (say nuclear and/or biotech mayhem fueled by climate crisis).

From Life 3.0 : being human in the age of artificial intelligence by Max Tegmark

"On June 4, 1996, scientists hoping to research Earth’s magnetosphere cheered jubilantly as an Ariane 5 rocket from the European Space Agency roared into the sky with the scientific instruments they’d built. Thirty-seven seconds later, their smiles vanished as the rocket exploded in a fireworks display costing hundreds of millions of dollars. The cause was found to be buggy software manipulating a number that was too large to fit into the 16 bits allocated for it. Two years later, NASA’s Mars Climate Orbiter accidentally entered the Red Planet’s atmosphere and disintegrated because two different parts of the software used different units for force, causing a 445% error in the rocket-engine thrust control. This was NASA’s second super-expensive bug: their Mariner 1 mission to Venus exploded after launch from Cape Canaveral on July 22, 1962, after the flight-control software was foiled by an incorrect punctuation mark. As if to show that not only westerners had mastered the art of launching bugs into space, the Soviet Phobos 1 mission failed on September 2, 1988. This was the heaviest interplanetary spacecraft ever launched, with the spectacular goal of deploying a lander on Mars’ moon Phobos—all thwarted when a missing hyphen caused the “end-of-mission” command to be sent to the spacecraft while it was en route to Mars, shutting down all of its systems."

Conclusions of Behave: the biology of humans at our best and worst by Robert M. Sapolsky.

A GREAT BOOK!

· It’s great if your frontal cortex lets you avoid temptation, allowing you to do the harder, better thing. But it’s usually more effective if doing that better thing has become so automatic that it isn’t hard. And it’s often easiest to avoid temptation with distraction and reappraisal rather than willpower.

· While it’s cool that there’s so much plasticity in the brain, it’s no surprise—it has to work that way.

· Childhood adversity can scar everything from our DNA to our cultures, and effects can be lifelong, even multigenerational. However, more adverse consequences can be reversed than used to be thought. But the longer you wait to intervene, the harder it will be.

· Brains and cultures coevolve.

· Things that seem morally obvious and intuitive now weren’t necessarily so in the past; many started with nonconforming reasoning.

· Cognition and affect always interact. What’s interesting is when one dominates.

· Genes have different effects in different environments; a hormone can make you nicer or crummier, depending on your values; we haven’t evolved to be “selfish” or “altruistic” or anything else—we’ve evolved to be particular ways in particular settings. Context, context, context.

· Biologically, intense love and intense hate aren’t opposites. The opposite of each is indifference.

· Adolescence shows us that the most interesting part of the brain evolved to be shaped minimally by genes and maximally by experience; that’s how we learn—context, context, context.

· Arbitrary boundaries on continua can be helpful. But never forget that they are arbitrary.

· Often we’re more about the anticipation and pursuit of pleasure than about the experience of it.

· You can’t understand aggression without understanding fear (and what the amygdala has to do with both).

· Genes aren’t about inevitabilities; they’re about potentials and vulnerabilities. And they don’t determine anything on their own. Gene/environment interactions are everywhere. Evolution is most consequential when altering regulation of genes, rather than genes themselves.

· We implicitly divide the world into Us and Them, and prefer the former. We are easily manipulated, even subliminally and within seconds, as to who counts as each.

· We aren’t chimps, and we aren’t bonobos. We’re not a classic pair-bonding species or a tournament species. We’ve evolved to be somewhere in between in these and other categories that are clear-cut in other animals. It makes us a much more malleable and resilient species. It also makes our social lives much more confusing and messy, filled with imperfection and wrong turns.

· The homunculus has no clothes.

· While traditional nomadic hunter-gatherer life over hundreds of thousands of years might have been a little on the boring side, it certainly wasn’t ceaselessly bloody. In the years since most humans abandoned a hunter-gatherer lifestyle, we’ve obviously invented many things. One of the most interesting and challenging is social systems where we can be surrounded by strangers and can act anonymously.

· Saying a biological system works “well” is a value-free assessment; it can take discipline, hard work, and willpower to accomplish either something wondrous or something appalling. “Doing the right thing” is always context dependent.

· Many of our best moments of morality and compassion have roots far deeper and older than being mere products of human civilization.

· Be dubious about someone who suggests that other types of people are like little crawly, infectious things.

· When humans invented socioeconomic status, they invented a way to subordinate like nothing that hierarchical primates had ever seen before.

· It’s not great if someone believes it’s okay for people to do some horrible, damaging act. But more of the world’s misery arises from people who, of course, oppose that horrible act . . . but cite some particular circumstances that should make them exceptions. The road to hell is paved with rationalization.

· The certainty with which we act now might seem ghastly not only to future generations but to our future selves as well.

Quote from Behave: the biology of humans at our best and worst by Robert M. Sapolsky (my bullets and highlights)

Evolution rests on three steps: (a) certain biological traits are inherited by genetic means; (b) mutations and gene recombination produce variation in those traits; (c) some of those variants confer more “fitness” than others. Given those conditions, over time the frequency of more “fit” gene variants increases in a population.

We start by trashing some common misconceptions.

First, that evolution favors survival of the fittest. Instead evolution is about reproduction, passing on copies of genes. An organism living centuries but not reproducing is evolutionarily invisible.* The difference between survival and reproduction is shown with “antagonistic pleiotropy,” referring to traits that increase reproductive fitness early in life yet decrease life span. For example, primates’ prostates have high metabolic rates, enhancing sperm motility. Upside: enhanced fertility; downside: increased risk of prostate cancer. Antagonistic pleiotropy occurs dramatically in salmon, who epically journey to their spawning grounds to reproduce and then die. If evolution were about survival rather than passing on copies of genes, there’d be no antagonistic pleiotropy.

Another misconception is that evolution can select for preadaptations—neutral traits that prove useful in the future. This doesn’t happen; selection is for traits pertinent to the present. Related to this is the misconception that living species are somehow better adapted than extinct species. Instead, the latter were just as well adapted, until environmental conditions changed sufficiently to do them in; the same awaits us. Finally, there’s the misconception that evolution directionally selects for greater complexity. Yes, if once there were only single-celled organisms and there are multicellular ones now, average complexity has increased. Nonetheless, evolution doesn’t necessarily select for greater complexity—just consider bacteria decimating humans with some plague.

The final misconception is that evolution is “just a theory.” I will boldly assume that readers who have gotten this far believe in evolution. Opponents inevitably bring up that irritating canard that evolution is unproven, because (following an unuseful convention in the field) it is a “theory” (like, say, germ theory). Evidence for the reality of evolution includes:

· Numerous examples where changing selective pressures have changed gene frequencies in populations within generations (e.g., bacteria evolving antibiotic resistance). Moreover, there are also examples (mostly insects, given their short generation times) of a species in the process of splitting into two.

· Molecular evidence. We share ~98 percent of our genes with the other apes, ~96 percent with monkeys, ~75 percent with dogs, ~20 percent with fruit flies. This indicates that our last common ancestor with other apes lived more recently than our last common ancestor with monkeys, and so on.

· Geographic evidence. To use Richard Dawkins’s suggestion for dealing with a fundamentalist insisting that all species emerged in their current forms from Noah’s ark—how come all thirty-seven species of lemurs that made landfall on Mt. Ararat in the Armenian highlands hiked over to Madagascar, none dying and leaving fossils in transit?

· Unintelligent design—oddities explained only by evolution. Why do whales and dolphins have vestigial leg bones? Because they descend from a four-legged terrestrial mammal. Why should we have arrector pili muscles in our skin that produce thoroughly useless gooseflesh? Because of our recent speciation from other apes whose arrector pili muscles were attached to hair, and whose hair stands up during emotional arousal.