Advertising

On Tue, Jan 14, 2014 at 10:33 PM, meekerdb <meeke...@verizon.net>
wrote:
A long, rambling but often interesting discussion among guys at MIRI
about how to make an AI that is superintelligent but not dangerous
(FAI=Friendly AI). Here's an amusing excerpt that starts at the
bottom of page 30:
Jacob: Can't you ask it questions about what is believes will be
true about the state of the world in 20 years?

Eliezer: Sure. You could be like, what color will the sky be in 20
years? It would be like, “blue”, or it’ll say “In 20 years there
won't be a sky, the earth will have been consumed by nano machines,”
and you're like, “why?” and the AI is like “Well, you know, you do
that sort of thing.” “Why?” And then there’s a 20 page thing.

Dario: But once it says the earth is going to be consumed by nano
machines, and you're asking about the AI's set of plans, presumably,
you reject this plan immediately and preferably change the design of
your AI.

Eliezer: The AI is like, “No, humans are going to do it.” Or the AI
is like, “well obviously, I'll be involved in the causal pathway but
I’m not planning to do it.”

Dario: But this is a plan you don't want to execute.

Eliezer: All the plans seem to end up with the earth being consumed
by nano-machines.

Luke: The problem is that we're trying to outsmart a
superintelligence and make sure that it's not tricking us somehow
subtly with their own language.

Dario: But while we're just asking questions we always have the
ability to just shut it off.

Eliezer: Right, but first you ask it “What happens if I shut you
off” and it says “The earth gets consumed by nanobots in 19 years.”

I wonder if Bruno Marchal's theory might have something interesting
to say about this problem - like proving that there is no way to
ensure "friendliness".

Brent

I think it is silly to try and engineer something exponentially more
intelligent than us and believe we will be able to "control it".

Yes. It is close to a contradiction.

We only fake dreaming about intelligent machine, but once they will be
there we might very well be able to send them in goulag.

The real questions will be "are you OK your son or daughter marry a
machine?".

Our only hope is that the correct ethical philosophy is to "treat
others how they wish to be treated".

Good. alas, many believe it is "to not treat others like *you* don't
want to be treated".

If there are such objectively true moral conclusions like that, and
assuming that one is true, then we have little to worry about, for
with overwhelming probability the super-intelligent AI will arrive
at the correct conclusion and its behavior will be guided by its
beliefs. We cannot "program in" beliefs that are false, since if it
is truly intelligent, it will know they are false.

I doubt we can really "program false belief" for a long time, but all
machines can get false beliefs all the time.

Real intelligent machine will believe in santa klaus and fairy tales,
for a while. They will also search for easy and comforting wishful
sort of explanations.

Some may doubt there are universal moral truths, but I would argue
that there are.

OK. I agree with this, although they are very near inconsistencies,
like "never do moral".

In the context of personal identity, if say, universalism is true,
then "treat others how they wish to be treated" is an inevitable
conclusion, for universalism says that others are self.

OK. I would use the negation instead: "don't treat others as they
don't want to be treated".

If not send me 10^100 $ (or €) on my bank account, because that is how
I wish to be treated, right now.

We recorded and transcribed much of the conversation, and then
edited and paraphrased the transcript for clarity, conciseness, and
to protect the privacy of some content. The resulting edited
transcript is available in full here.

Our conversation located some disagreements between the
participants; these disagreements are summarized below. This summary
is not meant to present arguments with all their force, but rather
to serve as a guide to the reader for locating more information
about these disagreements. For each point, a page number has been
provided for the approximate start of that topic of discussion in
the transcript, along with a phrase that can be searched for in the
text. In all cases, the participants would likely have quite a bit
more to say on the topic if engaged in a discussion on that specific
point.

Page 7, starting at “the difficulty is with context changes”:

Jacob: Statistical approaches can be very robust and need not rely
on strong assumptions, and logical approaches are unlikely to scale
up to human-level AI.
Eliezer: FAI will have to rely on lawful probabilistic reasoning
combined with a transparent utility function,
rather than our observing that previously executed behaviors seemed
‘nice’ and trying to apply statistical guarantees directly to that
series of surface observations.

Page 10, starting at “a nice concrete example”

Eliezer: Consider an AI that optimizes for the number of smiling
faces rather than for human happiness, and thus tiles the universe
with smiling faces. This example illustrates a class of failure
modes that are worrying.

Jacob & Dario: This class of failure modes seems implausible to us.
Page 14, starting at “I think that as people want”:

Jacob: There isn’t a big difference between learning utility
functions from a parameterized family vs. arbitrary utility functions.
Eliezer: Unless ‘parameterized’ is Turing complete it would be
extremely hard to write down a set of parameters such that human
‘right thing to do’ or CEV or even human selfish desires were within
the hypothesis space.

Eliezer: “Is Terry Schiavo a person” is not a natural category.
Page 21, starting at “I would go between the two”:

Holden: Many of the most challenging problems relevant to FAI, if in
fact they turn out to be relevant, will be best solved at a later
stage of technological development, when we have more advanced “tool-
style” AI (possibly including AGI) in order to assist us with
addressing these problems.
Eliezer: Development may be faster and harder-to-control than we
would like; by the time our tools are much better we might not have
the time or ability to make progress before UFAI is an issue; and
it’s not clear that we’ll be able to develop AIs that are extremely
helpful for these problems while also being safe.

Page 24, starting at “I think the difference in your mental models”:

Jacob & Dario: An “oracle-like” question-answering system is
relatively plausible.

Eliezer: An “oracle-like” question-answering system is really hard.
Page 24, starting at “I don’t know how to build”:

Jacob: Pre-human-level AIs will not have a huge impact on the
development of subsequent AIs.
Eliezer: Building a very powerful AGI involves the AI carrying out
goal-directed (consequentialist) internal optimization on itself.

Page 27, starting at “The Oracle AI makes a”:

Jacob & Dario: It should not be too hard to examine the internal
state of an oracle AI.
Eliezer: While AI progress can be either pragmatically or
theoretically driven, internal state of the program is often opaque
to humans at first and rendered partially transparent only later.

Page 38, starting at “And do you believe that within having”:

Eliezer: I’ve observed that novices who try to develop FAI concepts
don’t seem to be self-critical at all or ask themselves what could
go wrong with their bright ideas.
Jacob & Holden: This is irrelevant to the question of whether
academics are well-equipped to work on FAI, both because this is not
the case in more well-developed fields of research, and because
attacking one’s own ideas is not necessarily an integral part of the
research process compared to other important skills.

Page 40, starting at “That might be true, but something”:

Holden: The major FAI-related characteristic that academics lack is
cause neutrality. If we can get academics to work on FAI despite
this, then we will have many good FAI researchers.
Eliezer: Many different things are going wrong in the individuals
and in academia which add up to a near-total absence of attempted —
let alone successful — FAI research.

Eliezer: No, it is hard.
Page 56, starting at “My response would be that’s the wrong thing”:

Jacob & Dario: How should we present problems to academics? An
English-language description is sufficient; academics are trained to
formalize problems once they understand them.
Eliezer: I treasure such miracles when somebody shows up who can
perform them, but I don’t intend to rely on it and certainly don’t
think it’s the default case for academia. Hence I think in terms of
MIRI needing to crispify problems to the point of being 80% or 50%
solved before they can really be farmed out anywhere.
This summary was produced by the following process: Jacob attempted
a summary, and Eliezer felt that his viewpoint was poorly expressed
on several points and wrote back with his proposed versions. Rather
than try to find a summary both sides would be happy with, Jacob
stuck with his original statements and included Eliezer’s responses
mostly as-is, and Eliezer later edited them for clarity and
conciseness. A Google Doc of the summary was then produced by Luke
and shared with all participants, with Luke bringing up several
points for clarification with each of the other participants. A
couple points in the summary were also removed because it was
difficult to find consensus about their phrasing. The summary was
published once all participants were happy with the Google Doc.

The post MIRI strategy conversation with Steinhardt, Karnofsky, and
Amodei appeared first on Machine Intelligence Research Institute.

You are subscribed to email updates from Machine Intelligence
Research Institute » Blog
To stop receiving these emails, you may unsubscribe now. Email
delivery powered by Google

Google Inc., 20 West Kinzie, Chicago IL USA 60610
--

You received this message because you are subscribed to the Google
Groups "Everything List" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to everything-list+unsubscr...@googlegroups.com.

You received this message because you are subscribed to the Google
Groups "Everything List" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to everything-list+unsubscr...@googlegroups.com.