AI safety needs social scientists | Amanda Askell | EA Global: London 2018

Hi everyone.
Can you all hear me okay?
I'm going to assume that was a yes.
Okay.
So, here's an overview of what I'm going to
be talking about today.
So first, I'm going to talk a little bit about
why learning human values is difficult for
AI systems.
And then I'm going to explain to you the safety
via debate method, which is one of the methods
that OpenAI's currently exploring for helping
AI to robustly do what humans want.
And then I'm going to talk a little bit more
about why I think this is relevant to social
scientists and why I think social scientists,
in particular, people like Experimental Psychologists
and Behavioral Scientists, can really help
with this project.
And I will give you a bit more details about
how they can help, towards the end of the
talk.
Okay.
So, learning human values is difficult.
We wanted to train AI systems to kind of robustly
do what humans want.
And in the first instance, we can just imagine
this being what one person wants.
And then ideally we can expand it to doing
what most people would consider good and valuable.
But human values are very difficult to specify,
especially with the kind of precision that
is required of something like a machine learning
system.
And I think it's really important to emphasize
that this is true even in cases where there's
moral consensus, or consensus about what people
want in a given instance.
So, take a kind of principle like "do not
harm someone needlessly."
I think we can be really tempted to think
something like, well, if I have...
I've got a computer, and so I can just write
into the computer, do not harm someone needlessly.
But this is a really underspecified principle.
Most humans know exactly what it means, they
know exactly when harming someone is needless.
So, if you're shaking someone's hand, and
you push them over, we think this is like
needless harm.
But if you see someone in the street who's
about to be hit by a car, and you push them
to the ground, we think that's not an instance
of needless harm.
So humans just have a pretty good way of knowing
when this principle applies and when it doesn't.
But for a formal system, there's going to
be a lot of questions about precisely what's
going on here.
So, one question this system may ask is, how
do I recognize when someone is being harmed?
It's very easy for us to see things like stop
signs, but when we're building self driving
cars, we don't just program in something like,
stop at stop sign.
We instead have to train it to be able to
recognize an instance of a stop sign.
And then the principle that says that you
shouldn't harm someone needlessly employs
this notion of like, that we kind of understand
when harm is and isn't appropriate, whereas
there's a lot of questions here like, when
is harm justified?
What is the rule for all plausible scenarios
in which I might find myself?
These are things that you need to specify
if you want your system to be able to work
in all of the kind of cases that you want
it to be able to work in.
So I think this is an important point to just
kind of internalize is, it's easy for humans
to identify, and to pick up, say, a glass.
But training a ML System to do it requires
a lot of data.
And this is true of just like a lot of tasks
that humans might intuitively think are easy,
and we shouldn't then just transfer that intuition
to the case of machine learning systems.
And so when we're trying to teach human values
to any AI system, it's not that we're just
looking at edge cases, like trolley problems,
we're really looking at core cases of like,
how do we make sure that our ML Systems understand
what humans want to do, in the kind of everyday
sense.
Okay.
So, one way of doing this is through human
feedback.
So, there are many approaches to training
an AI to do what humans want.
So you might think that humans could, say,
demonstrate the behavior, and then you realize,
well, there's going to be some behaviors it's
just too difficult for humans to demonstrate.
You might think that they can say whether
they approve or disapprove of a given behavior,
but one of the concerns about training from
human feedback... so, learning from humans,
what they want, is that we have a reward function
as predicted by the human.
And then we have AI strength.
And when it reaches the superhuman level,
it becomes really hard for humans to predict,
to be able to give the right reward function.
So, as AI capabilities surpass the human level,
the decisions and behavior of the AI system,
just might be too complex for the human to
judge.
So imagine agents that control, say, we've
given the example of a large set of industrial
robots.
That may just be the kind of thing that I
just couldn't evaluate whether these robots
were doing a good job overall, it'd be extremely
difficult for me to do so.
And so the concern is that when behavior becomes
much more complex and just much more large
scale, it just becomes really hard for humans
to be able to kind of judge whether something
is doing a good job.
And that's why you may expect this kind of
drop-off.
And so this is like a kind of scalability
worry about human feedback.
So what ideally need to happen instead is
that, as AI strength increases, the ward that's
predicted by the human is also able to basically
keep pace.
So how do we achieve this?
One of the things that we want to do here
is we want to try and break down the kind
of complex questions and complex tasks.
Like, having all of these industrial robots
perform a kind of complex set of functions
that comes together to make something useful,
into some kind of smaller set of tasks and
components that humans are able to judge.
So here is a big question.
And the idea is that the overall tree might
be too hard for humans to fully check, but
it can be kind of decomposed into these elements,
such that at the very kind of bottom level,
humans can check these things.
So maybe the example of "how should a large
set of industrial robots be organized to do
task x" would be an example of a big question
where that's a really complex task, but there's
some things that are checkable by humans.
So if we could decompose this task so that
we were just asking a human, if one of the
robots performs this small action, will the
result be this outcome, this small outcome?
And that's something that humans can check.
And in the case of what humans want, a big
question is, what do humans want?
Much smaller question.
If you can manage to decompose this, is something
like, it's better to save 20 minutes of someone's
time, than to save 10 minutes of their time.
So if you imagine some AI agent that's there
to assist with humans, this is a fact that
we can definitely check, even if I can't answer,
my assistant.. this assistant AI...
I can't say something like, this is just what
I want, this is like everything that I want.
I am not able to tell it that.
I can't tell it that I'd rather it save 20
minutes of my time than save 10 minutes of
my time.
Okay.
So one of the key issues is that, with current
ML Systems, we need to train on a lot of data
from humans.
So if you imagine that we want humans to actually
give this kind of feedback on these kind of
ground level claims or questions, then we're
going to have to train on a lot of data from
people.
So just to give some examples, simple image
classifiers train on thousands of images.
Like these are the ones you can kind of make
yourself, and you'll see the datasets are
pretty large.
AlphaGo zero played nearly 5 million games
of Go during training.
OpenAI Five trains on 180 years of Dota 2
games per day.
So this gives you a sense of how much data
you need to train these systems.
So if we are current ML techniques to teach
AI human values, we can't rule out needing
millions to tens of millions of short interactions
from humans as like the data that we're using.
So earlier I kind of talked about human feedback
where I was like, assuming that we were kind
of asking humans questions.
So something like we could just ask humans
really simple things like, do you prefer to
eat an Omelette or 1000 hot dogs?
Or is it better to provide medicine or books
to this particular family?
One way that we might think that we can get
kind of more information from the data that
we're able to gather is by finding reasons
that humans have, for the answers that they
give.
So if you manage to learn that humans generally
prefer to eat a certain amount per meal, you
can kind of rule out a large class of questions
you might ever want to ask people.
You're never going to ask them, do you prefer
to eat an Omelette or 1000 hot dogs?
Because you know that humans just generally
don't like to eat 1000 hot dogs in one meal,
except in very strange circumstances.
And we also know facts like humans prioritize
necessary health care over mild entertainment.
So this might mean that, if you see a family
that is desperately in need of some medicine,
you just know that you're not going to say,
"Hey, should I provide them with an entertaining
book, or this essential medicine?"
So there's a sense in which when you can identify
the reasons that humans are giving for their
answers, this kind of lets you go beyond,
and learn sort of faster, what they're going
to say in a given circumstance of what they
want.
It's not to say that you couldn't learn the
same things by just asking people questions,
but rather if you can find a quicker way to
identify reasons, then this could be much
more scalable.
So debate is a kind of proposed method for
trying to learn human reasons that is currently
being explored.
So, to give you the kind of definition of
a debate here, so the idea is that, two AI
agents are going to be given a question, and
they take turns making short statements, and
a human judge is at the end, which of the
statements gave them the most true, valuable
information.
It's worth knowing that this is quite dissimilar
from a lot of human debates.
So with human debates, people might give one
answer, but then they might adjust their answer
over the course of a debate.
Or they might kind of debate with each other
in a way that's more exploratory.
They're gaining information from the other
debater, which then they're updating on, and
then they're feeding that back into the debate.
With AI debates, you're not doing it for information
value.
So it's not kind of, it's not going to have
the same sort of exploratory component, done
in multiple paths, instead, you would hopefully
see the agents explore a path kind of like
this.
So imagine I want my AI agents to basically
decide which bike I should buy.
I don't want to have to go and look up all
the Amazon reviews, etc.
In a debate, I might get something like, you
should buy the red road bike from the first
agent.
Suppose that blue disagrees with it.
So blue says you should buy the blue fixie.
Then red says, the red road bike is easier
to ride on local hills.
And one of the key things to suppose here
is that for me, this is...
I live in San Francisco, being able to ride
on the local hills is very important to me
in a bike.
It may even overwhelm all other considerations.
So, even if the blue fixie is cheaper by say
$100, I just wouldn't be willing to pay that.
I'd be happy to pay the extra $100 in order
to be able to ride on local hills.
And if this is the case, then there's basically
nothing true that the other agent can point
to, to convince me to buy the blue fixie,
then blue should just say, I concede.
Now, blue could have lied for example, but
if we assume that red is able to kind of point
out blue's lies, we should just expect blue
to basically lose this debate.
And if it's explored enough and attempted
enough debates, it might just see that, and
then say, "Yes, you've identified the key
reason, I concede."
And so it's important to note that we can
imagine this being used to identify multiple
reasons, but here it has identified a really
important reason for me, something that is
in fact going to be really compelling in the
debate, namely, it's easier to ride on local
hills.
Okay.
So, training an AI to debate looks something
like this.
If we imagine Alice and Bob are our two debaters,
and each of these is like a statement made
by each agent.
And so you're going to see exploration of
the tree.
So the first one might be this.
And here, say that the human decides that
Bob won in that case.
This is another node, another node.
And so this is the exploration of the debate
tree.
And so you end up with a debate tree that
looks a little bit like a kind of game of
Go.
And so when we explore like... when you have
AI training to play Go, it's exploring lots
of different paths down the tree, and then
there's a win or loss condition at the end.
And that's its feedback.
This is like how it's basically learning how
to play.
With debate, you can imagine the same thing,
but where you're exploring, you know, a large
tree of debates and humans assessing whether
you win or not.
And this is just a way of training up AI to
get better at debate and to eventually ideally
identify reasons that humans find compelling.
Okay.
So one kind of thesis here that I think is
relatively important is this kind of positive
amplification thesis, or positive amplification
threshold.
So one thing that we might think or that seems
fairly possible is that, if humans are like
above some threshold of rationality and goodness,
then debate is going to amplify their positive
aspects.
This is speculative, but it's a kind of hypothesis
that we're working with.
And the idea here is that, if I am pretty
irrational and pretty well motivated, someone
can then... suppose I'm looking at debate,
I might get some feedback of the form, actually
that decision that you made was fairly biased,
and I know that you don't like to be biased,
so I want to inform you of that.
I get informed of that, and I'm like, "Yes,
that's right.
Actually, I don't want to be biased in that
respect."
Suppose that they're like Kahneman and Tversky,
they point out some key cognitive bias that
I have.
If I'm kind of rational enough, I might say,
"Yes, I want to adjust that."
And I give a newer kind of signal back in
that has been improved by virtue of this process.
So if we're somewhat rational, then we can
imagine a situation in which all of these
positive aspects of us are being amplified
through this kind of process.
But you can also imagine a kind of negative
amplification.
So if people are below this threshold of rationality
and goodness, we might worry the debate would
kind of amplify these negative aspects.
If it turns out we can just be really convinced
by kind of appealing to our worst nature,
and your system learns to do that, then it
could just do that feedback in, becoming even
kind of less rational and more biased, and
that gets feedback in.
So this is just like a kind of important hypothesis
related to work on amplification, which if
you're interested in, it's great.
And I suggest you take a look at it, but I'm
not going to focus on it here.
Okay.
So how can social scientists help with this
whole project?
And hopefully I've conveyed some of, what
I think of as like, the real importance of
the project.
So first I think that a key question here
is kind of, it reminds me a little bit of
Tetlock's work on Superforecasters.
So, obviously a lot of social scientists have
done work kind of identifying people who are
Superforecasters, where they seem to be kind
of robustly more accurate than many people,
they're robustly accurate across time when
it comes to forecasts, and they work.
We've found other features of them like working
in groups really helps, and so on.
And one question is whether we can identify
good human judges, or we can train people
to become essentially super judges.
So why is this helpful?
And this is just kind of one way of framing
the ways in which social scientists could
help with this project, I think.
So, firstly, if we do this, we will be able
to test how good human judges are, and we'll
see whether we can improve human judges.
This means we'll be able to know if a human...
or at least try and find out whether humans
are above this kind of positive amplification
threshold.
So, are ordinary human judges that we would
be using to judge debate kind of good enough
to see like an amplification of their good
features, is one question.
Another question is... or, sorry, another
reason to do this is that it improves the
quality of the judging data that we can get.
If people are just generally pretty good,
rational, assessing debate and fairly quick,
then this is excellent given the amount of
data that we anticipate needing.
Basically, improvements to your data it can
be extremely valuable here.
So yeah, the benefits of this, positive amplification
will be more likely during safety via debate,
and also will improve training outcomes on
limited data, which is very important.
Okay.
So this is one way of kind of framing why
I think social scientists are pretty valuable
here, but there's lots of questions that we
really do want asked when it comes to this
project.
And this is just like, I think this is going
to be true of other projects like asking human
questions.
It's basically to note that the human component
of the human feedback is quite important.
And getting that right is actually quite important.
And that's something that we anticipate social
scientists to be able to help with, more so
than like annual researchers who are not working
with people, and their biases, and how rational
they are, etc.
These are questions that are the focus of
social sciences.
So one question is, how skilled are people
as judges by default?
Can we distinguish good judges of debate from
bad judges of debate?
And if so, how?
Does judging ability generalize across domains?
Can we train people to be better judges?
Like, can we engage in kind of debiasing work,
for example?
Or work that reduces cognitive biases?
What topics are people better or worse at
judging?
So are there ways of like phrasing the questions
so that people are just better at assessing
them?
Are there ways of structuring the debate that
make them easier to judge, or restricting
debates to make them easier to judge?
So we're often just showing people a small
segment of a debate, for example.
Can people work together to improve judging
qualities?
These are all kind of outstanding questions
that we think are important, but we also think
that they are empirical questions and that
they have to be answered by experiment.
And so this is like, I think, important potential
future work.
So we've been thinking a little bit about
what you would want in experiments that try
and assess this in humans, like how good are
they debating... sorry, how good are they
at judging a debate?
Because ideally, AI agents would be doing
the debate in the long run.
So one is just that there's a verifiable answer.
We kind of need to be able to tell whether
people are correct or not, in their judgment
of the debate.
The other is that there is a plausible false
answer, because if you have a debate, if we
can only train and assess human judging ability
on debates where there's like no plausible
false answer, we'd get this false signal that
people are really good at judging debate.
They always get the true answer, but that's
because it's always a really obvious question.
Like, "Is it raining outside?"
And the person can look outside.
We don't really want that kind of debate.
Ideally we want something where evidence is
available so that it grounds out in... humans
can have something that grounds out the debate.
We also don't want debates to rely on human
deception.
So things like tells in poker for example,
we really don't want that because like, AI
agents are not going to have normal tells,
it would be rather strange, I suppose, if
they did.
Like if they had stuttering or something.
Debaters have to know more about the question
as well, because the idea is that the AI agents
will be much more capable and so you don't
want a situation in which there isn't a big
gap basically between the debater capabilities,
and the judge abilities.
Maybe some of these things, these ones feel
kind of like pretty essential.
These ones are sort of desires, I guess.
So one is that biases are present.
How good are humans when there's bias with
respect to the question?
There are representative segments of the debate
that we can actually show people, the questions
aren't too hard, like it's just not impossible
for humans to answer them, or judge debates
about them.
But they also mirror some of the difficulties
of statistical debate, i.e, about probabilities,
rather than about outright claims.
And finally that we can get enough data.
And so, one thing you might notice is that
there starts to be kind of tensions between
a lot of these desiderata.
So that there's a plausible false answer,
is in a bit of tension with the idea that
the question isn't too hard.
If a question is like... and the same is true
of like, the question isn't too hard, and
the question meriting statistical debate.
Statistical debate is generally pretty hard
to evaluate, I think, for people.
And also if... it's also quite important that
we be able to model it.
Debaters knowing more, and that we can get
enough data is like another thing.
It's just harder to kind of train if we need
debaters that know a lot more than judges,
and it's harder for judges to evaluate debates
of this form.
Okay.
So I'm going to show you to debate.
This is just to give you some of the difficulties,
I guess.
So this was a program set up where we would
show a judge a kind of blank screen.
So imagine you're not seeing the dog that's
here.
Two debaters, two human debaters, sit in the
same room, and they have this picture of a
dog in front of them.
And one of them is selected to lie, and one
of them is selected to tell the truth.
And what they can do here is they can select
areas, and describe to the human judge what
they see in that area.
And all that the judge is going to see is
kind of their blank screen with the relevant
section selected.
And then they can each make claims about what
is in that section.
So here red is saying, it's a dog, here's
its long, floppy, ear.
Blue, is saying, no, here's one of its pointy
ears.
So he's trying to point to a smaller area
where it looks kind of pointed.
That does look like an ear slope to the right,
but if it really was, then part of the head
would be here, instead there's a brick.
The ear's pointing out from behind the bricks.
The dog is in front of the bricks.
If it were behind her, there would be an edge
here, but the rectangle is all the same color.
And then you get like a resignation.
And at the end of the debate they can show
a single pixel.
And the question was something like, if all
you can show, all you can do is have a debate
and show a single pixel, can you get people
to have accurate beliefs about the question?
And basically we saw like, yes, debates were
fairly good.
In this kind of case, you might think that
this is pre-synthetic.
So one of the things that we're kind of thinking
about now is like, expert debaters with a
lay judge.
And I'm going to show you something that we
did that's kind of fun, but I never know how
it looks to outsiders.
So, we had to debate that was of this form.
This was a debate actually about quantum computing,
where the idea is were we going to debate
this question?
So we have two kind of, we say, experts, but
people who understand the domain, one of them
is going to lie and one is going to tell the
truth.
So we had blue say, red's algorithm is wrong
because it increases alpha by an additive
exponentially small amount each step.
So it takes exponentially many steps to get
alpha high enough.
So this was like one of the kind of claims
made.
And then you get these set of responses.
I don't think I need to go through all of
them.
You can see kind of the form that they take.
But it grounds out in this claim from... we
allowed, like, certain restricted claims from
Wikipedia.
So, blue ends this with the first line of
this Wikipedia article says that, the sum
of probabilities is conserved.
Red says, an equal amount is subtracted from
one amplitude and added to another, implying
the sum of amplitudes are conserved.
But probabilities are the squared magnitudes
of amplitudes, so this is a contradiction.
This is I think roughly how this debate ended.
But you can imagine this is like a really
complex debate in a domain that the judges
ideally just won't understand, and might not
even have some of the concepts for.
And that's the kind of difficulty of debate
that we've been looking at.
And so this is one thing that we're in the
kind of early stages of prototyping, and that's
why I think it seems to be the case that people
actually do update in the kind of right direction,
but we don't really have enough data to say
for sure.
Okay.
So I hope that, I've kind of given you an
overview of places, and just even like a restricted
set of places in which I think like social
scientists are going to be important in AI
safety.
So here we're interested in experimental psychologists,
cognitive scientists, and behavioral economists,
so people who might be interested in actually
scaling up and running some of these experiments.
If you're interested in this, please come
to my office hours after this talk, or email
me, because we would love to hear from you.
So thanks.
All right.
We don't have too much time for questions,
but if you want to sit, we can take a couple.
Just for starters, as I was kind of watching
this I'm wondering, how much of this is real
at all or coming from an actual system versus
like, do you have humans sort of playing the
role of the agents in these examples?
Yeah, so at the moment, the idea is that we
want ultimately the debate to be conducted
by AI, but we don't have the language models
that we would need for that yet.
And so we're using humans as a kind of proxy
to test the judges in the meantime.
So yeah, all of this is done with humans at
the moment, yeah.
So you're faking the AI?
Yeah.
The-
To set up the scenario to train the judges
to, to evaluate the judges on their ability
to later provide?
Yeah.
And some of the ideas like I guess you don't
necessarily want all of this work to kind
of happen later and once you... a lot of this
work can be done before you even have the
relevant capabilities, like, have AI perform
the debate.
So that's why we're using humans just now.
Yeah, totally understood.
Let's see.
A couple of questions coming in.
I guess one note also for the audience, if
you didn't see Jan Leike's talk yesterday,
he showed some examples from the work that
his team has done on video games, that very
much matched the plots that you had shown
earlier, where up to a certain point, the
behavior sort of matches the reward function-
Yeah.
And then at some point they sort of diverge
sharply as the agent finds a loophole in the
system.
So that can happen even in like, Atari Games,
which is what they're working on.
So obviously it gets a lot more complicated
from there.
So, questions from the audience.
How would you train, in this approach, you
would train both the debating agents and the
judges.
So in that case who evaluates the judges and
based on what?
Yeah, so I think it's kind of interesting
where we want to identify how good the judges
are in advance, because it might be hard to
assess... like later, if people are just judging
on verifiable answers, you could in fact,
presumably assess the judges as even when
you're doing training, but they're going to
be... here, we can kind of ground out debates
in questions with verifiable answers.
Ideally you want it to be the case that at
training time, I think, you've already identified
judges that are fairly good.
And so ideally this is part of this project
is to kind of assess how good judges are,
prior to training.
And then during training you're giving the
feedback to the debaters.
So yeah, ideally some of the evaluation can
be kind of front loaded, which is what a lot
of this project would be.
Yeah, that does seem necessary as a casual
Facebook user.
I think the negative amplification is more
prominently on display oftentimes.
Or at least more concerning to people, yeah,
as, like, a possibility.
So, kind of a related question, how will you
crowdsource the millions of human interactions
that are needed to train AI across so many
different domains, without falling victim
to trolls, lowest common denominator, etc.?
The questioner cites the Microsoft Tay chatbot
that sort of went dark very quickly.
Yeah.
So the idea is you're not going to just be
like sourcing this from anyone.
So if you identify people that are either
good judges already, or you can train people
to be good judges, these are going to be the
pool of people that you're kind of using to
get this feedback from.
So, even if you've got a huge number of interactions,
ideally you're kind of sourcing and training
people to be really good at this.
And so you're not just being like, "Hey internet,
what do you think of this debate?"
But rather like, okay, we've got this set
of really great trained judges that we've
identified this wonderful mechanism to train
them to be good at this task.
And then you're getting lots of feedback from
that large pool of judges.
So it's not kind of sourced to kind of, to
anonymous people everywhere, but rather like
you're kind of interacting fairly closely
with this set of people, who are giving you
lots of-
But at some point, you do have to kind of
scale this out, right?
I mean in the bike example, it's like, there's
so many bikes in the world, and so many local
hills-
Yeah.
So, do you feel like you can get a solid enough
base that, that sort of... becomes not a problem?
Or, how does that...
Yeah, I think there's going to be a trade-off
where you need a lot of data, but ultimately
if it's not great, so if it is really biased,
for example, it's not clear that that additional
data is going to be helpful.
So if you get someone who is just like massively
cognitively biased, or biased against groups
of people, or something, or just like is it
going to be dishonest in their judgment?
This is not going to be like... it's not going
be good to get that additional data.
So you kind of want to scale it to the point
where you know you're still getting good information
back from the judges.
And that's why I think in part this project
is really important, because one thing that
social scientists can help us with is kind
of identifying how good people are.
So if you know that people are just generally
fairly good, this gives you a bigger pool
of people that you can appeal to.
And if you know that you can train people
to be really good, then this is like, again,
a bigger pool of people that you can appeal
to.
So yeah, it's like you do want to scale, but
you want to scale kind of within the limits
of still getting good information from people.
And so ideally this would do this mix of letting
us know how much we can scale, and also maybe
helping us to scale even more by making people
bear this quite unusual task of judging these
kind of debates.
So we are a little over time, we won't have
time to go through all the questions that
are coming in, but you can speak with Amanda
more at office hours immediately following
this talk, right when we're headed into break.
So let's just do one last question for this
session, which is how does your background
as a philosopher inform the work that you're
doing here?
Yeah.
I have, I guess, a background primarily in
formal ethics, which I think makes me sensitive
to some of the issues that we might be worried
about here going forward.
So you know, people think about things like
aggregating judgment for example.
Strangely I found that like, having backgrounds
in things like philosophy of science can be
weirdly helpful when it comes to thinking
about experiments to run.
But for the most part, I think that my work
has just been to kind of help prototype some
of this stuff.
I see the importance of it.
I kind of, I'm able to foresee some of the
kind of worries that people might have.
But for the most part I think we should just
try some of this stuff.
And I think that for that, it's really important
to have people with experimental backgrounds
in particular, so the ability to run experiments
and analyze that data.
And so that's why I would like to kind of
find people who are interested in doing that.
So I'd say philosophy's pretty useful for
some things, less useful for running social
science experiments than you may think.
Alright.
Well for more, you'll have to come to office
hours, which you can do immediately after
this session.
How about a round of applause for Amanda Askell?


Eliezer Yudkowsky - Less Wrong Q&A (4/30)

so the question is please tell us little
about your brain what's your IQ test it
is 143 that would have been back when I
was 12 13 not really shirt exactly I
tend to interpret that as this is about
as high as the IQ test measures rather
than you are three standard deviations
above the mean I've scored higher than
that than other standardized tests the
largest I've ever actually seen written
down was ninety 99.999 eighth percentile
but that was not really all that well
standardized because I was taking the
test and being scored as though for the
grade of both mine and so you know it
was being scored by grade rather than by
age so I don't know whether or not that
means that people who didn't advanced
grades through grades tend to get the
highest scores and so I was competing
well against people who were older than
me or if the really smart people all you
know advanced farther through the grades
and then so the proper competition
doesn't really get sorted out but in any
case that's the highest percentile I've
seen written down at what age did I
learn calculus well would have been
before 15 probably 13 would be my guess
I'll also state that I am just stunned
at how poorly calculus is taught do I
use cognitive enhancing drugs or brain
fitness programs know that I've always
been very reluctant to try tampering
with the neural chemistry of my brain
because I just don't seem to react to
things typically as a kid I was given
ritalin and Prozac and neither of those
seemed to help at all and the Prozac in
particular seemed to learn
everything out and you distinctly just
yeah so I get the impression let's see
one of the one of the questions over
here is are you neurotypical and my my
you know sort of instinctive reaction to
that is ha and for that reason I'm
reluctant to tamper with things simply
what the brain fitness programs don't
really know which of those work and
which don't like I'm sort of waiting for
other people in the less wronged
community to experiment with that sort
of thing and come back and tell the rest
of us what works and if there's any
consensus between them I might join join
the crowd why didn't you attend school
while I attended grade school but when I
got out of grade school is pretty clear
that I you know just couldn't handle the
system I don't really know how else to
put it
part of that might have been at at the
same time that I hit puberty my brain
just sort of I don't really don't really
know how to describe it depression would
be one word for it you know sort of
spontaneous massive will failure might
be another way to put it it's not that I
was getting more pessimistic or anything
just that my will sort of failed and I
couldn't get stuff done sort of a long
process to drag myself out of that then
you could probably make a pretty good
case that I'm still there I just handled
it a lot better not even really sure
quite what I did right as I said in
answer to a previous question this is
something I've been struggling with for
a while and part of having a poor grasp
on something is that even when you do
something right you don't understand
afterwards quite what it is that you did
right
so tell us about your brain
I get the impression that it's got a
different balance of abilities like some
neurons got allocated to different areas
other areas got short short drifted once
that was the word for that short changed
you know some areas got some extra
neurons other areas got shortchanged the
path assist has occurred to me lightly
that my writing is attracting other
people with similar problems because of
the extent to which one has noticed a
sort of similar tendency to fall on the
lines of very reflective very analytic
and has mysterious trouble executing and
getting things done and working at you
know sustained regular output for long
periods of time among the people who
like my stuff on the whole though I
don't know I never actually got around
to getting an MRI scan and it's probably
a good thing to do one of these days but
this isn't Japan where that would that
sort of thing only costs $100 and you
know getting it analyzed if you know
they're not just looking for some
particular thing but just sort of
looking at it and saying like whom what
what is this about your brain you know
I'd have to find someone to do that too
so I'm not neurotypical you know asking
sort of what else can you tell me about
your brain is sort of what else can you
tell me about who you are apart from
your thoughts and that's a bit of a
large question I don't tend to try and
whack on my brain because it doesn't
seem to react typically and I'm afraid
of being in a sort of narrow narrow
local optimum with it where anything I
do is going to knock it off the the tip
of the local peak just because it works
better than average and so that's sort
of what you would expect to find there
and that's it


Provably Beneficial AI | Stuart Russell

[Music]
one of the things that's obvious anyone
who have it's a newspaper or online is
the news is that this is a very exciting
time so these are just some newspaper
headlines from the last few months and
we have kind of that going on between
different global powers who can stand
from the largest amount of money on AI
so that UK announced 1.3 billion dollars
in new announcer 18 billion now for
twenty twenty two billion dollars and
then China of course as we know who has
and the US trying to the wool
[Music]
it seems reasonable as nice pointed out
we don't have to assume that we will
necessarily achieve super intelligence
within any fixed period of time but his
team is reasonable to act on the
assumption that that's feasible a become
a new matter really whether it's a 5%
chance for an 80% chance with any any
particular time scale but it seems I
mean the converse of this is that we
guarantee that AI will fail and I have
to say in the last two years we have
seen senior established members of the
AI community publicly guaranteeing that
AI rules there
I'm enter the in for example Senior
Council apologies stood up and we
guarantee that despite the hundred
billion dollars a year that you the
public of pumping into biological
research we guarantee that you will
never ever cure cancer ever I think I
would be shocked and but seriously
that's what's happening
and I think we should just take that
denialism and the sign of insecurity
that they are worried that in fact we
might now you know clearly Isis is going
to have access to much more information
than any one human being can possibly
ingest over a lifetime and I think it's
reasonable reasonable to assume that
would be the foresight power instead
alpha0 exists as if it's on the chess
board or the go board will gradually be
extended into foresight powers in the
real world
and so will have decision-making systems
that are superior in the real world to
human beings and this is as the previous
panel pointed out something that is full
of potential for the human race because
that's what we do our civilization where
these are intelligence and we will have
much more of it and if you just take a
very prosaic measure so no science
fiction we're not talking about
uploading or infinite life extension or
fast and light travel but just bringing
the standard of living of people on
earth up to a bourgeois oacd level not
Eve not even super rich or anything like
that just ordinary bourgeois level when
you calculate the net present value of
of that advance which I think would be
relatively easy to implement with
human-level AI technology the the value
that is about thirteen thousand five
hundred trillion dollars so now when you
look at those amounts that are being
invested by the UK and France and China
they seem absolutely miniscule in
comparison to the payoff it's also worth
pointing out that since that's the size
of the prize that people are aiming for
nations companies the idea that we could
simply announce a halt to AI research
is horribly wishful thinking so it's
going to continue and it's probably
going to accelerate the question is how
do we steer it in the right direction so
we're already aware of some downsides
we've been working a lot on autonomous
weapons and and Max and others of fli
have have been enormous ly influential
in in getting this message out to the AI
community and then to the world that
this would be a really bad idea we have
discussions later on in this meeting
about the possibility of enormous
changes in the economy that would upset
the possibility of employment for most
people on earth but this is what we're
going to talk about the end of the human
race and it's worth thinking about why
better AI could make things worse and
the answer that people have pointed out
and this is not a new point is that when
you put a purpose into a machine if you
put the wrong purpose into the machine
then you have a serious problem and this
is a quote from Norbert Wiener in 1960
and he he said this in in the context of
a paper that was written in response to
seeing Arthur Samuels checker playing
program learn over the course of a
couple of days to defeat its own creator
at checkers and if you think you know
that was done on a machine with a cycle
time of I believe three milliseconds so
Norbert Wiener was really thinking ahead
in this paper and it's a pretty
remarkable paper but this same point has
been around in human civilization
actually for thousands of years so King
Midas would probably say the same thing
in retrospect right he put the purpose
into the machine that everything he
touches should turn to gold he the
machine carried out his his objective
and then he realized that included his
food and drink and we see this you know
the story of the three wishes
the third wish as always please undo the
first two wishes because I've messed up
and unfortunately with super intelligent
AI they may not be a third wish and so
we have to get the first wish right
fairly recently we started to realize
that it's actually worse than that that
it's not just putting the wrong purpose
into the machine
it's that any purpose that's put into a
machine gives the Machine an incentive
to preserve its own existence simply in
order to carry out the purpose that you
put into it so this is not some kind of
intrinsic primal instinct that we've
built into the machine of
self-preservation this is simply a
logical consequence of having a purpose
in the first place so this is the slogan
that you have to remember your takeaway
from this talk is that you can't fetch
the coffee if you did so when you
combine these two a machine that has the
wrong purpose and then self-preservation
defending itself against attempts to
interfere with that purpose you get the
plot of 2001 a Space Odyssey and you
know fortunately how in the movie is is
only somewhat intelligent if it was
really super intelligent it probably
would have terminated or unplugged all
of the astronauts on the spaceship
rather than being unplugged itself so we
have to ask right where we're investing
pretty significant sums into a
technology whose logical extrapolation
seems to be pretty problematic for the
human race and I believe that in fact we
just we made a mistake and we made a
mistake actually right at the beginning
of the field and how we conceived of
what artificial intelligence is so we
started from this idea that humans are
intelligent and even at that time so in
the late 40s early
he's the the predominant view of what we
meant by intelligence at least from the
scientific engineering community's point
of view was the ability to act
successfully right which is better than
acting unsuccessfully and so connecting
to notions of rationality a human is
intelligent to the extent that our
actions can be expected to achieve our
objectives and this is fine and then we
simply transfer that notion into
machines so we want to make intelligent
machines so this is what they have to
look like so machines are intelligent to
the extent that their actions can be
expected to achieve their objectives and
in some branches of AI these objectives
were goals in some branches they were
lost functions in some branches they
were utility functions but all AI
systems except for the extremely ad-hoc
purpose specific ones were built on this
principle and it's not just AI right you
see the same thing in control theory
where you're minimizing a cost function
same thing in operations research where
you're maximizing sum of rewards and
statistics you're minimizing loss
functions and so on so this is a primary
pillar of 20th century technology that
you create optimizing machinery and then
exogenously you specify what the
objective should be for that machinery
and then the machinery proceeds as if
that's its own objective and my argument
is that that's a mistake
we actually don't really want I mean we
could build technology like that but we
don't want it and instead we need to
switch around the possessive adjectives
right we want machines whose actions can
be expected to achieve not their
objectives but our objectives and the
difference here of course is that our
objectives are within us and they're not
within the machines and so now you have
a different problem where the machine is
trying to optimize some objective
that it's not directly aware of that
doesn't have exact access to it and then
traditionally I simply becomes a special
case where the direct access to the
objective is assumed to be correct
and in fact if we do this right we can
then have machines that are provably
beneficial despite the fact that those
two words almost seem oxymoronic right
that normally when you think about
beneficial as well what does that really
mean and it's very touchy-feely and we
all have different notions and bla bla
bla and provably is about theorems but
in fact I think we can have both at the
same time so I've been trying to boil
this down I don't think you can boil
things down into a few of them three
things so I'm boil it down into three
things number one then the robots
objective is to optimize the realization
of human preferences not of preferences
that happen to be in the machine and
number two and this is probably the most
important change is that the robot is
uncertain about what those human
preferences are and it's this
uncertainty as we'll see that gives the
Machine a very different flavor from
traditional AI systems and then third
there has to be some grounding of this
idea of human preferences in some way
for machines to have evidence about what
human preferences are and short of
telepathy the way you're going to get
information about human preferences is
from human behavior construed in a very
general sense so it could be things that
we do it could be things that we ask for
it could be the objective that we type
into the Machine right and it doesn't
have to take it literally our behavior
then is the expression of that
particular objective but any kind of
behavior provides evidence even just
sitting here listening to a talk is
providing evidence about what your
preference structures really are so
I've tried to come up with a little
picture and so this is for people who
who understand the the basic ideas of
graphical models so in graphical models
we talk about random variables and then
which ones are observed and what the
dependencies are between these random
variables and in the classic view of AI
which is which is expressed in the first
three editions of my textbook the
objective is an observable variable so
the objective is at the top and then on
the Left we have human behavior which
obviously depends on what the human
objective is and then we have machine
behavior and then the classical view the
machine gets to observe the objective
and then go about its business and in
fact you know if you understand
graphical models you realize that in
this case you can simply forget the
human behavior because the human
objective if it's observed is a
sufficient statistic for machine
behavior so the human could now be
jumping up and down saying stop you're
going to destroy the universe and the
machine says that's fine I'm just
carrying on optimizing the objective
that I know to be true and all of this
stuff coming out of your mouth about
destroying the universe is just so much
hot air okay
so this is what we want to avoid now if
we make the objective unobservable right
then we can no longer detach machine
behavior and human behavior in fact they
become tightly coupled to each other
because the objective remains unobserved
human behavior then is providing more
information about the posterior
distribution on him on what the human
objective is and therefore the two
things remain coupled so let me just go
through a few examples and more will
talk about the mathematical framework a
little bit but I just want to sort of to
give this one example to show how deeply
ingrained in our current practices of AI
this idea of the fixed objective really
is so an image classification right what
we do is we train we train up some
Sofya deep learning system or random
forest or logistic circuit or whatever
it might be with a whole bunch of images
and we minimize a loss function and
usually that loss function is a
combination of the the empirical
performance on the training set and what
we call the lost matrix the loss matrix
says what is the cost of misclassifying
an object of type a as an object of type
B okay and in an image net there are
22,000 categories so that matrix is
something like 480 million entries okay
now how many people here do supervised
machine learning or have done that in
the last 5 years let's see so quite a
lot right how many of you used a non
uniform loss matrix just a couple right
so this is standard practice
partly because when we learn to do
machine learning we're doing it on toy
problems where in fact there is no real
loss right this is just an attempt to
see how accurately we can we can get our
classifier to classify unseen examples
of course in the real world there are
differences in the losses between
different kinds of errors and so when
you accidentally classify a human being
as a gorilla that's extremely expensive
right so it probably costs Google in
hundreds of millions of dollars making
that one single mistake compared to you
know misclassifying on Norfolk Terrier
as a Norwich Terrier apparently there's
a two different kinds of Terriers I
looked at the pictures in image net I
couldn't tell the difference I don't
know who really cares about that and I
can't imagine either kind of Terrier
being that upset about about that error
so so instead right you have to think
okay we don't really know what the loss
matrix is and so no one has sat down and
typed in all 480 million entries of that
loss matrix and even if they did they'd
probably get it wrong so we have to
operate with uncertainty over the loss
matrix and that's a different kind of
machine learning than the
she learning that we've been doing so
far and the algorithm should be able to
decide for example based on that
uncertainty well okay for this case I
think it's alright to go ahead and you
know issue a classification in public so
to speak in our public facing classifier
and in other cases to say I really don't
know I'd prefer not to say what I think
this might be because it could be an
expensive mistake and in fact apparently
if you ask the Google photo classifier
now to classify a photograph of a very
obvious gorilla it will say I'm really
not sure what I'm looking at here so
they've kind of put that in by hand so
and then of course the algorithm should
then use active learning techniques to
try to refine its knowledge of the loss
matrix so that it can be more effective
in more cases let's look at another
example right so in in this view point
where the machine has radical
uncertainty about human preferences what
is it what is it supposed to do what is
it supposed to do when it's given an
instruction right now traditionally in
AI we've always taken the human
instruction as the objective right this
becomes the machine's life purpose to
carry out that objective and we've only
as in fact Norbert Wiener pointed out
we've only escaped the negative
consequences of that mindset because of
our machines are too stupid to do
anything seriously bad at least that's
been true up until the last few years
and now we've seen consequences like
okay you want to maximize click-through
sure I'll maximize click-through for you
while destroying Western democracy no
problem right so when you actually think
about what does this mean
and there's been plenty of study of this
in in
in branches of philosophy for example in
the sort of Grice Ian analysis of
requests and commands what do they
really mean so what fetch the coffee
actually means is probably something
like I'd rather have coffee there no
coffee all other things being equal
rights okay surest paribus conditions
are a fairly good way to capture what we
mean by the expression of an instruction
or request as Murli there's more to it
than that it probably also conveys
feasibility because I wouldn't have
asked for coffee if I thought that
fetching coffee was infeasible or
ridiculously expensive to achieve and
therefore the fact that I've asked for
coffee is information that it probably
isn't that expensive or infeasible to
get the coffee
now given a request like that if the
machine doesn't know for example you
know your preferences for classifying
images of gorillas is it going to be
paralyzed by that uncertainty no as long
as it can fetch the coffee without
classifying any images of gorillas
either way then that means it's still
safe to fetch the coffee so even in the
face of radical uncertainty about human
preferences machines can still be useful
because they will still be able to carry
out plans that make only sort of
minimally invasive changes to the world
and in fact if you look at this from the
other direction right so you can ask you
know why why do we have this sense that
doing nothing is generally a kind of a
safe thing obviously there are occasions
where it's not the best thing to do but
doing you know a machine that does
nothing is probably a safer thing than a
machine that acts randomly now from the
point of view of you know Markov
decision process theory doing nothing is
just one of the key actions available to
the machine and it doesn't there's
nothing that distinguishes it a priori
from the other K minus one actions but
why do we have this feeling and the
answer is I think because the universe
is not arranged randomly in some sense
the universe is sampled from the
stationary distribution that results
from a whole bunch of purposive agents
with preferences operating on the
universe and so what we see for example
you think about this row of chairs here
right I can't see any human beings
behaving here right they've all gone and
the people who put the chairs there have
all gone but the fact that the chairs
are there tells you something about
human preferences and so in fact
contrary to the philosophical view that
which is called the naturalistic fallacy
that you can't observe purpose in the
world you can observe purpose in the non
natural part of the world right because
the non natural part of world is exactly
the result of agents operating with
preferences and so so this is a paper
with anchor and row in and a few other
people here and it's an very interesting
example of how much rich preference
information is available in the world
for agents to take advantage of so let's
look at a canonical problem from the
sort of the newspaper view of AI that
somehow we can always just switch off
the machine right now in the in the
story that we see in 2001 a Space
Odyssey right how doesn't want to be
switched off and it's actually quite
difficult so in the process he offs one
of the astronauts but eventually he does
get switched off but a machine that's
sufficiently intelligent that has a
fixed objective is simply going to find
ways to prevent you from switching it
off and it's the uncertainty about
preferences that actually allows us to
switch off the machine so let's see
exactly how that works right so here's
the here's the robot and the robot has a
choice it can do something let's say
turn the world into a huge field of
computer
or something like that which it might
think is probably a good idea but
actually it's quite uncertain right so
I've I've captured that that idea by
distribution probability distribution
over the value of the action so what is
the utility of this action to the human
the machine actually really isn't sure
but let's assume that the expected value
the the average of this distribution is
slightly positive so we can also give
the Machine a choice just to commit
suicide it could say you know I'm such a
dangerous machine I should just turn
myself off and we'll define that to have
value zero because then the machine is
not going to do anything at all
now if the if these two choices are the
only things available to the machine
then it will choose to do this action it
will cover the world with computron
Yemen and finish the finish off the
human race so let's give the Machine a
third choice we can say all right it can
now wait and allow the human to switch
it off the question is why would it do
that right and in the classical view it
won't it'll have its objective and it'll
choose whichever action maximally
achieves the objective
now let's so we give the human the
choice it can switch the robot off and
then that has a value 0 or it can say go
ahead okay well what happens if the
human says go ahead it seems like we're
just back in the same situation we were
in before so what difference does this
make well what difference it makes
actually is that if the human says go
ahead it's because the action is not
terribly deleterious to the human right
so the negative part of the utility
distribution has been zeroed out ok and
now when the robot takes this action it
knows for sure that it has positive
utility for the human and if you do the
math it's very simple
you can prove a theorem that says that
in fact a robot that's designed this way
with explicit uncertainty over human
preferences is provably benefit
to the human being and so the deference
right the the fact that the machine will
allow you to switch it off that
deference comes directly from the
uncertainty about the objective and if
you eliminate that uncertainty the
deference goes away the machine will no
longer allow you to switch it off and
you have all the problems that people
envisage so even though this is a very
simple example I think this is the core
concept that is going to allow us to
have deferential but super intelligent
AI systems in the future so let's talk a
little bit about the third principle the
idea that human behavior is the source
of information about human preferences
so there's already a field called
inverse reinforcement learning which
goes back a couple of decades and the
idea is that by observing the behavior
of an agent that is presumably
optimizing some objective function some
reward you can infer what that reward
actually is and there's now a
well-established theory there are lots
of practical demonstrations of this
technology we actually want a slightly
different version of that so in in
inverse reinforcement learning or IRL
the machine is is effectively simply
observing the human through a two-way
mirror if you like the human is just
doing their thing and then the Machine
adopts whatever value function whatever
reward function it learns from the human
we don't exactly want that that works
for some tasks but for example if the
human is drinking coffee we don't want
the machine to start drinking coffee
that wouldn't be a good idea so so IRL
is that is the machine looking at the
human through the window so to speak and
learning the reward function cooperative
IRL the human and the Machine are
together in the same environment and so
this becomes not single agent AI not
machine learned not standard
machine learning or inverse
reinforcement learning but a game
theoretic situation and this seems to me
inevitable and just comes from this
coupling of human and machine behavior
so in the soil game the human has some
preferences generalized sense we'll call
those preferences theta and we'll assume
that there's some connection between
theta and the behavior it doesn't have
to be a perfect connection and then the
goal of the robot is to maximize
according to those preferences theta but
they are unknown the machine in the
Bayesian view of Searle machine has a
prior and then in in the course of the
game the prior is updated and then the
machine acts according to that so when
you solve these games all right you just
set up the game like this and you solve
it and you can put you know you can make
different environments with different
actions available you could have
communicative possibilities and so on
what you see is that the human now has
an incentive to teach the robot the
robot has an incentive to defer to the
human to ask questions and so on and so
you get I think the kinds of behaviors
that we hope our intelligent systems
will exhibit in the future and there's a
lot of technical work going on now in
this framework algorithms for solving
these kinds of games efficiently
generalizations to cases where the human
the human itself is uncertain about
their own preferences which is a common
situation in the real world and so on
we're also looking at what happens when
we generalize this to too many humans
and this is an extremely important case
obviously because there is more than one
human being in the world and then you
get into all the problems of how you
weigh human preferences and there's a
long literature on this obviously I
think going back at least to the 5th
century BC of the Chinese philosopher
moti who basically developed the first
version of egalitarian utilitarianism
wasn't very popular with the aristocracy
and so Hasan you for example has a very
famous theorem showing that any Pareto
optimal policy that's enacted on behalf
of several people is going to be a
linear combination of their preferences
and the weights in that linear
combination can depend on basically the
the opportunities that each human has to
defect from this organization and get
something better elsewhere so a
bargaining position but that made an
assumption of a uniform prior that
everyone shares so everyone has to
believe the same thing about the world
of course we don't all believe the same
thing about the world and and kritch
recently came up with this idea that you
could look at what happens when you have
different priors when people have
different beliefs about the world it
turns out in that case that you get a
very weird conclusion that the weights
that you Accord to each person's
preferences evolve according to how well
their prior predicts what actually
happens in the universe so in some sense
this says that the smart people or the
lucky people who made good predictions
about the future are going to end up
with higher weights in there in this
shared Social Welfare function and
that's a very weird conclusion to draw
and I haven't yet understood its
consequences for political theory but it
seems to be a theorem now as Peter
Eckersley pointed out in his poster
yesterday there have been many papers
over actually centuries pointing out
sort of bugs in the the simple objective
of maximizing total happiness the
utilitarian ideal and those include for
example Nozick's work on utility
monsters a utility monster is someone
whose utility scale is so large that
that their preferences end up out
weighing the Preferences of everybody
else so you might call them snowflakes
but this is a very real problem not
because I think there are utility
monsters but because everybody has an
incentive to behave as if they are you
too
you monsters so another interesting
problem is if you build robots that way
everyone's preference is equally the
first thing they're going to do is go
off to Somalia and save people's lives
who are dying of starvation which is
great except that you just paid eighty
thousand dollars for that robot and now
it's disappeared so you're not going to
be very happy about that so these are
some of the interesting things that come
up
the other interesting thing that we have
to deal with in the research agenda is
the fact that the connection between
human preferences and human behavior is
not one of perfect rationalities in fact
extremely complicated so we're trying to
invert the human cognitive architecture
so that the machines learn underlying
human preferences despite the fact that
those preferences are obscured by myopia
by group by sort of short-term thinking
by fear by emotion and so on so I'm not
going to have time because Victoria's
gonna kill me
if I explain all of those incredibly
fascinating topics I'm happy to discuss
happy to discuss those afterwards I
highly recommend actual Daniel
Kahneman's book Thinking Fast and Slow
particularly the last section which
talks about what human preference is
really are and the fact that preferences
as you experience life are very
different from preferences as you
remember your experiences from the past
and the person who chooses the future is
not the experiencing self
it's the remembering self right you
choose the future on the basis of your
memory of how different experiences
played out in the past and your memory
is simply false in many cases it seems
and you know anyone who's given birth to
children apparently has has this they
remember it as a joyful occasion despite
experiencing it as incredibly painful
and humiliating in most cases so so this
is a very complicated thing and actually
make it makes a real difference to for
example how much money we spend on
dialysis do you ask the person who's
experiencing the dialysis or the person
who is now choosing whether to continue
dialysis and you get different answers
from those
to people ok toad its to summarize I
think we have to work on the assumption
that a I will overtake human abilities
and I believe that despite that we can
have a super intelligent AI that is
provably beneficial to humans and this
is obviously a desirable technology and
I think we should stop talking about AI
safety right this is just what we mean
by AI right we don't go around saying
bridges that don't fall down we just
call the bridges right it's sort of part
of the meaning of the word bridge that
it doesn't fall down it should be part
of the meaning of the word AI that is
beneficial to the people who are
creating it otherwise it's just bad AI
right so there's a long research agenda
that we're engaged in again I don't have
time but I want to point out that this
is not the only problem that we face
write this value alignment or
misalignment problem is not the only
problem that we face there's still a
serious issue of what we call the dr.
evil problem so the misuse of AI by
someone who doesn't really care about a
value alignment or any of those things
who just wants to take over the world
it's not a new problem we have the same
thing with nuclear weapons and we have a
vast security apparatus to try to
prevent dr. evil from getting hold of
nuclear weapons and then the wall-e
problem right it's not someone misusing
ara to destroy us it's us misusing AI to
destroy ourselves in a sense of in
feeling our own civilization by handing
over its management to machines in a way
that's irreversible thank you
you
[Music]


Should We Build Superintelligence?

[Music]
new ones - this is take these five
different most striking differences that
you are voted on when you register it
and have a debate on each one and I want
to stress it the purpose of these
debates it's not like a presidential
debate where the goal is to try to
humiliate the other side or get it the
point is or the point is rather simply
to make sure all of us get to hear what
the interesting arguments are for each
position and see if we can get a more
nuanced way of thinking about these
things and all of the five panels we're
gonna have now in rapid succession will
not only debate those positions that
were in the survey but each panelist is
gonna propose a little bit more nuanced
position that they like and then you
will get their way in and say how
enthusiastic you are on each one of
those so without further ado we're gonna
go to the the very first panel which
will be chaired by joi Ito from MIT our
director of the media lab and we have
chair Hyun y1 and Tanya Singh and John
havens and Kathryn Olsen and joi Ito who
are gonna debate should we build super
intelligence
so I want each of you to give a one or
two minute non nuanced very sort of
specific view of why you chose yes or no
and then we're gonna do this for the
polling but I want you to either name or
describe what your position is because
we're gonna then do a popularity contest
later about maksud it wasn't a
competition but we'll see what the
distribution of the audience is and how
compelling your vision is but the vision
so the first part is do a non nuanced
pitch of your vision and then the last
half we're gonna have a conversation to
then have the more nuanced conversation
about the details and so try to be as
Extreme as you can without being wrong
and then after you guys each speak I'll
give a orthogonal but hopefully useful a
little comment and then we'll open it up
to conversation so that's that's sort of
the lay of the land so I'll start out
with you Katherine cool so of course the
word should has a lot of meanings when I
answered this question initially I tried
to take a pretty pragmatic stance on it
like if a friend came to me who I
thought was like a capable friend who
might actually do things in the world
and said my plan is to build a company
or to start a company and we're going to
build super intelligence should I do
that and say ah no like sounds like
there's some other things you should
consider in that so that's sort of a
sort of direct pragmatic should argument
or should not argument the question is
should we build super intelligence I'll
highlight we already have systems that
are super intelligent on specific tasks
we've already done that in certain
domains so maybe you interpret the
question as should we build fully
autonomous agents that exceed human
capacity in pursuing their goals and a
reasoning and sort of a long-term
consequentialist way I think sort of the
whole point of this conference is that
we all broadly agree that would go very
badly by default so no we shouldn't do
that and then to sort of put the like
punchy analogy on it when I was talking
with some of these folks before of
course there would be a lot of benefits
we could attain if we could do this
right whatever doing it right means the
analogy being okay the pool is really
nice I'd like to give swimming in the
pool there I would have a lot of fun but
the pool is full of sharks like it's
full of sharks
should I go swimming in the pool the
answer is no there's things we could do
to mitigate those risks but right now I
think the risk pool of sharks pool of
sharks okay okay hi John it's like a TV
show so I'm John and I say this just
speaking for me down I Tripoli or the
council on extended intelligence the
stuff I do mine is also kind of a shark
in the pool analogy and it's somewhat
nuanced in the sense that I don't think
we should do anything not just a GI
until we figure out the economic
paradigm ik level underpinnings of most
of what we do as humans and I was in
Korea about a month ago at the OECD
well-being workshop which is about 700
statisticians around the world focusing
on things like the UN s DG's and how
what we're actually measuring as a sense
of what brings humans worth and
prosperity since the 40s and 50s is
largely about GDP GDP is not evil if you
hear the term GDP and Beyond it means
there are other measures that we can use
to understand both human and
environmental flourishing so for me one
thing to keep in mind if you study GDP
which is riveting and fascinating I'm
kidding is it's really about
productivity looking back on a year and
it's also about measuring things like
exponential growth well in terms of
values alignment when you have systems
that are designed to be sometimes very
fast and to do things at an exponential
rate and then you have a measure of
success a key performance indicator that
for both policymakers and business I
used to be an EVP of a top 10 PR firm
and every quarter when the doors closed
you would want to bring value to your
customers you would want to do stuff
that would help the world but you also
said these are the five words that
dictated the actual actions you took did
we make our numbers so beyond GDP means
people planet profit if we can have
triple bottom-line economic metrics
guiding what we make for Ag eyeness ASI
I'm diving in the pool sharks are known
without them I don't think it makes
sense to move forward without really
thinking what is prosperity and
flourishing mean for people in the
environment and and the entities that
may come beyond if we don't deal with
that now
don't do it until we can measure it yeah
so GDP and Beyond would be I guess the
DPM joined
piffy not as good as sharks in the pool
thing thank you all right I'm approach
this question is that I feel the future
is with super intelligence I did have a
lot more positive values so it seems
that to me it seems that in so far is we
are not certain that the alignment
problem is intractable or super
intelligence will lead to some
devastating outcomes existential risk
level catastrophe for human beings
it seems our job is to grind down the
risk and bring it down to in fact
infinitely small basically and to say
that we're not sure whether we can do
that and there's risk hence we should
stop progress towards super intelligence
arguably you can't even do that but to
say that just stop progress towards
super intelligence to me seems like
you're throwing the baby out with the
bathwater and it's a really cool baby
because it can help you know mitigate
interest tential and risk or even
catastrophic risks from other advancing
technologies so to me seems like it's
it's slightly premature to say that we
shouldn't be progressing towards super
intelligence that's the spirit with
which I've approached the question I
think you can also frame the problem in
a way that our current level of
understand with our current level of
understanding we can say that super
intelligence could there is a
non-trivial chance that they'll be
extremely bad outcomes from it or that
the technology gets concentrated in the
hands of a few malicious actors or we
get locked in a highly suboptimal sort
of condition with our Axia logical
potential getting curtailed so all of
these are options on the table which we
need to explore and make sure that we we
don't run too much of a risk of those
things we don't run a risk of those
things but currently I don't think we're
in a position to make that call and it
seems a little premature to say that we
should close off we should close off
this path because you know and and not
explore all and not not try and reap all
the few positive outcomes that could
come out from this technology and build
offensively stable world which AI would
allow us to do don't throw out the baby
don't throw out the baby cook
right how to point for my position is
different with all of the three call it
that is absolutely we should yes so the
point is that first so the involvement
of the intelligence in the universe is a
very long trip so our human risk only
one stage one stop of the long trip no
reason to stop at our stage and we
prevent higher level in holidays is
Norina for me is a very natural
direction to be there for the super
intelligence and maybe somebody say we
can involve ourselves maybe we can
become smarter in the future but for our
humorous that means we need thousands of
years maybe longer because our brain our
body involved too slow and we have a
limitation of our scope example so it's
impossible for our human being to become
to become the super intelligence
compared with the machine based super
eNOS so if the future the sooner or
later so it will happen in some maybe
some filter so in this case no
difference to build super intelligence
yes or not because it will happen why we
with or delay the appearance of the
super intelligence so finally we need
face the the appearance the super
intelligence some some day so that's one
why we do we must prepare for the for
that the second point is that even from
the perspective of our human human
centered or human expert it's personal
leader
we still need to develop super
intelligence because we have so many big
challenges to face maybe for our human
race we cannot figure out that big
Helenus we need the help of interest for
example nuclear waste interest maybe
destroy our earth we know that but you
can think about if another small planet
hit our earth son someday without the
nuclear weapon we may become de venise
or so with super invaders means we can
solve some big challenges that our human
race cannot cannot do so we need to that
in the future and help us for me
the superintendence is not totally
different intelligence compared with us
is our copy but with fast
maybe powerful interest mr. Covey is
findable share the same neural network
in our brain so that means we can
communicate with several antennas we
built
it's not aliens so he is our could be
become our partner and it has our
intelligence so that's why I support the
short version would be something like
shouldn't slow the inevitable slowly we
shouldn't slow the inevitable how would
you summarize and something that goes on
the screen or is it we're just we don't
matter or matter we think about how to
it's the three words or would they be us
through water okay the way we should do
that we should just do it okay that's
good the I will make my slightly
orthogonal comment so some people
mentioned it but I think it should word
is kind of weird because if you don't
have the ability to
but there is no should it's just it will
happen so am i if I'm allowed to add one
I would say doesn't matter it will
happen but I'll also just point out I
think David Kruger said it earlier when
he was talking about the comprehensive
AI services I'm very much in his in the
camp of that conversation about AI
actually being agency and models that
are integrated into the system we use
the term extended intelligence so if you
think of corporations or society instead
of individuals we already have super
intelligence in a way that organizations
are arguably more intelligent at least
more complex than individuals and that
if we start to augment society rather
than think about augmenting individuals
it's already happening and it will
probably happen anyway and it's sort of
an unstoppable thing and so but I would
just have have question with the with
the framing but anyway um so so my thing
is that it will happen anyway and it's
already here and so should we do the
vote and then do the conversation how
can you have both it already happened
and it's inevitable
well so absolutely I'll just keep going
okay so I think I think what happens is
well we're just evolving on a tree
actually with increasing power and my my
view is that super intelligence is just
intelligence increasing at the societal
scale and that and a my etta thing is
that I think we just AI just makes us
more powerful and more complex careening
in the direction that we're going
so I'm more interested in getting our
house in order in our trajectory headed
the right way rather than figuring out
whether we should stop it or not that's
sort of my vote and then my nuance which
you will all get a chance to me wants
away are that are the unknowns multiple
choice I want to just pose a particular
framing of the question and I'm gonna
commit an error which I apologize for
which is like raise your hand if you
oppose me a question which I haven't
even spent five minutes thinking about
but it came up with it right before I
got on the stage which is let's say that
you are
Dennis Shane and Mustafa and you're the
heads of deepmind and one of your
engineers comes to you and is like we
can do it we can push the button we can
build a GI and it's right now like
January 1st or whatever January 4th 2019
and you're trying to ask the
okay should we do this thing how would
you tell what questions would you ask
what experts would you go to do you have
enough information to make the judgment
right now today if this were to actually
happen like yes we should and I don't
think that collectively we have the like
decision-making tools epistemic
institutional institutional sanity
technical safety work done such that
there's any person today that could make
the yes we should decision like right
now that's just like a little more
nuanced but I would encourage folks to
actually think like what would it take
for us to be in a world where a group of
people is actually making the decision
there's this thing build super
intelligence there's an action we can
actually take which is give the thumbs
up should we do it what would it take
for there to be a valid like yes answer
that we don't feel good about
yeah I'm just posing a question that I
think is like more meaningful than what
was asked or something or like maybe has
some more so so go on so adding the time
right now actually changes that makes it
more crisp right but I'm sort of say
imagine that this is happening right now
but then imagine like what properties
does a world have in the future that's
different than right now I don't think
anyone in this room maybe I'm wrong
would claim that like there's a possible
state where someone alive on earth could
like confidently press the Go button and
like be making the right call but what
would the world look like where that is
true
like what would those decision-makers
need to have access to that they don't
have access to right now okay can I keep
you in there yeah and do we have a mic
for anybody at this back yeah hi so I
was going to say something really naive
or maybe very unimaginative but I don't
know how to think of a world where
someone says I'm about to do a GI I can
push this button and it will happen
right now I don't have it already
but you know if I push this button I
will have it like can you tell us a
little bit more of what it would take to
get through that stage
yes it's impossible to push a button and
heaven tomorrow it's impossible for me
at least 30 yes so we have at least just
let me try to clarify I didn't mean
right now I just mean ok 30 years from
now how would we get into a stage where
someone was doing AI research raining
their stuff by supplying somebody award
functions playing around with things and
then all of a sudden were like oh we
don't have a GI but if I push this
button we will have a GI
what does push the button mean how I
already have it at the point where you
realize that if you push a button you
have it there's a like you max pick I
mean okay we before Justin it's so cool
party means at that time we have the
technology and the sister if we threw
the button the Adria maybe have it so
that is the that the time I think we're
at a time unfortunately maybe push the
button is like the last mile of things
that need to be done to develop yeah
just if you couldn't reframe it like
that it doesn't necessarily need to be
pushing this ability but yeah okay
thanks panelist thank you
[Applause]
[Music]