Current work in AI alignment | Paul Christiano | EA Global: San Francisco 2019

so it is my pleasure to introduce Paul
Cristiano Paul works on AI alignment at
open AI previously he received a PhD in
learning theory from UC Berkeley he is
on the board of aught and he writes
about alignment at AI alignment comm
that's AI dash alignment comm he's here
today to speak about his current work in
AI alignment please give a warm welcome
to Paul Cristiano I should remind
everybody as well that we probably will
have a few minutes for Q&A at the end so
use your visible app to submit those
questions oh great to be here when I was
preparing this talk I initially intended
and perhaps advertised the talk there
was a survey of current work ongoing in
AI alignment as I was thinking about how
to give the talk I think I came to the
view to be more useful to instead
describe sort of some of the pieces of
AI alignment as I see it how they fit
together and how they relate to the
broader project of making AI go well so
I apologize to anyone who's profoundly
disappointed by the change of topic
anyone who saw me speak on Thursday this
might be very redundant and I won't be
offended if you disappear now and might
recommend it but that said let's dive in
so I'm interested broadly in the problem
of making humanities long term future
good by our lights I'm zooming in on
this sub problem of the effects of AI on
our long-term teacher so I'm just in the
problem of making AI have a positive
long run impact but even within that
problem there's sort of a large number
of different sub problems that we can
think about separately and I think are
worth having some conceptual clarity and
distinctions so that's why there's a lot
of empty space on this slide so I guess
the part of the problem I must think
about what I call alignment I use
alignment in a slightly different way
from other people I apologize for the
conceptual confusion what I mean by
alignment I'm going to call intent
alignment that's trying to build AI
systems that are trying to do what you
want them to do so in some sense this
might be like the minimal thing you'd
want out of your AI at least it is
trying that's what I mean by intent
alignment and have some one of these
pieces or one piece of making AI have a
positive long run impact right so we
want to avoid the situation where AI is
working at cross-purposes to us we want
to avoid the situation where as Ras
become more competent they're more
effectively doing stuff we do not want
them to do want them to at least be
trying so this is one driver of a going
well over the long term there are a lot
of other problems we could work on with
the same kind of intention
so another category of work is making AI
more competent right so some forms of
competence are mostly important over the
short run but even if we're sort of
exclusively focused on the long run
there are kinds of competence we care
about so an example is we would like our
AI systems to perform reliably right
that is if I have a system which is
well-meaning that doesn't mean the
system is reliable and my system makes
mistakes those can potentially be very
bad so if I'm interested in the
deployment of AI in high stakes
situations as if I imagine the AI system
which is like making decisions about
nuclear weapons I might think it's
really important for that system to not
mess up I think this is a really
important problem but it's worth
separating from the problem of building
AI that's trying to do what we wanted to
do I think some techniques apply to both
but again it's we're assuming on one of
those problems and asking how can we
solve this one yeah I think this is the
kind of problem we might hope will get
better as our air systems become more
competent so as AI improves air systems
are better at understanding the world
they're better at sort of doing what
they're trying to do we hope that like
not making mistakes is in that category
again there's separate research but it's
distinct from this problem of trying to
do the right thing I'm so I'm gonna walk
through a few more examples of
competence the one could work on from
the perspective of trying to make the
long-run impacts of AI positive I'm sort
of choosing the examples that feel
closest to intent alignment to be
clearest about what doesn't fall under
this heading of intent alignment again
because I think having that distinction
in mind is helpful for thinking about
work in that area so another example
it's maybe a little bit more surprising
is having our AI understand humans well
right so that is there's this potential
distinction between a system that is
well-meaning but a system which is
trying to do what I want it to do and a
system that knows me well if I imagine
hiring an assistant I could have an
assistant who is trying to do the right
thing but maybe doesn't understand me
very well or we don't have really high
bandwidth communication and this is
another important but separate problem
right that is there's a distinction
between that it's trying to get what I
wanted AI that really understands what I
want well and I mostly work on the well
meeting side of this rather than the
knows me well side and one of the hopes
that motivates that focus for me is that
you don't need a super deep
understanding of what humans want in
order to avoid catastrophic ly bad
outcomes
as if I imagine a scenario where any AI
system is like actively working against
humans like tiling the universe and
paperclips in the absurd example like
the very extreme case maybe in lesser
cases sort of not allowing human space
to like get what they want to figure out
what they want and then ultimately get
it right avoiding that case involves
sort of a minimal understanding of what
it is that humans want it involves
understand the humans don't want to be
killed humans don't you like to take all
their stuff and go run off these are
like important facts that may be factual
we hope on this like overall progress
towards just kid I happened kind of
early
so for philosophers I guess when I say
that in AI is trying to do what I want
I'm sort of using that phrase de dicto
rather than DeRay that is it's trying to
do what Paul wants it's not trying to do
the thing that Paul in fact wants that's
another example of competence which i
think is important which I want to keep
separate another example that's maybe
surprising is the distinction between
making Aria good and having our AI help
us make subsequent AI systems good so it
is we're sort of dealing with this
handoff to AI systems right now we're
gonna be building AI systems are going
to be making some decisions on our
behalf that AI system we build will be
helping us design future air systems or
at some point on its own designing
future air systems and it will face its
own similar problems as it does
subsequent handoffs so you can imagine a
case where we have built in the eye
that's doing what we want that AI system
we build messes up and builds another AI
system that's not doing what it wants
and I'm again distinguishing this
problem from the challenge we face right
now in this very first handoff and I'm
thinking about just this very first
handoff these problems are similar so in
the same way that I'm doing research
that I hope will help future humans
build AI systems that are aligned with
their interests I also hope that that
research will help future AI systems
build subsequent AI systems that are
aligned with their interests but I think
that our cognitive work right now can
sort of quickly become obsolete
once we're thinking about systems much
more sophisticated than us and so I'm
really focusing on this challenge sort
of the first challenge we face the first
step of this dynamic so these are all in
the category of making AI capable I
think sort of separating these from
alignment is important to expressing why
like I view alignment as this sort of
well scope problem that I'm optimistic
we can totally solve it's partly because
I'm willing to say that's like a
narrower problem than the overall
problem making I go well and many of
these problems on the competence side I
sort of hope will get better as AI
improves I'm sort of picking a Lyman in
part because I think it's
the aspect of the problem that's most
likely to not get better as AI improves
there's a whole nother category of work
rather than changing the way my AI
behaves or like changing what may I does
I can also try and cope with impacts of
AI systems so as an example AI systems
may enable new destructive capabilities
I might say that requires new approaches
to governance so it maybe makes us make
better bombs or make really big bombs
more cheaply maybe it allows us to like
design bio weapons something along these
lines that might require governance
changes but I want to keep those
separate from work on the character of
AI similarly like it could cause shifts
in the balance of power it can enable
certain kinds of misuse so maybe
criminals currently of a hard time
coordinating this becomes easier in a
world where requires only capital or
maybe totalitarian regimes have an
easier time sort of remaining in power
without supportive people if more and
more tasks are automated and there's a
lot of dot dot dots in this graph
because all of these nodes are sort of
covering only a very small part of the
total possible space I think these are
some of the big areas that people work
on between making your AI more competent
making your AI aligned so it's at least
trying to do the right thing coping with
impacts of AI these are all like
different projects that one could engage
in to try and make AI go well so I think
once I've said so many things are not AI
alignment may be a natural question is
how could we even possibly fail at this
problem like what could go wrong an
example of a thing that could go wrong
is when we train ML systems we often use
some measurable objective to say which
policies are doing better or worse and
you may end up with AI systems that
instead of trying to do what I want are
trying to optimize the kinds of
measurable objectives that they use
during training so that's an example of
a possible failure as if what I want is
not measurable and they have this
constraint imposed by the nature of the
technology which causes me to not have
systems that are trying to do it I want
another possible failures there might be
different kinds of values that behave
well on the historical cases I used to
train an AI system and so there might be
this degeneracy ml might give me in some
sense like a random draw from some
distribution of values and I might be
unhappy because most random draws of
values maybe aren't mine that's like a
little bit of intuition about how we
could possibly fail at the intent
alignment problem even if in some sense
it feels like I've set almost everything
to the side all of my work essentially
is focused on this problem intent
alignment I'm going to sort of zoom in
more we've left a lot
blank space at the bottom so to think
about dividing this problem further I'm
gonna use an abstraction I think this
comes from Eleazar though I'm not sure I
like this notion of an alignment tax
list a language comes from Eleazar what
I mean by this is so everyone would
prefer have AI systems that are trying
to do what they want them to do
I want my a AI to try and help me the
reason I might compromise is if there's
some tension between having the itit's
robustly trying to do what I want and
having the I that is competent or
intelligent and the alignment tax is
intended to capture that gap that costs
that I incur if I insist on alignment so
you might imagine I have two actors
Alice and Bob Alice really wants to have
an AI that is trying to do what she
wants it to do Bob is just willing to
have an AI that makes as much money as
it possibly can you might think that
even if making money is an instrumental
sub goal for Alice of getting what she
wants Alice wants her AI to make more
money in the service of helping her
achieve what she wants
Alice faces some overhead from insisting
that her AI is actually trying to do
exactly what she wants and the alignment
tax captures that overhead so the best
case is a world where we have no
alignment tax and in that world is sort
of no tension and no reason for anyone
to ever deploy an AI that's not aligned
the worst case is that there's no we
sort of have no way of building a lined
AI and so Alice is reduced to just doing
things herself by hand and now world is
maybe a giant alignment text sort of all
the value of AI I mean in the real in
reality were probably gonna be somewhere
in between those two extremes so we
could then imagine these two approaches
to making AI systems aligned one is
reducing the alignment tax making it so
that Alice has to incur less of a cost
if she insists on her AI being aligned
and the other is paying the alignment
tax so if I say I just accept that it's
gonna be hard to build a lined AI I
could just pay that cost or people could
pay that cost I'm mostly going to be
talking about reducing the alignment tax
but just to provide a little bit of
color and flushing out like the space of
options are paying the alignment tax you
can imagine the one class of strategies
is just caring enough to pay the tax so
if you're an ad developer if you're a
consumer you could pay some cost in
order to use aligned AI and that's like
a way that you can make the long-term
future better you can also try and
influence like what people are in a
position to make those choices so you
could hope that there are more people
who care more about the long-term future
in a position where they're making these
calls or you could hope that make those
people care more another option is to
try and coordinate so if Alice and Bob
are in this position they would both
prefer
systems that are doing what they want
but maybe they placed this trade-off
they could just both agree Alice could
say look Bob I will only deploy an AI
that's not doing exactly what I want if
you deploy such an AI if we just both
keep insisting on the line the eyes
we're sort of back where we started
roughly and maybe that can make the
alignment axe less painful you can
imagine there's some work I'm like
designing agreements that people might
sort of might make it less painful to
pay this tax
and then enforcing those agreements so
on the other side this is what I mostly
focus on how we can sort of do technical
work that reduces the alignment tax you
could imagine several different kinds of
approaches to this problem I'm going to
talk about two so one is the idea of
having some view about which algorithms
are easier or harder to align so for
example I might have a view that's like
we could either build AI by having
systems which perform inference and
models that we understand that have like
interprete beliefs about the world and
then act on those beliefs or I could
build systems by having opaque black
boxes and doing optimization over those
black boxes I might have to do that like
the first kind of AI is easier to align
so one way that I could make the
alignment axe smaller is just by
advancing that kind of AI which I expect
to be easier to align right so this is
not a super uncommon view amongst
academics it's also I guess maybe
familiar here because it describes
Miri's view various sort of takes this
outlook of like some kinds of AI just
look hard to align we want to build the
understanding such that we can build the
kind of AI that is easier to align
that's actually not the approach I'm
going to be talking about like not the
perspective I normally take AI no no
instead ask a question it's more like
let's just suppose that we have this
kind of algorithm suppose that we had
the black boxes is there some way that
we can design a variant of those
algorithms that's align a bowl so that
captures or works about as well as the
original unaligned algorithm but is
aligned so to be a little bit more
precise about that I sort of view the
goal most of my research I was starting
with an algorithm X that's potentially
on the line eg like deep reinforcement
learning and trying to design a new
algorithm a line of X which is intent
aligned nearly as useful as X and scales
as well as X so this was sort of the
situation I'd like to be in if you're in
the situation then Alice who wants to
have a line day I can just do the same
thing as Bob but every time Bob would
use this potentially unaligned algorithm
X Alice instead uses a line X it's kind
of the world I want to get to I think
the pass alien feature of this plan is
something like this idea of scalability
so you could imagine you sort of have
two options for aligning AI
one is that as AI improves we continue
to do ongoing work to ensure AI is
aligned and the alignment taks are then
consider work we have to do to make
these aligned ability is
state-of-the-art and the second approach
would be to scale ibly solve alignment
for a particular algorithm so i could
say i don't care how good maybe deep
reinforcement learning gets i know that
regardless i can sort of turn the crank
on this transformation end up with an
aligned version and now the alignment
tax depends on the overhead of that
transformation so I think in general
option two is sort of great if it's
possible but it's very unclear if it's
possible maybe this depends on sort of
how broad I mean how broad a category I
mean by algorithm so I'm not hoping to
have a solution which works for any
possible kind of AI algorithm and sort
of going to look at particular
algorithms and say can i format a line
version of this algorithm and hope that
I sort of as new algorithms appear I'm
able to align them so maybe in that
category the problem now breaks up by
the type of algorithm I want to align
right so for example I'm going to talk
about planning as an algorithmic
ingredient so I an agent can plan in the
sense of searching over actions or
spaces of actions to find actions that
are anticipated to have good effects and
then taking the actions that it predicts
will have good effects that's like an
algorithm it's a really simple
algorithmic building blocks it
introduces potential alignment if
there's a mismatch between what I want
and the standard which an AI uses to
evaluate predicted affects also
potential introduces mismatches like if
there's some implicit decision theory
and that planning algorithm which I
think doesn't capture the right way of
making decisions and so on so I then
have this problem can I find a version
of planning which is just as useful as
like the planning I might have done
otherwise but is now aligned i'm scales
just as well similarly there's this nice
algorithm deduction where i sort of
start from a set of premises and then
use valid inference rules to come up
with new beliefs and i might ask is
there some version of deduction that
avoids alignment failures maybe the
alignment pillars in deduction or a
little bit more subtle i won't be
talking about them and then another
algorithm which sort of looms large
right now and is the main subject in my
research is learning which is kind of
like planning it at the meta level so
instead of searching over actions to
find one that's good I'm gonna have some
way of evaluating is a possible policy
good eg by like playing a bunch of games
with that policy and seeing does it win
and I'm going to search over policies eg
like over weights for my neural network
to find a policy which performs well and
I'm gonna use that one for making
decisions in the future that's what I
mean by learning and it's again
introduces this challenge of is there
some way to
take that algorithm which is potentially
underlined and make an aligned version
of learning that's we're gonna be
talking about for the rest of the time
that's like the focus on my research I
think I'm going to again borrow from URI
and talk about some distinction in this
problem which i think is important and
really helps organize thinking between
outer alignment and inner alignment so
roughly speaking what I mean by outer
alignment is finding an objective that
incentivizes a line behavior so when I
do learning I had some way of evaluating
which policies are good and which
policies are bad I might pick a policy
which is good according to that
objective a first part of the problem is
designing that objective so that it
captures what I care about well enough
that a policy that actually performs
well on the objective is actually good
so here my failure mode looks something
like you get what you measure or a good
hearts law where I have behavior that's
not in fact good it but looks good
according to this proxy that I'm using
it looks good according to the standard
I'm using to evaluate I'm so for example
I have an advisor that adviser is giving
me advice maybe I pick policies for
getting advice based on how good the
advice looks that they produce and I end
up with bad advice that's just optimized
to look good to me that's like the
failure mode for outer alignment and now
the problem is designing an objective
that captures what I want well enough
there's a second half of the problem
which is maybe a little bit more subtle
to refer to as inner alignment which is
making sure that the policy that we end
up with is robustly pursuing the
objective that we use to select it so
what I mean by this is well maybe I'll
start with the analogy and this analogy
Mary likes a lot where humans were
selected over lots of generations to
produce large numbers of descendants but
in fact humans are not primarily
motivated day to day by having lots of
descendants so humans instead of this
complicated mix of values we care about
like art and joy and flourishing and so
on which happened to be well enough
correlated on the evolutionary
environment that becoming better at
pursuing the things we want caused us to
have more descendants but you put a
human in some very novel situation they
won't keep taking the actions that
promote having the maximum number of
descendants they're like conditions
under humans under which humans would
say look this action will cause me to
have more descendants but I don't care
it'll be like really unfun I'm gonna do
the fun thing instead from our
perspective that's good because we like
fun from evolutions perspective maybe
that's a bummer if it was like trying to
get very large numbers of descendants
so you can imagine facing a similar
problem in the learning setting so I
selected a policy which performed well
on the distribution but it may be the
case that that policy is trying to do
something right there might be different
policies which are different values
which leads to reasonably good behavior
on the training distribution but then
cause in some new situation do something
different from what I want let's swim in
by inner alignment again my research
mostly focuses on outer alignment they
I'm gonna say a little bit about the
techniques I'm most excited about for
inner alignment so one example is
adversarial training so here the idea is
if I'm concerned that my system may fail
in some cases they're unlike the cases
that appear during training I can ask an
adversary to construct cases on which my
system is particularly likely to fail
and then train on those and then we can
just sort of repeat this loop every time
I have a new agent I ask me adversary ok
now come up with a situation where this
would fail so the notion of fail here is
sort of a free parameter that we can
choose in the case of alignment fail
means it's not trying to do what we want
that would be a failure of alignment and
we're going to ask the adversary to
generate cases where the system isn't
trying to do what we want in practice
today fail most often means something
like the behavior of your model is
sensitive to a small perturbation of the
input but the basic idea is the same
between these two cases there's an open
question of like how much the techniques
which help in one case will help in the
other but the basic framework the schema
is the same and this is a topic which
sort of an active focus of research and
machine learning not normally from the
alignment perspective normally from this
robustness reliability perspective of
finding system which works well reliably
there are other possible approaches so
another is I would sort of like to
understand what the policy I've learned
is doing if I understand that sort of
even in a very crude way I could hope to
be able to see like ah this policy is
getting a good objective on the trading
distribution but in fact would do
something very weird in different
conditions so it's like not really
trying to optimize the objective and so
in some strange situation it would do
something contrary to my interests or
some novel situation again people mostly
study that question today in the context
of it would sure be nice to understand
something about what these models we
learning are doing but the same kinds of
research they're doing you could hope
will be helpful for understanding is
this learn to model doing something bad
would it potentially do something bad in
a novel situation
another example is verification
we're instead of just considering the
behavior of my policy on particular
examples I can try and sort of have
inputs I can try and quantify over
possible inputs I'm gonna try and test
like is there a situation directly test
is there a situation where this would do
badly or can I demonstrate that does
well in every case like I said I mostly
work on out of alignment and here again
I'm gonna make had another pairwise
distinction dividing the problem into
two pieces sort of the easy half and the
hard half are like the the warm-up and
full problem depends how you want to
look at it one setting is the case where
we had access to some expert that
understands the task we want our AI to
do very well so imagine I have some
teacher who is able to demonstrate the
intended behavior is able to evaluate
the intended behavior understands what
the AI should be doing in this case we
have lots of lots of options for
constructing an objective that will
cause our AI to do the right thing right
so one option is I could dress just
choose policies that produce behavior
that looks very similar to the teachers
behavior so if I have a teacher that
does what I want I want to get an agent
that does what I want
I'll just loop over agents until I find
one that seems to do the same kind of
thing the agent does where the teacher
does another option is to just have the
teacher say like I have my my policy I'm
training I have my agent who's trying to
learn I can have my teacher look at its
behavior and say that was good or that
was bad and then search for policies
that produce behaviors the teacher
thinks are good and sort of both the
cases of imitation and learning from
feedback maybe one of my main
difficulties is I have access to a
teacher but I want to be able to
efficiently use the data the teacher
provides maybe the teachers expensive
each time I run them and this is the
kind of difficulty the looms really
large in the case where I already have
an expert who's able to do the task
being efficient about how I use that
experts time being careful about like
how that gets transmitted to the agent
another option is I are L so trying to
look at the behavior of the teacher and
infer what values are what preferences
the teacher seems to be satisfying and
then use those so take those and ask an
agent to optimize those would you can
sort of view imitation and learning from
feedback as special cases of some more
general paradigm this is gonna involve
some assumption that relates the
preferences of the teacher to their
behavior eg like an approximate
optimality assumption right so I said
that the sort of easier cases one where
we have a teacher and the teacher is
able to exhibit or understand the
intended behavior the case we care about
in the long run is where we want to
build a systems that are able to make
decisions that a human couldn't make or
they understand things about their City
mission that no available teacher
understands and so the case of learning
from a teacher is maybe practically the
more immediately relevant one right we
have humans who are very good at doing
tasks today we mostly want our AI
systems to do what humans do but cheaper
in the long run we care a lot about from
the perspective of long-run alignment
risk we care a lot about the case we
want to do things that human couldn't
have done or human can understand so
maybe this is more like a demand that
people concerned about long-run impacts
focus on relative to people who care
about practical applicability so I'm
gonna talk about three different
approaches to this problem which again
not exhaustive so one approach that I
think is maybe the most common implicit
expectation of people in the ml
community is to treat the case where you
have a teacher as like a training set
and then to try and train a model that's
going to extrapolate from that data to
the case where you don't have access to
a teacher so you hope if your model
generalize isn't in the right kind of
way or if you train it on a sufficiently
broad distribution of cases it will
learn the right thing and extrapolate
I'm not gonna say much about this I'm
like very scared of this I think like
the record so far is not super great on
this kind of extrapolation I think it's
a reasonable thing to ask like might
this be very different as Aria systems
get more intelligent another option
would be to treat the learning from
teacher cases of warm up to take the
same kind of approach so we could do
inverse reinforcement learning in the
learning from teacher case where we try
and back out what their values were we
can also try and it in a case where
we're going far beyond the teacher and
say can we infer what the teacher
actually wanted in some deeper sense can
we understand which of the teachers
behaviors are artifacts of what they
want versus what they value can we pull
those apart and then say now we're gonna
get what they want but without the
limitations the teacher had again I
think this is sort of a plausible
project but it looks really hard right
now and then the third approach which
maybe also looks hard but is the one I'm
most excited about and let's focus on is
to treat learning from a teacher as a
building block and say supposing we
could solve that problem well is there
some way we can use that to directly get
what we want in the long term so that we
might say if we could just build a
sequence of better teachers then we can
train a sequence of better and better
agents without ever having to actually
go beyond the behavior of the teacher so
the idea here is we're asking where can
we get a better teacher working a
teacher who understands enough to train
the agent
and one place we can get traction is the
hope that if you take ten people they
can solve harder problems than one
person can solve so do ten people and
you ask them to like decompose a problem
break it into pieces divide up the work
they can solve problems that would have
been beyond at least slightly beyond the
abilities of a single person and we
could hope that the same applies to the
AI systems we train the sort of a
cartoonish caricature would be rather
than taking our initial teacher to be
one human and say let's learn from that
one human to behave as well as a human
we start by learning from a group of
humans and we say great is there some
way we can have this group do more
sophisticated stuff than one human could
do now we can have an AI which is not
human level but is the level of like a
human group and then we can try and sort
of take that as our building block and
recurse like iterate on that procedure
so now we can form a group of these AI
systems each AI was already as good as a
human group maybe this group of the eyes
is now better than any group of humans
or it's as good as like a much larger
group of humans and I can use that as
the t-shirt to train another AI system
I'm sort of if this dynamic consistently
goes in the right direction so putting
together this group is smarter this
group does more what we want than the
individual then I can hope to just train
a sequence of better and better AI
systems which continue to do what we
want because you to be aligned with like
their original intention okay so this is
a broad overview of sort of how I see
the aligning problem different parts of
the hundred problem relating to each
other on how they fit into the bigger
picture there's gonna be I think
especially tomorrow a bunch of talks
about technical alignment topics so they
got one PM mary is having office hours
Mira's like in this bucket or I put them
in this bucket maybe if anyone from
using the audience they can object--
I've liked advancing a line of
algorithms where I was giving me a
thumbs up that is trying to understand
instead of just accepting they're gonna
have these crazy opaque black boxes
let's try just like push on like
different approaches that don't go
through the same potential risks I'm
Kris oh wow
colleague from open era is gonna be
talking about transparency so how can we
look inside neural nets and understand
what they do
andreas Julie Miller from odd is going
to be talking about this case we want to
surpass human experts so how can we
delegate cognitive work when we can't
evaluate the outputs and is gonna be
very similar to sort of the perspective
I'm taking here and then if i PM
Geoffrey in a man to ask also colleagues
from Oakland they are going to be
talking about sort of debate has a
strategy for surpassing human experts
where we ask a human not to do work but
to evaluate given to
conflicting claims or two experts are
both trying to do the work which of them
is doing it better can we have them like
check each other's errors so those are a
bunch of things on the docket for
tomorrow which you think fit into this
picture in different ways and when I'm
thinking about these kinds of issues
it's really helpful for me to have sort
of a big picture in mind of how things
work together what the separate problems
are and how they relate cool so that's
all I have to say and I think that
leaves us just a few minutes for
questions thank you all right
thank you very much first question since
you mentioned Eleazar my understanding
of his kind of personal trajectory going
through these sorts of problems was to
get kind of stuck on what humans want
and even if that is you know and wonder
if that is even a coherent if there is a
coherent answer to that or if what
humans want to sort of inherently
contradictory and and thus sort of not
really answerable by systems you blew
right past that so are you not worried
about that or dude how do you think
about that problem yeah so I guess in
this talk I sort of zoomed in enough
that I left many parts of the tree
really unflushed out and in particular
didn't talk at all about like how you
actually or any of the ambiguity and
like what humans want or like doing what
who wants I think those are important
questions
I don't currently feel like they're the
central difficulties or at least central
difficulties to be resolved but like
someone with me with the kind of
perspective I'm taking I think Eleazar
is probably in like a broadly similar
place so I think now maybe he'd be more
he sort of treated it's like hazardous
to focus on that aspect of the problem
and one more talk about like look you
want that it's just gonna like make you
a sandwich can you have an eye that's
like really just trying to make you a
sandwich as well as it possibly can just
to get out I think he also thinks that's
like the most likely step to fail like
is rayon trying to do like basically the
right kind of thing which is related I
think in the context of systems like
deponent systems he's very concerned
about like it's kind of in a reliable
case or like the way in which humans are
it's not even like vaguely trying to do
what evolution wanted extent the view
evolution it's like something trying
like building humans mm-hmm I obviously
can't speak for Eliezer but this
a couple of questions that have come in
from the app one person is wondering do
you think it is possible to give AI as
the ability to understand humans very
well without also giving them the
capability to make things go very badly
yeah I think there are a lot of subtle
questions and trade-offs and like the
rest of this tree like what capabilities
do you think are good versus bad
capabilities I think some people would
have the view that like this
understanding humans thing is actually
like on net negative because puts them
like compared to helping us resolve some
kinds of problems
it like gives them more ability to
influence the world or the trajectory of
civilization I don't have a strong view
on that I think I'm sort of persuaded by
the primal foster case the like if
you're trying to build a system to help
you like it seems good for two more
understand what you want
I'd say that's like again sort of not
the aspect of the problem I've been
focusing on in significant part because
I'm sort of optimistic that both on the
bad for better or worse like very
sophisticated AI systems will understand
humans well enough till I could be able
to yeah both push the world in various
directions and also understand this very
crudely of what we want okay
another question from the audience you
mentioned that the track record for a
eyes extrapolating from a teacher is not
great can you give some examples of
where that's fallen down yeah so I guess
in deep learning some people care a lot
about training a system on one kind of
case and then having it work immediately
on a new kind of case I'd say our record
so far has not been great on that and
the big question by okay so even by what
cases like I very dumb case the sort of
learning algorithms like have given at
it sorts the list you want to sort a
list and you do it like an end to end
way just say grates gonna be big neural
network takes a simple endless produces
another list you're like probably not
basically not going to see
generalization from like a thing that
source list of link you know 1 2 4 8 16
32 64 hundred 28 you didn't run around
256 it's gonna let go crazy you sort of
have to make kind of strong
architectural assumptions or like you
have to give to really impose something
about structure the task into the model
in order to have that kind of
generalization occur that's like one
example I think like in general people
have been surprised and like one of the
big criticisms of deep learning is
little like doesn't do sort of you train
in one case it works well in that case
it doesn't tend to work well yet in
other cases and the big question is just
like is that is that sort of fundamental
is that like a thing the statistics will
always have something it will go away as
they become more powerful
they get closer to like transformative
AI awesome well for more on these
questions you'll have to track Paul down
at office hours I believe you'll have
later today thanks three at 3:00 p.m.
perfect how about a round of applause
for Paul Cristiano
[Applause]
you


Provably Beneficial AI | Stuart Russell

you
[Music]
I'm going to talk about the reasons why
we're here to some extent and then what
I'm doing about it so the reason why
we're here is in some sense the
extrapolation of the talk that we just
heard that progress is occurring is
occurring at a rapidly increasing rate
and absent some other disaster that we
could bring upon ourselves I think we
have to make the prudent assumption that
AI systems will be able to make
decisions in a broad sense better than
we can and what that really means is
that they're using more information than
we can individually take into account in
making a decision and they will be able
to look further ahead into the future
and so just as alphago could beat any of
us at playing go
you take the go board and you expand it
out to the world then we are going to be
dealing with superior decision making
capabilities from machines and this is
potentially an incredibly good thing
because everything we have everything
that's worthwhile about our civilization
comes from our intelligence and so if we
have access to significantly greater
sources of intelligence that we can use
then this will inevitably be a step
change in our civilization and of course
with any such powerful technology we
expect there to be some downsides the
possibility of using this and in
military fear has been raised and is
already occurring
but I'm not going to talk about that
there are sessions later on Eric talked
about the question of employment this is
another big theme of the conference I'd
like to talk about this one the the
possible end of the human race that's a
very lurid way to describe it and
fortunately the press are not here at
least if they are here they have to keep
quiet so why why are people talking
about this right what's wrong with
taking a technology that has all kinds
of beneficial uses and making it better
what what's the
problem and you can go back to a speech
given a little while ago if a machine
can think it might think more
intelligently and than we do and then
where should we be even if we could keep
the machines in a subservient position
for instance by turning off the power
I've highlighted turning off the power
because we'll get to that answer later
on a strategic moments we should as a
species be greatly humbled and this new
danger is certainly something that that
should give us anxiety so this was
actually a speech of Alan Turing in 1951
given on BBC Radio 3 as we now call it
so it's a very inchoate fear right
there's no real specification of why
this could be a problem just this sort
of general unease that making something
more intelligent than you could humble
your species and the gorillas here they
are having a meeting to discuss this is
their version of our meeting and they're
having this discussion and they say look
yeah you're right you know our ancestors
made these humans a few million years
ago and you know we have these inchoate
fears then and they turned out to be
true our species is humbled right but we
can actually be more specific than that
so here's another quote if we use to
achieve our purpose is a mechanical
agency with whose operation we cannot
interfere effectively we better be quite
sure that the purpose put into the
machine is the purpose which we really
desire there's a more specific reason
why there's a problem that you will put
some objective and the machine will
carry it out and it turns out not to be
the right one this is from a paper by
Norbert Wiener in 1960 written actually
in response to to the work by Arthur
Samuel showing that his checkered
playing program could learn to play
checkers much better than he could but
this could equally have been King Midas
talking you know 2,500 years ago
realizing that when you get exactly what
you say you want it's often not what you
really want and it's sometimes too late
so this is sometimes called the value
misalignment problem that AI systems
will be
be good at achieving an objective which
turns out not to be what we really want
and so if you said okay great well let's
look at everything we know about how to
how to design the objectives to avoid
this problem
unfortunately you find that there really
isn't very much to go on but all of
these fields that that are based on this
idea of optimizing objectives which is
not just AI but you can all make some
statistics and operations research and
control theory they all have this same
problem they assume that the objective
is just something that someone else
brings along to the game and then the
game is that we optimize the objective
and you know economists certainly notice
that things like profit and GDP which
are the sort of the official objectives
turn out not to be always the things
that we really want to be optimizing but
they haven't really figured out what to
do instead
and then Steve Omohundro and others
pointed out that there's a yet another
problem which is that whatever objective
you give to a machine it will need to
stay alive in order to achieve it our
and so this if there's a little takeaway
from the talkest you can't search the
coffee if you're dead right so if you
ask the machine to fetch the coffee
staying alive is a sub-goal affecting
the coffee and if you if you interfere
with the machine and its attempt to get
the coffee it will prevent you from
interfering it if you try to turn it off
it will take countermeasures to being
turned off because its objective is to
get the coffee so if you combine that
with value misalignment now you have a
system that has an objectives that you
don't like because you specified it
wrong and now it's defending itself
against any attempts to switch it off or
to change what it's doing then you get
the problem that you know science
fiction has talked about it's not a
spontaneous evil consciousness where the
machine wakes up and hates humans it's
just this combination of unfortunate
circumstances that arise from having a
very very powerful technology okay so
lots of people have said well you know
this is all rubbish
all right everything I've said is
complete nonsense and one of the first
responses in fact there are many
responses I've written the paper where I
list about 15 of these of these
responses and I think they're all kind
of defensive knee-jerk reactions that
haven't been thought through so you find
for example people in the AI community
saying having said for 60 years of
course we will get to human-level AI
despite what all those skeptics from the
philosophy community and everything all
those other people don't know what
they're talking about of course it will
get the human level AI and as soon as
you point out that that's the problem is
well of course we'll never get to
human-level AI but I just want to point
out that you know in history there have
been these occasions where other
powerful technologies have been stated
to be impossible so here's Ernest
Rutherford on September 11 1933 he gave
a speech in Leicester addressing the
British Association for the Advancement
of science and he said that essentially
there's no chance that we'll ever be
able to extract energy from atoms they
knew the energy was in there they could
calculate how much but they said there's
no possibility we'll ever be able to get
it out and even Einstein was extremely
doubtful that we would ever be able to
get anything out of out of the atoms and
then the next morning lady lard read
about this speech in the time and went
for a walk and invented the neutron
induced nuclear chain reaction so so
there's there's there's only been like a
few of these giant tech technological
step changes in our history and and this
one took 16 hours so so to say that you
know maybe this is the fifth or sixth
one that we're talking about to say that
it's never going to happen and to be
completely confident that we therefore
need to take no precautionary measures
whatsoever seems a little rash
okay so there's lots of other arguments
I'm not going to go through them all
I'll just just page through them I did
want to mention the last one the last
one which is it's very pernicious which
is um so it doesn't show up very well
but don't mention the risks it might be
bad for funding
so I've seen this not quite frequently
in recent years and if you just look at
what half of a nuclear power I go back
to the 50s and 60s where they were
trying to gain acceptance for nuclear
power there was every attempt to play
down all possible risks say it's
completely safe it'll be you know too
cheap to meter the electricity will be
free there's no pollution there's no
possibility of there ever being an
accident and what that leads to is a
lack of attention to the risks which
then leaves the Chernobyl which then
destroys the entire nuclear industry so
history shows it's exactly the other way
around that if you suppress the risks
you will destroy technological progress
because then the risk will come to pass
okay
so I hope that you are now convinced
that there is a problem and so what are
we going to do about it because that's
the other thing right okay yes I agree
with you this is the George Bush
response okay yeah global warming is
going to happen but it's too late to do
anything about it now what are we going
to do about this so the work I'm going
to talk about now is happening under a
new center that max mentioned the center
for human compatible AI at Berkeley
which is funded by the open philanthropy
project and what we're basically trying
to do is to change the way that we think
about AI away from this notion of pure
intelligence that the pure optimizer
that can take any objective you like and
just optimize it and naturally look at a
more comprehensive kind of system which
is guaranteed to be beneficial to the
user in some sense there's a lot of
other work that I don't have time to
talk about many of these new centers and
also those professional societies have
started to become very interested in
these problems as well as the funding
agencies and industry so the work at the
center is based on three simple ideas
the first of all that the robots only
objective should be to maximize the
realization of human values and the
second point is the robot doesn't know
what those are but nonetheless its
objective is to do this
so these two points together it turns
out actually makes a significant
difference to how we design AI systems
and the properties that they have so
obviously if the robot has no idea about
what human values are and never
discovers what they are that's not going
to be very useful to us so it has to
have some means of learning and the best
source of information about human values
is human behavior that the standard idea
a longstanding idea in economics for
example that our actions reveal our
preferences and this allows for a
process that results in value alignment
and joshua mentioned on one of his slide
a fairly old idea 20 years old now
inverse reinforcement learning which is
the opposite of reinforcement learning
or the dual so in reinforcement learning
we provide a reward signal on the system
has to figure out how to behave in
inverse reinforcement learning we
provide the behavior in other words the
Machine sees our behavior and has to
figure out what is the reward function
that's being optimized by this behavior
so in economics this is known as
structural estimation of MVPs which is a
somewhat of a mouthful in control theory
inverse optimal control so it's a fairly
an idea that sprung up independently in
several different disciplines and
there's now a fairly well advanced
theory lots and lots of papers
demonstrations that this technique can
be successful in learning lots of
different kinds of behaviors so it isn't
quite what we want for one thing we
don't want the robot to to learn our
value function and adopt it so if it
sees me drinking coffee I don't want the
robot to want coffee because that's not
the that's not the right thing that we
want to happen we want the robot to know
that I want coffee and to have the
objective of getting me coffee whatever
it might be so a slight generalization
is cooperative inverse reinforcement
learning which is which is a two-player
game in generally there will be many
humans and many robots but to start with
one human and one robot and the human in
some sense knows their own value
function but only in that they can act
approximately
according to it doesn't mean they can
explicate it and write it down and give
it to the robot but they have there's
some connection between their value
function and their actions the robot
doesn't know what it is but its
objective is as I said before to
maximize the human value function so
when you write down simple instances of
this game then you can solve it
mathematically you can look at how the
system's behave as they play this game
and as you would hope some nice things
happen so the robot now has an incentive
to ask questions first so it doesn't
just do whatever it thinks is best it
can ask you know is this a good idea I
can ask you know which of these two
things might I do and the human now has
an incentive to teach the robot because
by teaching the robot the robot will
become more useful and so the behavior
of both parties is considerably changed
by being in this game so I want to look
at one particular instance of this game
which we call the off switch problem and
the opposite problem arises because
based on the argument of instrumental
goals the idea that you can't fetch the
coffee if you're dead any attempt to
switch off a robot you know is going to
result in countermeasures by the robot
and this seems like a problem that's
almost unavoidable right we said you
know for pretty much any objectives very
hard to think of an objective that you
can carry out better after you're dead
than before it and so this is a
fundamental problem and Turing's
assumption that we could just switch off
the super intelligent machine is kind of
like saying well you know if you're
worried about losing the game of chess
you can just beat the blue just play
better move right it's not as easy as
that
but there is an answer which is that the
robot should not be given a specific
objective we want the robot to be humble
in the sense the robot should know that
it does not know the true objective and
then it's single-minded pursuit of the
objective and it's self defense against
any interference will actually evaporate
so if if the human is
going to switch off the robot why would
it do that the reason is that the robot
is doing something that the human
doesn't like the robot of course thinks
that what it's doing is what the human
likes but it acknowledges that there's a
possibility that it's wrong and so if
the human is going to switch the robot
off the robot learns in some sense for
being switched off that what it was
doing was undesirable and therefore
being switched off is better for its
objective which is to optimize the human
value function and so now the robot
actually has a positive incentive to
allow itself to be switched off it does
not have a positive incentive to switch
itself off so I won't commit suicide but
it will allow the human to switch it off
and this is a very straightforward
analog of the theorem of the non
negative expected value of information
but in some sense the humans action is
switching off is a form of information
and the robot welcomes that if it
happens so this leads actually to sort
of a rethinking a little bit of of how
we go about doing AI that that
uncertainty and objectives turns out to
be quite important and it's been ignored
even though uncertainty and all kinds of
other parts of AI has been studied
intensively since the early 80s
uncertainty in the objectives has been
almost completely ignored one reason is
that in the standard formulation of
decision problems Markov decision
processes and so on uncertainty in
objectives is actually provably
irrelevant because you're trying to
optimize an expected reward and if
there's uncertainty over the reward then
you can simply integrate over the
uncertainty and your behavior will be
exactly the same as if you knew the
expected value of that reward but that
theorem only holds if the environment
contains no information about the reward
so as soon as the environment can
provide more information then that
theorem is invalid and clearly if what
you care about is the human value
function and a human is in your
environment and the human can act then
those actions provide information about
the reward function similarly the
human one particular kind of action is
the provision of reward signal so
reinforcement learning can occur by
humans providing a reward signal now
let's go and look at a little bit of a
little bit of history a reward signal
and so here here are some well-known
experiments on what Kauai aheading so a
rat will actually sort of circumvent its
normal behavior and will actually starve
itself to death if you give the ability
to the rat to basically provide reward
signals directly either by chemicals or
by electrical stimulation even though
it's starving to death it will do that
instead of eating so a human actually
will behave the same way these are some
very interesting experiments in 1950s so
in any real situation unlike the
mathematical model of reinforcement
learning where the reward signal is
provided exhaustion ously sort of as it
were by God in the real world
someone has to provide the reward signal
you if you're providing the reward
signal are part of the environment and
the reinforcement learning agent will
hijack the reward generating mechanism
and if that's you then it will hijack
you and force you to provide a maximal
reward but this actually just results
from a mistake the masam and it's
interesting that by looking at it from
this different perspective from the
perspective of cooperative inverse
reinforcement learning we realize that
the standard formulation of
reinforcement learning is just wrong the
signal given to the agent is not a
reward it is information about the
reward and you just change the
mathematical formulation to have that
definition instead and then the
hijacking becomes completely pointless
because if you hijack something that's
providing information all you do is get
less information you don't get more
reward you get less information and so
we can avoid wire heading by
reformulating RL to have a information
base rather than reward based signals so
that leads to a general approach that
we're taking within the sensor that when
we define a formal problem what we are
going to do is build agents that are
designed
mathematically to solve that problem and
then we want to understand do those
agents behave in ways that make us happy
so we are not trying to solve the
problem that someone else is building a
an AGI from generally intelligent agent
and then we are going to somehow defend
against it like that's not the right way
to think the right way you think is to
find a formal problem build agents that
fold it and they can solve it
arbitrarily well they can be arbitrarily
brilliant but they are fall with all
that problem F and then show that the
human will benefit from having such a
such a machine okay so this is a
difficult problem there is a lot of
information about human behavior in our
history everything we write down is a
record of human behavior so there's a
massive amount of data that we're not
really using that can be used to learn
about what the human value system is so
that's good there's also a strong
economic incentive so Google found out
for example if you if you write down
your value function and say that the
costs of misclassification of one type
of object as another are all equal for
every type of object and everything you
could miss classify it as that value
function is wrong and then you lose a
lot of a lot of reputation as a result
you can also imagine that you know a few
years down the road mistakes in
understanding human value functions will
cause a very significant backlash
against in the industry that produced
that mistake so this is very strong
incentive to get this value system right
even before we reach the stage of having
superhuman AI there are some reasons for
pessimism which I always translate into
reasons for working hard and that is
that humans are very complicated they
are not optimizes they are very
complicated systems that there are lots
of them and they're all different and
some of them you don't really want to
learn from and so on these are problems
where we need social science to help us
and it makes things much more difficult
but also much more interesting so I
recommend that we work on practical
projects that we don't simply speculate
about what a GIS might be like and and
and write sort of interesting but
ultimately
our implementable ideas about how we
might control them that we actually look
at practical projects on real systems so
I think with an authentic we'll probably
be looking at intelligent personal
assistant other people might look at
things like smart homes where clearly
there are things that could go wrong and
there are there are incentives to get
this right early on it would be nice to
have simulation environments where in
fact real simulated disasters could
happen so we can get more of a sense
this will be a sort of a generator of
ideas about what can go wrong and then
how we might try to address it so I
think yan is getting impatient so we're
really aiming at a change in the way
that AI defines itself so we shouldn't
be talking about safe AI or beneficial
AI any more than a civil engineer talks
about building bridges that don't fall
down it's just part of the definition of
a bridge that it doesn't fall down it
should be part of the definition of AI
that is beneficial that it's safe and
this is not a separate AI safety
community that's nagging the real AI
community this should be what they I
community does intrinsically and in a
state of a business we want social
scientists to be involved we would like
to understand a lot more about the
actual human value system because it
really matters we're not building AI to
benefit bacteria we're doing AI to
benefit us and hopefully it'll make us
better people that will learn a lot more
about what our value systems are or
could be and by making that a bit more
explicit it'll be easier for us to be
good so to speak so Weiner
going back to his paper from 1960 and
going back to the slightly more gloomy
color of the earlier part of the talk he
pointed out that this is incredibly
difficult to do but we don't have a
choice right we can't just say oh this
is too far off in the future too
difficult to make any predictions we
have to you know just continue as if
nothing was going to happen
the problem is bigger and more difficult
but that still doesn't mean we should
ignore it thank you
Thanks I was very sure you liked your
formulation of the problem as a
two-player cooperative game and
addressing the question about switching
off AI possibly I don't think I
understood it entirely though how is
that particular choice by humans
different than other choices if you say
I want to move the bishop here I want to
push the throttle forward I want to
drive off the bridge I want to take the
red pill how does the AI know oh this is
one where I let the human overrule and
this other one maybe I shouldn't let the
human decide is there a distinction
between that choice and other choices if
you're uncertain yeah there is I mean I
think and it also depends on your on
your model of the human so if you if you
have a self-driving car it has an off
switch but if a self-driving car is
transporting a you know a two-year-old
you may want you may want to disable the
off switch it's like that's the right
thing to do so you can calculate the
expected value to the human of of having
an off switch that type six right now
sometimes yeah sometimes the officer
should be disabled so so the whole the
whole evaluation is to look at the value
of the human plus the robot system is
that is the human better off with the
robot or better off not with the robot
and we only want robots where the human
is provably better off and sometimes
that means the robot shouldn't allow
itself to be switched off so we know a
robot anesthesiologist that's keeping
you alive while you're unconscious right
is not relying on any of the decisions
that you're making right and you want to
you want to trust it completely while
you're unconscious so it depends on the
circumstance but the off switch problem
right that curing originally proposed is
a intelligent human who thinks they can
switch off the machine but turns out
they can't one more question
Andy Andy McAfee MIT this is an on/off
switch question you brought up the great
example of nuclear power as a case where
maybe under discussion of risks got the
field into trouble the next example in
your slide was GMOs which seems like a
really interesting case in
click the opposite direction where the
scientific consensus about safety there
is at least as strong as the consensus
for chemical human human caused global
warming and yet a lot of that that
technology is not diffusing and a lot of
people are made much worse off because
of over emphasis on the risk could you
comment on why you added GMOs to that
slide and what you conclude from it yeah
I I I thought I thought through this
this case and partly so in fact the the
Prime Minister's office in the UK asked
me to go talk to them because they were
worried that AI might be subject to the
same negative outcome that happened with
with GMOs in Europe I don't think
there's any real danger of that I mean
the the level of investment in the the
acceleration of investment in AI is is
enormous and it's also it's a different
type of thing it's very hard to ban AI
which is really people writing formulas
on whiteboards it's not it's not a
particular organism or or a particular
chemical that you could put on the field
or not put on the field but I think what
happened with GMOs is that the industry
went into a defensive mode where they
would deny any and all possible risks
and so it ended up not being of it not
being seen as an honest discussion that
was going on it was very much the
industry versus the people who had
doubts or questions were trashed people
were mr. industry with funding other
people outside a supposedly outside to
write negative articles smearing the
people who are doing the research but
might raise questions about GMOs so it
was a they adopted the classic technique
that for example the tobacco industry
adopted and I think that resulted in
skepticism
but they're right but it was the
techniques that they adopted in the
discussion right to deny risk to say
that all these questions are answered
trust us we know well in fact there were
things they didn't know and it's turned
out that the some of the negative sex
people predicted haven't come to pass
but it wasn't that they had already done
all the science they just simply denied
that anything could possibly go wrong
so I think that it was more of a
political failure and the political
failure was not that they were they were
right and everyone else was stupid it's
that they adopted this approach that
everyone else was stupid and that they
were right and that didn't work
[Music]


Interactions between the AI Control Problem and the Governance Problem | Nick Bostrom

[Music]
so the first one was the question if
human-level AI is developed then what
are the likely outcomes and I'm going to
try to go over this fairly quickly to
spend more time on the second question
but we heard about them is talking about
the alphago project earlier and I think
that can be used to illustrate one
possible scenario here so I want to zoom
in on one specific aspect of this which
is the timeline so we had the the first
game with the European champion in
October 2015 and at least at all at that
point said based on its level of play
that I saw in in this initial practice
match I think I will win the game by a
near landslide so Lisa doll is very
optimistic in October of 2015 then we
have February 2016
oops this so he becomes a little bit a
little bit disconcerted here I've heard
that google's deepmind is surprisingly
strong and getting stronger but I'm
confident that I can win at least this
time then a little bit later in mark
March 9th this is after the first the
first game of the best-of-five I was
very surprised because I didn't think I
would lose then the following day after
the second game is now even more sorry I
was quite speechless he says that he
admits that the third game is not going
to be easy
finally he admits that and then let's
see I should have practiced from this
it's it soon gonna be over March 12th I
felt kind of powerless so the
interesting thing here is I mean
obviously there's all the interesting
engineering and the creative
problem-solving that winds into
achieving this but also notice the
timescale here so it's about half a year
from being confident that it just can't
compete with the best humans to being
completely overwhelmed and as we heard
alphago has continued to improve at the
rapid keep since then so my answer to
the question of what happens if like I
wonder what will go wrong next if human
level a is developed then what are the
likely outcomes I think is super
intelligence and possibly within a
relatively short time frame now you can
then ask what are the likely outcomes of
super intelligence and I'm not going to
go into this I'm going to refer you to
do it in my book which talks a lot about
this and
I'm but I think there's a range of
outcomes and I actually I spent a lot of
time in them sorry I'm I feel partly
responsible here's I spent a lot of time
in the book talking about what could go
wrong because it seemed to me when I was
writing the book especially a few years
ago that let's thank you well place that
there it seemed it seemed more important
at that point to to develop more
fine-grained understanding of what could
possibly go wrong so we could make sure
we know where the pitfalls are and we
can avoid them whereas it seemed we
could get by with a fairly vague general
sense of the enormous upside but I think
that maybe some people reading the book
mistook the number of pages allocated to
different topics for the probability
estimate of that actually coming to pass
I would say that I'm actually more
positive about AI and its potential then
then some people might have come away
believing so now to the second question
which I want to sort of spend the rest
of the time on what can we do now to
maximize the probability of a good
outcome so there are basically three
things we need we need to solve
intelligence like to get the good
outcome of super intelligence we need to
create super intolerance so that
involves a lot of difficult technical
challenges then we'll need to figure out
how to solve the technical control
problem and and the governor's problem
so I'm just going to note with regard to
solving intelligence that it's a very
difficult problem but there is also a
lot of progress being made and huge
resources being poured into this and
it's a really exciting time in this
field but the latest nipson in barcelona
I think had like 6500 people attending
which is up something like 40 percent
from the year before which is in turn up
from the air before and there is it's
just a giant magnet right now for
talented people coming of age in
computer science or neighboring fields
so that will require a lot of work but I
feel too
some extent we've got it covered it's
like good progress there we are
basically doing what needs to be done in
order to make progress in AI the world
is having its act together in that
respect now on the second problem
solving scalable control developing
control methods that can ensure that
even if we develop arbitrarily powerful
AI systems in the future that they will
still be safe still be aligned with
human values we are much farther behind
but we are a lot further ahead than we
were when this group met two years ago
in Puerto Rico there has been actually a
quite remarkable change in the landscape
what we have for example now is various
technical research agendas so we've
moved from just having the sense of
really powerful AI maybe it could be
dangerous we need to figure out how to
control it to having now a set of topics
that you could imagine assigning a PhD
student to work on this and they would
be able to figure out how to produce
some technical papers and begin to make
incremental progress on this so this is
a big step forward we also have some
work starting to come out from this and
we have the beginnings of a research
community it's still very small but but
there are now several different research
groups that are working on this doing
technical work gradually refining that
and testing it and so I want to I want
to give a shout out at this point to
some of the people who have helped
enable this by providing funding Elon
Musk also for the future of humanity
Institute Alexander Thomas and John
Talon thank you very much this is like
super gay
a third thing that has happened on the
technical front is the growing
acceptability of this and the
collaborations that have started
springing up so it's not now just a
separate little community of a few
outsiders who think we should do work on
this but deepmind has been a pioneer in
taking the technical control problem
seriously joined more recently with by
open AI which is also starting to
produce work in this and technical
research agendas and and coordinating a
collaborating we have regular research
seminars with deep lines and so forth
and I think that that's very very
healthy these these communities really
need to fuse and that's beginning to
happen that's really exciting so that
leaves us with the third problem that we
need to solve the AI governance problem
broadly construed so this is a big
complex topic and I want to just pick
out one strand one one thought within
that to discuss with you today which is
a slightly difficult and problematic
aspect I want to talk about openness in
AI development and how we should think
about that so we've got a number of
papers recently that that addresses
various aspects of this some is my own
work some is with colleagues Alan that's
our character in Stuart Armstrong Carl
Shulman who are here in the audience and
I'm just going to be able to scratch the
surface but as a first step to think
about this we want to distinguish
different things you could be open about
in terms of advances in the technical
control problem you could be open about
your thinking about that and that that
seems clearly good like if you develop a
way to keep AI safe you want to share it
with everybody else so that their areas
can also be safe values in terms of what
what the goal ultimately also and
objectives are of your organization it
seems positive for there to be openness
about that because it would tend to
differentially benefit organizations
that have beneficial goals that other
people
prove off the more you can see what
different groups are ultimately trying
to achieve them the more people can
support the ones that are trying to
achieve something good so I'll put the
green checkmark on that - I'm not going
to talk much about capability heads
complex is showing its own but I want to
I want to talk about openness with
regard to the science with regard to the
source code the platform's training data
environments things that directly help
push the technical capabilities forward
because here I think that the picture is
a little bit more complex as to the
desirability of this over different time
scales I want to make this observation
which is that to the extent that there
is openness in the development that
would in expectation reduce the gap
between the leading developer and the
nearest following developer day there
are various factors that could create
dispersion among different development
teams that might have access to
different amounts of hardware that might
have access to a different amount of
data but that might also have different
algorithms because they have found
different ideas so if you equalize one
of those that will tend to reduce the
spread so that perhaps while you might
imagine in in a low openness scenario
that that the leader might be a few
years ahead of the nearest follower in a
high openness scenario where they can
always see exactly each other's most
recent algorithmic advances and
scientific ideas that this person would
shrink perhaps instead that would be a
few months and you could imagine in the
limiting case whatever what I had
exactly the same source code there would
be no dispersion from from from
differences in that and it would just be
a matter of who has the most hardware
let's say our data so so that that is
one property of openness that I think is
significant now this has some positive
consequences for a start it probably
helps advance the field more rapidly in
the near term science advanced is best
when
results are shared and so that people
can build on them thinking about the
long-term outcomes it also has the
potentially beneficial
result of reducing the risk that one
small group monopolizes all the benefit
kind of spreads the capability more
around makes it harder for any one group
to kind of achieve as monopoly situation
however there are some important
qualifications here or concerns
potential downsides to openness so
suppose it turns out that safety solving
the control problem requires some
significant amount of extra work after
the AI is basically completed like maybe
there is some kind of safety module that
you can only install after the other
modules are in place or maybe you have
all of the safety thing installed but
you want some period of time when you
can test it in a controlled environment
without sort of pouring on the gasoline
and allowing it to reach its full
potential maybe you would really like to
have four months after you basically
have your system finished before you
kind of bring it up to superintelligence
or release it into the natural
environment now in a maximally open
scenario where you basically have an
inability for the lead developer to have
any significant advantage over the
nearest followers it seems impossible in
this scenario for that extra safety work
to be done maybe the lead developer is
responsible and decides to pause but
that just means somebody else
Sushi's right by so it looks in this
scenario that that extra safety required
to make extra safety work requires to
make the AI safe will just not happen in
the limit in case we have a large number
of competitors that are within days of
one another so in an unfavorable snorer
if the world is such that you require
this extra safety word that can only be
done at the end maximum openness
[Music]
bracketing the question of compute power
it seems to lead to a catastrophic
outcome or here's an alternative
supposition suppose that safe operation
of the AI initially require requires
accepting some performance penalty like
maybe the initial safety techniques are
not that great it requires
maybe a human supervisor to inspect
every minute exactly what the ice plans
are and approving them and you go slowly
or there are other things that that cost
increase the computational cost by 10x
or something like that then again if you
have a world where a lot of different
developers independently have the
capability of deciding to down-regulate
their safety investment the lead will
then go to however cuts to most corners
and invests least in safety and they
will zoom ahead and potentially be able
to shape the future with their unsafe AI
so again you have a very bad outcome a
third so these are discounted
possibilities it surprises if one of
these is true the third is if what I
call it the vulnerable world hypothesis
happens to be true in the post a I
transition what's the vulnerable world
hypothesis they thought is this the
hypothesis is that there is some level
of technology at which Athens strongly
dominates defense in the sense that any
small group with reasonable competent
people with access to that hypothetical
technology would be able to talk some
action that would lead to the
destruction of the world independently
of what other people do after that
action is taken so we don't know whether
the vulnerable world hypothesis is true
but there might be a lot of technologies
above the ones we have currently
developed and if at one level of
technology it turns out that this this
property holds that oftens is just it's
not just a lot easier to destroy them to
defend with at that level of technology
then there is this world vulnerability
that human civilization will encounter
once we reach that level of
technological development you could
imagine different possible you know
maybe maybe in biotech at some
particular level of advancement it turns
out to be easier to concoct some some
doomsday agent than to create the the
therapeutic or the vaccine or nanotech
or maybe some as yet unconceived
technology might have this unfortunate
property like we were lucky with nuclear
weapons that they are really hard to
build if if you could have unleashed the
power of the atom by some easy method
like baking sand in the microwave
and then maybe human civilization would
have ended in 1946 we were fortunate at
that time but you know you keep
inventing these new thinks maybe the
next nuclear weapon will not require
plutonium or highly enriched uranium but
some some easy thing so again if it
should turn out that this vulnerable
world hypothesis is true particularly in
such a way that with advanced AI you are
able to reach this technology that
enables the extremely destructive
morality then again it seems problematic
if there are many independent actors
that have access to that technology so
[Music]
while openness on safety measures values
seem really positive openness with
regard to these technical ingredients is
more problematic what should we do in
light of this well so I think that for
now
openness makes a lot of sense we are not
yet at the level of AI were where these
concerns would really kick in so we can
continue to do the academic frolicking
and exploration and curiosity and like
moving things forward in an open manner
but when thinking about past forward
trajectories future governance
institutions this looks like a very
desirable property a conditional
stabilization that is if it at some
point starts to look as if the world is
vulnerable in one of these senses or as
if solving the scalable control problem
requires doing extra work after the AI
is finished or as if we'd initially
developed control methods there will be
a performance penalty if it turns out
that the world is like that then we want
to be in a position where we can
stabilize this competitive race and
avoid the race dynamic like we want to
have that option in the world because it
looks like a real possibility that the
world could be like that so hmm
that suggests that what we might want to
do now even as we continue open
scientific development is to try to lay
the foundations
for a collaborative approach later which
would involve perhaps coordination
between their leading AI research groups
at some future point where it started to
look like the potentially concerning
forms of AI were getting close at that
point rather than each of them racing to
the finish line it would be much better
if they could coordinate form some joint
venture cross invest may be ideally pull
the resources and go forward together so
that they extend their lead over the non
included laggards a little bit extra
lead time and then don't race against
one another that means perhaps that we
should now be working to create the
ability not to share science and
algorithms until it is safe to do so so
we can continue to share them but we
want to sort of build up the capability
of making a different choice later which
might mean for example when you're
recruiting new scientists into your
organization saying that yes you're
perfectly happy free to publish as you
see fit but if there might come a future
time when we might have to stop the free
publication because sort of starts to
make that possible and working on other
things in the organizations like
cybersecurity so that your algorithmic
advances can't just be stolen at a later
time if it turns out that that stopping
openness becomes important and also to
try to develop ways to credibly commit
to sharing the benefits and the
influence the one thing that reduces the
temptation to race is if you could be
confident that if somebody else wins the
race it's still going to be good for you
so it's not a win or or die situation
but that it's good whoever wins that
would also reduce the intensity of the
racing dynamic so that you could get
some of the benefits of the open
development which is that we all get to
share in the outcomes it's not going to
be monopolized or captured by a small
clique but without the downsides which
is the sort of blind race towards the
precipice thank you very much
[Music]


0L - Theory [rough early thoughts]

well we're going to be diving in to
trying to figure out how transformers
mechanistically work
but
i'm sure you'll be shocked to learn that
uh transformers are pretty complicated
to think about and so rather than going
and starting uh with uh
full-on large transformers and
especially um the kind of really large
uh language models that we see
in modern analog nlp and we're gonna
start with a couple of videos uh
studying smaller simplified versions of
the transformer and work our way up and
in particular
we're going to be starting right now
with a zero layer transformer which is
really the the simplest model that you
can sort of conceive of that um bears
any resemblance at all to a transformer
and um despite being so simple there
will be some small takeaways that are
useful and so we're going to briefly
briefly talk about the zero layer
transformer
so a serial layer transformer really
just has two steps uh we're going to do
a token embedding and then we're going
to do an unembedding to get the watches
so for the token embedding um we're
going to go and think of the token as a
one-hot vector
and then we're going to multiply by w e
the word embedding
and that will give us
the
the token embedding and then we'll
multiply by the under-bending matrix and
then like adjust the logics so two steps
that's the entire thing that's the
entire model and so we're just going in
from the previous token predicting the
next token
uh by going and multiplying those
through those two matrices
and we can just write those out
if we want as a product
and so that that w-e-w
matrix
has to be representing um the bi-gram
statistics the the frequency is just
that empirically one token follows
another
and those bipartisan statistics in
particular needs to go and represent
bi-gram vlog likelihoods right because
we're going to go and feed it into a
soft mac so we want to have the log
likelihoods
um
and it'll probably be an approximation
because uh it has to be low rank
probably but the embeddings that we're
using are much smaller than a vocabulary
size so it's an approximation of that
but uh when we see that product that
that's what it means
and that actually right there is
everything useful i have to say on zero
there are transformers um but it is i
think a genuinely useful statement
because um when we stuck in larger
transformers all the way up to very
large transformers every equation we see
or at least the overall equation for the
transformer will always have a term that
looks exactly like that w-u-w-e and when
we see it we should immediately suspect
that it's gonna be doing some kind of
bi-gram statistic-ish like thing and we
should think back to the humble
zero-layer transformer and remember that
so okay that's what we have to say on
zero later zero layer transformers um
and
uh in our next video we'll dive into one
layer of attention only transformers