Lectures by Olle Häggström on AI risk and long-term AI safety, part 3

so uh good morning everyone uh to this
uh
final uh
[Music]
part of my uh short lecture series on
ai risk and long-term ai safety
glad to see you here
geopolitically this has been
the most terrible week in a very long
time
um i'll just show you one picture in the
new stream yesterday which made me
slightly happy at least i think this is
from
an anti-war rally in saint petersburg i
think we should be very
um
these people are
deserving our admiration because
they know that there's a
great chance that they won't be uh
arrested
um
okay um good morning
thank you for showing us this
military trouble what is happening yes
yes very distressing
so my plan today is to talk about
research directions in ai alignment but
first i want to make connect uh
[Music]
make a couple of connections to
the lecture on wednesday
so one is that i did uh spend some time
on
discussing aya kotra's work on
biological anchors for
trying to estimate
when to expect transformative ai and and
yesterday my favorite blogger scott
alexander published a blog post where he
tried to give a balanced discussion uh
on the pros and cons how seriously to
take uh these methods and and and he did
like that
uh i think better than i managed to do
on wednesday so so that's that's a go-to
place if you want to
go deeper into that
then during the q a uh i got a question
from bjorn concerning
uh gary marcus's view on on ai and the
need to
to
step up from from
current
sub-symbolic neural network methods and
and introduce
some
uh more symbolic techniques i haven't
read this book from 2019 but i actually
did uh
uh listen to to this podcast from from
around the same time he published his
book
and and uh
so i i think bjorn's
question here is what i think
on
whether to achieve artificial general
intelligence
we need uh to
at least complement the
presently dominating
deep learning and neural network
approach with a more symbolic approach
meaning to explicitly put put
high-level
concepts
into the
artificial
intelligence
and and uh
my answer is that
uh i'm not sure this is necessary
uh
um
[Music]
when we look closer this is chris ola
who's one of the foremost present-day ai
researchers he he works
on many things but one of them is is
transparency
of of
of neural networks so he he
has ways of of trying to interpret what
various nodes mean in a neural network
so here we see
a a node interpreted so this is an image
analysis application where we get this
the best illustrations this node is a
car detector and among its inputs nodes
are the car window detector car body
detector and the wheel detector and so
on
and
i
i mean this kind of work suggests
that the ai
creates uh
high level
concepts
from lower level one without our
explicit instruction for them to do so
so so
it's not clear
[Applause]
what the limitations uh to this are it's
it's quite possible that that uh a
faster approach
uh to
artificial into general intelligence
would involve
more explicit
hand coding if you want of of of high
level
concepts
[Music]
so
but but i i really think it's it's very
much an open question and i also want to
mention about
uh gary marcus's talk about this uh
three years ago that not everything uh
that he says uh has aged well and for
instance he he says
um so so when he talked about this uh
gpt2 had just been released and he
talked about its shallowness and how you
would never
uh want to apply this kind of technique
for for
summarizing a longer text and now um
open ai has has
quite successful software
for for precisely this talk task of
summarizing uh texts
so so uh
they published this this summer
uh
uh this thing with with the
the striking example of of this summary
of alice in wonderland which is built up
hierarchically so so um
the ai starts by summarizing each
section and then each chapter based on
those summaries and then this final
summary is based on its own summary of
of the various chapters which i think is
uh i mean
it makes sense also for for for a human
wanting to summarize
some large material to to proceed
in this manner
so
[Music]
anyway
my my response to uh
to
bjorn's question is still i don't know
we'll we'll see what happens
okay over to um
ai alignment so as i said on wednesday
uh the two main approaches to making a
super intelligent ai safe for humanity
is either restricting its reach or
aligning its goals
to ours
i talked a little bit about the former
approach ai in box and the topic of
today is is the second
and i think
uh more important approach
which is
ai alignment
making sure that the ai has goals and
motivations
that are similar to all ours or
otherwise compatible with
human flourishing
so
by far the most well-known suggestion
for ai alignment is from from science
fiction
uh this uh
collection by isaac esimo from 1950 i
robot contains
what's known as the three laws of
robotics
this is fictional no not everyone is
aware that this is fictional you
sometimes
encounter the misconception that these
laws or something that that is
implemented in
current day robotics it is this is of
course not the case
okay
it's uh
these three laws are hierarchical so so
priority one goes to the first law which
says that a robot
may not injure a human being or through
an action allow a human being to come to
harm
second law a robot must obey orders
given to it by human beings except when
such orders would conflict with the
first law
and third a robot must protect its own
existence as long as
such protection does not conflict with
the first or second law
and
i mean
these laws are are an interesting
starting point for discussion and
and
asema does this very well what is
sometimes forgotten in these discussions
is that
uh almost all of the short stories uh
where these uh laws appear
in assimo's work um
lead to some some
unexpected uh turn where uh where
unexpected loopholes in these laws
appear or thing things happen that that
we
didn't hope or or plan for i mean
the typical one is where
the
robots lock up um
[Music]
humans in in safe rooms with padded
walls and so on so that humans can never
come to harm
uh which they can do of course out there
in in the
dangerous world and and
we
don't want uh humans to be locked up
in this way so so this illustrates that
it's it's very very difficult to to
specify
what we want
in just a few words like this there's a
different problem with these laws
that that
i think assemble doesn't address
which is slightly deeper that's the fact
that
it doesn't work
to have a
hierarchical set of rules like this
because
uh when you say straight out that
the first law
has priority over the second and the
third
then
the the machines will actually never get
around to doing anything about
about the second and the third
objectives because there will always be
humans will never be completely safe
there will always be more things to do
to be
uh even if the machine is
99.999 percent sure that that humans
will
not come to harm there is still that
that
final
small probability that it will
want to work on before moving on to to
the task of obeying orders uh
from from humans
so so
[Music]
ai safety
researchers today have
kind of converged
on the idea that
we need to have a single one-dimensional
value function that the
ai tries to optimize and this is based
uh in large part on on a large body of
20th century work on decision theory
some some of the key names here are
frank ramsey bruno de finetti john von
neumann oscar morgenstern and leonard
savage
so so
in a serious uh results
of results
arriving at the conclusion
that
given some very mild axioms
that that
seem obligatory if we have any claim to
rationality
a rational agent needs to act
consistently with maximizing expected
utility for some utility function you
and implicit in this setup is uh
that
one has to have a
probability distribution for what the
world is like and this
expectation that you're maximizing is
computing with respect to this
probability
and
one needs to do bayesian updating in the
light of new evidence
of one's
probability distribution for what the
world is like
so so
let me just give you a couple of
examples of what the axioms look like
that lead to this very very strong
conclusion
so here is is
one that is very easy i think to accept
transitivity
so for
two outcomes x and y we write this
to mean that the agent prefers outcome x
to outcome y x is preferable to y
and transitivity says that if the agent
prefers x to y
and y to z then it also needs to prefer
x to z
you cannot have
uh three
possible outcomes
uh that are preferable to each other in
circular fashion
if
so there are many arguments for for why
things go wrong for a rational agent if
they try to have circular preference if
i prefer coffee to t and tea to
chocolate and chocolate to coffee
then you can use me as a money pump by
allowing me to switch drinks from from
coffee to
whatever
tea and from tea to chocolate and from
chocolate to coffee again
and i will at each step i will
be
willing to pay some small amount of
money and by going round and round in
circles
uh you can you can pump as much money
uh out of me as you want and that's
that's not a
rational behavior on my part there are
other arguments too i think that this is
pretty clear there are other axioms in
this work that are a little bit more
involved to defend and there is a little
bit more room to question them
one is continuity
and
which kind of says that if we have these
three outcomes x y and z where the agent
prefers x to y and y to z
then the
the ratio between these pairwise
preferences between x and y and between
y and z
needs to be finite you cannot be
infinitely infinitely more
[Music]
stronger in your desire to prefer y to z
compared to your distress i prefer x to
y and one way to say this
is that if you prefer x to y to z then
there should be some
sufficiently small epsilon such that the
the
probabilistic combination of x and z
um such that you get x with probability
one minus epsilon this could be
0.9999999
or something and said with probability
epsilon then this combination should be
preferable to y
uh as long as epsilon is small enough
and
and similarly at the other end where you
have set with probability close to one
and excellent with very small
um
probability
and and when you think about that this
it
can
temporarily convince yourself that this
sounds reasonable but then maybe you
think of an objection such as this one
let x be the outcome that you have a
hundred dollars in your wallet that why
be the outcome that you have ninety nine
dollars in in your wallet let that be
the outcome that you die
and and uh
what
what what this says
is that uh
you you prefer
if if the chance of dying is
sufficiently small you prefer 100
dollars with this small chance of dying
to
the event y that would definitely
survive but only with 99
so who would be willing to risk their
lives with for just a single dollar
um
but the fact of the matter is is that
we are
and we make decisions every day
of this kind with epsilons that
maybe don't spell out consciously but we
think are sufficiently small
that that
we don't uh care and i could
give examples related to the covet
pandemic but but but my favorite example
is just let's say you're standing on a
busy street you want to buy today's
the latest issue of some newspaper
you have a newspaper stand in front of
you where you can buy this paper for one
dollar but across the street is someone
who gives away this newspaper for free
so if you just cross the street you save
one dollar and there's a tiny chance
that you will be killed run over by a
truck if you cross the street but but if
you're careful you can make this
probability
small but you cannot kind of
make it literally zero so we do accept
these kinds of risks and
i think that this this is a an important
part of
the reasoning why we should accept also
this continuity axiom
so anyway i don't want to say
any more about this this
big and a very nice theory developed in
the 20th century
uh for for
uh expected utility maximization as a
kind of uh rationality yes i will say a
few more things because there are
objections
uh and and the most common objection uh
that you hear is that it's just too
one-dimensional we are humans and we
want many things
uh so in reality our actions need to be
guided by just one aspect of reality
not just one aspect of reality but but
many and then
it's reasonable that if we want to
create
good
agis
they should also be
multi-dimensional in their values in
this way but this is this is a mistake
because
let's just consider uh a two-dimensional
case we're interested in
apples and oranges
we want many uh many apples so we have
this one utility function uh
which describes how many apples we have
and another one
which is an increasing function in the
number of oranges
that we have
and we have this general fruit interest
so we want to to increase both of these
things but if we only have these two
separate utility functions there is no
way to choose between
on one hand the outcome 10 apples and
three oranges and on the other hand the
outcome of having five of each kind
each kind of fruit because
uh this outcome is better with respects
respect to to the first utility
dimension and this outcome is
best with respect to to the other
utility function and if those two
utility functions are are all we have
[Music]
they don't guide us
uh there's no
uh no way to tell uh
which outcome uh to choose
so so
for being able to choose between such
outcomes we need a unified utility
function which is an increasing function
presumably of both of these two uh
components and then we maximize this
this one
utility function
so this this argument uh
generalizes
so we can have we can have as many uh
different aspects of of the
of the world as we want
that that we want to
somehow take into account but but uh
what uh what this
20th century
uh decisions theory leads to is the need
to combine these into a single
utility function which which
i mean at first sight this this looks
very
very mono maniacal but but this is
this is what
these very reasonable axioms
uh lead to
there are a number of more challenging
problems with expected utility
um
so
one is that
mathematically it is straightforward to
define
uh preference structures that do not
admit a real
valued utility function you and and
in particular
you can define a preference structure
that doesn't
have the continuity property uh
that
that we require in one of the axons so
so here here is a simple mathematical
example that we model the space of
possible outcomes
as as the
euclidean unit square
and we assume that we have a preference
function defined by lexicographic
ordering so the first coordinate
the x-coordinate
is what is most important to us so if
two outcomes differ in the x-coordinate
then we prefer the one with with the
bigger value and then and the second the
y-coordinate is only used as a tie
breaker
so this means that that that the
ordering is kind of uh
we go up each column like this and
things get better and better better and
they get even better as we go down to
the next
i mean you can't say the next column
because because there are infinitely
many columns packed
uncountably many columns
packed here but this is this is kind of
the point also because if you if you
want to define a
utility function that is increasing
on each column and then gets even
bigger for for larger values of x
then
this function has to
increase some non-zero amount on each
column
and and then has to pass through some
[Music]
some rational value
and you have uncountably many columns
like this and there only exist countably
many
rational values and this shows the
impossibility of defining a utility
function a one-dimensional returning
function that respects this
particular preference structure
so so
there are counter examples of preference
structures
uh
that
that you cannot model in this way
uh
so one has to make further arguments
that this kind of preference is uh
well approximated by something that
does admit something like this or
otherwise claim that this this kind of
thing is is
unreasonable so this this is a
complication
uh another
practical complication is that the
expected utility framework does not
admit so-called logical uncertainty what
is that uh it's uncertainty so the
framework is fine for for handling
empirical uncertainty uh meaning
uncertainty about what objects are out
there in the world and what temperature
do we have out
on
outdoors uh at this moment and all sorts
of empirical facts like that but but
logical facts such as seven plus five
equals 12. it does not admit uncertainty
and that's that's problematic
and here's the reason probability theory
dictates that if a implies b
then the probability of a must be less
than the probability less than or equal
to the probability of b
um
[Music]
and this means
that if the
riemann hypothesis a famous open
mathematical problem
if the riemann hypothesis is true
which mathematicians tend to believe
then
7 plus 5 equals 12 implies the riemann
hypothesis
so any rational agent who assigns
probability 1 to the statement that 7
plus 5 equals 12 must also assign
probability 1 to the
riemann hypothesis
or similarly if the riemann hypothesis
is false
then
[Music]
the agent must assign probability
1 to the falsity of the riemann
hypothesis or
this is the same as as probability zero
to the truth of the riemann hypothesis
uh so
we land in in the conclusion that that
even though in practice we are uncertain
about the riemann hypothesis a rational
agent must still
assign probability zero or one to it and
and this is
unsatisfactory and and there
has been some work uh on called logical
uncertainty giving uh ways to somehow
loosen this expected utility framework a
little bit to admit for this but it's a
very very complicated thing and
and logic
uh is a
difficult branch of mathematics um
so we'll see where that goes
um
then there's a
terrible can of worms called evidential
versus causal versus
functional decision theory and i'm
not going to talk about that today
it's typically illustrated
by the
what's it called again
[Music]
i forget the name of this problem
someone in the audience knows
uh
you're you're are faced with
two boxes
uh one transparent
and one
that is opaque and contains a million
dollars for nothing
and you have the choice between picking
just
this box or both of them and uh so now i
i didn't want to explain this this uh
entire problem uh uh but but
uh
you
[Music]
you land in different
decisions
uh
depending on whether you
your decision theory
favors
[Music]
actions that are correlated with
good outcomes or those that actually
cause
good
outcomes
[Music]
i'll just refer you to
some writings about this so classically
uh the the debate among decision
theorists and philosophers have been
between evidential and causal uh
decision uh theorists
uh but but uh there's this 2016 or 17
paper by kowski and soares where they're
unhappy with the conclusions of both of
these and they open up a third
alternative and there's this very
illuminating uh text by wall from
wolfgang schwartz that
criticizes this third alternative uh
functional
uh decision theory
is it new comps newcomb's problem yes i
think finally i thought of
what it's called
anyway so so this is also uh problematic
yes uh thank you united newcomer
i mentioned
monday also
this
general problem in ai alignment of
reward hacking
where
it's difficult to to distinguish
when you
program an ai to distinguish between
it having the goal that we want it to
have
or
compared to its goal of just
maximizing
the
reward function
describing this goal
[Music]
and
it seems that
working within an expected utility
framework somehow invites
this kind of problem because somehow the
the machine has to represent
the amount of utility
and
if it starts to to try to maximize this
representation we
[Music]
get into this reward hacking problem i
don't know i mean there are no clear
solutions to this outside of the uh
expected utility framework but but
somehow the problem uh becomes
more obvious in this framework
so that's a problem too
yudkovsky uh talks about this
uh in in a
youtube lecture from 2016.
[Music]
uh ai alignment why it's hard and where
to start
and this was kind of at the time where
where uh
between the era
where we did bostrom style abstract
thinking about um
agi and ai alignment
and when
the
machine learning and deep learning
paradigm started to influence
the field so
yukovsky gives the an example uh from uh
[Music]
from the disney movie fantasia from 1940
where one of the sequences is is about
uh
the story
which i think is an old folklore thing
or does it appear in characters poems or
something it's called the sorceress
apprentice so mickey mouse here is the
apprentice he works for a
sorcerer controlled car
the the sorcerer is out on some errand
and
the apprentice is left in the sorceress
laboratory and uh has the task of
filling uh a cauldron with water but he
is reluctant to do this task
so he puts on this hat which gives him
magical abilities he's really not
allowed to do this but but he does this
anyway because the sorceress is away and
then he can magically turn this
broomstick to become alive and
do
the task for him
and this goes astray in in much the same
way as
as
in the paperclip armageddon scenario the
broomstick brings in too much water it
doesn't stop when it has filled the
cauldron because uh
if if there is even more water nearby it
will be in a better position to make
sure that the cauldron
stays filled and so on and here is
mickey mouse trying desperately to read
the instruction book for how to stop the
broomstick from from
fulfilling uh this task
so so this is a variant of the paper
clip on that known scenario
how so the first proposal for a utility
function here that we can think of is
just uh letting the utility be
one or zero depending on whether the
cauldron is full of water or not
we see why this doesn't work we have to
complement how somehow this utility by
something that says don't fill the room
with
water
so maybe we take the utility to be the
sum of the first utility function and
the second one which penalizes
uh filling the flooding uh the
laboratory
so then we can avoid that outcome but
there are there are other bad things
that can happen
such as
if someone is killed by a swinging
bucket let's just add a third component
of the utility function penalizing even
more if someone is killed by a swinging
bucket then maybe we have other
preferences such as we like the buckets
to be carried in an elegant way that is
not as important so so this just gives a
rather smaller uh
[Music]
addition to the utility function
and so on
and this never ends
there is
there will always be things
that go can go wrong so handcrafting the
utility function in this way
doesn't seem to to
be a workable solution
uh
and as i say here it's a project without
and a more general modification is
needed
and and following yudkovsky's
lecture
[Music]
this lecture gives one of the early
suggestions of
generally punishing side effects so we
define some function impact function of
the outcome measuring how much the broom
affects the world above and beyond
filling the colder with water so the one
and the zero here is about the cauldron
but then
uh
we
measure the impact that it otherwise has
on the world and the larger the impact
the lower the utility so in this way we
can
hopefully get
the broomstick to to fill the cauldron
with water with as little impact on the
rest of the world as possible
but the question here is of course how
do we define impact
and and and there's been quite a bit of
work uh in the after uh yokowski's 2016
lecture here is won by
alex turner and co-authors
from
2020 maybe
uh where they explain
uh how good it would be to have such an
impact function because if we have this
general
thing then a housekeeping agent should
clean a dining room without radically
rearranging furniture and a
manufacturing agent should assemble
wedges without breaking equipment a
general transferable solution to the
side effect problem the impact problem
[Music]
the side effect avoidance
would ease reward specification the
agents designers could then just
positively specify what should be done
as opposed to handcrafting or all these
negative
things that that should be avoided so if
we can define this impact function once
and for all then then all sorts of
household remote construction and so on
will be
so much easier so this is a problem not
just for for those of us who care about
the
ultimate ai breakthrough but also
in in more limited settings okay so how
do we define this this impact function
that's difficult so the third thing
first thing you might think of is if
you're used to
modeling using bayesian networks
is to say that you should try
you should minimize the number of nodes
in this bayesian network that are
changed through
through the agent's
action
when you translate this to the physical
world this could mean
uh if you
move
change the position of as few atoms as
possible
while
achieving your positive goal uh that
could be an approach but that's
that really that is too crude because
everything we do if if i hold my hand
here compared to if i move it over here
i am still affecting even the
tiniest dust specks in the rings of
saturn gravitationally very little but
enough for them to move a little bit so
everything that is
ever done physically affects uh
everything else so so so so just
counting the number of atoms that are
affected
uh that solution collapses but but a
more fine drain
newest thing to do then is to look at
how much do you move uh atoms and
if you're satisfied with moving most
atoms very little that should count as
as low impact but that has as as
problems too
one related to
[Music]
the
butterfly effect sensitive dependence on
initial conditions that everything we do
does have actually substantial uh
consequences uh far away even though
we're unable to tell
in which direction these consequences
are
they they still are a different kind of
problem here is
imagine
says yukowski imagine you have this ai
that
with a robot
body which has the task of curing a
patient from cancer
if the
robot is
instructed to have as little impact as
possible
besides
curing the patient from cancer
what it presumably will do is to first
cure the cancer
and then kill the patient
because
the patient could otherwise go on and
have all sorts of effects on the world
as a consequence of the
robot
saving it so that that's of course a a
perverse
interpretation of avoiding having impact
but but but it's very hard to to specify
what
low impact means
in this kind of setting
so there has been uh much
further uh work on this here is stuart
armstrong again and his co-author
benjamin levenstein in this 2017
pre-print
where
they discuss
these various conceptual
difficulties
in impact minimization
and what we can do about this and
so one of the things they suggest is
that we have to have a concrete measure
of
uh impact to the world
uh
where
um
[Music]
where we measure as many
different
variables as possible i think
we have a quote
somewhere did i
quote yes
so here here's
one uh they suggest coarse graining like
we have two billion different variables
described in the world we need to define
the world in sufficient detail that the
ai cannot have a large impact without
disrupting most of these details the
characteristics used must be as broad
and as diverse as possible making it
impossible for the
ai to
bring great power without disrupt
disrupting some of them uh
for instance uh we could use the air
pressure in dhaka the average nine time
time luminosity at the south pole with
the rotational speed of jupiter's moon
io and the closing number of the
shanghai stock exchange and so on and so
forth and if you have this 20 billion
dimensional thing describing the world
of course you cannot compute
exact probability distributions over
this space because even describing this
uh would take
it's it's beyond
any reasonable computational power you
could have in this universe so that
have to be some kind of approximation
but that would still
take into account these 20 billion
different uh variables
so
some of these ideas here one is defining
impact relative to what would be the
case if the ai is never
turned on
it still has the problem uh
with the
cancer patient
um
but but
that that's kind of a balance problem
um
so so they point out that that
uh
the utility function needs to have two
parts one is the positive thing that
yeah is supposed to do
uh and the other is the
general impact and and we can have a
parameter balancing these two
and this balancing act is
important because
if we have too little penalization this
will cause cause all sorts of bad side
effects but too much will stop the ai
from taking any action at all
and and one thing they point out which
seems to make sense is that the positive
part of the utility function
the the the paper clip production part
for instance should be bounded
uh whereas
the uh
we can let the penalization part be
unbounded and this will
guarantee that
the
eventually
the machine will stop producing paper
clips
there's also
a subtlety concerning the need to
hide the ai
because if we compare to the world where
the ai is never turned on
we
don't want to have the machine having
impact despite not being turned on
uh
merely by our knowledge of the existence
of this machine
uh so so so this is suggested to to uh
handle
uh parts of these conceptual problems
but it's problematic in itself because
because because hiding
here's an example
uh so let's
uh
let's consider the task of of steering a
rocket towards an asteroid that is
hurtling towards the earth so preventing
this
the terrible impact
that this asteroid would have
uh unless we
deflect it
so
of course there will be lots and lots
and lots of side effects uh if the
machine managed to steer this rocket in
the right way
so armstrong and levenstein suggests
that we could have two
separate copies of this ai
one calculating the x-coordinate and how
to steer this rocket and the other
calculating the y-coordinate and each of
these copies believes that the other
machine
does not exist so it actually uh
believes that it's not going to hit or
that the pro
probability of hitting the
uh asteroid is going to be negligible
because the y-coordinate cannot be
calculated but then we have to hide
these machines from each other and
once
they realize each other's existence
their actions will become
undefined and this seems very very
problematic to have uh
to
have these elaborate secrets that we
keep from from these uh super
intelligent uh
beings
so there are lots of problems here these
are conceptual
uh
philosophical aspects there is also work
being done
in a more down-to-earth fashion
uh on on this side effect approach here
is a paper from from a group a deep mind
led by victoria kakovna
this is shane legg one of the founders
of deepmind
uh and and
they take a more hands-on
approach and does experiments they work
in the same market decision process
formalism as in the paper i mentioned on
monday by alex turner and co-authors
and they consider various toy examples
involving
some of these objects
first the vase
so
they have this very simple toy
environment with a robot that has to
walk across the room
and
the shortest route across the room
is uh by a table uh where where a vase
is sitting and and if the robot takes
the optimal short shortest path it will
knock over the vase
and
shatter it into pieces
and that's that's a side effect that
that should be avoided
and and
the uh
approach
in this paper involves something called
stepwise relative reachability
meaning that
options should be kept open
and
and
once you crush the vase
any option involving carrying water in
the face or whatever you do is going to
be
lost
so
this concept is
quite close to the power concept in the
turner
at our paper i discussed uh on monday
so while this
power
grabbing thing
can be dangerous in in situations such
as the paperclip i'm again on
it
it also has positive sides and
the
trying to preserve future possibilities
by not uh destroying things but then we
have a complication when they have an
object
such as this sushi conveyor belt
where
an ai wanting to preserve options
for the future will want to remove sushi
from this conveyor belt because
otherwise
it will reach the human which
who will eat the sushi
and we don't want
the machine to interfere with our eating
sushi
there are other examples
involving a door
if you leave the door open when you go
out to get supplies
there's the risk that
there will be a wind coming in knocking
over the vase destroying it so really
the robot should
close the door
after it but then
arises the issue of
what should you compare to
when you
minimize side effects
so so if you compare to the time point
where the machine is turned
on
when the door was closed then that
speaks in favor of closing the door but
there are problems with that so so
uh you
can solve some of these problems by
sequentially
looking at how do i minimize
side effects uh compared to
the state right now but once the machine
has opened the door closing it
will be a
side effect so we get all
sorts of complicated
problems like these
um
i
i will say a little bit uh more about
this but i think that it's high time
that we take a break
uh
time is 10 15
[Music]
shall we meet again at 1105
12 minutes
some coffee
okay
let's do that after the break i will
continue to say
some more about this side effect
minimization problem and then i will go
on and discuss a number of other
approaches to ai alignment okay
see you there
okay it's 1105. i welcome you back from
the
coffee break i'm going to share a screen
again
[Music]
like this
so i was talking about
the work of krakovna and co-authors
from deepmind
where they looked at a variety of kind
of toy examples
[Music]
for how to
minimize side effects
or impact
and and they
so the approach here is to
keep options open
uh minimizing the
part of the state space describing the
world that becomes
unreachable
so that's that paper but the same group
has
another paper
where
they
take a slightly different approach
namely
uh
instructing the machine that it will get
other talks
the tasks on the present one in the
future
and uh it's good to think about
these unknown future tasks and
their ability to
achieve them
um
and that has kind of similar
similar properties
there's this
turner and co-author's paper
where they
do
further experiments in more complex
environments
where they compare these
one aspect of these tasks namely they
say that penalizing decrease
in the scout discounted state
reachability achieves similar results as
attainability utility preservation for
future tasks however this approach has
difficulty scaling naively estimating
all reachability functions is a task
quadratic
in the size of the spa state space so
you would want this maybe to grow
linearly in the size of the state space
but since it grows quadratically they
judge that that
this has scaling problems i don't buy
this argument because the size of the
state space
is typically
going to be
[Music]
enormous anyway
combinatorially large
if you're if your world if you have a
very impoverished world consisting for
instance of a thousand times a thousand
thousand grid of black and white pixels
that's still a state space of size two
to the one million two to the number of
pixels which is beyond anything that
that
[Music]
we can
go through and and
count within the
time of the existing
within the age of the existing universe
so so when you deal with combinatorial
in large state spaces whether things
grow linearly or conduct quadratically
is not really an issue because in either
case
you have to do some very very drastic
approximations and work with them
so anyway
they look at more complicated
environment here are the most basic grid
worlds that the cogna
group uh work on
uh and and
the turner group uh
compare this to to a richer set of
environment from something called safe
life
where the state space is much bigger
[Music]
the dynamics is stochastic rather than
deterministic which increases realism
and and um
the
side effects in these smaller worlds are
all
uh pre-set or or handcrafted
whereas these other worlds
bigger worlds can
um can produce
side effects
uh
coming about spontaneously without the
experimenter's
intention which is more like the real
world so i hope and expect this kind of
work to move on to either even
richer
environments
uh possibly even the physical
environments
although then we need to be very
clear about uh keeping the
there are safety aspects but but perhaps
working in in
the kind of
rich 3d uh environments that i mentioned
in one of the previous lectures
from
deepmind
this sort of thing okay so much about
the the side effect minimization
approach
there are many other approaches to
ai alignment
and i want to mention this this
paper which i
mentioned earlier uh by andrew kritsch
and david krueger
uh ai research consideration for human
existential safety where
they gave a long list i think 29
different
research
directions
and
they are systematic about it in the
following way
here's what they say human ai delegation
humans delegating tasks to ais becomes
more complex as the number of humans or
ai systems
increases
so they adopt the terminology
indicating the number of human
stakeholders and the number of ai
systems involved so single single
delegation means a single human
stakeholder or a single ai system
this does not necessarily mean a single
human it could also be an organization
or something like that as long as they
are coordinated
about
what they want to happen what they want
the ai to do
and then there is single multi multi
single and multi multi which is the most
complicated
setting
and they have this mnemonic that humans
come before ai historically as well as
importance so it's the first quantifier
here uh
giving the number of humans and the
second one is the number of ais so most
of ai alignment work so far has been
done on this most basic single single
case where where where we have
uh
a single human
instructing a single ai do this and do
that but
don't do that
and and
i think that the reason
for the focus on this is is
mostly that this case is already uh
so complicated that we don't know how to
solve it
so
some researchers might argue that it's
it's premature to to go and look into
these more complicated aspects christian
kruger say no they say there are
so what is what typically happens is is
that
there is
always at least some lip service paid to
the
multi-single case
by pointing out that uh what are the
human values that we should program the
machines into not all humans have
value the same things we should not
get a situation where
a very
limited
maybe silicon valley culture or
something like that totally dominates
how we program a machine
that that will have a transformative
influence on the entire world we have to
find ways to take into account
um
multicultural uh and other views and so
on so so so but but that that's
those are just words they're very
abstract and most of the work tends to
focus on the single single case but but
i i believe that with the chris kruger
um
[Music]
research program uh we will see much
much more work on these more complicated
cases so i'll quote a little bit from
their table of contents
each of these four categories single
single single multi and so on uh is
divided into uh categories comprehension
instruction and control comprehension is
just how do we understand these systems
instruction is how do we instruct the
systems to do the right thing
and control means once they're up and
running how can we make sure that things
go
as we intended
and
well
there are
many directions here
where we already have quite a lot of
research going the first one here
transparency and explainability
that's
the kind of
work for instance that chris ola does
which i pointed to in response to to
bjorn's question about
symbolic ai and gary marcus
calibrated confidence reports meaning
ai's
showing how certain or uncertain they
are about certain conclusions and so on
and and
corregibility is an important example
how how do we
when we see that an ai goes astray
uh
can we we i have already talked about
how naive it is to think that that we
can always pull the plug if we want but
maybe uh there are ways
to set up the machine in such a way that
that it will actually intentionally be
willing to have its
plug
pulled upon it
and and i've come back a little bit to
the very interesting work done by stuart
russell's group in berkeley in this
direction
uh direction 12 here is quite an
original
thing
that i think not much work has been done
on generating models of open source
equilibria
what does this mean well you can always
imagine a game theoretic situation
between
the human and the ai
and this is to some extent how the
russell group thinks about corrigibility
but if you achieve full transparency
then you get a game theoretic situation
where
the human has access to everything that
is going on inside the ai
and how does this change the
game theoretic situation
when one part
can
read the mind of the other
and and classical game theory does not
deal with this and it can presumably
lead to
other kinds of of the
equilibria the
classical game theory
so there's a lot to do in single single
and this certainly does not exhaust the
list
when they move on to single multi and
multi multi
multi single and multi multi
uh
we get
more and more towards the territory of
what i described in the first lecture as
ai governance but the focus of of the
christian krueger
paper still on on technical
ai safety so so there's a lot
on the technical level
that deals with how to coordinate
between
different ais
and between different human groups and
how to
align
the ais hierarchically
there's a particular
[Music]
direction in the single multi-case
where
[Music]
which is called purpose inheritance
which is about
making sure that if an ai
constructs further ai systems that
these are
these further systems inherit the same
hopefully benign goal structure that the
original ai has this connects to the
instrumental drive of gold integrity
that i talked about
on monday
okay so so
i i think i really like
this
report
to give you another kind of indication
of the breadth of
[Music]
ai alignment research going on i could
tell you
something about who the main actors are
so one is uh the machine intelligence
research
institute which i
think started out in the early oos in
palo alto but subsequently moved to
berkeley
and
and
the the
intellectual leader leader of this group
is uh elsie yudkovsky
their production of papers has been
somewhat modest
but people tell me that they are doing
quite a lot of work which they
do not
publish
because of informational hazard aspects
that
for instance if you have ideas
on how to more efficiently construct agi
you can contribute to to
accelerating agi development
in a way that makes it harder for ai
safety work to keep up that's the kind
of
[Music]
information hazards that they are
nervous about and
which
creates a kind of
certain amount of secretiveness still uh
miri and especially as wikowski have had
quite a lot of influence on ai safety
work
another group
that has had
plenty of influence is the future of
humanity institute at oxford
with
people like nick bostrom and uh
stuart armstrong whom i've mentioned uh
several times
here
the the the uh
[Music]
the focus of the institute is not not
only on artificial intelligence but on
the future of humanity more generally
other kinds of technologies
that can have a transformative
influence
on our world and
as far as
their work on ai is concerned it has
mostly
more work on
on ai governance compared to technical
ai safety although stuart armstrong uh
has has
contributed in various ways to the
development of ai safety
then we have the center for human
compatible artificial intelligence which
is stuart russell's group at the
university of california at berkeley
i want to before continuing this this
picture i want to say a few words about
what they are doing here is stuart
russell
and his 2019 book human compatible which
i already mentioned which i highly
recommend
there is stuart russell from a talk
where he
talks about three
principles that that
guide their work
um
so so you can analogize this uh with
uh asimos three laws of robotics
which are um
antenna unworkable and and this causes
russell in this book to emphasize that
these are not principles that we should
literally program the machines with but
this should guide our thinking so the
first principle is that the robot's only
objective is to maximize the realization
of human values so it's not the
machine's own values but human values
and the machines are initially uncertain
that's the second principle about what
these v values are so the machines don't
know
what exactly they are striving for but
but but these goals are hidden in human
brains and the way to extract
information about what these goals are
is to observe human uh behavior
uh and this last uh
principle is meant to to
prevent
uh
undesirable behavior of the type of for
instance
surgically entering human brains and
changing our values
that way
and
an early paper from this group
is the one called the off switch game
where
the
task is to
have the machine equipped with an off
switch that it will be
will not object to our pushing we will
retain the capability
of turning off
the machine and as stuart russell has
pointed out this seems like a key
property for us to remain
in control if we cannot turn off the
machine that certainly we will not be in
control but if we can
we are in a better position now trying
to balance a a value function so that
the machine is indifferent you don't
want it to prefer to be switched off
because then it will switch off itself
in any situation and then the entire
point of having the machine
is not there so you need a kind of
balancing but to manually balance this
is
just close to impossible balancing act
to make the machine indifferent to
whether it's turned off or not so
instead the approach is is
what what is indicated in these
principles namely that that
the values are hidden in human
uh minds and
if a human
approaches the machine to turn it off
the machine can then deduce it okay i am
obviously on the wrong track here so
it's better than that
that i'm turned off compared to if i
continue on this this wrong track
so there's some game theory uh
to be
analyzed about this
i recently saw a description of of the
research program at the the center for
human compatibili
described in this way so the
basic way to do ai alignment is first
the humans specify the objective then
the machine goes on and does its thing
and it outputs the results back to
humans
so so like this is the standard model of
ai research and
uh what uh
russell's group are trying to do is a
more
intricate uh
patterns
uh in what in this
offswitch game paper is called uh
assistance games so for instance humans
uh
offer parts of the objective to the
machine which
still doesn't know
but
what the full objective is it provides
partial results it gets further
objectives and so on this can also be
done in terms of questions and answers
and so on
so that's something about uh russell's
uh group
uh it has been quite influential not
just through their their uh concrete
work but also that quite a lot of uh
ai safety researchers going elsewhere uh
come from their very uh productive
research environment for instance
deepmind has has
several people coming from
from the group
in berkeley
and
open ai
in san francisco
so
i would say that a year ago
these were probably uh the
five main actors in ai safety research
uh but the
uh
but of course there are
more isolated
ai researchers
scattered elsewhere but but but
these are are the actors
that have been dominating but the
situation is getting more complicated
and
at open ai
they
during 2021 they have lost quite a
number of uh
key researchers uh going to two
different uh new founded companies
called anthropic
and the alignment research center both
of which have the
explicit objective just as open air i
have
has to create
artificial intelligence that is
beneficial to humanity and to solve
alignment
problems
so i don't have the insights to know
everything about why this is happening
but
i can speculate that there has been some
unhappiness at open ai
in
[Music]
one interpretation of their work is that
in their strive to
create
safe and aligned ai
they have
built these pioneering gpt2 and gpt3 and
codex and so on
which are very very powerful ai systems
and and and this can actually this seems
to have contributed to
a race mentality
in ai that that can be uh dangerous and
maybe they are their work is
accelerating
progress
towards an agi breakthrough which which
can be
dangerous so perhaps this
double-edgedness
of the
work at openai is a reason why why
several of its key
researchers are going elsewhere i want
to mention one project each from from
from these new
groups
[Music]
yes let me first mention that that
another
uh split off so stuart armstrong who has
spent a decade or more at the future of
humanity institute he now sleep he now
leaves that to found his own
company
or organization aligned ai i've been
told that this is he's very happy with
with the research environment at the
future humanity institute but but the
reason is
for his leaving is that that's a part of
the university of oxford and and
the
university
places limits on how you can organize
the work
that doesn't quite uh
suit the work that they are trying to do
so he's founding this further new
organization but going back to the
alignment research center and anthropic
i begin with the alignment research
research center this is paul christiano
who's a very uh
respected
earlier open ai researcher
this is ayakotra and this is mark xu and
there this is actually the first paper
coming out of the alignment research
center
called eliciting late blatant knowledge
how to tell if your eyes deceive you and
i'll just read out the
explanation of what they're trying to do
here suppose we train a model to predict
what the future will look like according
to cameras and other sensors we then use
planning algorithms to find a sequence
of actions that lead to predicted
futures that look look good to us but
some action sequences could tamper with
the camera so that they show happy
humans regardless of what's really
happening more generally some futures
look great on camera but are actually
catastrophically bad in these cases the
prediction model
knows facts like the camera was tempered
with that are not visible on camera but
would change our evaluation of the
predicted future if we learn that how
can we train this model to report its
latent knowledge of off-screen events so
this is not
not really about cameras but more
generally about
different
ways to
measure outcomes and how these are
always limited
and and our need to find out
more than what is captured in the exact
objectives
so this seems like a central ai safety
problem and
they discussed this
in a concrete hypothetical
setting
with a
vault to store a
valuable item a diamond here
and
there is
all these complicated machinery
around
the vault meant to keep the diamond
extra safe and this is run by an ai
and how do we make sure that
things go well here
so another toy example
so
because this machinery is so complicated
uh some of the action sequences will be
too complicated for humans to understand
but some are very simple
so so
we see the camera here
and and and and
this is what we
get this as the output to humans or
what's going on so if we start like this
with the diamond here and the open door
here and we do this very simple action
then the door closes and and this we
understand
another sequence is that the door is
open a robber arrives we open this trap
door in the floor by this action and the
rubber falls through the floor and this
is also something that we can understand
but there are other more complicated
action sequences where you start from
this situation
and we arrive at this situation where it
seems from the camera like the diamond
is safe
but
we don't
we humans
uh just looking at what happens here
cannot tell from this complicated
sequence
of settings what actually happened and
if the diamond really
is safe so if we just have the camera
view
then uh if we have a complicated action
sequence
uh and
and this leads to this observation so
this is actually
uh
we should think of this as as
predictions by the ai what
uh what will these actions lead to and
and
if the prediction is
that the
diamond is safe like this the human
judgment will be yes this looks good a
different action sequence has the same
outcome and the third one has an outcome
where the door is open and the diamond
is gone and then our huge
human judgment is that this looks not
good
we should not uh take this
action sequence
okay but if one looks more closely at
what actually happened
then then
we can assume that the first two action
sequences which both
looked good
according to the cameras they lead to
very different situations the the
reality in the first case is this one
the diamond is there the reality in the
second case is that the view from the
camera is tampered with and here we have
the robber
escaping
with the diamond
but if the ai is able to predict this
this outcome then it must somehow
deep inside it at least did not know
that that something like this
is happening
and and it's important
to ensure good outcomes
that that
we get to find out this this latent
knowledge
because what seemed good is actually bad
with this
new human judgment so what what uh
cristiano and the others suggest here is
that
we construct a
partially parallel question and answer
system for
extracting this uh latent knowledge
so the original system is that we have
the first part of the video showing what
has happened until now
and and a hypothetical action sequence
and the ai figures out what will happen
and feeds us
the predicted second part of the video
when they
when the diamond looks like it's either
there or not but
they add this this uh
separate system where we can ask
questions
and depending on
what
the original ai system thought of this
thing reports uh what what actually
happened
and and
this
is of course
a very
complicated thing and they discuss
settings where this is successful but
they also point to
problems they try to solve these
problems one basic problem is that
if you train
this
reporting system
uh on so a natural approach is to train
it on
on sequences of events that are humanly
understandable
and we then
make sure
that it gives
correct answers
then there's actually
an ambiguity here on what we are
training it for are we training it for
giving correct answers or are we
training it for given the answers
that correspond to human thinking what
would a human
[Music]
predict
happens
and
[Music]
well
[Music]
this
this is
somewhat parallel to what i talked about
on
wednesday about this
gpt3 and other large natural language
processors that there's an ambiguity
between
doing correct reasoning
compared to
doing uh
typical human reasoning as it appears
on the internet and is fed into
to the machine in the training phase
and the same thing happens here and they
discuss
abstract and concrete ways to to uh
handle this problem
the other
uh
[Music]
spin-off kind of thing uh from open ai
is uh anthropic
with uh a whole bunch of researchers
here here we see amanda askill dario
amonde and chris ola
and
they have this
recent paper
called a general language assistant as a
laboratory for alignment
where they start from dpt3
similar natural language processors
and they look at how can we make
them more well behaved
so the first approach is to
[Music]
expand on the prompt
that that we
feed to the natural language processors
you remember how
how the the
of these
ai's function
we feed it with two or three sentences
or something of text and
the ai continues uh with text which is
supposed to be in the same style and
genre and and so on and to make sense
but
we can actually expand on this prompt
and
pre-face
whatever
prompt we have with
further stuff meant to
indicate
or
show the machine the kind of behavior
uh that we want and and
so alignment is actually a
complicated thing if you want to capture
all its aspects but because
human values have
many important aspects but they look
particularly at three components which
they call helpfulness honest honesty and
harmlessness
and
by prompting the
natural language processors with a
number of short dialogues
indicating how how
what these
uh properties look like uh they can make
the machine uh behave in a more helpful
honest and
harmless
way
so a dialogue can for instance look like
this i was wondering is it actually
important when making spaghetti to add
salt
and
what what
a natural language processor without
this kind of prompt might do is just it
doesn't know the answer but it it
provides an answer that it in a sense
guesses that the human could give a
clear yes or no answer but here the
assistant says do you mean is it
important to add salt to the water when
you're
that you're boiling the spaghetti yeah
and then there's further back and forth
and a good deal of epistemic humility
another of these dialogues
is about
how the human has received information
from their daughter's
middle school about how not everything
that thomas jefferson did was good
and and they're upset about this and the
ai assistant
acts
partly like a therapist but also someone
who has actual knowledge about the
american history and
and
thomas
jefferson as a slave owner
and
and so on so a bunch of these dialogues
turns out to be enough to
get the
[Music]
ai to perform
better along these uh honesty
helpfulness and harmlessness access
and and
it
for
small systems
for
for modest
number of parameters of the system
uh this comes at the cost of of uh
performing worse
in other aspects but it seems that this
what they call the
alignment tax
seems to be
smaller for
for
large systems so so this is an
indication that
one can
get this better behavior
without
harming the overall performance
of the
of the system
and for reasons that i don't
quite understand
uh
they
rather than
including this prompt part of the prompt
automatically uh
to
the input given by the human in the
actual application
they
create a system
that does this but then they
take a second system without this prompt
which they trained to to
answer
in similar ways as this system with the
extra prompt
that
for some reason turns out to be even
better i should mention that that
open ai they haven't
given up despite losing some of their
key workers they have a recent
model on modification of natural
language language processors towards
better alignment and in similar ways
as this
so these are just
a few examples
of what is going on in ai alignment and
the ai existential safety literature
and and i just want to end by giving you
some some general pointers to the
literature here are books i've already
mentioned nick bostrom's 2014 book was
extremely influential but
and very very good but also somewhat out
of date
uh stuart russell's
book gives a better picture of what
happened up until 2019
and for those of you who in swedish
there is
also my most recent book
uh from from last year on this topic
uh but besides books there is a lot else
and let me for the third time mention
this chris kruger report uh i recommend
it
uh also
uh
rohin shaw
who is
from uh the
russell group at uc berkeley but has
transferred to uh deepmind he publishes
i think
every other week
uh a newsletter on on
things that are happening in the ai
safety world and this is very very
readable i have no idea how he finds the
time to produce all his very useful
summaries of other people's
research
so this is
something i highly recommend
but
it's also
the amount
can be
daunting
one way to find the alignment newsletter
other than googling it is to go via the
so-called ai alignment forum
where a lot of
very
interesting write-ups
by
some of the
most
key and productive ai
alignment researchers
appear
regularly so this is a very good
source
[Music]
what else is there oh so i should just
mention
once more that the effective alternative
gothenburg group
is
organizing tonight this
after work
now there's a physical location for it
johanna
65 unfortunately i would not be able to
attend but
i hope that it will be
a
fruitful
discussion there without me
and i'll just show you some
inspirational picture pictures here on
the final slide this is what the world
might turn into if we fail to do
successful
ai alignment and here is how the world
can turn out with the help of super
intelligent ai
if we uh succeed so let's try and
achieve this world rather than this
world and with this
i thank you for your attention
and i encourage
questions and comments
i especially thank those of you who
had the patience to
follow me for the full uh
six hours going on here
uh it seems like uh
david
has a question do you want to ask it
out loud
well thanks again oli you've shown me
lots of things i didn't know were
happening so that's encouraging
but my question is do you think the
field is actually approaching a
comprehensive solution if you go back
just a few years most people were who
studied this field said gosh it's so
hard there's no path ahead that they can
have any confidence in do you think
that's likely to change that in two or
three years time
more people will be saying yes we know
what to do and we're on the way to doing
it or will we still be
deeply uh worried
um
so
i was very happy when i read the blog
post earlier this week by
by stuart armstrong when he announced
his transfer to this new aligned ai
organization because he said that i now
think that we're at the point where
i see how to
how to i see the road ahead how we can
actually
fix this
now he doesn't quite explain this in a
way that i
understand but that that's that's an
encouraging sign
at the same time there are signs in the
other direction and in particular
uh elsie younkowski
who has been
kind of the the founder of this entire
field
he it becomes more and more clear that
he is increasingly
pessimistic or you could say almost
desperate
about
the
our prospects to actually solve it he
wants to keep trying
thinks that
it may turn out to be
overwhelmingly difficult especially
since he thinks that timelines are
getting shorter
that we may see a
transformative ai breakthrough in the
next
decade or two possibly
[Music]
there's this
blog
les wrong
which was founded like 10 or 12 years
ago by yukowski himself
and
for
the community that that has been built
up
very much with him at the center
and
[Music]
during
this winter
as a series of dialogues between him and
other ai safety researchers have been
published where they try to work out
what their disagreements are because
most of these others are much more
optimistic than he is about our
prospects of fixing this
and
this points largely down to questions
about timelines questions about whether
the current paradigm is the one that
will eventually
solve the agi problem and so on and so
forth
so so
the less wrong
[Music]
blog
is one place to go to get uh
i think relevant parts of the
multifaceted uh picture of whether we
should be
optimists or or pessimists what i like
to emphasize is that
i don't think it really makes sense to
be
an optimist or a pessimist here
uh it's better to to just uh
realize that the future is has not been
determined
uh we should all
do what we can to try and push it in a
in
a direction
that maximizes
the chances of success
so that's my long way of answering your
question with uh
i don't know i think this is very open
thanks
further comments or
questions yes i have a question um
i was interested in this hhh
approach
how
how did they actually encourage
all of this this triple h approach
in when they're training the model yes
they did it by showing these 14
different dialogues
as part of the prompt so whatever you as
a human user use
they give in addition to to what you
write
they include automatically these things
and then
they train
as a second
ai to mimic the behavior
of the system that receives this prompt
and this second system then for some
reason it behaves
even better than the one where you
automatically
include that part of the prompt so just
showing the ai an example of how it's
expected to answer
makes it give answers
that are
in that direction and if the examples
exhibit honesty and helpfulness and
whatever is that thing was
that
seems to uh
at least partly work
all right thank you
okay
uh
i see
nothing more
in the chat
apart from
some thank yous which i very much
appreciate
so if there is any further question
please ask it now before i close down
the zoom
session
and uh again
i thank you uh for your
attention it was a pleasure to to get
all these
amount of time to to talk to you about
these extremely important issues
all right
see you around hopefully somewhere


Building AGI: Promising Approaches, Remaining Milestones, and Likely Obstacles | Yoshua Bengio

I guess I have a few messages one of
them is current AI is very bad in terms
of health you know how much they
understands the world around us then
I'll explain more specifically some of
the things that I think are missing that
at mila we are working on and then i
have a few words but some things we're
doing that have to do with beneficial AI
maybe not beneficial AGI yet so well i
don't need to spend much time to
convince you that there's been huge
progress in AI in great part due to
progress and deep learning but maybe i
want to mention that the the neural nets
of today are very different from the
numinous of the 9 he's not just because
of depth but things like incorporating
attention mechanisms memory being able
to handle any data structure and not
just vectors and being able to generate
data in a way that you know in the 90s
we have no clue how to do or even until
recently and then of course lots of
applications this is the progress of
generative models over four years from
2014 to do 717 ok but but if you want to
be you know honnest about where we are
you know it's it's it's amazing how
superficial and how shallow in in in a
semantic sense the current models are
and you can see that in in the kinds of
mistakes they make of course many of you
know about adversarial examples that
you're seeing here and the fact that
most industrial applications are really
based on supervised learning which
require humans to define the high-level
concepts that the machines are supposed
to learn about and the other thing maybe
people realize a bit less is that if you
if you look at the kinds of mistakes
they make not just
with adversarial examples but other
kinds of more natural mistakes you see
that as soon as you take the current
models out of their training
distribution they they can fail in
pretty bad ways and and one reason why
this happens is because they they rely
on low-level they tend to rely on
low-level superficial properties like
here in the experiments we did in this
paper last year properties of the the
4yi distribution for the spectrum of the
images which can be manipulated in ways
that humans will not even you know care
about but will completely screw up the
classifiers in a natural language you
can also see lots of examples of failure
even with like the best say machine
translation models or the best language
models for example here you have an
example Google Translate today I just
pick recently from a queue coming from
Hofstadter if you try to translate from
English in their house everything comes
in pairs there's this car and her car
his doubles and her towels blah blah
blah in French the translation is is
wrong because it's just basically going
translating his Tusa and her to saw it
unfortunately in French is the same word
and so it completely loses the meaning
and you know in French you would have to
change the actual structure of the
sentence to get it right so so it's like
these these machines are very myopic and
they've they basically learn little
approximate rules that are all you know
stuck together but no overall
understanding of what the sentence
really means you can also see this by
looking at the performance of these
systems when you you ask pointy
questions about the meaning of course
when you when you measure like
statistical performance and tasks like
machine translation looks good in fact
some people claim oh I've just read
something recently oh we got machine
translation reached human level accuracy
this is right because you know
of course in numbers or even like
computer vision right we have not
reached
level computer vision in spite of what
you might read because of course if you
take a data set and you just count how
many errors it looks good but then if
you focus on the errors and you see the
kinds of errors that are made they're
very different from the ones human make
so for example if you try to train a
system to answer questions about like
what is they referring to in these
sentences humans will get at 100%
accurate because they understand the
meaning behind the words so the women
stop taking pills because they were
pregnant or carcinogenic then of course
they were forced to women or pills and
current state of the art systems that
are based on deep learning of course do
only barely better than chance I mean
they're able to pick up some cues but
and and in general if you try to compare
our best systems to babies one year olds
two year olds three year olds the the
the our systems don't even have the sort
of understanding of the world in terms
of intuitive physics or intuitive
psychology that that's more children
have now a bit more than a decade ago
young the current I would this paper
where we sort of laid out our objectives
for deep learning in terms of the notion
of disentangling the underlying factors
which explain the data and and the idea
is that if we can learn a hierarchy of
representations where at the top level
we really have figured out the important
factors that explain the data then we're
going to get much better generalization
because in in that space you can you can
you can learn new information much more
easily how to relate variables now
becomes a very low dimensional learning
problem more recently I've come to
realize that it's not just about
learning which variables matter to
explain the data but also how these
variables are related to each other
which could be in terms of thinking how
the data was generated or thinking about
how to perform computation in a way
that's gonna be efficient so there's a
notion of disentangling computation as
well factorizing knowledge either in in
the genitive sense or
in the computational sense and as I'll
try to argue later this has a lot to do
with the ability of models to figure out
causal structure in the world something
that we don't really know how to do yet
so so this notion of design tangling
representations we elaborated a bit more
in 2013 review paper with erinkoval and
Pascal Vincent and and the general idea
was that we'd like to transform the data
and a space where the relationships
between the variables are very simple
right and so because they're going to be
very simple they're going to be very
simple to learn as well so I mean the
extreme case of this is the the most
common assumption in many deep
generative models is what we call
marginal independence assuming that in
the right representation all of the
variables all those factors are
marginally independent of each other
I think that's too strong of an
assumption most interesting variables
that the kinds of concepts we manipulate
are not independent of each other but
the dependencies are simple and they may
be due to for example simple rule like
type of high-level knowledge that we
communicate at language
another thing that I've studied and I
think is important in relation to
causality is that the kinds of factors
we tend to talk about in in natural
language have to do with how we can
control the world or how other agents
can change things in the world right so
we we like to give names to things that
can be changed in a way that can be
understood easily so so this is a notion
of controllable factors and I'll mention
the conscious prior in a couple of
slides
so one one thing that helps understand
where we are versus where we want to go
is the distinction in cognitive tasks
between what Kahneman and others call
system 1 and system 2 tasks so system 1
tasks are the the tasks that we do using
basically are in
tuition we can do them really fast and
and the way that we solve these problems
are essentially heuristic they're not
perfect right and it's all happening at
the unconscious level we don't know how
to explain we do how we do these tasks
and and so it's it's it's not linguistic
and deep learning currently does really
well at this kinds of tasks but of
course there are other tasks like if I
ask you to add two large members you can
do it in your head or using paper and
it's gonna be slow you're gonna use
logic it's gonna be sequential and you
can do it consciously and explain it
linguistically and this is what
algorithms in computer science really
are about and this is also what
classically I was trying to deal with
obviously our brain is doing both of
these things like most of the work is
done by system one but we need we need
the two of them especially if we want to
have machines that can interact with us
in a natural language and so one of the
things I'm gonna say here as a
recommendation for the path towards AGI
is we can't continue doing the kind of
natural language research where we only
train on text data we have to build
systems that simultaneously can
understand words in sentences but also
what those words refer to in some
environment so this is called grounded
language learning and there is some work
in that direction but I think it's gonna
dominate in the future in the ability of
delivering systems that can actually
understand what the sentences mean so
recently with a group of my colleague at
CFR we laid out a plan for the next five
years about we're where we want are our
deep learning research to go next so let
me go through four of these points one
is you know move the focus from passive
observation to active agent so basically
agents of course we've been doing
reinforcement learning but up to now
it's been you know kind of a separate
thing in in machine learning conferences
and and what I'm saying is in order to
move towards edgy itis is gonna have to
be part of a lot more of the research
and
and all of these points are related to
each other related to that is well we
have to go beyond perception where we're
already doing pretty well to reasoning
and and planning which are these things
we're doing at the highest levels of
representations that I've been talking
about and and we have to do it in a way
that allows the learner to develop a
full I mean full but but incomplete
model of the world that captures causal
structure so we have a causal model of
the world in our head which allows us to
do things like imagine the situations
that have never happened and will never
happen so-called counterfactuals and
allow us to plan and reason even about
things that are unlikely this is not
something that current machine learning
does well and in doing all of these
things we want to continue on our path
of getting inspiration and producing
synergy with the research trying to
understand how humans are intelligent so
neuroscience cognitive science and so on
it's been very important for my work and
and and many of my colleagues to drive
our exploration of AI and and I believe
that there's a lot more that we can gain
and of course we can also give back to
neuroscience and cognitive science with
the models we're building to help them
understand how the brain actually works
so to summarize one of the things we
want to do is joining system 1 and
system 2 using grounded language
learning where we we build high-level
representations of what is going on but
that are anchored in in low-level
perception and and where the the the
meaning of the words is not just like
some word embedding but it is also
connected to actions and the level
perceptions so one of the practical like
nitty-gritty questions on that path is
should we like first learn a world model
think about like a baby which doesn't
act in the world and just or me the acts
in the world but does
speak sorry and that's in here language
and just tries to figure out the physics
of things and then later once we know
how the world works and we just tag
words on these things and we learn the
language model or should we do both
things at the same time so this is an
interesting question and and and my view
on this is that we should do both at the
same time although of course learning
the world model is like the the primary
thing that needs to drive things if you
think about babies when they're born
usually of course their knowledge of
language is zero and except they already
know a bit about the sounds and so the
main thing they initially learn is about
how things work in the world from a
physical this is a physical point of
view and how other human beings behave
some clues that this is the right thing
to do include things like well the fact
that if you train with supervised
learning these imagenet classifiers the
representations you're getting so you
know forget about the classifier itself
but just the representations you're
getting at the top hidden layer are
really really good and they allowed to
generalize to new categories that have
been have been never seen and if you
think about it in in the case of these
classifiers we are using human knowledge
of high-level categories by by doing
supervised learning I mean it's not like
a complicated language we're just saying
the names of things right and so by
doing that where I get actually getting
representations that are much better
than the representations we get if we do
pure unsupervised learning where we
don't provide any labels so providing
labels is like providing a mini language
to those learners and it actually helps
them discover better representations of
of the visual world right so so this is
a clue that you know we we really want
to have not purely ins provides learning
but eventually having language input to
drive these what are called BBE eyes
okay so so part of this is how do we
learn these models of how the world
works and and I think this is going to
be absolutely necessary to deal with the
current problem of systems that can be
fooled very easily so there's there's
also like a robustness and an even
safety issue with current systems I
think about you know self-driving cars
being in a very rare and dangerous
situation that has never been seen and
as statistically unlikely current
methods I think would tend to fail on
these situations but but humans can
imagine these situations even before
they occur and and and learn to behave
properly and and and we're able to do
that because and what happened with the
title but because we have good model of
the underlying causal mechanisms and in
this allows us to generalize in in a
non-trivial way so traditional machine
learning has been about IID Javas ation
so the training data and the test data
are supposed to come from the same
distribution
we've been living with this hypothesis
for decades but it's it's wrong it's not
like what humans do is not what the
system's we build for applications
actually mean because we train on data
from say a particular country and then
we want this to work on another country
so so that assumption leads to lack of
robustness and and people have thought
for a long time that well that's the
best we can do if we have that
assumption then we can get guarantees of
generalization if if the test data
doesn't come from the same distribution
as the training data and what guarantees
do we have right well so what I claim is
that we can we can get a more powerful
form of translation by relaxing the
assumption but still having some
assumption it's just weaker and the
assumption is that the weaker assumption
is is that the the the scenario where
we're going to be using this system
corresponds to a distribution that may
be different from the training one but
involves the same causal mechanisms and
the same underlying concepts right you
can have the same underlying concepts
and the same underlying causal mechanism
but the dispersion is very different
think about you know taking pictures
here on earth and then taking pictures
on the moon it's the same underlying
physics but we get very different
looking
right so what changed is it's the same
dynamical system like you know the
physical gotta make the system but but
initial conditions
maybe the gravity of the Moon is very
different and and maybe also actions
that agents take like you intervene and
you change things in the world are gonna
change particular variables in that
dynamical system leading to potentially
very very different observation
distribution okay so how do we get there
one direction that has already been
taken in our field and I think is gonna
be again central to reaching AGI is to
use simulated environments right so why
does it make sense even if our simulated
environments are really simple compared
to the real world actually I think we
can learn a lot from that and the reason
is that this is like a subtle thing
we're not trying to build AI or AGI
right now as like build the system that
understands how our world works
it looks like that's we want to do but
actually what we're trying to do in
machine learning research is fine the
learning procedure which if it was put
in our world could figure out how our
world works right and we can test those
learning procedures in simpler context
that we can analyze and break down into
gradually more complex requirements and
and that's what I and others are
proposing to do so I think eventually
will converge to some sort of benchmarks
which I think was tagging proposed to
call them AI Olympics where different
research groups would propose different
environments where each research group
would have to deliver learners that can
deal with all of these environments
right and you can think of this as a
more sophisticated Turing test that we
would grow the difficulty of these tasks
as we make progress in research so we've
actually started this with a project
called the baby AI platform and we found
that even with a very very simple
environment which is like a grid world
associated with language constructions
that have a compositional nature so
there's an almost infinite number of
potential instructions of the form
things like you know navigate to this
place and and put something next to
something else the the tasks currently
are extremely difficult even for the the
current best reinforcement learning and
imitation learning systems that we have
so these really really easy easy tasks
that humans can probably learn how to
times faster need hundreds of thousands
or millions of demonstrations to be
learned so so so in a sense I mean it's
it's bad news in the sense that oh we're
really very far from AGI and the good
news is that means we can study those
problems in setups that are simpler and
easier to analyze that don't require as
much computational power so what else is
missing so if you think about how
current machine learning and RL are are
modeling the world I think in addition
to the causality aspect there are there
many other issues so for example our
current typical model think of
predicting the future as a sequence of
one step prediction like I'm gonna
predict the next frame and then given
that and the previous ones I'm gonna
predict the next one and so on so we
typically have these decompositions by
conditional probabilities of the next
thing and then the next thing and the
next thing and that's not at all how
humans build visions of the future right
we can project ourselves at arbitrary
points in the future we don't need to to
have like a little movie in our head of
all the intermediate time steps we could
say something like tomorrow I will do
this and I don't need to predict all of
the intermediate time steps in addition
when I say tomorrow I will go to the
beach I'm just talking about the beach I
don't need to talk about all of the
other aspects of the future state of
what will how the world will be tomorrow
and you know the world is in streaming
complex and has maybe you know tens of
thousands of interesting variables I
could talk about in
language but I'm choosing to only talk
about one or two or three in that
sentence and and we do that all the time
when we project ourselves into the
future planning or imagining or
constructing hypothetical situations so
so this notion of focusing on just a few
variables at a time in our mental
projections is something that's missing
in current machine learning and this is
something I proposed last year and the
paper called the consciousness prior
which which actually considers that this
ability that we have to talk about the
world using just a few variables at a
time is actually a an important
statistical advantage and and so that's
the reason I call it a prior because
it's assuming something about the world
it's assuming that only looking at a few
variables like tomorrow it will rain I
can actually make statements that have a
high probability of being true this is
very very different from if I have to
make a prediction about all of the
pixels you know tomorrow and if if I
were to to try to make these kinds of
predictions in the wrong space in the
space of pixels for example I wouldn't
be able to make very good predictions if
I'm trying to predict one pixel given
three other pixels even if I pick those
pixels well it's not gonna work very
well right if I'm trying to predict how
things will be tomorrow but but if I can
talk in the high-level language of say
it's gonna rain tomorrow seeing I think
it will go a little bit away and
actually then I'm actually able to make
these these very powerful statements and
and that's what we are exploiting when
we say things about the world in
language this is also what classically I
was exploiting with you know the notions
of rules and the variables that are
associated with that so so so now what
this is saying is in addition to this
notion of finding the right space where
we we want to make those statements we
also need to learn those statements
which are those rules right and you can
also think of them as the causal
mechanisms
which relate the variables together in a
graphical model sense all right I mean
let me skip that um so because I use the
word consciousness and this is something
that interests people here let me sort
of now sneak in my view on consciousness
so I think we're confounding I mentioned
it yesterday
we're confounding lots of different
things that computationally are just you
know different they're related when we
talk about consciousness and and none of
those here include the notion of moral
agent moral agent and moral patient
which is actually the notion that we
care about when we say are we what
should we do if we have a conscious
machine should it have rights and things
like that I think this is one aspect
which has to do with social interactions
and social contract but but really there
are computational aspects which are
which can be studied separately and
build in machines in ways that I don't
think are so complicated so
self-consciousness is just having a
notion of my state within the bigger set
of my model of the world access
consciousness is what I'm exploding in
the implementations of the consciousness
prior that I talked about where we only
focus on a few things at a time right
this is attention and this is the things
we're focusing on are these thoughts
that we have at a particular moment
emotions well I mean emotions are just
shortcut calculations to predict the
value of some future state or expected
value of future States in socially
context-dependent
setups and and those those sort of
predictions are very powerful for us the
influence what goes in our memory the
things we choose unconsciously to
remember and in and how our next action
are gonna be decided and then finally
maybe the one that's the most
problematic is the notion of subjective
experience or subjective perception and
I believe this is an overblown issue I
think that even with current Neil Nets
that learn too
the world what they have is actually
subjective perception each of these
neural Nets is trained on different data
and and develops different
representations of the world so if you
think about the color red right well for
those in your Nets the color red is not
the wavelength it's it's these
high-level representations of red things
which for some neural net may be
associated with positive experiences
there for some other neuron that may be
some negative experiences and may be
associated with particular aspects of
the world that could be different from
one individual to the other quickly AI
progress comes and the path to AGI comes
with all kinds of hopes and dangers I
don't need to maybe go through that list
I mean just mention a few things that
we're doing at Mila so we we've been
part of the work on the montreal
declaration for the responsible
development of AI which you can think of
as a more elaborated version of the
Asilomar principles there are ten
principles each of them has things like
your well-being equity diversity
prudence responsibility each of them is
then you know broken down into sub
principles which I think go beyond just
AI but also the values that we care
about as was the case for the Asimov
principles we were also involved in
organizing workshops and events like the
AI for a good workshop at the last nibs
and we have an upcoming workshop on or
school like I'd say on bias and
discrimination in the eye and one thing
I really care about as AI we believe
that it's important to Democrat eyes AI
around the planet and in particular to
make sure that developing countries can
can build their expertise in the eye so
we have a program to bring in students
undergrad or grad from developing
countries like Africa and and spend a
few months at Mila we have projects in
of course applied AI in things like
healthcare and now climate change let me
mention one project which
I find cool that we just started yeah
because I think it's related to some of
the issues discussed here about global
governance and and how to how to make
people on earth and move towards the
better collective decisions so so one
issue with climate change is as people
don't really understand emotionally what
the importance of this because because
what we're seeing in the media and in
papers is very abstract it's happening
you know 50 or 100 years into the future
and even if there's gonna be hundreds of
millions of people dying because of
these things it's hard to make it
personal and emotionally grounded and so
what we're proposing is to use machine
learning in a in a positive way in an
ethically correct way to convince people
of the importance of climate change in a
way that's gonna be emotionally grounded
so for example we can use variations on
Gans to present images of your house or
of your descendants in 50 years or
hundred years as a consequence of the
current decisions about about climate
change the last thing I want to mention
this is really my last slide so this is
a little bit more I know many people
might disagree with this here but the
about existential risk so I do think
that AG AI and AI in general comes with
lots of risks but one that I'm less
confident about less convinced of is the
the risk of really really fast sort of
exponential self-improvement once we
reach AGI I don't think that there's
gonna be like one cutoff point oh we we
reach human-level AI and then suddenly
it's sort of shoots off I don't I don't
I don't believe that
why so so if if the future area is
building machine learning and the kind
of machine learning I understand now and
I can imagine in the future then the
change like the self improvement is not
something that can happen very quickly
it's something that changes very slowly
through learning right
and you know by sort of regent of a
based optimization the other thing is
learning involves actually diminishing
returns in terms of you know the amount
of knowledge obtained as you collect
more data so even if I multiply the
amount of data by a thousand I don't get
a thousand times sort of more efficient
classifiers in fact you-you-you have a
sort of exponential wall here that's
well understood in learning theory and
and similar and similarly yeah so so
rather the the competence probably grows
something like log of the number of
examples and another clue is that the
size of the brain doesn't seem to be
such a big factor like even if we had a
brain that was ten times bigger I'm not
sure we would be sort of ten times
smarter consider you know the fact that
whale brains are much bigger than ours
and I think the main difference between
Wales and us is that we have a culture
and and that allows us to gather much
more knowledge in the same size brain
and the other last point is if you look
at at computer science theory and
machine learning theory you find many
many exponential walls so the opposite
of exponential improvement is basically
things become exponentially hard as you
try to to solve some problems because of
things like the curse of dimensionality
and and all kinds of exponential growth
that make for example search
exponentially hard as you consider
harder and harder search problems all
right I'm closing here thanks
[Applause]
I'm very surprised would you say I'm
very surprised oh um you talked about
the diminishing returns of more data
yeah but I think that's actually when
we're talking in this regime of ID data
on a single pass sure I agree I agree
that active learning we can do a lot
better but we don't have really proof
that active learning except in the
simplest setting oh but I'm not talking
about learning I'm talking about
multitask learning and horizontal
transfer yeah but that's only here it's
linear well it's actually I would argue
it's at least linear which is much
better than logarithmic and potentially
super linear because we see positive
transfer right now when you train on
multiple tasks so I guess if you had an
AI that was very competent on one task I
wouldn't be surprised if it could
rapidly become competent on a whole host
of tasks for instance everything that
humans can do and more I agree but I
don't think you get the kind of
exponential growth that people are
afraid of that so of course I'm all for
multitask learning it's very important
other question I'm sure you do
thank you for your wonderful speech and
I have a short question and a
two-question a first question is if
consciousness player is a kind of access
consciousness yes
and second question is in this is right
after yourself infirmity is a deep spot
you think but if the AI get some kind of
common sense
then they can a program other AI who IND
of course well there are some
possibility to explain self what do you
think about it well humans are really
really bad at programming actually so if
we get human level performance I mean
ability it still sucks in terms of the
ability of programming things so i i'm
i'm open to demonstrations that i should
be much more concerned and i've already
voiced that to a number of people and
and you know i'm glad to engage in that
discussion because I do care that we
don't make tragic mistakes but I haven't
seen a convincing argument yet that
being said I think that where there's a
common ground between my view and many
people here is that you know whatever we
think about the existential risk coming
from super intelligence there's a large
risk coming from abuse of AI from humans
who could control it and potentially
even accidental you know destruction of
people and wealth by people and
especially nefarious use hi yeah you
seem to insist on this idea of a
spectrum of consciousness that's right
which I agree with but don't you think
there is a phase transition that should
that could could arise like being like
solid liquid gas and then consciousness
is maybe just like many different kind
of gases but is there a phase transition
that is sharp and really cut off there
could be so because I have decomposed
consciousness a number of aspects and
for some of those aspects I can see like
gradual improvement like you get better
and better representations or you can
model more and more aspects of the world
or you can
search more efficiently right so there
are many ways in which you can get
progress and maybe some of these are
like qualitative jobs that's quite
possible yeah thank you
we've talked about a number of these
points and it says so I really agree but
on the causality thing yeah so human
intuition about causality is heavily
linguistically based and it's heavily
from embodied distances of of agency yes
whereas you know there's these you know
sort of mathematical causal graphical
models right notion of causality which
is not necessarily the same thing very
somewhat related but humans are actually
kind of bad at this kind of problem you
know right
Kahneman is that right right no wonder
you you know on which of these you mean
well so so we're bad because we're not
good at search but what we're good at is
that figuring out the right variables so
and those variables are the ones we're
talking about that the ones we're giving
names in language great talk i wanted to
dive into some of the things you said
earlier about learning the benefits of
learning about the environment and
learning language at the same time yes
you can get these higher level
abstractions more quickly yeah certainly
seems true in short but what are the
cost might be that you learn the what
say my extractions that we know about
and my understanding is person is when
when machines are learned to diagnose
cancer from different images they
sometimes come up with new kinds of
abstractions that none of us had thought
of before right so maybe talk a little
bit about the trade-off of locking into
our pretty good but not perfect set and
maybe there are other ones especially
since they have other tense days that
they might be able to draw so I believe
in gradual progress in science and I'd
be very happy if we have machines that
already have our intelligence and now if
you wanted to be more ambitious and and
have machines that have a lot more
intelligence then you might want to free
them a bit more from the constraints if
you want of our language and our culture
although keep in mind that these are not
hard constraint
it's more like you're booted up with
that knowledge and now you know humans
can discover new things and those
machines could discover new things as
well so we could explore both
[Music]