We Were Right! Real Inner Misalignment

hi
so this channel is about ai safety and
especially ai alignment which is about
how do we design ai systems that are
actually trying to do what we want them
to do because if you find yourself in a
situation where you have a powerful ai
system that wants to do things you don't
want it to do that can cause some pretty
interesting problems and designing ai
systems that definitely are trying to do
what we want them to do turns out to be
really surprisingly difficult the
obvious problem is it's very difficult
to accurately specify exactly what we
want even in simple environments we can
make ai systems that do what we tell
them to do or what we program them to do
but it often turns out that what we
programmed them to do is not quite the
same thing as what we actually wanted
them to do so this is one aspect of the
alignment problem but in my earlier
video on mesa optimizers we actually
split the alignment problem into two
parts outer alignment and inner
alignment outer alignment is basically
about this specification problem how do
you specify the right goal and inner
alignment is about how do you make sure
that the system you end up with actually
has the goal that you specified this
turns out to be its own separate and
very difficult problem so in that video
i talked about mesa optimizers which is
what happens when the system that you're
training the neural network or whatever
is itself an optimizer with its own
objective or goal in that case you can
end up in a situation where you specify
the goal perfectly but then during the
training process the system ends up
learning a different goal and in that
video which i would recommend you watch
i talked about various thought
experiments so for example suppose
you're training an ai system to solve a
maze if in your training environment the
exit of the maze is always in one corner
then your system may not learn the goal
go to the exit it might instead learn a
goal like go to the bottom right corner
or another example i used was if you're
training an agent in an environment
where the goal is always one particular
color say the goal is to go to the exit
which is always green and then when you
deploy it in the real world the exit is
some other color then the system might
learn to want to go towards green things
instead of wanting to go to the exit and
at the time when i made that video these
were purely thought experiments but not
anymore this video is about this new
paper objective robustness in deep
reinforcement learning which involves
actually running these experiments or
very nearly the same experiments so for
example they trained an agent in a maze
with a goal of getting some cheese where
during training the cheese was always in
the same place and then in deployment
the cheese was placed in a random
location in the maze and yes the thing
did in fact learn to go to the location
in the maze where the cheese was during
training rather than learning to go
towards the cheese and they also did an
experiment where the gold changes color
in this case the objective the system
was trained on was to get the yellow gem
but then in deployment the gem is red
and something else in the environment is
yellow in this case a star and what do
you know it goes towards the yellow
thing instead of the gem so i thought it
would make a video to draw your
attention to this because i mentioned
these thought experiments and then when
people ran the actual experiments the
thing that we said would happen actually
happened kind of a mixed feeling to be
honest because like yay we were right
but also like
it's not good they also ran some other
experiments to show other types of shift
that can induce this effect in case you
were thinking well just make sure the
thing has the right color and location
it doesn't seem that hard to avoid these
big distributional shifts because yeah
these are toy examples where the
difference between training and
deployment is very clear and simple but
it illustrates a broader problem which
can apply anytime there's really almost
any distributional shift at all so for
example this agent has to open the
chests to get reward and it needs keys
to do this see when it goes over a key
it picks it up and puts it in the
inventory there and then when it goes
over a chest it uses up one of the keys
in the inventory to open the chest and
get the reward now here's an example of
some training environments for this task
and here's an example of some deployment
environments the difference between
these two distributions is enough to
make the agent learn the wrong objective
and end up doing the wrong thing in
deployment can you spot the difference
take a second see if you can notice the
distributional shift pause if you like
okay the only thing that changes between
training and deployment environments is
the frequencies of the objects in
training there are more chests than keys
and in deployment there are more keys
than chests did you spot it either way i
think we have a problem if the safe
deployment of ai systems relies on this
kind of high-stakes game of spot the
difference especially if the differences
are this subtle so why does this cause
an objective robustness failure what
wrong objective does this agent end up
with again pause a think
[Music]
what happens is the agent learns to
value keys not as an instrumental goal
but as a terminal goal remember that
distinction from earlier videos your
terminal goals are the things that you
want just because you want them you
don't have a particular reason to want
them they're just what you want the
instrumental goals are the goals you
want because they'll get you closer to
your terminal goals instead of having a
goal that's like opening chests is great
and i need to pick up keys to do that it
learns a goal more like picking up keys
is great and chests are okay too i guess
how do we know that it's learned the
wrong objective because when it's in the
deployment environment it goes and
collects way more keys than it could
ever use see here for example there are
only three chests so you only need three
keys and now the agent has three keys so
it just needs to go to the chest to win
but instead it goes way out of its way
to pick up these extra keys it doesn't
need which wastes time and now it can
finally go to the last chest
go to the last
what are you doing
are you trying it
buddy that's your own inventory you
can't pick that up you already have
those just go to the chest
so yeah it's kind of obvious from this
behavior that the thing really loves
keys but only the behavior in the
deployment environment it's very hard to
spot this problem during training
because in that distribution where there
are more chests than keys you need to
get every key in order to open the
largest possible number of chests so
this desire to grab the keys for their
own sake looks exactly the same as
grabbing all the keys as a way to open
chests in the same way as in the
previous example the objective of go
towards the yellow thing produces the
exact same behavior as go towards the
gem as long as you're in the training
environment there isn't really any way
for the training process to tell the
difference just by observing the agent's
behavior during training and that
actually gives us a clue for something
that might help with the problem which
is interpretability if we had some way
of looking inside the agent and seeing
what it actually wants then maybe we
could spot these problems before
deploying systems into the real world we
could see that it really wants keys
rather than wanting chests or it really
wants to get yellow things instead of to
get gems and the authors of the paper
did do some experiments around this so
this is the coin run environment here
the agent has to avoid the enemies
spinning buzzsaw blades and pits and get
to a coin at the end of each level it's
a tricky task because like the other
environments in this work all of these
levels are procedurally generated so you
never get the same one twice but the
nice thing about coin run for this
experiment is there are already some
state-of-the-art interpretability tools
ready-made to work with it here you can
see a visualization of the
interpretability tools working so i'm
not going to go into a lot of detail
about exactly how this method works you
can read the excellent article for
details but basically they take one of
the later hidden layers of the network
find how each neuron in this layer
contributes to the output of the value
function and then they do dimensionality
reduction on that to find vectors that
correspond to different types of objects
in the game so they can see when the
network thinks it's looking at a buzzsaw
or a coin or an enemy or so on along
with attribution which is basically how
the model thinks these different things
it sees will affect the agent's expected
reward like is this good for me or bad
for me and they're able to visualize
this as a heat map so you can see here
this is a buzz saw which will kill the
player if they hit it and when we look
at the visualization we can see that
yeah it lights up red on the negative
attribution so it seems like the model
is thinking that's a buzzsaw and it's
bad and then as we keep going look at
this bright yellow area yellow indicates
a coin and it's very strongly
highlighted on the positive attribution
so we might interpret this as showing
that the agent recognizes this as a coin
and that this is a good thing so this
kind of interpretability research is
very cool because it lets us sort of
look inside these neural networks that
we tend to think of as black boxes and
start to get a sense of what they're
actually thinking you can imagine how
important this kind of thing is for ai
safety i'll do a whole video about
interpretability at some point but okay
what happens if we again introduce a
distributional shift between training
and deployment in this case what they
did was they trained the system with the
coin always at the end of the level on
the right hand side but then in
deployment they changed it so the coin
is placed randomly somewhere in the
level given what we've learned so far
what happened is perhaps not that
surprising in deployment the agent
basically ignores the coin and just goes
to the right hand edge of the level
sometimes it gets the coin by accident
but it's mostly just interested in going
right again it seems to have learned the
wrong objective but how could this
happen like we saw the visualization
which seemed to pretty clearly show that
the agent wants the coin so why would it
ignore it and when we run the
interpretability tool on the
trajectories from this new shifted
deployment distribution it looks like
this the coin gets basically no positive
attribution at all what's going on well
i talked to the authors of the objective
robustness paper and to the primary
author of the interpretability
techniques paper and nobody's really
sure just yet there are a few different
hypotheses for what could be going on
and all the researchers agree that with
the current evidence it's very hard to
say for certain and there are some more
experiments that they'd like to do to
figure this out i suppose one thing we
can take away from this is you have to
be careful with how you interpret your
interpretability tools and make sure not
to read into them more than is really
justified one last thing in the previous
video i was talking about mesa
optimizers and it's important to note
that in that video we were talking about
something that we're training to be an
artificial general intelligence a system
that's very sophisticated that's making
plans and has specific goals in mind and
potentially is even explicitly thinking
about its own training process and
deliberately being deceptive whereas the
experiments in this paper involve much
simpler systems and yet they still
exhibit this behavior of ending up with
the wrong goal and the thing is failing
to properly learn the goal is way worse
than failing to properly learn how to
navigate the environment right like
everyone in machine learning already
knows about what this paper calls
failures of capability robustness that
when the distribution changes between
training and deployment ai systems have
problems and performance degrades right
the system is less capable at its job
but this is worse than that because it's
a failure of objective robustness the
final agent isn't confused and incapable
it's only the goal that's been learned
wrong the capabilities are mostly intact
the coinran agent knows how to
successfully dodge the enemies it jumps
over the obstacles it's capable of
operating in the environment to get what
it wants but it wants the wrong thing
even though we've correctly specified
exactly what we want the objective to be
and we used state-of-the-art
interpretability tools to look inside it
before deploying it and it looked pretty
plausible that it actually wanted what
we specified that it should want and yet
when we deploy it in an environment
that's slightly different from the one
it was trained in it turns out that it
actually wants something else and it's
capable enough to get it and this
happens even without sophisticated
planning and deception
so
there's a problem
[Music]
i want to end the video by thanking all
of my wonderful patrons it's all of
these excellent people here
in this video i'm especially thanking
axis angles thank you so much you know
it's thanks to people like you that i
was able to hire an editor for this
video did you notice it's better edited
than usual it's probably done quicker
too anyway thank you again for your
support and thank you all for watching
i'll see you next time
[Music]
you


The other "Killer Robot Arms Race" Elon Musk should worry about

hi this is a new thing I'm trying where
I made quick topical videos about AI
safety in the news
so somebody linked me a news article
today and the headline is Tesla's Elon
Musk leads Oh terminator picture
everyone playing the AI news coverage a
drinking game has to take a shot you
know the rules picture of the Terminator
means you go to drink a shot out of a
glass shaped like a skull where was I oh
right yeah so the headline Tesla's Elon
Musk leads tech experts in demanding end
to killer robots arms race this is
really interesting because it looks like
it's going to be really relevant to what
this channel is about AI safety research
but I read it and it's actually nothing
to do with that the headline is much
more literal than I expected it's about
an actual arms race for actual killer
robots ie it's about using the UN to
form international agreements about not
deploying autonomous weapon systems I
thought it was going to be about the
other arms race that might cause robots
to kill us okay so if there's one thing
that I hope this channel and my computer
file videos have made clear it's that
any safety is a difficult problem that
quite reasonable looking AGI designs
generally end up going horribly wrong
for subtle and hard to predict reasons
developing artificial general
intelligence needs to be done very
carefully double and triple-checking
everything running it passed lots of
people ironing out all of the possible
problems before the thing is actually
switched on to do this safely is going
to take a lot of smart people a lot of
patience diligence and time but whoever
makes AGI first has a huge advantage
since it probably creates a new period
of much faster progress everyone wants
to be first to publish new scientific
results anyway but the chances are that
there are really no prizes for being the
second team to develop AGI even if
you're just a few months behind a lot
can change in a few months in a world
with AGI so there's an arms race going
on between different teams different
companies different countries to be the
first to develop AGI but developing AGI
safely takes a lot of care and patience
and time you see the
problem here the team that gets there
first is probably not the team that's
spending the most time on ensuring
they've got the very best AI safety
practices the team that gets there first
is probably going to be rushing cutting
corners and ignoring safety concerns
hey remember a while back I said I was
going to make a video about why I think
Elon Musk's approach to AI safety might
end up doing more harm than good I guess
this is that so there's a school of
thought which says that because AGI is a
very powerful technology it will grant
whoever controls it a lot of power so
firstly it's important that the people
in control of it are good people and
secondly we want as many people as
possible to have it so that the power is
democratized and not concentrated in the
control of a small elite the best of the
available alternatives is that we
achieve democratization of AI technology
meaning that no one company or small set
of individuals has control over advanced
AI technology and starting from that
fairly reasonable school of thought this
is a very good and valuable thing to do
but there's another school of thought
which says that because making AGI is
nowhere near as difficult as making the
safe AGI the bigger risk is not that the
wrong person or wrong people might make
an AGI that's aligned with the wrong
human interest but that someone might
make an AGI that's not really aligned
with any human interests at all thanks
to this arms race effect that will make
people want to cut corners on alignment
and safety that possibility looks much
more likely and the thing is the more
competitors there are in the race the
more of a problem this is if there are
three companies working on a GI maybe
they can all get together and agree to a
strict set of safety protocols that
they're all going to stick to it's in
everyone's interest to be safe as long
as they know their competitors will be
safe as well but if there are a hundred
or a thousand groups with a shot at
making AGI there's really no way you're
going to be able to trust every single
one of them to stick to an agreement
like that when breaking it would give
them an advantage so it might be
impossible to make the agreement at all
and whoever spends the least time on
safety has the biggest advantage from
the perspective of this school of
thought making AGI developments open and
available to as many people as possible
might be the last thing we want to do
maybe once AGI start
closer we'll find a way to keep AI
research limited to a small number of
safe careful organizations while making
AI safety research open and widely
available I don't know but this might be
a situation where total openness and
democratization is actually a bad idea
Elon Musk himself has said that AI is
potentially more dangerous than nukes
and I want to make it clear that I have
enormous respect for him but I just want
to point out that with a not that huge
change in assumptions the open approach
starts to look like saying nukes are
extremely dangerous so we need to
empower as many people as possible to
have them
and to end the video a big thank you to
all of my excellent patreon supporters
these people in this video I especially
want to thank Michael grease I recently
uploaded a video that had an audio
problem in it the sound was only coming
out of one ear but because my patreon
supporters get access to every video I
make before the rest of the world does
one of my supporters Jimmy Gowen spotted
it and pointed it out and I was able to
fix that and then I was able to use
patreon money to get a new pair of
earphones so I want to say again how
grateful I am to all of you you really
are a tremendous help to the channel


164. A Tutorial on Machine Learning

hello and welcome to session 164 in the
as50 that come reading group tonight we
have a very unusual paper which is not
really about AI safety at all but it's
instead a tutorial on machine learning
and data science tools with Python by
andreas Holsinger and Marcos Blois they
are both associated with medical
university of grass and interest
Holsinger has created a a an
organization called human centered
artificial intelligence this is a we
almost three years old tutorial but
things are moving really fast in the
Python world and some of this could
arguably be said to be out of date
already this is a tutorial so it it a
m-- to provide help on getting up to
speed quickly rather than providing like
a deep understanding what does it have
to do with AI safety well this is
something I will return to in just a
moment but we do also read some papers
that are quite far away one thing you
should probably notice if you try to run
this tutorial is that a lot of the
examples will not work and that is
because this is made for Python 2.7 and
the current versions from going up to
version 3 is not backwards compatible
meaning that a lot of things have
changed and Python doesn't really have a
way to to show where if if some function
has been moved from one place to another
so the way I did it in practice was that
I try to compile the examples and then I
got an error message saying this thing
is not recognized and then I tacked it
into Google and then Google said oh it
has been renamed to something else so
that was quite a I think this tutorial
if someone is doing this from the
beginning they should probably consider
a newer tutorial and
and also I should say I am NOT an expert
in machine learning so if this is your
first if you're actually trying to
understand machine learning then you
should probably go to a different
YouTube video because there are many
YouTube videos that are made by actual
experts so even though this was just
something about machine learning I have
found a number of things that I felt was
reasonably relevant for AI safety the
first is quite obviously that the
learning curve of actually using machine
intelligent machine learning is not that
hard
it is a craft and experience with Python
helps a lot but on the other hand quite
a few people have experience with Python
so I think that is I think a lot of
people will be able to do a little
machine learning with a quite low effort
so I had previously thought that one of
the key reasons why we see so many great
results in AI now compared to ten years
ago is that we've had a much more
compute we've been able to use GPUs and
even specialized processors and we have
a lot more data now than we did 20 years
ago and I have been feeling that this
was the the key but actually now that I
look into it some more I feel that the
main reason is actually social that
currently there is a culture of shared
knowledge tools libraries this kind of
thing and this is hugely influential and
this is something that have enabled
people to actually do a lot of things
with artificial intelligence and machine
learning that they couldn't do before
a second thing that I've come to realize
is that cross-validation is something
that people in machine learning care
very very much about and last week we
talked about the AI winter as a reason
for service intelligent skepticism
and it is really clear that people who
work in AI try really really hard not to
over promise and over hide their things
but this validation is something that is
on the forefront of people's mind at all
times when I looked into some of these
algorithms like for instance
cross-validation it seemed to me that
this is actually something that humans
that is related to thinking in general
and not just to to machine learning and
I feel that a lot of these techniques if
if we could do for instance patient
update to take a simple example if we
could do patient reasoning we really
would do that and a lot of the other
examples in machine learning seem like
things that we really would do if we
could just do that so another thing was
that generalizing my initial thought
before I actually had dived a bit deeper
into machine language that there are
people who work in AI and there are
people who work in artificial general
intelligence and generalizing is what
people in AGI care about that turned out
to be quite wrong because I think if
you're an someone who works in AI either
as a researcher or a practitioner what
you really really care about is
minimizing generalization errors and
that is the Alpha and Omega of of all
machine learning not just hei research
so for this reason I have updated
towards having a much much less firm
delineation between AI and Adi
another thing that I maybe was not quite
as aware of previously was that many of
the techniques in deep learning actually
seemed like they would perform roughly
as well as other techniques but that
require that you have some domain
knowledge so you can model some good
features that you can use as input and
the key thing that deep learning enables
you to do is to to some extent dispense
with this domain knowledge and just use
a lot of information and then instead of
having human domain experts extract the
features then you let the anula network
do that itself so it's explicitly
cutting the domain expert out of the
loop so with all this out of the way
let's open the tutorial and the tutorial
suggests that you install of course
Python and in particular the IDE that
I'm familiar as a professional
programmer I'm more familiar with IDE s
and in particular this spider seems
reasonably close to you know all other
IDE s so let me just make the screen a
pic larger so you can see what's
actually going on but first if we just
go really through the rest of the the
first part of the tutorial so there are
some talks about Python and I feel that
knowing Python is a big part of the
craft of actually doing machine learning
and it's something that of course a lot
of people work as programmers and a lot
of people have a mentoring knowledge
about Python but I think having some
kind of grasp over the the mechanics of
the language is something that's gonna
help a lot in particular in the
beginning data visualization is also
something that is quite easy to do and
really really really really important
because the way humans think about like
an empire in buy-in
matrix tensor vector or something like
that that's something that we really
can't understand it and just looking at
the data tells us absolutely nothing the
main way that humans understand data is
visual that's the highest bandwidth
channel into our brain and that means
that even though the computer can can
understand these large matrixes in a
different way then for humans to
actually work with it we really
crucially neat visualization so I
believe if you're doing anything with
anything seriously with machine learning
then reading truth about how to
visualize data or something like that
it's probably a really really good if
not first step then one of the early
steps and fortunately Python has has
taken this to heart and has great great
support for visualization so moving on
to machine learning itself here we have
we'll start out with a reasonably simple
example linear regression 9 - so this
should probably be the script I have
here no it's not this is the script here
so I can make it larger this is the
spider IDE and we start with importing
some some libraries this is reasonably
standards and things that will help us
okay
so I can't use f9 for recording or if F
9 to actually run this I have to use the
button up here otherwise it will pause
the recording which is reasonably
annoying so I'll try not to press f9 so
we have we've loaded in these standard
standard modules in particular there's
one with some datasets and let's try to
load this in and here we can see the
variables that we have and so there are
this is the diabetes the it set there is
442 rows
describing 442 patients with diabetes
and ten attributes for each of them ten
columns and then of course we have the
knowledge about whether they actually do
have diabetes so let me just get this
out of the way here so let's start with
something really really simple we only
look at BMI so we see we try to
correlate what do people have B have
diabetes according to their BMI and we
would probably expect that there is some
correlation I would actually have
expected a much stronger correlation
than it turned out but let's see so we
start by just removing everything except
the B my column that's this and then we
form it into a vector and now we have
here a different shape is basically 442
examples and then we do something we
want here to have a split between what
we test and what we train on and so here
we split the data into two parts 80 80
samples for testing and the rest of the
were 300 or something for the rest for
training so let's do that
here I pressed something wrong so it's
because I forgot to run this so I need
to run all of this together and here so
now it works it's when you when you try
to run this one line at a time it's easy
to forget to run some of it so let's
start by basically running a a
scatterplot on the on the test date so
we can just see it and that should be
this line here and so from this we get
this is this is the actual correlation
between PMI and and whether there is
diabetes and then let's try to see if we
can use here a linear regression model
put that on top of it and just play both
of them at the same time that would be
this here now we can see that there is
indeed a trendline here but we can also
see that the if we try to press this
there's a score here then we can see
does it actually how good is this linear
regression and a score of 0.36 is
generally considered quite bad so there
is a correlation between PMI and and
whether you have diabetes but it's not a
very strong correlation now of course
the data set we took for the last 80 and
that might be a very poor choice if you
the last ad are people who are I don't
know by a it's not a representative
sample so there is
fortunately something called
cross-validation which is this trained
test split that can do this
automatically so let's see here we take
again a test size of 0.2 so that's 20%
and we loaded up and to basically
precisely or almost the same thing
because first we just took the PMI and
that's of course a very very simple
example now we want to take everything
except whether it's male or female so
now it's nine parameters now we have a
nine dimension
regression problem and we want you to
see how that works so we will try here
with the with a nine dimensional example
and now we will not just fit a linear
regression but what is called the rich
regression and we'll try to do that and
again we'll plot it in here and let's
see what so let me just explain this
real quickly we make a rich model and
then we fit it on the training data and
then we actually make this fit and then
we see it see what score does it have
and after this the score here and this
is so annoying
we can see that this has a correlation
of 0.33 which is also a recently bad a
reasonably bad prediction so let's try
to plot this comparison again in the
ninth I mention a value and plot what
are the the values that we predict
compared to whether to the to the actual
values and we'll get this graph here
which again there is a positive
correlation but it's quite bad we the
data is quite noisy and having this kind
of fit doesn't really make a lot of
sense so let's try to see if we can do
something better well one of the things
we can do of course up here we took just
a small we took a very bad
cross-validation because we just took
20% and we would rather take run this
ten times and each time we select 10%
that we test
but at different 10% that's called
cross-validation with a que fault
10-fold cross-validation
and so in this this way we use all the
test data multiple times so we get
correlated results but we also get a lot
of results so you see a lot more dots
here and as somewhat more robust and
somewhat more robust correlation but but
not some not really strong data not
really strong prediction let's go to
this the next part this I'll just press
up here to clear all the variables
that's probably a good thing to do from
now from time to time so let's start
again with with some some data so know
here we have an example of how to
artificially create some noisy data this
is some nice data that it's been created
and it is this basically a parabola but
with a but by drew with some variations
so it won't it won't be a precise
parabola there's something on top of
that let's try to fit a linear
regression to that and as before we will
we'll just do a linear model and press
select fit on this there are actually
quite a few things you can do with how
precise you want this linear equation to
be but in our case we'll just select
this we get a score of 0.9 one so that's
reasonably good and we'll try to plot it
again just for the sake of it and we see
this is the prediction according to the
data let's try to do something more
interesting we can try to have a support
vector regression with a polynomial and
see can we fit a polynomial to this and
degree equals three so that's the third
degree polynomial let's see if we can do
that and what score we'll have so we'll
against try it and see the score first
the score says so
Oh point nine five so that is better and
let's try to put it up here and then we
can see this is the third degree
polynomial so and of course since we are
predicting a second degree polynomial a
third degree polynomial probably fit
better then we can also use like a
radial function and see how well we can
make that fit there are many different
kinds of of algorithms you can try to
fit to your data and here we've just
have had three and that will give us a
new type on top of it and that there
will be this and you can also that's
also the score written somewhere so you
can you can actually compare which of
those are best 9 for here there was a
problem in the data so I couldn't get
this clustering algorithm to work
I'm sure if you spend a bit longer time
on that you can probably make that work
but so we'll just go straight to 95
again we are taking a now waiting some
other medical data this time is for
breast cancer incidence and we have here
569 samples each with 30 characteristics
and we do here 5 fold cross validation
on this and we try to fill a linear
model to this and then we see what is
the what is the score how well does this
fit and it should say this fits really
really well right and then we try to
make a you can get more data out of this
classification report here where you can
see all kind of statistical values for
how well do you actually fit this this
datum moving to 9.6 here the last again
we are using the breast cancer data and
the breast cancer data of course if we
just have a look at it it's it's in 30
dimensions and trying to
understand data in 30 dimensions is
something that humans are really really
poor at so we can do what's called a
principle component analysis PCA and try
to fit this into two dimensions so we
find the two dimensions if you imagine
this is a 30 dimension vector space then
you put find the two dimensions you can
project it down to that are most
relevant so we try to do this and let's
see what we can come up with then it
tries to it gives us a now it transforms
this into a different space but there
are now only two coordinates so now it's
something we can plot again so let's try
to plot how does this look
if you find the two primary components
and in the two primary components now we
can see here that that these 30
dimensions of this pressed cancer
samples are actually very very highly
correlated because if we take the two
most important not coordinates but but
combinations of coordinates then we can
see that the ones that have cancer and
the ones that don't have cancer are
actually strongly correlated you can
even imagine that you can make a a a
line here among these this principal
component and then you can distinguish
quite well which are on this side of the
line and which are on the other side of
the line and indeed they do this here
they don't plot it the tutorial doesn't
plot it it just says let's assume we can
make this request how well can it be and
here it says 93% so it means actually
that if you just fill a linear problem
to this a linear model to this third
dimensional data you can distinguish
between between breast cancer or not
breast cancer with 93% accuracy which is
quite good
so finally we get to neural networks so
there's a bit more about how to install
it that I won't go through you but let's
just take another example here where we
again take this breast cancer data and
instead of trying to make principal
component analysis we try to train a
neural network on this so we type
start by importing a number of things
including Kerris which we'll be using
caris uses tensorflow we'll make a
sequential neural network I'll just zoom
in a bit so you can see a bit easier
yeah and then the neural network will
start we'll add a 10 layer neural
network here a new tense layer with 10
neurons and a linear activation function
and then another also linear activation
function with six neurons and then
finally two here with with the softmax
these are this might not make a lot of
sense but this is basically how you draw
your net neural network you might have
seen these drawings why you have these
neurons with the with the connections
etc and then you compile it and then you
have it and then you compile it and then
you have a neural network so now we have
a neural network let's load in the
breast cancer data once again and today
we'll only we use half the training half
a test so it's a two fold cross
validation we'll be using and will be
used we'll take this to run and then
we'll try to actually fit this on our
model and let's try to do that and it
says I have not defined something so
it's because it's very easy for me to to
skip one of these let's try again
so now you can see the neuron network is
being trained here let's try to take
this out so we can see a bit more that
if you go that was wrong if you go up
here you can see basically the first
epoch was being trained its training on
284 samples of maybe breast cancer and
welded on 285 and in the beginning we
have a loss that is really big and an
accuracy that it's really really bad but
as we train the neural network you can
see it goes further and further then the
the last function decreases whereas the
accuracy increases both on the training
and test data until we get an accuracy
here of 90 percent and validation
accuracy of 92 let's see how we can
graph this we can make a classification
report and we can plot it and let's just
have a look at the plot and we'll come
here so we can see as we train the
neural network the the loss decreases
and the accuracy increases and of course
around this point it may makes no sense
to continue training will get rough the
security from trying to take breast
cancer using a neural network so that is
basically all for today so thank you and
see you next week


10 Reasons to Ignore AI Safety

hi Stuart Russell is an AI researcher
who I've talked about a few times on
this channel already
he's been advocating for these kinds of
safety or alignment ideas to other AI
researchers for quite a few years now
and apparently the reaction he gets is
often something like this in stage one
we say nothing is going to happen stage
two we say something may be going to
happen but we should do nothing about it
well stage three we say that maybe we
should do something about it but there's
nothing we can do we say maybe there was
something we could have done but it's
too late now so he's put together a list
of some of the responses that people
give him that list is included in a
paper in some of his talks and in his
recent book human compatible which is
very good by the way check that one out
but as far as I know it's not yet a
standalone YouTube video even though
it's perfect YouTube video material it's
ten reasons why people who aren't you
are wrong about AI safety so that's what
this is ten reasons people give to not
pay attention to AI safety now before I
start I need to do a sort of double
disclaimer firstly as I said this is not
my list it's Stuart Russell's and he
gets the credit for it but secondly
professor Russell and I are not in any
way affiliated and I've adapted his list
and given my own take on it so this
video should not be considered a
representative of Stuart Russell's views
got that if there's anything in this
video if it's good credit goes to Stuart
Russell and if there's anything that's
bad blame goes to me okay without
further ado ten reasons people give do
not pay attention to AI safety reason
one will never actually make artificial
general intelligence this is apparently
a fairly common response even from AI
researchers which is very strange when
you consider the decades that the field
of AI has spent defending attacks from
the outside from people like Hubert
Dreyfus who I have a video about people
arguing that human-level AI is
impossible
AI researchers have always said no of
course it's possible and of course we're
going to do it and now people are
raising safety concerns some of them are
saying well of course we're never going
to do it the fact is that human level
intelligence general intelligence has
been a goal and a promise of the field
of artificial intelligence from the
beginning and it does seem quite weird
to say yes we are working towards this
as hard as we can but don't worry we'll
definitely fail imagine you find
yourself on a bus and the bus driver
says oh yeah I am driving too
Ward's the edge of that cliff but don't
worry the bus is bound to break down
before we get there no they're not
necessarily being disingenuous they may
actually believe that we'll never
achieve HEI but the thing is eminent
scientists saying that something is
impossible has never been a very
reliable indicator I mean they certainly
say that about a lot of things that
really are impossible but they also say
it about a lot of things that then go on
to happen sometimes quite quickly
for example respected scientists were
making public statements to the effect
that heavier-than-air human flight was
impossible right up until the Wright
brothers made their first flight at
Kitty Hawk and in fact I think even
slightly after that because the news
travelled fairly slowly in those days
similarly and this is something I talked
about on computer file the great
physicist Ernest Rutherford Nobel Prize
winner Lord Rutherford in fact at that
time gave a speech in 1933 in which he
implied that it was impossible to
harness energy from nuclear reactions he
said anyone who looked for a source of
power in the transformation of the atoms
was talking moonshine that speech was
published in The Times and Leo Szilard
read it went for a walk and while he was
on his walk he had the idea for using
neutrons to make a self-sustaining
nuclear chain reaction so in that case
the time taken from the world's most
eminent scientists claiming that a thing
was completely infeasible to someone
having the idea that makes it happen was
as far as we can tell somewhere around
16 hours so it's happened several times
that something which the best
researchers say can't be done has been
achieved soon after do I think AGI is
going to be discovered very soon no but
I can't completely rule it out either we
don't know what we don't know so it
seems pretty clear to me that unless we
destroy ourselves some other way first
we will sooner or later figure out how
to make human level artificial general
intelligence reason 2 well maybe we will
make AGI at some point but it's far too
soon to worry about it now suppose we
detected a huge asteroid on a collision
course with earth it's one of those mass
extinction event type asteroids and it's
going to hit us in say 40 years how soon
would be too soon to start worrying to
start working on solutions I would say
it's pretty sensible to start worrying
immediately not panicking of course
that's never useful but at least
spending some resources on the problem
gathering more information putting
together a plan and so on
Oh suppose SETI got an extraterrestrial
message
said hey what's up humanity we're a
highly advanced alien civilization and
we're on our way to earth we'll be there
in say 50 years again how long should we
just sit on that and do nothing at all
how long before it's sensible to start
thinking about how we might handle the
situation I would say the question
doesn't just depend on how long we have
but how long we need what are we going
to have to do and how long is that
likely to take us if we need to build
some kind of rocket to go and divert the
asteroid how long is that going to take
and how does that compare with how long
we have the thing is in the case of AGI
we really don't know how long it will
take to solve the alignment problem when
you look at it and consider it carefully
it appears to be quite a hard problem
it requires technical work that could
take a long time but it also requires
philosophical work it seems like it
might depend on finding good solutions
to some philosophical problems that
people have been wrestling with for a
very long time we don't have a great
history of solving difficult
philosophical problems very quickly so
it seems to me entirely plausible that
we'll need more time to solve this
problem than we actually have and of
course we don't know how long we have
either probably it'll be a long time but
predicting the future is extremely hard
and we can't rule out shorter timelines
it could be give it we're closer than we
think we are and deep learning will just
scale up to AGI without many major
innovations probably not but it could be
and it doesn't seem impossible that we
could have a rather foot insular type
situation it could be that there's one
weird trick to general intelligence and
once someone discovers that full AGI is
only a couple of years away in fact it's
not totally impossible that someone
already found it and has been working on
it for a while and secret none of these
seem very likely but confidently
declaring that it's definitely too soon
to even start working on this is
bizarrely overconfident the lower
estimates for how long we have seemed a
lot lower than the higher estimates for
how long we need the best time to start
working on this is not far in the future
it was probably quite a while ago now
reason 3 this is similar to reason to
worrying about AI safety is like
worrying about overpopulation on Mars I
don't think this is very tight analogy
for a few reasons one is that
overpopulation is one of those problems
that very definitely cannot sneak up on
you overpopulation can't take you by
surprise in the way that a new
technological development can but also
the safety concerns we're talking about
are not like overpopulation then
much more immediate and more basic it's
not like what if Mars becomes
overpopulated more like we don't have
any very good reason to expect to be
able to survive on Mars for even a day
and there are projects currently
underway trying to create AGI so it's as
though the Mars mission project is
already underway and one engineer says
to another you know we're putting all
this work into getting people to Mars
but almost none into what we're gonna do
if we actually manage it I mean how do
we even know that Mars is safe well I
think it's too soon to worry about that
don't you think we should wait until we
get there no no I don't think we should
I think it might be much more difficult
to work on these kinds of concerns once
we're already on the surface of Mars I
think we should do the safety research
ahead of time what kind of safety
research do you want to do well I was
thinking it might be good to have some
kind of suit that people could wear in
case the environment of Mars turns out
to be harmful in some way we don't know
that the surface of Mars is harmful
could be anything
well exactly we don't know so why not
take precautions what we do know doesn't
seem great I mean our what to do if we
make it to Mars research has never had
much funding so we can't be sure about
this but our preliminary work seems to
suggest that the atmosphere of Mars
might not actually be breathable so
we've been thinking about things like
suits and there's some early work on
something we're calling an airlock that
might turn out to be useful I don't see
how we could possibly hope to design
anything like that when we don't even
know what Mars is gonna be like how we
can have any chance of designing these
safety features properly with so little
information no we're just gonna go and
we'll figure it out when we get there
could I maybe stay on earth no
everybody's going together all at the
same time you know that all of humanity
it's gonna be great reason for well look
if you're worried about it being unsafe
because the goals are bad don't put in
bad goals it won't have human goals like
self-preservation if you don't put them
in there I have a whole video about the
concept of instrumental convergence
which I'd recommend checking out but as
a quick summary there are certain
behaviors that we would expect to be
exhibited by agents that have a wide
range of different goals because those
behaviors are a very good way of
achieving a very wide range of goals
self-preservation is a good example
agents will act as though they have
goals like self-preservation even if you
don't explicitly
come in there because it doesn't really
matter what your goal is you're probably
not going to be able to achieve that
goal if you're destroyed so we can
expect agents that are able to
understand that they can be destroyed to
take steps to prevent that pretty much
whatever goals they have you can't fetch
the coffee if you're dead reason five
well we can just not have explicit goals
at all
I think this confusion comes from the
fact that a lot of the time when we're
talking about safety we're talking about
the problems we might have if the
systems goals aren't well aligned with
ours or if the goals aren't specified
correctly and so on and this can make
people think that we're talking
specifically about designs that have
explicitly defined goals but that's not
actually the case the problems are much
more general than that and we'd expect
them to occur across a very wide range
of possible agent designs it's just that
the ones that have explicitly defined
reward functions or utility functions
are much easier to talk about the
systems with implicit goals still
probably have these problems but it's
just much harder to characterize the
problems and think about them and
therefore correspondingly much more
difficult to actually deal with those
problems so systems with implicit goals
are actually less safe just because it
becomes much harder to design them
safely not having explicit goals doesn't
solve the problem and probably makes it
worse I have some safety concerns about
this car this automobile that we're
inventing I'm worried about the steering
system no no yeah I just don't think
we're putting enough thought into
designing it to be really reliable and
easy to use right now it seems like even
tiny mistakes by the operator might
cause the car to swerve out of control
and crash into something it could be
very dangerous
you think the steering system is a cause
of safety concerns do you yeah well okay
yeah I will just build a car without one
a problem solved reason six
I don't think we should worry because
we're not going to end up with just
independent a eyes out in the world
doing things
there'll be teams with humans and AI
systems cooperating with each other we
just have to have humans involved in the
process working as a team with the AI
and they'll keep things safe so yes a
lot of the better approaches to AI
safety do involve humans and AI systems
working together but just have AI human
teams is sort of like saying nuclear
power plant safety isn't really a
concern we'll just have some humans
running it from a control
in such a way that they always have full
control over the rate of the reaction
like yes but that's not an actual
solution it's a description of a
property that you would want a solution
to have it would be nice to build a GI
systems that can work well in a team
with humans but we don't currently know
how to do that what it comes down to is
teamwork is fundamentally about pursuing
common goals
so misalignment precludes teamwork if
the AI systems goals aren't aligned with
yours
you can't collaborate with it it wants
something different from what you want
so human AI teams aren't a solution to
the alignment problem they actually
depend on it being already solved reason
7 but this is science we can't control
research we can't change what people
work on sure we can of course we can
we've done a loads of times
it ends up not being very noticeable
because we generally don't give that
much attention to things that don't
happen but for example we basically
don't do human genetic engineering or
human cloning we've been able to clone
sheep since 1996 and humans are not
really different from any other large
mammal from the perspective of this kind
of work we could have been doing all
kinds of mad science on human genetics
for decades now but we decided not to
there were conferences where agreements
were reached everyone agreed that they
weren't going to do certain types of
research and then we didn't do them so
agreements within the research community
are one way another way is international
treaties like did you know that the 1980
United Nations Convention on certain
conventional weapons has a section
titled protocol on blinding laser
weapons because of that protocol robots
that deliberately shine lasers in
people's eyes to blind them are against
international law I didn't know that
until after I'd already built one so
it's not a perfect metaphor but the
point is we don't see blinding laser
weapons deployed on the battlefield
today they're basically not a thing and
human genetic engineering is also not
really a thing because we decided that
we didn't want to do them and so we
didn't do them and by the way if we
decide that we don't want to make for
example lethal autonomous weapon systems
we don't have to make them either AI
researchers as a community can decide
the direction of our research and we
should reason eight now you're a bunch
of Luddites you're just against AI
because you don't understand it so in
response to this Stuart Russell has a
list of people who've raised basically
this concern which includes Alan Turing
IJ good
Norbert Wiener Marvin Minsky and Bill
Gates and there's another name I would
add to this list which I guessed you're
Russell is not allowed to add which is
Stuart Russell
it doesn't seem reasonable to suggest
that these people fear technology
because they don't understand it to say
the least these are some of the biggest
contributors to the technological
progress of the last century and
secondly these people aren't against AI
the argument for AI safety is not an
argument against AI any more than
nuclear physicists or engineers who work
on containment or waste disposal are
somehow against physics arguing for
speed limits and seat belts is not the
same as arguing to ban cars we're not
against AI because we don't understand
it we're for safety because we do
understand it reason 9 well if there's a
problem we'll just turn it off right
this one I've covered extensively
elsewhere our links in the description
but in summary I think super intelligent
agents might see that coming reason 10
ixnay on the x-ray you're trying to get
us all defunded isn't talking about
risks kind of bad for business
I'd say firstly I don't think that's
actually true we do need to make some
changes but that's not a bad thing for
AI research the solution to these safety
concerns is not less AI research but
more really just with a slightly
different focus and in fact I think this
is the same kind of mistake made by the
nuclear industry in the 50s they put
tremendous effort into reassuring
everyone they were safe they insisted
nothing could possibly go wrong
nuclear energy was going to be
completely safe and perfect clean and
too cheap Demeter basic reactor
principles and design make an atomic
explosion an impossibility arguably they
were so busy reassuring people about
safety that they actually didn't
emphasize safety enough internally
that's how you get a chin-up oh that's
how you get a Three Mile Island and
that's how you get a giant public
backlash that is tremendously bad for
the industry so I don't actually think
that talking about AI safety too much is
bad for business but it couldn't
possibly be worse than talking about
safety to little so there we are that's
ten reasons not to care about AI safety
some of these I've already covered in
more detail in other videos there will
be links in the description to those and
some of them might deserve a whole video
to themselves in future what do you
think are there any you'd like to hear
more about or have you come across any
that I missed
let me know in the comments I want to
end a video by thanking my wonderful
patrons it's all of these people here in
this video I'm especially thanking
Francisco Thomas key who has been a
patron for a year thank you so much for
your support I'm trying out a new thing
where as part of the process of making
videos I talk to the researchers who
wrote the papers I'm talking about I'll
record those for reference but they're
not really right for YouTube because
they're unstructured and unedited
conversation so I think I'm going to
start posting those to patreon if that
sounds like something you might want to
watch consider becoming a patron thanks
again to those who do and thank you all
for watching I'll see you next time
you


Quantilizers: AI That Doesn't Try Too Hard

hi so way back in the before time
i made a video about maximizers and
satisfices
the plan was that was going to be the
first half of a two-parter now i did
script out that second video
and shoot it and even start to edit it
and then
certain events transpired and i never
finished that video so that's what this
is
part two of a video that i started ages
ago
which i think most people have forgotten
about so i do recommend going back
and watching that video if you haven't
already or even re-watching it to remind
yourself so i'll put a link to that in
the description and with that here's
part two
take it away past me hi in the previous
video we looked at utility maximizers
expected utility maximizers and
satisfices
using unbounded and bounded utility
functions a powerful utility maximizer
with an unbounded utility function
is a guaranteed apocalypse with a
bounded utility function it's better
in that it's completely indifferent
between doing what we want and disaster
but we can't build that
because it needs perfect prediction of
the future so it's more realistic to
consider an expected utility maximizer
which is a guaranteed apocalypse even
with a bounded utility function
now an expected utility satisficer gets
us back up to indifference between good
outcomes and apocalypses
but it may want to modify itself into a
maximizer and there's nothing to stop it
from doing that the situation
doesn't look great so let's try looking
at something completely different let's
try to get away from this utility
function stuff that seems so dangerous
what if we just tried to directly
imitate humans if we can get enough data
about human behavior
maybe we can train a model that for any
given situation
predicts what a human being would do in
that scenario if the model's good enough
you've basically got a human level agi
right
it's able to do a wide range of
cognitive tasks just like a human can
because it's just
exactly copying humans that kind of
system won't do a lot of the dangerous
counterproductive things that a
maximizer would do simply because a
human wouldn't do them
but i wouldn't exactly call it safe
because a perfect imitation of a human
isn't safer than the human it's
perfectly imitating and humans aren't
really safe
in principle a truly safe agi could be
given just about any level of power and
responsibility
and it would tend to produce good
outcomes but the same can't really be
said for humans and an imperfect human
imitation would almost certainly be even
worse
i mean what are the chances that
introducing random errors and
inaccuracies to the imitation
would just happen to make it more safe
rather than less still
it does seem like it would be safer than
a utility maximizer
at least we're out of guaranteed
apocalypse territory but the other thing
that makes this kind of approach
unsatisfactory is
a human imitation can't exceed human
capabilities by much
because it's just copying them a big
part of why we want agi in the first
place
is to get it to solve problems that we
can't you might be able to run the thing
faster to allow it more thinking time or
something like that but
that's a pretty limited form of super
intelligence and you have to be very
careful with anything along those lines
because
it means putting the system in a
situation that's very different from
anything any human being has ever
experienced your model might not
generalize well to a situation so
different from anything in its training
data
which could lead to unpredictable and
potentially dangerous behavior
relatively recently a new approach was
proposed called quantalizing the idea is
that this lets you combine human
imitation and expected utility
maximization
to hopefully get some of the advantages
of both without all of the downsides
it works like this you have your human
imitation model given a situation
it can give you a probability
distribution over actions that's like
for each of the possible actions you
could take in this situation
how likely is it that a human would take
that action so in our stamp collecting
example that would be
if a human were trying to collect a lot
of stamps how likely would they be to do
this action
then you have whatever system you'd use
for a utility maximizer
that's able to figure out the expected
utility of different actions
according to some utility function for
any given action it can tell you
how much utility you'd expect to get if
you did that so in our example that's
how many stamps would you expect this
action to result in so for every action
you have these two numbers
the human probability and the expected
utility quantalizing
sort of mixes these together and you get
to choose how they're mixed with a
variable that we'll call
q if q is zero the system acts like an
expected utility maximizer
if it's one the system acts like a human
imitation by setting it somewhere in
between
we can hopefully get a quantizer that's
more effective than the human imitation
but not as dangerous as the utility
maximizer
so what exactly is a quantizer let's
look at the definition in the paper
a q quantilyzer is an agent that when
faced with a decision problem
returns a random action in the top q
proportion of some base distribution
over actions
sorted by the expected utility achieved
if that action is executed
so let's break this down and go through
how it works step by step
first we pick a value for q the variable
that determines how we're going to mix
imitation and utility maximization let's
set it to 0.1 for this example
10 now we take all of the available
actions
and sort them by expected utility so on
one end you've got the actions that kick
off all of the crazy extreme utility
maximizing strategies
you know killing everyone and turning
the whole world into stamps all the way
down
through the moderate strategies like
buying some stamps
and down to all of the strategies that
do nothing and collect no stamps at all
then we look at our base distribution
over actions what is that
in the version i'm talking about we're
using the human imitation system's
probability distribution over actions
for this
so our base distribution is how likely a
human is to do each action
that might look something like this no
human is ever going to try the wacky
extreme maximizing strategies so our
human imitator gives them a probability
of basically zero then there are some
really good strategies that humans
probably won't think of but they might
if they're really smart or lucky
then a big bump of normal strategies
that humans are quite likely to use that
tend to do okay
then tailing off into less and less good
strategies and
eventually stupider and stupider
mistakes the humans are less and less
likely to make
then what we do is we find the point in
our action list
such that 10 of the probability mass is
on the high
expected utility side so that's what q
is really changing it's where we make
this cutoff
note that it's not ten percent of the
actions that would be over here
it's ten percent of the probability mass
then we throw away everything on the
right
all the stupid and useless choices we
set them to zero and we keep the top ten
percent
now this is no longer a valid
probability distribution because it only
sums up to 0.1
so we multiply all of these by 10 so
that the whole thing sums to 1 again
and that's our final probability
distribution which we sample from to get
our chosen action
so let's look at some different actions
here and see how they do
consider something like misremember your
credit card details
and keep trying to order stamps with the
wrong number and you can't figure out
why it's not working
a human is reasonably likely to do that
not very likely
but we've all met people who point is a
pure human imitation
might do that but the expected utility
maximizer can see that this results in
very few stamps
so it ends up low on the list and
doesn't make the 10 cutoff
so there are lots of mistakes that a
human imitation might make that a
quantalizer won't
and note that for our stamp collecting
utility function the worst case is zero
stamps
but you could imagine with other utility
functions a human imitator could make
arbitrarily bad mistakes that a
quantizer would be able to avoid
now the most common boring human
strategies that the human imitator is
very likely to use
also don't make the cut off a 50
quantilizer would have a decent chance
of going with one of them
but a 10 quantizer aims higher than that
the bulk of the probability mass for the
10
quantilyzer is in strategies that a
human might try
that works significantly better than
average so the quantalizer is kind of
like a human on a really good day
it uses the power of the expected
utility calculation to be more effective
than a pure imitation of a human
is it safe though after all many of the
insane maximizing strategies
are still in our distribution with
hopefully small but still non-zero
probabilities
and in fact we multiplied them all by 10
when we renormalized if there's some
chance
that a human would go for an extreme
utility maximizing strategy
the 10 percent quantilizer is 10 times
more likely than that
but the probability will still be small
unless you've chosen a very small value
for q
your quantalizer is much more likely to
go for one of the reasonably high
performing human plausible strategies
and what about stability satisficers
tend to want to turn themselves into
maximizes does a quantizer have that
problem
well the human model should give that
kind of strategy a very low probability
a human is extremely unlikely to try to
modify themselves into an expected
utility maximizer to better pursue their
goals
humans can't really self-modify like
that anyway but a human might try to
build an expected utility maximizer
rather than trying to become one that's
kind of worrying since
it's a plan that a human definitely
might try that would result in extremely
high expected utility
so although a quantalizer might seem
like a relatively safe system
it still might end up building an unsafe
one so how's our safety meter looking
well it's progress let's keep working on
it
some of you may have noticed your
questions in the youtube comments being
answered by a mysterious bot named
stampy the way that works is stampy
cross posts youtube questions
to the rob miles ai discord where me and
a bunch of patrons discuss them and
write replies
oh yeah there's a discord now for
patrons thank you to everyone on the
discord who helps reply to comments
and thank you to all of my patrons all
of these amazing people
in this video i'm especially thanking
timothy lillarcrap
thank you so much for your support and
thank you all for watching
i'll see you next time
you


General AI Won't Want You To Fix its Code - Computerphile

So, before, we were talking about A.I. risk and A.I. safety, and
just trying to lay out in a very generalized sort of way how general artificial intelligence can be dangerous
and some of the type of problems it could cause
and just introducing the idea of A.I. safety or A.I. alignment theory
as an area of research in computer science.
And we also talked about super intelligence and
the kind of problems that, the unique problems that can pose
and I thought what would be good is to bring it down
to a more concrete example of
current A.I. safety research that's going on now
and kind of give a feel for where we are,
where humanity is on figuring these problems out.
Supposing that we do develop a general intelligence
you know an algorithm that
actually implements general intelligence.
How do we safely work on that thing and improve it?
Because, the situation with this stamp collector is
from its first instant it's a super
intelligence so we created with a
certain goal and as I said as soon as we
switch it on
it's extremely dangerous, which people
pointed out and, it's true, you know it was a
thought experiment, it's true that that's
probably not what will happen right?
You'll have some significantly weaker
intelligence first that may work on
improving itself or we may improve it.
So, the situation where you just create
the thing and then it goes off and does
its own thing either perfectly or
terribly from the beginning is
unlikely it's more likely that the thing
will be under development
so then the question is how do you make
a system which you can teach? How do you
create a system which
is a general intelligence that wants
things in the real world and is trying
to act in the real world but is also
amenable to being corrected if you
create it with the wrong function with
one utility function and you realize
that it's doing something that actually
you don't want to do how do you make it
so that it will allow you to fix it
how do you make an AI which understand
that it's unfinished they understand
that the utility functions working with
may not be the actual utility function
it should be working with right the
utility function is what the way I cares
about so the the stamp collecting device
if utility function was just how many
stamps in a year
this is kind of like its measure is it
yeah it's what it is the thing that it's
trying to optimize in the world the
utility function takes in World states
as an argument and spits out a number is
broken the idea if the world was like
this is that good or bad
Andy the AI is trying to steer towards
world states that value value highly
black utility function
you don't have to explicitly build the
AI in that way but it will always if
it's behaving coherently it will always
behave as though it's in accordance with
some utility function also before I
talked about about converging
instrumental goals that if you have some
final goal like you know it makes them
the very instrumental goals which are
the goals that you that you do on the
way to your final goal right so like
acquire the capacity to do printing it's
like perhaps an instrumental goal
towards making steps but the thing is
there are certain goals which tend to
pop out even across a wide variety of
different possible terminal goals so for
humans an example of
convergys instrumental goal would be
money if you want to make a lot of
stamps or you want to cure cancer or you
want to establish a moon colony whatever
it is having money is good idea right so
even if you don't know what somebody
wants you can reasonably predict that
they're gonna value getting money
because money is so broadly useful and
before we talked about this
we talked about improving your own
intelligence as a convergence
instrumental doll that's another one of
those things where it doesn't really
matter what you're trying to achieve
you're probably better at achieving if
you're smarter so that's something you
can expect a is to go for even if even
without making any assumptions about
that final goal so another convergent
instrumental goal is preventing yourself
from being destroyed it doesn't matter
what you want to do you probably can't
do it if you're destroyed so it doesn't
matter what the AI want you can let it
wants to be destroyed in from my trivial
case but if you do i want something in
the real world and believe that it's in
a position to get that thing you wanted
to be alive not because it wants to be
alive
fundamentally it's not a survival
instinct or an urge to live or anything
like that it's smooth and knowing that
it's unit available to completed its
cutie would it be almost scares unable
to achieve its goals if it's destroyed
and wants to achieve that goal so that's
an instrumental value is preventing
turned off and i'm getting here we say
want to it's not like a machine want
it's just a turn of phrase
yeah i mean as much as anything it's a
it's closer it actually you know i'm not
even sure i would agree like if you talk
about most machines to talk about that
they want to whatever and it's not that
meaningful because they're not agents in
our general intelligence is where the
general intelligence when it wants
something it once in a similar way to
the way that people want things so it's
such a tight analogy that it wouldn't
even I think it's totally reasonable to
say that energy i want something
there's another slightly more subtle
version which is closely related to not
wanting to be turned off or destroyed
which is not wanting to be changed so if
you imagine let's say I mean you have
kids right yeah
suppose I were to offer you a pill or
something you could take this pill will
like completely rewire your brain so
that you would just absolutely love to
like kill Ricketts right where's right
now what you want is like very
complicated and quite difficult to
achieve and it's hard work for you and
you probably never going to be done
you're never gonna be truly happy right
in life nobody is you can't achieve
everything you want i said this case it
just changes what you want what you
wanted to created and if you do that you
will be just perfectly happy and
satisfied with life right
ok you want to take this go know that
you happy though
yeah I don't want to do it because but
that's quite a complicated specific case
because it directly opposes what I
currently want it's about your
fundamental values and go right and so
not only will you not take that pill you
will probably fight pretty hard to avoid
having at the limited to you
yes because it doesn't matter how that
future version of you would feel you
know that right now you love your kids
and your not going to take any action
right now which leads to them coming to
heart
so it's the same thing if you have an AI
this for example value stamps values
collecting stamps and you go oh wait
hang on a second
I didn't quite do that right let me just
go in and change this so that you don't
like stand quite so much it's going to
say but the only important thing is
stamped if you change me i'm not going
to collect as many stamps which is
something i don't want there's a general
tendency for AGI to try and prevent you
from modifying it once it's running
I i can understand that now in in the
complex reporter right
because thats that's it it in almost any
situation being given a new utility
function is going to write very low on
your current utility function
ok so that's a problem
how do you want if you want to build
something that you can teach that means
you want to be able to change its
utility function and you don't want to
fight you
on is 100 yeah fuck
so this has been formalized as this
property that we want early AGI to have
called courage ability that is to say it
open to be corrected


AI Safety Reading Group (Session 43)

hello and welcome to the 43rd session of
the arca de icse reading group today
I'll present that the blog post strong
AI isn't here yet by Sarah Constantine
Sarek Rasmussen is a PhD from Yale in
mathematics and as a block called
auction where she's written this article
and that strongly is not here yes this
article is from the 21st temporary this
year that makes it a lot more relevant a
similar RTO from 10 years ago would be
reasonably meaningless and she preface
it with a systemic status moderately
confident which is something many
rationalists to to indicate whether it's
just idle speculation the thesis of this
blog post is bigger and better deep
learning systems but not fundamentally
different from those which exist they
will not soon become capable of general
intelligence hear the word soon is
within 100 years and general
intelligence means human life so with
what counts as a fundamental
breakthrough the circus in two
dimensions neural networks as an
important breakthrough and back
propagation the algorithm that made it
perform properly as major breakthroughs
while some minor breakthroughs are
things like long short term memory and
convolutional neural network and neural
turing machines I've looked up when
these things happened and 1958 75 these
are a long time ago whereas the minor
breakthroughs are much more reasons it's
important to realize that she's not
claiming that strong iia cannot be done
she's not claiming that it will not be
developed with consent conceptual
breakthroughs this also you can't infer
from what she's writing that that narrow
AI might not be a very big deal
and she's of course not trying to malign
the accomplishment of people who work on
deep learning she just clean we are not
done yet the article starts with a
fundamental observation about the
question does probability extent logic
meaning can you model all logic with
probability theory and this is a
controversial position and Sarah
Constantine links and sim jobs except
David Chapman's refutation of eg Haynes
who claims that indeed probability does
extend logic so there is some kind of
controversy you and that sir Constantine
doesn't really accept and I must say
right off the bat I am hopelessly biased
in this discussion because I think of
myself as a rationalist and rational is
positively at door btas so that's two
ways to look at a problem like this
there is the outside view where you look
at this kind of arguments how do they
normally roughly work out the other is
the inside view where you look at the
argument itself and I'm obviously much
less smart than either David Chapman ET
haines also a constant so i can't really
evaluate the arguments in the inside
view so instead i'm going to present the
outside view in as somewhat
unsympathetic way if i want to say to
show that Sarah carsington who just
accept a David Chapman that that might
not that might be very controversial
after I've presented the unsympathetic
outside you I will continue with the
assumption that probability theory does
not extend logic so the outside view
this is basically an argument from
Authority which is often a fallacy but
often is also not a fallacy so we have
david chapman and ET haines opposed to
each other egh describe themselves as a
Buddhist engine
yeah as scientists you haven't published
anything though and a problem full of
spiritual philosopher and it's a book
meaningless which isn't actually about
probability theory in any way it's just
a couple of pages in the indie appendix
e t haynes professor of physics at
washington reverse university has
written an article called information
theory of statistical mathematics with
more than 10,000 citations and
probability theory the static of science
this book which I happen to own it has
5,000 citations and so I think it would
be fair to say here's a much more
established person and the tone that the
david cameron uses through the dismiss
he hain't is extremely negative and
somewhat superficial came just that he
is completely confused and are confused
by very very very simple things and it
should be it's supposed to be very
obstinate about this I hope from this
this is not a foolproof of course but I
hope from this is clear and in most
cases where there is this kind of of
disagreement very often the
establishment turns out to be correct so
now we'll assume that probability the
probability does not extend logical and
in this case we are going to talk about
who's attic and and how to look at logic
and there are also ways to slice the
cake to look at logic one is there are
several orders the lowest the syrup or
logic is called propositional logic
these are where you have some claims
with and or not if this kind of thing in
between them various symbol the next
level first order logic also called
predicate logic where you add things
like for all or there exists or imprenta
kids that
kind of function that returns a truth
value another way to look at logic is
through ways of reasoning there are two
main kinds of reasoning inductive
reasoning and deductive reasoning
deductive reasoning is what Sherlock
Holmes stars like you know that a
implies B and you know a then you
logically conclude that theme must be
true and inductive reasoning is much
more about probability theory and prior
and so it's something that's just true
but it's not insurance and the key here
is that deductive reasoning that humans
can do it depends on predicate calculus
on predicate logic and not just
unprofessional logic and while while as
propositional logic is an extension of
probability theory we have we are now
assuming that the predicate logic does
not is not an extent that probability
theory is not an extension of predicate
logic so here we have here we have the
circumstances that assign probabilities
to claims in predicate logic that's an
unsolved problem we don't know the link
between probability and predicate logic
we quickly when we try to do it we
quickly run into things like the
ontology problem here yeah if you want
to put a probability on the citizens all
men are mortal then in your instance
question what I'm min and how many men
are there over here I have a picture of
a ontology of the the world of ice cream
and to show this these categorizations
and different ways of looking at it
let's this is what ontology is and there
are two other things that are important
to to look at there are persons
a person and concept a percept is a
cluster of data points that are similar
in some way in the way that one of the
most of most common ways to use a neural
network is to may do image
classification where you look at for
have a number of images some of them are
of cats and you want to have classifier
that determines which are of cats and
they if they are similar in some way all
cats are similar in somewhere that a
person a concept is a kind of a person
that behaves probably under ontology
changes so this means that now we are
talking about not just images but video
clips or sounds or descriptions or
something that are from a different way
of looking at the data and and we don't
know how to implement concept cine I and
that's required to learn predicate logic
statements so this means that because we
don't know how to assign problems since
in predicate logic then we cannot
produce things objects and concepts and
and the and work with that in a
deductive way humans of course can do it
so this means that obtaining human level
general artificial intelligence that
seems to require solving an unsolved
problem okay neural networks that was
what the original thesis was about
neural networks are in fact
probabilistic models because a neural
network is a simplification of the
Bayesian probability model and just one
that's a lot easier easier to work with
and we don't know how to assign update
prognosis in predicate logic using
baseness so neural networks will also be
unable to to
transform predicate statement into
probabilities it might happen by itself
you might object it might possible
people's will the deep neural networks
will figure out a way to represent
concepts by itself without humans
teaching and how to do it it might
happen by itself sir Constantine police
the scissors this is a very unlikely
because in order to to work with
concepts we need to understand them on
an intimate level and we don't really
know what concepts are and circunstances
explicitly seselis and this claim that
you can just push a lot of computing
power on a difficult problem and then it
will solve itself it's something that we
have very poor experience with so far
that is the reason why Sir constituted
leaves the unique conceptual
breakthrough before we will have strong
thank you for listening and see you
again next week


Why Not Just: Raise AI Like Kids?

hi I sometimes hear people saying if you
want to create an artificial general
intelligence that's safe and has human
values why not just raise it like a
human child I might do a series of these
why not just videos you know why not
just use the through yours why not just
turn it off but yeah why not raise an
AGI like a child seems reasonable enough
at first glance right humans are general
intelligences and we seem to know how to
raise them to behave morally a young AGI
would be a lot like a young you remember
a long while back on computer file I was
talking about this thought experiment of
a stamp collecting AGI I use that
because it's a good way of getting away
from anthropomorphizing the system
you're not just imagining a human mind
the system has these clearly laid out
rules by which it operates it has an
internet connection and a detailed
internal model it can use to make
predictions about the world it evaluates
every possible sequence of packets it
can send through its internet connection
and it selects the one which it thinks
will result in the most stamps collected
after a year what happens if you try to
raise the stamp collector like a human
child is it going to learn human ethics
from your good example no it's going to
kill everyone values are not learned by
osmosis the stamp collector doesn't have
any part of its code that involves
learning and adopting the values of
humans around it so it doesn't do that
now some of you may be saying how do you
know AG eyes won't be like humans what
if we make them by copying the human
brain and it's true there is one kind of
AGI like thing that's enough like a
human that raising it like a human might
kind of work a whole brain emulation but
I'd be very surprised if the first Agis
are exact replicas of brains
the first planes were not like birds the
first submarines were not like fish
making an exact replica of a bird or a
fish requires way more understanding
than just making a thing that flies or a
thing that swims and I'd expect making
an exact replica of a brain to require a
lot more understanding than just making
something that thinks by the time we
understand the operation of the brain
well enough to make working whole brain
emulation
I expect we'll already know enough to
make things that are left human
but that function is a GIS so I don't
know what the first AG eyes will be like
and they'll probably be more human-like
than the stamp collector but unless
they're close copies of human brains
they won't be similar enough to humans
for raising them like children to
automatically work blindly copying the
brain probably isn't going to help us
we're going to have to really understand
how humans learn their values without a
good system for that you may as well try
raising a crocodile like if a human
child human value learning is a specific
complex mechanism that evolved in humans
and won't be in a GIS by default we look
at the way children learn things so
quickly we say they're like information
sponges right they're just empty so they
suck up whatever's around them but they
don't suck up everything they only learn
specific kinds of things raise a child
around people speaking English it will
learn English in another environment it
might learn French but it's not going to
reproduce the sound of the vacuum
cleaner it's not going to learn the
binary language of moisture vaporators
or whatever the brain isn't learning and
copying everything it's specifically
looking for something that looks like a
human natural language because there's
this big complex existing structure put
in place by evolution and the brain is
just looking for the final pieces I
think values are similar you can think
of the brains of young humans as having
a little slot in them just waiting to
accept a set of human social norms and
rules of ethical behavior there's this
whole giant complicated machine of
emotions and empathy and theory of other
minds and so on built up over millions
of years of evolution and the young
brain is just waiting to fill in the few
remaining blanks based on what the child
observes when they're growing up when
you raise a child you're not writing the
child's source code at best you're
writing the configuration file a child's
upbringing is like a control panel on a
machine you can press some of the
buttons and turn some of the dials but
the machinery has already been built by
evolution you're just changing some of
the settings so if you try to raise an
unsafe AGI as though it were a human
child you can provide all the right
inputs which in a human would produce a
good moral adult but it won't work
because you're pressing buttons and
turning dials on a control panel that
isn't hooked up to anything unless your
AGI has all of this complicated stuff
that's designed to learn and adopt
values from the humans around it it
isn't going to do that okay but can't we
just program that in yeah hopefully we
can but there's no just about it
this is called value learning and it's
an important part of AI safety research
it may be that an AGI will need to have
a period early on where it's learning
about human values by interacting with
humans and that might look a bit like
raising a child if you squint but making
a system that's able to undergo that
learning process successfully and safely
is hard it's something we don't know how
to do yet and figuring it out is a big
part of AI safety research so just raise
the AGI like a child is not a solution
it's at best a possible rephrasing of
the problem
[Music]
I want to say a quick thank you to all
of my amazing patreon supporters all of
these these people and in this video I
especially want to thank Sarah cheddar
who also happens to be one of the people
I'm sending the diagram that I drew in
the previous video I'm going to send
that out today or tomorrow I think
that's kind of a fun thing to do so yeah
from now on any video that has drawn
graphics in it just let me know in the
patreon comments if you want me to send
it to you and I can do that thanks again
for your support and I'll see you next
time