Predicting AI: RIP Prof. Hubert Dreyfus

I recently learned that professor Hubert
Dreyfus the philosopher and outspoken
critic of the field of artificial
intelligence has died at the age of 87
he published a lot of philosophy and did
a lot of teaching but one of the things
he's best known for and most relevant to
this channel was his work on artificial
intelligence and its limits so I'm going
to talk about that a little bit today
and focus on one of his arguments in
particular so in the 50s and 60s AI was
the next big thing
computers were tackling problems they'd
never been able to approach before
people were very excited about what was
possible and artificial intelligence
researchers were confident that true
human level intelligence was right
around the corner their reasoning was
something like this thinking consists of
processing facts using logic and rules
what you might call symbol manipulation
you have some things you know you know
the relationships between them you know
the rules of logic and you can use those
to reason about the world computers can
do this kind of symbol manipulation very
well and they're getting better over
time
so computers are able to think and
they're getting better at thinking all
the time
Dreyfuss saw some problems with this and
one of those problems was that a lot of
human thinking doesn't seem to boil down
to symbol manipulation at all it's only
one of the ways we can think it's
perhaps the most salient because it's
related to language and conscious
thought which is the part of the mind
that we're most aware of but actually a
lot of the processing seems to be
happening much slower down when you look
at something you don't think is this has
four legs and a flat top and a back so
it's a chair and this thing has four
legs and a flat top and Novac so it's a
table because I've learned the rules of
what a table is and what a chair is and
now I'm applying those rules now you
just look at the thing and you know that
it's a table the object identification
is done before your conscious mind is
even aware of anything and it's not like
your unconscious mind is doing this kind
of rules based thinking in the
background it's much fuzzier than that
and what would the rules be for
identifying chairs anyway we do need a
back do you need four legs or like any
legs at all I mean this is still a chair
is this thing is this
or this even something as simple as a
chair is really hard to pin down you
need a lot of rules and it would take a
human a long time to evaluate them and
that isn't how the brain works you can
imagine any of our ancestors who thought
well this thing has four legs but it has
a tail so my rules say that it's an
animal the pointy ears and sharp teeth
suggest it's maybe a cat and it's size
suggests perhaps a lion or a tiger
looking at the stripes on the side I can
like you're dead before you're done
counting the legs so Dreyfuss thought
there were problems with the assumptions
that AI researchers were making about
the mind he saw that the model of
cognition that the AI researchers were
using didn't capture the complexity of
human thought and so their reasons for
thinking that computers could do the
things they claimed they could do
weren't very good and he went on to
publish some work including a book
called what computers can't do which
argued that a lot of the things that AI
researchers claimed computers would soon
be able to do were actually impossible
things like pattern recognition natural
language vision complex games and so on
these things didn't just boil down to
symbol manipulation so computers
couldn't do them so how did the AI
researchers react to this philosopher
coming along and telling them that they
were foolishly attempting the impossible
well they did the obvious thing which is
to ignore him and to say unkind things
about it there's a wonderful paper
called the artificial intelligence of
Hubert L Dreyfus which a link to in the
doobly-doo check it out back in the day
before the internet you know people had
to do their flame wars on typewriters
was a different time so having dismissed
him completely they carried on trying to
apply their good old-fashioned AI
techniques to all of these problems for
decades until they find we had to admit
that yeah it wasn't going to work for a
lot of these problems so about some
things at least Dreyfus was right from
the start but the interesting thing is
that since then new techniques have been
developed which has started giving
pretty good results in things like
pattern recognition language translation
high complexity games and so on the kind
of things Dreyfus said computers flatly
couldn't do people say that we moved
away from this good old-fashioned AI
approach which I don't think it's really
true we didn't stop using those take me
at least on the problems that they work
well on we just stopped calling it AI
but the point is the new techniques were
what you might call sub symbolic you
could make a neural network and train it
to recognize tables and chairs and
tigers you can look through the source
code of that system and you won't find
anywhere a single symbol which means
legs or ears or teeth or anything like
that it doesn't work by symbols in the
way that the logic and rules based
approaches of the sixties do so Dreyfus
was right that the AI techniques of the
time weren't able to tackle these
problems but the thing he didn't expect
to happen was computers being able to
use symbols to implement these non
symbolic systems they can tackle the
problems so to oversimplify the AI
researchers said thinking is just symbol
manipulation computers can do symbol
manipulation therefore computers can
think and Dreyfus said thinking is not
just symbol manipulation computers can
only do symbol manipulation that all
computers can't think I don't think
either of those is really right one of
them underestimates what the human mind
can do the other underestimates what
computers can do I think what I take
away from all of this is you can come up
with a simple model that seems to
explain all the important aspects of
some complex system and then it's very
easy to convince yourself that that
model fully covers all of the
complexities and capabilities of that
system but you have to be open to the
possibility that you're missing
something important and the things are
more complex than they seem now you
might say well both sides of this
disagreement made a similar kind of
mistake and they're both wrong I don't
see it that way at all though I mean
someone who says the earth is flat is
wrong someone who says it's a sphere is
wrong as well
it's an oblate spheroid it's bigger
around the equator but then it's not
really an oblate spheroid either they're
perfectly smooth which the earth is not
so you could say that all of those views
are wrong but some of them are clearly
more wrong than others so at the end of
the day can we make computers do any
kind of thinking that humans can do some
people think that now that we have all
these new approaches we have deep
learning and so on and we're able to
start doing the kind of non symbolic
thinking that Dreyfuss pointed out was
necessary we can add that on to the
rules and logic stuff and then we're
nearly done and we're going to ride this
current wave of breakthroughs all the
way up to true
general intelligence and maybe we will
but maybe we won't maybe there's some
third thing that we also need and it's
going to take us several decades to
figure out how to get computers to do
that as well maybe there's a fourth
thing or a fifth thing but I don't think
there's a hundredth thing I don't even
think there's a tenth thing I think
we'll get there sooner or later but I've
been wrong before
so here's the Hubert Dreyfus a great
thinker and a man well ahead of his time
was he right overall probably too soon
to tell but I think he's truly deserving
of our admiration and respect for being
loudly and publicly less wrong than
those around him which is probably the
best any of us can hope for
to end the video with a quick thank you
to my amazing supporters on patreon and
I especially want to thank Fabian
Concilio who sponsors me for $10 a month
I actually have quite a few people
sponsoring me at that level but I
thought I'd thank each one in their own
video though the waiting list is getting
kind of long now I think I might
increase the dollar amount on that
reward level to keep the wait times down
anyway I hope you enjoyed the
behind-the-scenes PC build video I put
up and looking to have some more behind
the scenes stuff going up fairly soon oh
and we hit a target which means I can
get a new lens for this camera which
should really increase the quality and
range of what I can do here so thank you
again and I will see you next time


Training AI Without Writing A Reward Function, with Reward Modelling

hi what is technology don't skip ahead I
promise I'm going someone with this so
you could have some kind of definition
from a dictionary that's like technology
is machinery and equipment made using
scientific knowledge something like that
but where are the boundaries of the
category what counts for example pair of
scissors technology I think most people
would say no although it does meet the
definition perhaps scissors used to be
technology but now I think they're too
simple they're too well understood I
think once we've really nailed something
down and figured out all of the details
people stop thinking of it as technology
I think in order to be technology
something has to be complex and
unpredictable maybe even unreliable
YouTube for example is definitely
technology as is the device you're
watching this on ok why does this matter
I guess part of my point is the exact
definitions are really difficult and
this generally isn't much of a problem
because language doesn't really work by
exact definitions maybe it's hard to
specify exactly what we mean when we use
a word like technology but to paraphrase
something from the US Supreme Court you
know it when you see it and that's good
enough for most uses the reason I bring
this up is sometimes people ask me about
my definition of artificial intelligence
and actually think that's pretty similar
you could say that AI is about trying to
get machines to carry out human
cognitive tasks but then arithmetic is a
cognitive task does that make a
calculator artificial intelligence
you know sorting a list is a cognitive
task I don't think most people would
call that AI playing a perfect game of
noughts and crosses used to be
considered AI but I don't think we'd
call it that these days
so to me AI is about making machines do
cognitive tasks that we didn't think
they could do maybe it's because it's
about making machines do human cognitive
tasks and once machines can do something
we no longer think of it as a human
cognitive task this means that the goal
posts are always moving for artificial
intelligence some people have complained
about that but I think it's pretty
reasonable to have that as part of the
definition so that means that the goal
of AI research is to continue to expand
the range of tasks that computers can
handle so they can keep surprising us it
used to be that AI research was all
about figuring out and formalizing
things so that we could write programs
to do them things like arithmetic
sorting lists and playing noughts and
crosses these are all in the class of
problems that you might call things we
can specify well enough to write
programs that do them and for a long
time that was all that we could do that
was the only type of problem we could
tackle but for a lot of problems that
approach is really really hard like
consider how would you write a program
that takes an image of a handwritten
digit and determines what digit it is
you can formalize the process and try to
write a program it's actually kind of a
fun exercise if you want to get to grips
with the old school like computer vision
and image processing techniques and once
you've written that program you can test
it using the M NIST data set which is a
giant collection of correctly labeled
small images of digits what you'll find
is if you do well then this thing will
kind of work but even the best programs
written this way don't work that well
they're not really reliable enough to
actually use someone is always going to
come along with a really blunt pencil
and ruin your programs accuracy and this
is still a pretty easy problem I mean
what if you wanted to do something like
letters as well as numbers now you have
to differentiate between oh and zero and
one and I and a lowercase L it's forget
about it's never going to work and even
that is a relatively simple problem what
if you're trying to do something like
differentiating pictures of cats from
pictures of dogs this whole approach is
just not going to work for that but
there is a fact that we can exploit
which is that it's a lot easier to
evaluate a solution than to generate a
solution for a lot of these problems
I've talked about this before
I couldn't generate a good rocket design
myself but I can tell you that this one
needs work it's easier to write a
program to evaluate an output than to
write one to produce that output so
maybe it's too hard to write a program
that performs the task of identifying
handwritten numbers but it's pretty easy
to write a program that evaluates how
well a given program does at that task
as long as you have a load of correctly
labeled examples you just keep giving it
labeled examples from the data set and
you see how many it gets right in the
same way
maybe you can't write a program that
plays an Atari game well but you can
easily write a program that tells you
how well you're doing you just read off
the score and this is where machine
learning comes in it gives you ways to
take a program for evaluating solutions
and use it to create good solutions all
you need is a data set with a load of
labeled examples or a game with a score
or some other way of programmatically
evaluating the outputs and you can train
a system that carries out the task
there's a sense in which this is a new
programming paradigm instead of writing
the program itself you write the reward
function or the loss function or
whatever and the training process finds
you a set of parameters for your network
that perform well according to that
function if you squint the training
process is sort of like a compiler it's
taking code you've written and turning
it into an executive all that actually
performs the task so in this way machine
learning expands the class of tasks that
machines can start to perform it's no
longer just tasks that you can write
programs to do but tasks that you can
write programs to evaluate but if this
is a form of programming it's a very
difficult one
anyone who has programmed in C or C++
will tell you that the two scariest
words you can see in a specification are
undefined behavior so how many folks
you're a little bit afraid of undefined
behavior in their source code everybody
and machine learning as a programming
paradigm is pretty much entirely
undefined behavior and as a consequence
programs created in this way tend to
have a lot of quite serious bugs and
these are things that I've talked about
before on the channel for example reward
gaming where there's some subtle
difference between the reward function
you wrote and the actual road function
that you kind of meant to write and an
agent will find ways to exploit that
difference to get high reward to find
things it can do which the reward
function you wrote gives a high reward
too but the reward function you meant to
write wouldn't have or the problem of
side-effects where you aren't able to
specify in the reward function
everything that you care about and the
agent will assume that anything not
mentioned in the reward function is of
zero value which can lead to a having
large negative side-effects there are a
bunch more of these specification
problems and in general this way of
creating programs is a safety nightmare
but also it still doesn't allow machines
to do all of the tasks that we might
want them to do a lot of tasks are just
too complex and too poorly defined to
write good evaluation functions for for
example if you have a robot and you want
it to scramble you an egg how do you
write a function which takes input from
the robot senses and returns how well
the robot is doing it scrambling an egg
that's a very difficult problem even
something simple like getting a
simulated robot to do a back flip it's
actually pretty hard to specify what we
about this well normal reinforcement
learning looks like this you have an
agent and an environment the agent takes
actions in the environment and the
environment produces observations and
rewards the rewards are calculated by
the reward function that's where you
program in what you want the agent to do
so some researchers tried this with the
bakflip task they spent a couple of
hours writing a reward function it looks
like this and the result of training the
agent with this reward function looks
like this
I guess it's that's basically a back
flip I've seen better
something like evaluating a back flip is
very hard to specify but it's not
actually hard to do like it's easy to
tell if something is doing a back flip
just by looking at it it's just hard to
write a program that does that so what
if you just directly put yourself in
there if you just play the part of the
reward function every time step you look
at the state and you give the agent a
number for how well you think it's doing
it back flipping people have tried that
kind of approach but it has a bunch of
problems the main one is these systems
generally need to spend huge amounts of
time interacting with the environment in
order to learn even simple things so
you're going to be sitting there saying
no that's not a back flip no that's not
a back flip either that was closer nope
that's worse again and you're gonna do
this for hundreds of hours nobody has
time for that so what can we do well you
may notice that this problem is a little
bit like identifying handwritten digits
isn't it we can't figure out how to
write a program to do it and it's too
time-consuming to do it ourselves so why
not take the approach that people take
with handwritten numbers why not learn
our reward function but it's not quite
as simple as it sounds back flips are
harder than handwritten digits in part
because where are you going to get your
data from four digits we have this data
set M list we have this giant collection
of correctly labelled images we built
that by having humans write lots of
numbers scanning them and then labeling
the images we need humans to do the
thing to provide examples to learn from
we need demonstrations now if you have
good demonstrations of an agent
performing a task you can do things like
imitation learning and inverse
reinforcement learning which are pretty
cool but there are subject for a later
video but with backflips we don't have
that I'm not even sure if I can do a
back flip and that wouldn't help
wait really I don't have to do it no we
don't need a recording of a human
backflipping we need one of this robot
backflipping right there physiology is
different but I don't think I could
puppeteer the simulated robot to
backflip either that would be like
playing co-op on nightmare mode so we
can't demonstrate the task so what do we
do well we go back to the Supreme Court
exactly defining a back flip is hard
doing of actually this hard but I know a
back flip when I see one so we need a
setup that learns a good reward function
without demonstrations just by using
human feedback without requiring too
much of the humans time and that's what
this paper does it's called deep
reinforcement learning from human
preferences and it's actually a
collaboration between open AI and deep
mind the paper documents a system that
works by reward modeling if you give it
an hour of feedback it does this that
looks a lot better than two hours of
reward function writing so how does
reward modeling work well let's go back
to the diagram in reward modeling
instead of the human writing the reward
function or just being the reward
function we instead replace the reward
function with a reward model implemented
as a neural network so the agent
interacts with the environment in the
normal way except the rewards it's
getting are coming from the reward model
the reward model behaves just like a
regular reward function in that it gets
observations from the environment and
gives rewards but the way it decides
those rewards is with a neural network
which is trying to predict what reward a
human would give okay how does the
reward model learn what reward a human
would give well the human provides it
with feedback so the way that works is
the agent is interacting with the
environment you know trying to learn and
then the system will extract two short
clips of the agent flailing about just a
second or two and it presents those two
clips to the human and the human decides
which they liked better which one is
more backflipping and the reward model
then uses that feedback in basically the
standard supervised learning way it
tries to find a reward function such
that in situations where the human
prefers the left clip to the right clip
the reward function gives more reward to
the agent in the left clip than the
right clip and vice-versa
so which clip gets more reward from the
reward model ends up being a good
predictor of which clip the human world
which should mean that the reward model
ends up being very similar to the reward
function the human really wants but the
thing I like about this is the whole
thing is happening asynchronously it's
all going on at the same time the agent
isn't waiting for the human it's
constantly interacting with the
environment getting rewards from the
reward model and trying to learn at many
times faster than real time and the
reward model isn't waiting either it's
continually training on all of the
feedback that it's got so far when it
gets new feedback it just adds that to
the data set and keeps on training this
means the system is actually training
for tens or hundreds of seconds for each
second of human time used so the human
is presented with a pair of clips and
gives feedback which takes just a few
seconds to do and while that's happening
the reward model is updating to better
reflect their previous feedback at God
and the agent is spending several
minutes of subjective time learning and
improving using that slightly improved
reward model so by the time the human is
done giving feedback on those clips and
it's time for the next pair the agent
has had time to improve so the next pair
of Clips will have new hopefully better
behavior for the human to evaluate this
means that it's able to use the humans
time quite efficiently now to further
improve that efficiency the system
doesn't just choose the clips randomly
it tries to select clips where the
reward model is uncertain about what the
reward should be like there's no point
asking for feedback if you're already
pretty sure you know what the answer is
right so this means that the user is
most likely to see clips from unusual
moments when the agent has worked out
something new and the reward model
doesn't know what to make of it that
maximizes the value of the information
provided by the human which improves the
speed the system can learn so what about
the usual reinforcement learning safety
problems like negative side effects and
reward gaming you might think that if
you use a neural network for your reward
signal it would be very vulnerable to
things like reward gaming since the
reward model is just an approximation
and we know that neural networks are
very vulnerable to adversarial examples
and so on and it's true that if you stop
updating the reward model the agent will
quickly learn to exploit it to find
strategies that the reward model scores
highly but the true reward doesn't but
the constant updating of the reward
model actually provides pretty good
protection against this and the way that
the clips are chosen is part of that if
the agent discovers some crazy new
illegitimate strategy to cheat and get
high reward
that's going to involve unusual novel
behavior which will make the reward
model uncertain so the human will
immediately be shown clips of the new
behavior and if it's reward gaming
rather than real progress the human will
give feedback saying no that's not what
I want the reward model will update on
that feedback and become more accurate
and the agent will no longer be able to
use that reward gaming strategy so the
idea is pretty neat and it seems to have
some safety advantages how well does it
actually work is it as effective as just
programming a reward function well for
the back flip it seems like it
definitely is and it's especially
impressive when you note that this is
two hours of time to write this reward
function which needs a lot of expertise
compared to under one hour of rate and
clips which needs basically no expertise
so this is two hours of expert time
versus one hour of novice time now they
also tried it on the standard magico
simulated robotics tasks that have
standard reward functions defined for
them here it tends to do not quite as
well as regular reinforcement learning
that's just directly given the reward
function but it tends to do almost as
well and sometimes it even does better
which is kind of surprising they also
tried it on Atari games now for those it
needed more feedback because the task is
more complex but again it tended to do
almost as well as just providing the
correct reward function for several of
the games also there's kind of a fun
implementation detail here they had to
modify the games to not show the score
otherwise the agent might learn to just
read the score off the screen and use
that they wanted to rely on the feedback
so it seems like reward modeling is not
much less effective than just providing
a reward function but the headline to me
is that they were able to train these
agents to do things for which they had
no reward function at all like the back
flip of course they also got the cheetah
robot to stand on one leg which is a
task I don't think they ever tried to
write a reward function floor and in
enduro which is an Atari game a racing
game they managed to train the agent
using reward modeling to stay level with
other cars even though the games score
rewards you for going fast and
overtaking them and what all this means
is that this type of method is again
expanding the range of tasks machines
can tackle it's not just tasks we can
write programs to do or tasks we can
write programs to evaluate or even tasks
we're able to do ourselves all that's
required is that it's easy to have
great outputs that you know good results
when you see them and that's a lot of
tasks but it's not everything consider
for example a task like writing a novel
sure you can read two novels and say
which one you liked more but this system
needed 900 comparisons to learn what a
back flip is even if we assume that
writing a novel is no more complicated
than that does that mean comparing 900
pairs of AI generated novels and a lot
of tasks are like this what if we want
our machine to run a company or design
something complex like a cities
transportation system or a computer chip
we can't write a program that does it we
can't write a program that evaluates it
we can't reliably do it ourselves enough
to make a good data set we can't even
evaluate it ourselves without taking way
too much time and resources so we're
screwed
right not necessarily there are some
approaches that might work for these
kinds of problems and we'll talk about
them in a later video
[Music]
I recently realized that my best
explanations and ideas tend to come from
actual conversations with people so I've
been trying a thing where for each video
I first have a couple of video calls
with patreon supporters where I try sort
of running through the idea and seeing
what questions people have and what's
not clear and so on so I want to say a
big thank you to the patrons who helped
with this video you know you are I'm
especially thanking Jake Eric and of
course thank you to all of my patrons
who make this whole thing possible with
their support which reminds me this
video is sponsored by nobody know I
actually turned down a sponsorship offer
for this video and I'll admit I was
tempted because it's a company whose
product I've used for like 10 years and
the offer was thousands of pounds but
they wanted me to do this whole
60-second long spiel and I just thought
no I don't want to waste people's time
with that and I don't have to because
I've got patreon so thank you again to
all of you if you like learning about AI
safety more than you like learning about
mattresses and VPNs you might want to
consider joining those link in the
description thanks again for your
support and thank you all for watching
hi there my knees


229. The Case For Aligning Narrow Superhuman Models

and welcome to session 229
in the asa.com reading group tonight
we'll be discussing
the uh the post on alignment forum
called the case for aligning narrowly
superhuman models
by ajaya kocher
we've seen some previously previous work
by jayakotra
her work on ai timelines and explaining
iterated distillation and amplification
currently she's a senior research
analyst at oregon philanthropy
this my post is from uh
and we will go through miri's comments
on this almost certainly in the next
session but
we usually have some algorithms that are
trained on data
and then we um
we obtain some kind of model that we use
for uh
taking actions or predictions and this
kind of thing
in this case we're talking about super
human models i'm not entirely sure
i like the word superhuman in this case
because we also care about
uh capabilities that are strictly below
the superhuman level and we also care
about those that are
at roughly the human level at expert
human level and
at a level where humans can't even
evaluate
we are talking about narrow uh narrow
models so
something like playing go would be an
example of a very narrow model
um but um but some of these
techniques we're talking about are
actually surprisingly general like give
advice
so narrow shouldn't be interpreted
in too narrow a sense if you get what i
mean and then finally aligning
that's uh what we're trying to do to
figure out how
these models can be used to their full
potential to do what we actually want
an example would be if someone goes to
gt3
and say i am sick i have these symptoms
what should i do
in this case dbd3 has some novel
some knowledge about this because it has
digested a lot of information
from doctors and it might actually be
able to reason
that the uh specific disease is
tuberculosis or whatever
and but it doesn't want to tell us that
if instead it wants to create the
where you have this improvisation game
where it's just trying to
predict the next word because that's
what the qt3 actually cares about
so the main objective of this kind of
research
is to figure out how to give the best
advice
when we know less than the model about
what the real advice is
and also have some kind of systematic
way of figuring out
whether the model is doing the best it
can
a big reason why this kind of research
is something to be optimistic about
is because we are actually doing the
thing that we want to be really good at
just in a smaller sense yeah so we want
to be really good at aligning
uh strongly broadly general superhuman
models
and aligning narrow models are kind of
the same thing
we'll get a lot of practice on doing the
thing that we actually want to get good
at
and on the outside that's a really uh um
[Music]
this will this experience will be
helpful so
we take a big alignment project and we
scale it down
and then we try to find some small
solutions um
for this problem so an example of a
narrowly superhuman model would be
something like other ghost zero
but that's not really an interesting
point of view from a
line from an alignment point of view
this is really interesting
because you can just specify the
objective really simply
it's just to win at go and we care about
the ones
where we sometimes distinguish between
the outer alignment problem and the
inner alignment problem
in this case the outer alignment problem
is very easy for alpha
serum and we want something that is more
fussy where we can't write it down
so we need some kind of human feedback
so in general a way to generate a
project
from this uh in in this general area
is to choose some kind of fussy domain
um
something where we have some kind of
intuition or expectation
that the language ones we have are in
fact superhuman
right now we just can't get that out of
them yet
then we want to have some kind of to
design a reward learning scheme
where we find a way to train the ai
that allows us to reach above the level
of the trainers
and then once we have done that we need
to
to see that this is actually something
that will continue to scale
and something that will also work when
we um get to bigger models
with the ones that we actually want to
end up
aligning so we can't just hard code
knowledge or something like that
and then finally once we have these
models we need to look into pathologies
way that some sometimes the models do
wrong things maybe um
they start to lie or something like that
and we need to understand that
and ensure they don't lie so
the uh the example uh jaya gives
of something that is close to this is
the paper by paul christiano
learning to summarize from human
feedback it's not just
uh procrastination opening up
this almost follows this formula but not
quite
in that the fussy domain of um yeah
summarizing
um is uh somewhat easier and
some of these the people also don't
quite get as far into as we want
but so what kind of projects fit into
this agenda
and which don't it's not binary really
but
some things that means that it counts
more is if we're doing something like
fine-tuning an existing large model
because that's where we expect this
capability to actually resize
it needs to be functioning enough that
we are dealing with humans
and it needs to be something that is
genuinely helpful
on the other hand if you are if you have
an obvious training signal
some either either just something you
can specify or
algorithmically code then it probably
doesn't fit into this agenda
if you only optimize for usefulness and
nothing else
and no generality then also probably
doesn't count
and if you just work on doing the model
larger then that doesn't count either
within this agenda
making the model larger is not always a
negative thing sometimes it can be
positive and similarly
it's rather complicated i'm not quite
sure i agree with this
we can't go out and say it is always
evil because that would imply that um
and that's probably neither true or
helpful to say
but on the other hand i don't really
think it's very complicated
question in that i believe that almost
all
from almost all material circumstances
uh making the model larger
theme is that could not be done right
now
is one that she terms sandwiching so
that's one
where the uh the model is decided
they're not super german
right now because we have one layer
where the
humans are more cable than the model and
then we have a model
and under that we have some other group
of humans who are less stable than the
model
and the idea of the same teaching
project is to figure out how to help the
lower part
of the
um train the model to be as good as
the more capable humans
and this is of course also something
that will help us out in that
the models are improving and it might be
possible
uh to even go above what the more
capable humans have
in some way
so how does this uh reduce long-term
risk extension
this is basically what we want to do
just
a bit more narrow uh so uh
practicing as close as possible to the
thing you want to get good at
is probably really valuable and it's
likely that
since um this kind of work will allow us
to iterate
um then that is likely to be uh
far more effective than doing uh some of
the other conceptual research can be a
bit
uh pie in the sky in that you don't get
feedback on whether these things are
actually working
since you don't actually have a bottle
to work with
that's also likely to help us and you
might see some of the pathologies
earlier
the treacherous turn would be a
prominent example of something that we
could see
before it gets really bad
will give us the alarming community
as a whole some know how about how to
deal with this
and some infrastructure some
organizations and community and the hope
is
that if all the models that
if we succeed on mostly aligning
the relatively weak models we have now
then once we actually get
super human general intelligences we
will have much more
ability to align them since we've
practiced so much
and of course if we discover a potential
solution or maybe the real solution
then we will be able to have a number of
test cases to see
if we believe that this is a full
general solution then we'll see be able
to see does it work
in these kind of specific cases
we might also see something that looks
conventionally
impressive um at this point in
in the post a culture writes this as uh
only a positive thing and i thought that
that was a
a very dangerous thing that we could see
conventionally impressive results in
demos
um later she uh clarifies this could
actually also be a problem
but um but there are definitely upsides
like if alignment researchers can show
something that is really impressive
then that is good for their careers and
get good for
the field of alignment
so why not do other kinds of alignment
research
in a geocachers model
she devised all kinds of research into
conceptual research
some grid worlds and gains and
mainstream
machine learning safety and compared to
this she believes that
aligning narrowly superhuman models
is a more real media and also more
tractable
so for instance the grid worlds and
games
uh are usually very small in that
and that can give some bad habits for
how you look into whether it's doing
something good or bad because
if the model is small enough then the
researcher can actually see that
directly and we bring some more indirect
methods for working with ais that are
potentially smarter than us
so in this case we get some discipline
and that we
already at this point have to use some
methods that need to scale beyond
what we can truly just
directly
also a kind of research where progress
seem much more
legible in that conceptual research in
particular
can be very very illegible um
so that's another reason to pursue this
kind of research
the potential is also very large
in that you could imagine if this is
really successful
then the the task of aligning these
models could be
a um a big part of the work that
eventually goes into making these large
models so
we can also have more people involved in
this in that
if you consider a conceptual like
working at
the machine intelligence research
institute would be something
that not many people can do there is i
believe
actually that miri already have um
sufficient funding and just have a
really hard time
finding people who have the specific uh
taste or intuition
for how to do this kind of conceptual
research and i think just
it might just be a fact of the world
that most people can't do that
whereas lining their early superhuman
models might be
far more possible
mainstream machine learning safety is
very technical
and mathematically dense and if as we
expect
they are actually not really considering
the real problems
then having the alignment
community becoming larger
is another reason why that's good
so what kind of skills uh
and tasks are required to align narrowly
superhuman models
well a lot of software engineering and
machine learning engineering
needs to take place we need to have a
lot of work dealing with
humans in that human feedback
is likely to be an essential part of it
and there will be a lot of problems that
are
hard work but not but doesn't require
the very best minds in some way
aj's intuition is that if we have proper
institutions for getting people
up to speed on this and sufficient
funding
then it might be possible for someone
who doesn't really know much about
ai safety but is strong on a software
engineer to start
working productively with this within a
half to a full year or something like
that
at the am are these a number of possible
objections and responses to these
objections
then the first is how would this address
the treacherous turn
and it's just quite uncertain
and there's some weakness of of this
research agenda
that it doesn't explicitly address us
there are many research agendas which
does not
for instance measuring machine learning
safety
doesn't really help either there are
some ways
you can imagine that transparency would
be a part of alignment and
in that case um that's something that
would help
um we might see many treacherous turns
somewhere in particular if we go looking
for them
which is something that would be
a big part of this particular um
a big part of this particular research
agenda um
and of course if we build some
institutions that
work hard with these kind of problems
then having those
ready when the first mini treacherous
turns happen
will be very beneficial
doesn't this feel suspiciously close to
just profit maximizing
well somewhat but there are a number of
key differences
for instance if the most cost effective
thing to do
is to make the models bigger which it
seems to be in many cases
then fine-tuning them might be um
and trying to align them might not be
the most commercially valuable thing to
do
we are not trying to find the easiest
problem that we can
uh but rather the ones where this
research intended fits best where
alignment
is most interesting and um
we want to learn general models even if
the main specific
techniques are available so that's
another reason why
this differs from just profit maximizing
so we want to have a community where uh
most people who are doing this work is
doing it for altruistic reasons
and this is legible to analogous
and and things like that
so if we are closely but not quite
optimizing for usefulness
then that doesn't seem obviously
neglected because a lot of people want
something that's useful
well in adjacent estimate most
mainstream machine learning and ai
projects are either extremely domain
specific
or they're focused on just scaling big
generic models
so this kind of thing in between that we
are want to do
might be somewhat detected still um
an example would be the paul christian
paper learning to summarize from human
feedback
which was from christiano's obviously
motivated by long-term
existential risk um and that's something
that we would not have seen
um created or at least not have seen as
early created
were not for this kind of uh
motivation and it matters a lot who is
doing this research
and why they're doing this research not
just that it gets done
um and right now the aich community has
a problem in that there isn't really a
very strong community on
uh like mentoring and um uh
no building origins um and no one to
uh like a pipeline for how to uh
get into this kind of technical
alignment work
um again says that she
strongly suspects that something like
this sandwiching
is just not being done in any commercial
setting
then there is the objection that this
would increase investment in scaling ai
and this was actually quite close to one
objection that was raised
um in the discussion of the previous
video
last session um so will this increase
investment
well right now we are seeing investments
booming
and we expect this will continue and
since ai is in general uh
much larger than ai alignment it and
it's growing faster
the jr gets gives us two orders of
magnitude
and that means that even if adding one
dollar to the alignment
builds up a height then it needs to
build up less than
this 100 times as much
according to ai investment in order for
this to shift the balance between
capability work
and alignment work
and there's a good reason that to
believe that if
this is successful then this will shift
and substantial number of
marginal dollars from making the models
larger
to making the more fine-tuned if we can
show that this actually works
but here aj knows that
this kind of research could generate
exciting demos and that's certainly
something that is possible
and something that potentially could
could increase investment in scaling ai
so this research agenda
could be paired down right there is uh
the last part of the project generation
uh formula was uh to stamp out the
pathologies
and why not just focus on that
well this is also a valuable thing to do
but if you don't have the other
alignment
this is basically just robustus
reliability work
and
this is something that is hard to do in
a general sense
in that one a lot of reliability work
assumes
that there is a human judge which at
some point can decide whether
an action is good or bad and it's not
really neglected there is a lot of work
being done
because obviously if you want to build
autonomous weapons then reliability is
really important
uh fairness accountability and
conspiracy of
yeah under which a lot of robustness
work is being done
also if you want to build a subculture
or brain around a life
then that's not easier than focusing on
the same as others are doing right
so um adjacent sentence my best guess is
that it would actually be harder to tell
which people working in the robustness
space
are optimizing for reducing long-term
existential risk from ai versus for
profit
i think that's probably true but it's
also a very very small configuration
i'm a bit confused about why idea thinks
this is this is really important
because sure it matters
who and why they do the research but it
also matters a lot that the research is
being done
so another option we could do instead of
this research agenda is to focus
on evaluating and testing the existing
candidate long-term solution that we
already have
for instance paul christiano's agenda um
so um i i tried to describe paul
christiana's agenda really crisply and
i couldn't actually find a really crisp
description of
paul christiana's agenda right we have
the iterative distillation and
amplification
we have the aic via debate we have a
number of
these kind of headings but um i i
haven't actually seen
a really crisp description of this
agenda and
and that might actually be an important
problem because
right now are somewhat influx they are
not described there's a lot
and there are also other problems with
it in that
we uh we might be more interested in
finding new solutions
rather than testing the ones that we
already have
so what is the current state of opinion
on this word
in this section uh um describes what
other people think
about this which is of course always a
bit uh
a bit dangerous but um i think she tries
to do it fairly
so if along people who are alignment
researchers
working on large models they are
probably really uh
enthusiastic about this perhaps the best
kind of work they personally
could do um i think this
might be somewhat sociological um
paul cristiano is really optimistic
about this
because it's so scalable and this is
something that
matters a lot for cristiano people who
are working on conceptual research
are perhaps less excited about this
research agenda
than paul cristiano um rohingya
is more optimistic than the machine
intelligence research institute
but still not convinced that this is
actually literally the best
some people are relatively cautiously
optimistic including esa caskey and evan
um we will get more into in particular
india's
classics comments uh next week and i
should also say that relative
optimistic in this case might mean that
uh
i expect that uh elias here koski
believes that
most other alignment research is
actually wasted so if this is
even being relatively optimistic that
might also
uh be consistent with him saying that
this is almost certainly a waste of time
so what are the takeaways what are the
next steps
well
in particular whether this could be
harmful uh whether there are other
things that might be more interesting
and in particular how to deal with the
treacherous term
because this is this seems like one of
the weaknesses of this particular
research agenda
that we don't really have a strong
attack on literature
and he believes that if you are in a
position where you can start traffic
like this
then you should start and as examples of
people who might be a
principal investigator so the people at
university who decide
what to research and see researchers who
have the freedom to
just do them practice do whatever they
want
they should also be able to um
people in this kind of situation should
basically just start
but open philanthropy is not so listing
uh
grant applications for people who want
to do this right now and
they instead say people who might be in
a position
to do this kind of research in a couple
of years should be on the lookout for
opportunities
but there are no precise additional
opportunities right now that is all for
today
thank you and see you next week


AI That Doesn't Try Too Hard - Maximizers and Satisficers

hi so way back when I started this
online air safety videos thing on
computer file I was talking about how
you have a problem when you maximize
just about any simple utility function
the example I used was an AI system
meant to collect a lot of stamps which
works like this the system is connected
to the Internet and for all sequences of
packets it could send it simulates
exactly how many stamps would end up
being collected after one year if it
sent those packets it then selects the
sequence with the most stamps and sense
that this is what's called a utility
Maximizer and it seems like any utility
function you give this kind of system as
a goal it does it to the max utility
maximizers
tend to take extreme actions they're
happy to destroy the whole world just to
get a tiny increase in the output of
their utility functions so unless the
utility function lines up exactly with
human values their actions are pretty
much guaranteed to be disastrous
intuitively the issue is that utility
maximizers have precisely zero chill to
anthropomorphize horribly they seem to
have a frantic obsessive maniacal
attitude we find ourselves wanting to
say look could you just dial it back a
little can you just relax just a bit so
suppose we want a lot of stamps but like
not that many it must be possible to
design a system that just collects a
bunch of stamps and then stops right how
can we do that well the first obvious
issue with the existing design is that
the utility function is unbounded the
more stamps the better with no limit
however many stamps it has it can get
more utility by getting one more stamp
any world where humans are alive and
happy is a world that could have more
stamps in it so the maximum of this
utility function is the end of the world
let's say we only really want a hundred
stamps so what if we make a bounded
utility function that returns whichever
is smaller the number of stamps or 100
getting a hundred stamps from ebay gives
100 utility converting the whole world
into stamps also gives 100 utility this
function is totally indifferent between
all outcomes that contain more than a
hundred stamps so what does a Maximizer
of this utility function actually do now
the system's behavior is no longer
really specified it will do one of the
things that results in a hundred utility
which includes a bunch of perfectly
reasonable behaviors that the programmer
would be happy with
and a bunch of apocalypse is and a bunch
of outcomes somewhere in between
if you select at random from all courses
of action that result in at least 100
stamps what proportion of those are
actually acceptable outcomes for humans
I don't know probably not enough this is
still a step up though because the
previous utility function was guaranteed
to kill everyone and this new one has at
least some probability of doing the
right thing but actually of course this
utility Maximizer concept is too
unrealistic even in the realm of talking
about hypothetical agents in the
abstract in the field experiment our
stamp collector system is able to know
with certainty exactly how many stamps
any particular course of action will
result in but you just can't simulate
the world that accurately it's more than
just computationally intractable it's
probably not even allowed by physics
pure utility maximization is only
available for very simple problems where
everything is deterministic and fully
known if there's any uncertainty you
have to do expected utility maximizing
this is pretty straightforwardly how
you'd expect to apply uncertainty to
this situation the expected utility of a
choice is the utility you'd expect to
get from it on average so like suppose
there's a button that flips a coin and
if its tail's you get 50 stamps and if
it's heads you get 150 stamps in
expectation this results in a hundred
stamps right it never actually returns
100 but on average that's what you get
that's the expected number of stamps to
get the expected utility you just apply
your utility function to each of the
outcomes before you do the rest of the
calculation so if your utility function
is just how many stamps do I get then
the expected utility of the button is
100 but if your utility function is
capped at a hundred for example then the
outcome of winning one hundred and fifty
stamps is now only worth a hundred
utility so the expected utility of the
button is only 75 now suppose there were
a second button that gives either eighty
or ninety stamps again with 50/50
probability this gives 85 stamps in
expectation and since none of the
outcomes are more than 100 both of the
functions value this button at 85
utility so this means the agent with the
unbounded utility function would prefer
the first button with its expected 100
stamps but the agent with the bounded
utility function would prefer the second
button since its expected utility of 85
is higher than the
buttons expected utility of 75 this
makes the bounded utility function feel
a little safer in this case it actually
makes the agent prefer the option that
results in fewer stamps because it just
doesn't care about any stamps past 100
in the same way let's consider some
risky extreme stamp collecting plan this
plan is pretty likely to fail and in
that case the agent might be destroyed
and get no stamps but if the plan
succeeds the agent could take over the
world and get a trillion stamps an agent
with an unbounded utility function would
rate this plan pretty highly the huge
utility of taking over the world makes
the risk worth it but the agent with the
bounded utility function doesn't prefer
a trillion stamps to a hundred stamps
it only gets 100 utility either way so
it would much prefer a conservative
strategy that just gets a hundred stamps
with high confidence but how does this
kind of system behave in the real world
where you never really know anything
with absolute certainty the pure utility
Maximizer that effectively knows the
future can order a hundred stamps and
know that it will get 100 stamps but the
expected utility maximize it doesn't
know for sure the seller might be lying
the package might get lost and so on so
if the expected utility of ordering a
hundred stamps is a bit less than 100 if
there's a 1% chance that something goes
wrong and we get 0 stamps then our
expected utility is only 99 that's below
the limit of 100 so we can improve that
by ordering some extras to be on the
safe side maybe we order another 100 now
our expected utility is 99.99 still not
a hundred so we should order some more
just in case now we're at 99.9999 the
expected value of a utility function
that's bounded at 100 can never actually
hit 100 you can always become slightly
more certain that you've got at least
100 stamps better turn the whole world
into stamps because hey you never know
so an expected utility Maximizer with a
bounded utility function ends up pretty
much as dangerous as one with an
unbounded utility function ok what if we
try to limit it from both sides like you
get a hundred utility if you have a
hundred stamps and zero otherwise now
it's not going to collect a trillion
stamps just to be sure it will collect
exactly 100 stamps but it's still
incentivized to take extreme actions to
be sure that it really does have a
hundred like turning the whole world
into elaborate highly
stamp counting and recounting machinery
getting slightly more utility every time
it checks again it seems like whatever
we try to maximize it causes problems so
maybe we could try not maximizing maybe
we could try what's called satisficing
rather than trying to get our utility
function to return as higher value as
possible and expectation what if we set
a threshold and accept any strategy that
passes that threshold in the case of the
stamp collector that would look like
look through possible ways you could
send out packets calculate how many
stamps you'd expect to collect on
average with each strategy and as soon
as you hit one that you expect to get at
least 100 stamps just go with that one
this satisficer seems to get us to about
where we were with the pure utility
Maximizer with a bounded utility
function it's not clear exactly what it
will do except that it will do one of
the things that results in more than a
hundred stamps in expectation which
again includes a lot of sensible
behaviors and a lot of apocalypses and a
lot of things somewhere in between since
the system implements the first
satisfactory strategy it finds the
specific behavior depends on the order
in which it considers the options what
automated use well one obvious approach
is to go with the simplest or shortest
plans first after all any plan that
takes over the world probably requires
much more complexity than just ordering
some stamps on eBay but consider the
following plan get into your own source
code and change yourself from a
satisficer into a Maximizer all you're
doing there is changing a few lines of
code on your own system so this is a
pretty simple plan that's likely to be
considered fairly early on it might not
be simpler than just ordering some
stamps but that's not much reassurance
the more challenging the task we give
our AGI the more likely it is that it
will hit on this kind of self
modification strategy before any
legitimate ones and the plan certainly
satisfies the search criteria if you
change yourself into a Maximizer that
Maximizer will predictably find and
implement some plan that results in a
lot of stamps so you can tell that the
expected stamp output of the become a
Maximizer plan is satisfactorily high
even without knowing what plan the
Maximizer will actually implement so
satisficers kind of want to become
maximizes which means that being a
satisficer is unstable as a safety
feature it tends to uninstall itself so
to recap a powerful utility maximized
with an unbounded utility function is a
guaranteed apocalypse with a bounded
utility function it's better in that
it's completely indifferent between
doing what we want and disaster but we
can't build that because it needs
perfect prediction of the future so it's
more realistic to consider an expected
utility Maximizer which is a guaranteed
apocalypse even with a bounded utility
function now an expected utility
satisficer
gets us back up to in difference between
good outcomes and apocalypses but it may
want to modify itself into a Maximizer
and there's nothing to stop it from
doing that so currently things aren't
looking great but we're not done people
have thought of more approaches and
we'll talk about some of those in the
next video
I want to end the video with a big thank
you to all of my wonderful Patriots
that's all of these great people right
here in this video I'm especially
thanking Simon strand card thank you so
much you know thanks to your support I
was able to buy this boat for this I
bought a green screen actually but I
like it because it lets me make videos
like this one that I put up on my second
channel where I used GPT to to generate
a bunch of fake YouTube comments and
read them that video ties in with three
other videos I made with computer file
talking about the ethics of releasing AI
systems that might have malicious uses
so you can check all of those out
there's links in the description thank
you again to my patrons and thank you
all for watching I'll see you next time


Experts' Predictions about the Future of AI

hi there's a lot of disagreement about
the future of AI but there's also a lot
of disagreement about what the experts
think about the future of AI I sometimes
hear people saying that all of this
concern about AI risk just comes from
watching too much sci-fi and the actual
AI researchers aren't worried about it
at all when it comes to timelines some
people will claim that the experts agree
that AGI is hundreds of years away
prediction as they say is very difficult
especially about the future and that's
because we don't have data about it yet
but expert opinion about the future
exists in the present so we can do
science on it we can survey the experts
we can find the expert consensus and
that's what this paper is trying to do
it's called when will a I exceed human
performance evidence from AI experts so
these researchers from the future of
humanity Institute at the University of
Oxford the AI impact project and Yale
University ran a survey they asked every
researcher who published in ICML or nips
in 2015
those two are pretty much the most
prestigious AI conferences right now so
this survey got 352 of the top AI
researchers and asked them all sorts of
questions about the future of AI and the
experts all agreed that they did not
agree with each other and Robert Aumann
didn't even agree with that there was a
lot of variation in people's predictions
but that's to be expected
and the paper uses statistical methods
to aggregate these opinions into
something we can use for example here's
the graph showing when the respondents
think will achieve high level machine
intelligence which is defined as when
unaided machines can accomplish every
task better and more cheaply than human
workers so that's roughly equivalent to
what I mean when I say super
intelligence you can see these gray
lines show how the graph would look with
different randomly chosen subsets of the
forecasts and there's a lot of variation
there but the aggregate forecast in red
shows that overall the experts think
we'll pass 50% chance of achieving high
level machine intelligence about 45
years from now well that's from 2016 so
more like 43 years from now and they
give a 10% chance of it happening within
nine years which is seven years now so
it's probably not too soon to be
concerned about it a quick side point
about surveys by the way in a 2010 poll
44% of Americans said that they
supported homosexuals serving openly in
the military in the same poll 58% of
respondents said
they supported gay men and lesbians
serving openly in the military
implicitly fourteen percent of
respondents supported gay men and
lesbians but did not support homosexuals
something similar seems to be going on
in this survey because when the
researchers were asked when they thought
all occupations would be fully automated
all defined as for any occupation
machines could be built to carry out the
task better and more cheaply than human
workers they gave their 50% estimate at
a hundred and twenty two years compared
to forty five for high-level machine
intelligence these are very similar
questions from this we can conclude that
Aix PERTs are really uncertain about
this and precise wording in surveys can
have a surprisingly big effect on the
results figure two in the paper shows
the median estimates for lots of
different a AI milestones this is really
interesting because it gives a nice
overview of how difficult a AI
researchers expect these different
things to be for example human level
Starcraft play seems like it will take
about as long as human level laundry
folding also interesting here is the
game of go remember this is before
alphago the AI experts expected go to
take about 12 years and that's why
alphago was such a big deal it was about
eleven years ahead of people's
expectations but what milestone is at
the top what tasks do the AI researchers
think will take the longest to achieve
longer even than high-level machine
intelligence that's able to do all human
tasks that's right it's AI research
anyway on to questions of safety and
risk this section is for those who think
that people like me should stop making a
fuss about AI safety because the AI
experts all agree that it's not a
problem first of all the AI experts
don't all agree about anything but let's
look at the questions this one asks
about the expected outcome of high-level
machine intelligence the researchers are
fairly optimistic overall giving on
average a 25% chance for a good outcome
and a 20% chance for an extremely good
outcome but they nonetheless gave a 10%
chance for a bad outcome and 5% for an
outcome described as extremely bad for
example human extinction 5% chance of
human extinction level badness is a
cause for concern moving on this
question asks the experts to read
Stewart Russell's argument for why
highly advanced AI might pose a risk
this is very close
related to the arguments I've been
making on YouTube it says the primary
concern with highly advanced AI is not
spooky emergent consciousness but simply
the ability to make high quality
decisions here quality refers to the
expected outcome utility of actions
taken now we have a problem
one the utility function may not be
perfectly aligned with the values of the
human race which are at best very
difficult to pin down to any
sufficiently capable intelligent system
will prefer to ensure its own continued
existence and to acquire physical and
computational resources not for their
own sake but to succeed in its assigned
tasks a system that is optimizing a
function of n variables where the
objective depends on a subset of size K
less than n will often set the remaining
unconstrained variables to extreme
values if one of those unconstrained
values is actually something we care
about the solution may be highly
undesirable this is essentially the old
story of the genie in the lamp or The
Sorcerer's Apprentice or King Midas you
get exactly what you asked for not what
you want so do the AI experts agree with
that
well 11% of them think no it's not a
real problem 19 percent think no it's
not an important problem but the
remainder 70% of the AI experts agree
that this is at least a moderately
important problem and how much do the AI
experts think that society should
prioritize AI safety research well 48%
of them think we should prioritize it
more than we currently are and only 11%
think we should prioritize it less so
there we are
AI experts are very unclear about what
the future holds but they think the
catastrophic risks are possible and that
this is an important problem so we need
to do more AI safety research
I want to end the video by saying thank
you so much to my excellent patreon
supporters these people and in this
video I'm especially thanking Jason hice
who's been a patron for a while now
we've had some quite interesting
discussions over a patreon chat been fun
so thank you Jason and thank you all for
watching I'll see you next
[Music]


Quantilizers: AI That Doesn't Try Too Hard

hi so way back in the before time
i made a video about maximizers and
satisfices
the plan was that was going to be the
first half of a two-parter now i did
script out that second video
and shoot it and even start to edit it
and then
certain events transpired and i never
finished that video so that's what this
is
part two of a video that i started ages
ago
which i think most people have forgotten
about so i do recommend going back
and watching that video if you haven't
already or even re-watching it to remind
yourself so i'll put a link to that in
the description and with that here's
part two
take it away past me hi in the previous
video we looked at utility maximizers
expected utility maximizers and
satisfices
using unbounded and bounded utility
functions a powerful utility maximizer
with an unbounded utility function
is a guaranteed apocalypse with a
bounded utility function it's better
in that it's completely indifferent
between doing what we want and disaster
but we can't build that
because it needs perfect prediction of
the future so it's more realistic to
consider an expected utility maximizer
which is a guaranteed apocalypse even
with a bounded utility function
now an expected utility satisficer gets
us back up to indifference between good
outcomes and apocalypses
but it may want to modify itself into a
maximizer and there's nothing to stop it
from doing that the situation
doesn't look great so let's try looking
at something completely different let's
try to get away from this utility
function stuff that seems so dangerous
what if we just tried to directly
imitate humans if we can get enough data
about human behavior
maybe we can train a model that for any
given situation
predicts what a human being would do in
that scenario if the model's good enough
you've basically got a human level agi
right
it's able to do a wide range of
cognitive tasks just like a human can
because it's just
exactly copying humans that kind of
system won't do a lot of the dangerous
counterproductive things that a
maximizer would do simply because a
human wouldn't do them
but i wouldn't exactly call it safe
because a perfect imitation of a human
isn't safer than the human it's
perfectly imitating and humans aren't
really safe
in principle a truly safe agi could be
given just about any level of power and
responsibility
and it would tend to produce good
outcomes but the same can't really be
said for humans and an imperfect human
imitation would almost certainly be even
worse
i mean what are the chances that
introducing random errors and
inaccuracies to the imitation
would just happen to make it more safe
rather than less still
it does seem like it would be safer than
a utility maximizer
at least we're out of guaranteed
apocalypse territory but the other thing
that makes this kind of approach
unsatisfactory is
a human imitation can't exceed human
capabilities by much
because it's just copying them a big
part of why we want agi in the first
place
is to get it to solve problems that we
can't you might be able to run the thing
faster to allow it more thinking time or
something like that but
that's a pretty limited form of super
intelligence and you have to be very
careful with anything along those lines
because
it means putting the system in a
situation that's very different from
anything any human being has ever
experienced your model might not
generalize well to a situation so
different from anything in its training
data
which could lead to unpredictable and
potentially dangerous behavior
relatively recently a new approach was
proposed called quantalizing the idea is
that this lets you combine human
imitation and expected utility
maximization
to hopefully get some of the advantages
of both without all of the downsides
it works like this you have your human
imitation model given a situation
it can give you a probability
distribution over actions that's like
for each of the possible actions you
could take in this situation
how likely is it that a human would take
that action so in our stamp collecting
example that would be
if a human were trying to collect a lot
of stamps how likely would they be to do
this action
then you have whatever system you'd use
for a utility maximizer
that's able to figure out the expected
utility of different actions
according to some utility function for
any given action it can tell you
how much utility you'd expect to get if
you did that so in our example that's
how many stamps would you expect this
action to result in so for every action
you have these two numbers
the human probability and the expected
utility quantalizing
sort of mixes these together and you get
to choose how they're mixed with a
variable that we'll call
q if q is zero the system acts like an
expected utility maximizer
if it's one the system acts like a human
imitation by setting it somewhere in
between
we can hopefully get a quantizer that's
more effective than the human imitation
but not as dangerous as the utility
maximizer
so what exactly is a quantizer let's
look at the definition in the paper
a q quantilyzer is an agent that when
faced with a decision problem
returns a random action in the top q
proportion of some base distribution
over actions
sorted by the expected utility achieved
if that action is executed
so let's break this down and go through
how it works step by step
first we pick a value for q the variable
that determines how we're going to mix
imitation and utility maximization let's
set it to 0.1 for this example
10 now we take all of the available
actions
and sort them by expected utility so on
one end you've got the actions that kick
off all of the crazy extreme utility
maximizing strategies
you know killing everyone and turning
the whole world into stamps all the way
down
through the moderate strategies like
buying some stamps
and down to all of the strategies that
do nothing and collect no stamps at all
then we look at our base distribution
over actions what is that
in the version i'm talking about we're
using the human imitation system's
probability distribution over actions
for this
so our base distribution is how likely a
human is to do each action
that might look something like this no
human is ever going to try the wacky
extreme maximizing strategies so our
human imitator gives them a probability
of basically zero then there are some
really good strategies that humans
probably won't think of but they might
if they're really smart or lucky
then a big bump of normal strategies
that humans are quite likely to use that
tend to do okay
then tailing off into less and less good
strategies and
eventually stupider and stupider
mistakes the humans are less and less
likely to make
then what we do is we find the point in
our action list
such that 10 of the probability mass is
on the high
expected utility side so that's what q
is really changing it's where we make
this cutoff
note that it's not ten percent of the
actions that would be over here
it's ten percent of the probability mass
then we throw away everything on the
right
all the stupid and useless choices we
set them to zero and we keep the top ten
percent
now this is no longer a valid
probability distribution because it only
sums up to 0.1
so we multiply all of these by 10 so
that the whole thing sums to 1 again
and that's our final probability
distribution which we sample from to get
our chosen action
so let's look at some different actions
here and see how they do
consider something like misremember your
credit card details
and keep trying to order stamps with the
wrong number and you can't figure out
why it's not working
a human is reasonably likely to do that
not very likely
but we've all met people who point is a
pure human imitation
might do that but the expected utility
maximizer can see that this results in
very few stamps
so it ends up low on the list and
doesn't make the 10 cutoff
so there are lots of mistakes that a
human imitation might make that a
quantalizer won't
and note that for our stamp collecting
utility function the worst case is zero
stamps
but you could imagine with other utility
functions a human imitator could make
arbitrarily bad mistakes that a
quantizer would be able to avoid
now the most common boring human
strategies that the human imitator is
very likely to use
also don't make the cut off a 50
quantilizer would have a decent chance
of going with one of them
but a 10 quantizer aims higher than that
the bulk of the probability mass for the
10
quantilyzer is in strategies that a
human might try
that works significantly better than
average so the quantalizer is kind of
like a human on a really good day
it uses the power of the expected
utility calculation to be more effective
than a pure imitation of a human
is it safe though after all many of the
insane maximizing strategies
are still in our distribution with
hopefully small but still non-zero
probabilities
and in fact we multiplied them all by 10
when we renormalized if there's some
chance
that a human would go for an extreme
utility maximizing strategy
the 10 percent quantilizer is 10 times
more likely than that
but the probability will still be small
unless you've chosen a very small value
for q
your quantalizer is much more likely to
go for one of the reasonably high
performing human plausible strategies
and what about stability satisficers
tend to want to turn themselves into
maximizes does a quantizer have that
problem
well the human model should give that
kind of strategy a very low probability
a human is extremely unlikely to try to
modify themselves into an expected
utility maximizer to better pursue their
goals
humans can't really self-modify like
that anyway but a human might try to
build an expected utility maximizer
rather than trying to become one that's
kind of worrying since
it's a plan that a human definitely
might try that would result in extremely
high expected utility
so although a quantalizer might seem
like a relatively safe system
it still might end up building an unsafe
one so how's our safety meter looking
well it's progress let's keep working on
it
some of you may have noticed your
questions in the youtube comments being
answered by a mysterious bot named
stampy the way that works is stampy
cross posts youtube questions
to the rob miles ai discord where me and
a bunch of patrons discuss them and
write replies
oh yeah there's a discord now for
patrons thank you to everyone on the
discord who helps reply to comments
and thank you to all of my patrons all
of these amazing people
in this video i'm especially thanking
timothy lillarcrap
thank you so much for your support and
thank you all for watching
i'll see you next time
you


Empowerment: Concrete Problems in AI Safety part 2

hi this is part of a series about the
paper concrete problems in nai safety
which looks at preventing possible
accidents in AI systems last time we
talked about avoiding negative side
effects and how one way of doing that is
to create systems that try not to have
too much impact to not change the
environment around them too much
this video is about a slightly more
subtle idea than penalizing impact
penalizing influence so suppose we have
a robot it's a cleaning robot so it's
got a mop and a bucket and an apron I'm
trying something new here there with me
so the robot knows that there's a mess
over here that it needs to clean up but
in between the robot and the mess is the
server room which is full of expensive
and delicate equipment now if an AI
system doesn't want to have a large
impact it won't make plans that involve
tipping the bucket of water over the
service but maybe we can be safer than
that we might want our robot to not even
want to bring the bucket of water into
the server room to have a preference for
going around it instead we might want it
to think something like not only do I
not want to have too big of an impact on
my surroundings I also don't want to put
myself in a situation where it would be
easy for me to have a big impact on my
surroundings how do we formalize that
idea well perhaps we can use information
theory the paper talks about an
information theoretic metric called
empowerment which is a measure of the
maximum possible mutual information
between the agents potential future
actions and the potential future State
that's equivalent to the capacity of the
information channel between the agents
actions and the environment ie the rate
that the agents actions transmit
information into the environment
Manen bits the more information an agent
is able to transfer into their
environment with their actions the more
control they have over their environment
the more empowered the agent is so if
you're stuck inside a solid locked box
your empowerment is more or less zero
none of the actions you can take will
transmit much information into the world
outside the box but if you have the key
to the box your empowerment is much
higher because now you can take actions
that will have effects on the world at
large you've got options people have
used empowerment as a reward for
experimental AI systems and it makes
them do some interesting things like
picking up keys avoiding walls
even things like balancing an inverted
pendulum or a bicycle you don't have to
tell it to keep the bike balanced it
just learns that if the bike falls over
the agent actions will have less control
over the environment so it wants to keep
the bike upright so empowerment is a
pretty neat metric because it's very
simple but it captures something that
humans and other intelligent agents are
likely to want we want more options more
freedom more capabilities more influence
more control over our environment and
maybe that's something we don't want our
AI systems to what maybe we want to say
clean up that mess but try not to gain
too much control or influence over your
surroundings don't have too much
empowerment that could make the robot
sink if I bring this bucket of water
into the server room
I'll have the option to destroy the
service so I'll go around to avoid that
empowerment okay so now we're at that
part of the video what's wrong with this
why might it not work pause the video
and take a second to think well there
are a few problems one thing is that
because it's measuring information we're
really measuring precision of control
rather than magnitude of impact as an
extreme example suppose you've got your
robot in a room and the only thing it
has access to is a big button which if
pressed will blow up the moon that
actually only counts as one bit of
empowerment the buttons either pressed
or not pressed the moon is exploded or
not two choices so one bit of
information one bit of empowerment on
the other hand if the robot has an
ethnic cable that's feeding out lots of
detailed debug information about
everything the robot does and that's all
being logged somewhere that's loads of
information transfer loads of mutual
information with the environment so
loads of empowerment the robot cares way
more about unplugging the debug cables
than anything to do with the button and
then you have another possible problem
which is perverse incentives okay so
this button is only one bit of
empowerment nowhere near as big a deal
as the debug cable but the robot still
cares about it to some extent and wants
to avoid putting itself in this
situation where it can plow up the moon
however if it finds itself already in a
situation
it has one bit of empowerment because of
this button the easiest way to reduce
that is by pressing the button once the
buttons press to the moon is blown up
the button doesn't work anymore so the
robot then has basically zero bits of
empowerment it's just in a box with an
unconnected button and now it's content
but it's managed to make itself safe it
finally has no influence over the world
so yeah in this admittedly contrived
scenario an empowerment reducing robot
will unplug its debug cable and then
blow up smooth that's not safe behavior
what did we think this might be a good
idea well it just makes the point that
even very simple information theoretic
metrics can describe interesting
abstract properties like influence over
the environment so maybe doing something
a little bit cleverer than just
penalizing empowerment might actually be
useful a more sophisticated metric a
better architecture around it you know
there could be some way to make this
work so this is an area that's probably
worth looking into by AI safety
researchers so that's all for now next
thing in the paper is multi agent
approaches which should be really
interesting make sure to subscribe and
hit the bell if you want to be notified
when that's out also make sure you're
subscribed to computer fault because I'm
probably going to make some new videos
there as well since some of the multi
agent stuff is closely related to the
stop button problem that I already
talked about so it might be nice to put
those together thanks for watching I
hope to see you next time in this video
I want to thank Eastern flicked who
supported me on patreon sinb April thank
you and thank you again to all of my
wonderful patreon supporters all of
these people I've been setting up a room
in my house to be a full time studio
I might make a behind-the-scenes video
about that soon oh and I've got these
pictures that I drew while making this
video which I have no use for now does
anyone want them got the incense weird
sunglasses but yeah I can probably post
them to support it any one more
100 it's time click I was close


The other "Killer Robot Arms Race" Elon Musk should worry about

hi this is a new thing I'm trying where
I made quick topical videos about AI
safety in the news
so somebody linked me a news article
today and the headline is Tesla's Elon
Musk leads Oh terminator picture
everyone playing the AI news coverage a
drinking game has to take a shot you
know the rules picture of the Terminator
means you go to drink a shot out of a
glass shaped like a skull where was I oh
right yeah so the headline Tesla's Elon
Musk leads tech experts in demanding end
to killer robots arms race this is
really interesting because it looks like
it's going to be really relevant to what
this channel is about AI safety research
but I read it and it's actually nothing
to do with that the headline is much
more literal than I expected it's about
an actual arms race for actual killer
robots ie it's about using the UN to
form international agreements about not
deploying autonomous weapon systems I
thought it was going to be about the
other arms race that might cause robots
to kill us okay so if there's one thing
that I hope this channel and my computer
file videos have made clear it's that
any safety is a difficult problem that
quite reasonable looking AGI designs
generally end up going horribly wrong
for subtle and hard to predict reasons
developing artificial general
intelligence needs to be done very
carefully double and triple-checking
everything running it passed lots of
people ironing out all of the possible
problems before the thing is actually
switched on to do this safely is going
to take a lot of smart people a lot of
patience diligence and time but whoever
makes AGI first has a huge advantage
since it probably creates a new period
of much faster progress everyone wants
to be first to publish new scientific
results anyway but the chances are that
there are really no prizes for being the
second team to develop AGI even if
you're just a few months behind a lot
can change in a few months in a world
with AGI so there's an arms race going
on between different teams different
companies different countries to be the
first to develop AGI but developing AGI
safely takes a lot of care and patience
and time you see the
problem here the team that gets there
first is probably not the team that's
spending the most time on ensuring
they've got the very best AI safety
practices the team that gets there first
is probably going to be rushing cutting
corners and ignoring safety concerns
hey remember a while back I said I was
going to make a video about why I think
Elon Musk's approach to AI safety might
end up doing more harm than good I guess
this is that so there's a school of
thought which says that because AGI is a
very powerful technology it will grant
whoever controls it a lot of power so
firstly it's important that the people
in control of it are good people and
secondly we want as many people as
possible to have it so that the power is
democratized and not concentrated in the
control of a small elite the best of the
available alternatives is that we
achieve democratization of AI technology
meaning that no one company or small set
of individuals has control over advanced
AI technology and starting from that
fairly reasonable school of thought this
is a very good and valuable thing to do
but there's another school of thought
which says that because making AGI is
nowhere near as difficult as making the
safe AGI the bigger risk is not that the
wrong person or wrong people might make
an AGI that's aligned with the wrong
human interest but that someone might
make an AGI that's not really aligned
with any human interests at all thanks
to this arms race effect that will make
people want to cut corners on alignment
and safety that possibility looks much
more likely and the thing is the more
competitors there are in the race the
more of a problem this is if there are
three companies working on a GI maybe
they can all get together and agree to a
strict set of safety protocols that
they're all going to stick to it's in
everyone's interest to be safe as long
as they know their competitors will be
safe as well but if there are a hundred
or a thousand groups with a shot at
making AGI there's really no way you're
going to be able to trust every single
one of them to stick to an agreement
like that when breaking it would give
them an advantage so it might be
impossible to make the agreement at all
and whoever spends the least time on
safety has the biggest advantage from
the perspective of this school of
thought making AGI developments open and
available to as many people as possible
might be the last thing we want to do
maybe once AGI start
closer we'll find a way to keep AI
research limited to a small number of
safe careful organizations while making
AI safety research open and widely
available I don't know but this might be
a situation where total openness and
democratization is actually a bad idea
Elon Musk himself has said that AI is
potentially more dangerous than nukes
and I want to make it clear that I have
enormous respect for him but I just want
to point out that with a not that huge
change in assumptions the open approach
starts to look like saying nukes are
extremely dangerous so we need to
empower as many people as possible to
have them
and to end the video a big thank you to
all of my excellent patreon supporters
these people in this video I especially
want to thank Michael grease I recently
uploaded a video that had an audio
problem in it the sound was only coming
out of one ear but because my patreon
supporters get access to every video I
make before the rest of the world does
one of my supporters Jimmy Gowen spotted
it and pointed it out and I was able to
fix that and then I was able to use
patreon money to get a new pair of
earphones so I want to say again how
grateful I am to all of you you really
are a tremendous help to the channel


What's the Use of Utility Functions?

okay so in some of the earlier computer
file videos I talked about utility
functions or objective functions and we
got a lot of different comments relating
to that idea one thing people said was
well surely this kind of monomaniacal
following of a single utility function
at the cost of everything else is really
the cause of the problem in the first
place why even use a utility function or
maybe have several conflicting ones that
interact with each other or something
like that some people asked why do we
assume that an AI will have a utility
function in the first place aren't we
making a pretty strong assumption about
the design of the AI when in fact we
don't know how it would be implemented
humans don't have explicit utility
functions that they consult when they're
making their decisions a lot of
different AI designs people are working
on now don't have utility functions
coded into them explicitly so why make
that kind of unwarranted assumption so
before we get into that let's just go
over what a utility function is okay so
here's the earth or the universe it can
be in any one of several different
states so let's just look at three
possible world states in this world I'm
enjoying a pleasant cup of tea in this
world I've run out of milk so the tea
isn't quite how I'd like it to be and in
this world I'm being stung by noon two
wasps we want some way of expressing
that I have preferences over these world
states some of them are better for me
than others so a utility function is a
function which takes as an argument a
world state and outputs a number saying
broadly speaking how good that world is
for me how much utility I get from it so
in this example perhaps a nice cup of
tea is worth 10 a a mediocre cup of tea
is worth nine and the wasps are minus a
thousand but Rob you might say that's
way too simple I care about all kinds of
things and what I what I love about the
world is is complex and nuanced you
currently steal everything down to just
a single number on each world state note
with that attitude you can and you kind
of have to but let's just forget about
the numbers for now and talk about
preferences
let's make some basic assumptions about
your preferences the first one is that
you do have preferences given any two
states of the world you could decide
which one you would prefer to happen or
you could be indifferent but there's
this basic trilemma here for any pair of
world states a and B either a is
preferable to B or B is preferable to a
or you're indifferent between a and B it
doesn't matter to you which one happens
always exactly one of these things is
true hopefully that should be obvious
but just think about what it would mean
for it not to be true like what would it
mean to not prefer A to B not prefer B
to a and also not be indifferent between
DNA similarly what would it mean to
prefer A to B and simultaneously prefer
B to a if you're faced with a choice
then between a and B what do you do the
second basic assumption is transitivity
so you have this relation between States
is preferable to and you assume that
this is transitive which just means that
if you prefer A to B and you prefer B to
C then you prefer a to C again this
seems intuitively pretty obvious but
let's look at what it would mean to have
intransitive preferences let's say I
prefer being an Amsterdam to being in
Beijing and I prefer being in Beijing to
being in Cairo and I prefer being in
Cairo to being in Amsterdam what happens
if I have these preferences let's say I
start out in Amsterdam I prefer being in
Cairo so I get on a plane and I flied to
Qatar now I'm in Cairo and I find
actually I prefer being in Beijing so I
get on the plane I fly to Beijing I'm
now in Beijing and I say oh you know
actually I prefer to be in Amsterdam so
I slide around stir done and now I'm
back where I started and hey what do you
know I prefer to be in Cairo so you can
see that if your preferences are
transitive you can get sort of stuck in
a loop where you just expend all of your
resources flying between cities or in
some other way changing between options
and this doesn't seem very smart so if
we accept these two pretty basic
assumptions about your preferences then
we can say that your preferences are
coherent you may have noticed there
something else that has these two
properties which is the greater than
relation on numbers for any two numbers
a and B either a is greater than B B is
greater than a or a and B are equal and
if a is greater than B and B is greater
than C then a is greater than C the fact
that preferences and numbers share these
properties is relevant here so if your
preferences are coherent they'll define
an order overworld States that is to say
given your preferences you could take
every possible world state and arrange
them in order of how good they offer you
there will be a single ordering
overworld States you know there aren't
any loops because your preference is a
transitive now if you have an ordering
of world States there will exist a set
of numbers for each world state they
correspond to that ordering perhaps you
could just take them all in order and
give each one a number according to
where it falls in the ordering so those
are your utility values for any coherent
preferences there will be a set of
utility values that exactly represents
it and if you have a utility value on
every world state well there will be
some function which takes in world
States and return to their utility
values and that's your utility function
so if you have consistent preferences
you have a utility function but Rob you
may say I don't have consistent
preferences I'm a human being my
preferences are all over the place
that's true human beings do not reliably
behave as though they have consistent
preferences but that's just because
human intelligence is kind of badly
implemented our inconsistencies don't
make us better people it's not some
magic key to our humanity or secret to
our effectiveness or whatever it's not
making us smarter or more empathetic or
more ethical it's just making us make
bad decisions talking about utility
functions is actually a way of assuming
very little about the design of an AI
other than assuming that it has coherent
goal directed behavior it doesn't matter
how its implemented if it's effective at
navigating the world to get what it once
it will behave as though it has a
particular utility function and this
means if you're going to build an agent
with coherent goal directed behavior
you'd better make sure it has the right
utility function
[Music]
just wanted to say thank you to my
patreon supporters the three people who
somehow managed to support me before I
even mentioned in the video that I was
setting up a patreon and I especially
want to thank Chad Jones who's pledged
$10 a month thank you so much it really
means a lot to me that there are people
out there who think what I'm doing is
worth supporting
so thanks again