What Can We Do About Reward Hacking?: Concrete Problems in AI Safety Part 4

hi welcome back to this video series on
the paper concrete problems in AI safety
in the previous video I talked about
reward hacking some of the ways that can
happen and some related ideas like
adversarial examples where it's possible
to find unexpected inputs to the reward
system that falsely result in very high
reward partially observed goals where
the fact that the reward system has to
work with imperfect information about
the environment incentivizes an agent to
deceive itself by compromising its own
sensors to maximize its reward wire
heading where the fact that the reward
system is a physical object in the
environment means that the agent can get
very high reward by directly physically
modifying the reward system itself and
good hearts law the observation that
when a measure becomes a target it
ceases to be a good measure there's a
link to that video in the doobly-doo and
I'm going to start this video with a bit
about responses and reactions to that
one there are a lot of comments about my
school exams example of good hearts law
some people sent me this story that was
in the news when that video came out a
school kicked out some students because
they weren't getting high enough marks
it's a classic example in order to
increase the school's average marks ie
the numbers that are meant to measure
how well the school teaches students the
school refused to teach some students
this is extra funny because that's
actually my school that's the school I
went to anyway some people pointed out
that the school exams example of good
hearts law has been separately
documented as Campbell's law people
pointed out other related ideas like the
Cobra effect perverse incentives the law
of unintended consequences there are a
lot of people holding different parts of
this elephant as it were all of these
concepts are sort of related and that's
true of the examples I used as well you
know I used a hypothetical super mario
bot that exploited glitches in the game
to give itself maximum score as an
example of reward hacking but you could
call it wire heading since this score
calculating part of the reward system
sort of exists in the game environment
and it's sort of being replaced the
cleaning robot with a bucket on its head
was an example of partially observed
goals but you could call it an instance
of good hearts law since amount of mess
observed stops being a good measure of
amount of mess there is once it's used
as a target so all of these examples
might seem quite different on the
surface but once you
look at them through the framework of
reward hacking you can see that there's
a lot of similarities and overlap the
paper proposes ten different approaches
for preventing reward hacking some will
work only for certain specific types for
example very soon after the problem of
adversarial examples in neural networks
was discovered people started working on
better ways to train their nets to make
them resistant to this kind of thing
that's only really useful for
adversarial examples but given the
similarities between different reward
hacking issues maybe there are some
things we could do that would prevent
lots of these possible problems at once
careful engineering is one you're AI
can't hack its reward function by
exploiting bugs in your code if there
are no bugs in your code there's a lot
of work out there on ways to build
extremely reliable software like there
are ways you can construct your programs
so that you're able to formally verify
that their behavior will have certain
properties you can prove your software's
behavior with absolute logical certainty
but only given certain assumptions about
for example the hardware that the
software will run on those assumptions
might not be totally true especially if
there's a powerful AGI doing its best to
make them not true of course careful
engineering isn't just about formal
verification there are lots of different
software testing and quality assurance
systems and approaches out there and I
expect there's a lot a AI safety can
learn from people working in aerospace
computer security anywhere that's very
focused on writing extremely reliable
software it's something we can work on
for AI in general but I wouldn't rely on
this as the main line of defense against
reward hacking another approach is
adversarial reward functions so part of
the problem is that the agent and the
reward system are in this kind of
adversarial relationship it's like
they're competing the agent is trying to
trick the reward system into giving it
as much reward as possible when you have
a powerful intelligent agent up against
a reward system that's a simple passive
piece of software or hardware you can
expect the agent to reliably find ways
to tricks subvert or destroy the reward
system so maybe if the reward system
were more powerful more of an agent in
its own right it would be harder to
trick and more able to defend itself if
we can make the reward agent in some
sense smart
or more powerful than the original agent
it could be able to keep it from remote
hacking though then you have the problem
of ensuring that the reward agent is
safe as well the paper also mentions the
possibility of having more than two
agents so that they can all watch each
other and keep each other in check
there's kind of an analogy here to the
way that the legislative the executive
and the judiciary branches of government
keep one another in check ensuring that
the government as a whole always serves
the interests of the citizens but
seriously I'm not that hopeful about
this approach firstly it's not clear how
it handles self improvement you can't
have any of the agents being
significantly more powerful than the
others and that gets much harder to make
sure of if the system is modifying and
improving itself and in general I don't
feel that comfortable with this kind of
internal conflict between powerful
systems it kind of feels like you have a
problem with your toaster incinerating
the toast with a flamethrower so you add
another system that blasts the bread
with a powerful jet of liquid nitrogen
as well so that the two opposing systems
can keep each other in check
instead of systems that want to hack
their award functions but figure they
can't get away with it I'd rather a
system that didn't want to mess with its
reward function in the first place this
next approach has a chance of providing
that and approach the paper calls model
look ahead you might remember this from
a while back on computer file you have
kids right suppose I were to offer you a
pill or something you could take and
this pill will like completely rewire
your brain so that you would just
absolutely love to like kill your kids
right whereas right now what you want is
like very complicated and quite
difficult to achieve and it's hard work
for you and you probably never gonna be
done you're never gonna be truly happy
right in life nobody is you can't
achieve everything you want I said this
case it just changes what you want what
you want is to call you kids and if you
do that you will be just perfectly happy
and satisfied with life right okay you
want to take this pill I don't want to
do it and so not only will you not take
that pill you will probably fight pretty
hard to avoid having that pill
administered to you yeah because it
doesn't matter how that future version
of you would feel you know that right
now you love your kids
and you're not gonna take any action
right now which leads to them coming to
harm so it's the same thing if you have
an AI that for example values stamps
values collecting stamps and you go oh
wait hang on a second I didn't quite do
that right let me just go in and change
this so that you don't like stamps quite
so much it's gonna say but the only
important thing is stamps if you change
me I'm not gonna collect as many stamps
which is something I don't want there's
a general tendency for AGI to try and
prevent you from modifying it once it's
running in almost any situation being
given a new utility function is gonna
write very low on your current utility
function so there's an interesting
contrast here the reinforcement learning
agents we're talking about in this video
might fight you in order to change their
reward function but the utility
maximizes we were talking about in that
video might fight you in order to keep
their utility function the same the
utility maximizers reasons that changing
its utility function will result in low
utility outcomes according to its
current utility function so it doesn't
allow it to change but the reinforcement
learners utility function is effectively
just to maximize the output of the
reward system so it has no problem with
modifying that to get high reward model
look EDD tries to give a reinforcement
learning agent some of that
forward-thinking ability by having the
reward system give rewards not just for
the agents actual actions and it's
observations of actual world states but
for the agents planned actions and
anticipated future world states so the
agent receives negative reward for
planning to modify its reward system
when the robot considers the possibility
of putting a bucket on its head it
predicts that this would result in the
messes staying there and not being
cleaned up and it receives negative
reward teaching it not to implement that
kind of plan there are several other
approaches in the paper but that's all
for now
[Music]
I want to end with a quick thanks to all
my wonderful Patriots these people and
in this video I'm especially thanking
the Guru of vision I don't know who you
are or why you call yourself that
but you've supported the channel since
May and I really appreciate it anyways
thanks to my supporters and that I'm
starting up a second channel for things
that aren't related to AI safety I'm
still going to produce just as much AI
safety content on this channel and I'll
use the second channel for quicker fun
stuff that doesn't need as much research
I want to get into the habit of making
lots of videos quickly which should
improve my ability to make quality AI
safety content quickly as well if you
want an idea of what kind of stuff I'm
gonna put on the second channel check
out a video I made at the beginning of
this channel titled where do we go now
there's a link to that video in the
description along with the link to the
new second channel so head on over and
subscribe if you're interested and of
course my patrons will get access to
those videos before the rest of the
world just like they do with this
channel thanks again and I'll see you
next time the interests of Pacific right
take 17


Are AI Risks like Nuclear Risks?

hi in the previous video I talked about
this open letter which has been signed
by a lot of prestigious people which
talks about how there are risks and
possible problems associated with AI and
it says we need to do more thinking
about the future of AI technologies and
the impact they're going to have on
society but it's worth noting that in
those 8,000 or so signatories there's
quite a broad range of opinions about
the specific nature of these problems
which problem is most important what the
timescales are so to say all of these
people are concerned in some sense with
AI safety is not to say that they all
agree with each other and the document
the open letter links to lists a whole
bunch of different things and I'm going
to talk about some of those now so the
first thing is economic impact things
like technological unemployment you know
what if AI put someone out of a job or
the effects on economic inequality in
the sense that if the AI technologies
produce a huge amount of new wealth not
everyone is going to benefit from that
wealth equally and this can increase
inequality which can cause other
problems also some people are concerned
that even if we manage to set up a
situation where people don't need to be
employed in order to get the resources
that they need to live there's a
question of in a world in which nobody
has to work what are people actually
doing with their time it just measure
how do people get a feeling of
achievement and ideally they shouldn't
just have a feeling of achievement they
should have actual achievement but it's
hard to have actual achievement in a
world in which you're outperformed in
every way by machines there's concern
about AI ethics things like driverless
vehicles this has been done to death but
that's just because it's an interesting
problem that we're not sure how to deal
with yet so you know you have a
self-driving car it's in some scenario
where it has to hit one person or the
other and then the question is how did
it make that decision this is a
philosophical question right it's an
instance of the trolley problem in real
life so there are really two questions
here the first is the ethical one how
should we design our AI systems to make
ethical decisions in these situations
the interesting thing to me about this
is that humans are routinely in these
situations right car crashes happen
regularly but we don't have time to make
ethical decisions if you're in this type
of scenario in which you are forced to
hit someone and you have to choose no
one's going to blame you for choosing
one person or the other because you're
in the middle of a car crash like almost
by definition you have no time to think
whereas with a self-driving car the
decision of how
you want your car to behave needs to be
made beforehand with all the time in the
world and no excuses so what's new isn't
the decision itself so much as its
having enough time to think also a
sidenote prediction I'm calling it now
when self-driving cars are common we
will have a problem with morons
deliberately jumping in front of them
for fun anyway that's one question
the other question is legal what should
the law be about this who's liable the
person in the car the person who owns
the car the company that wrote the
software and practically speaking the
way the software actually gets written
will be determined by the legal question
on the ethical one there's concerns
about military use of this technology
autonomous weapons systems and so on the
ethics of that and some people are
worried about discrimination in machine
learning systems that these systems we
build to process people's data and make
decisions about insurance premiums
hiring decisions loads all kinds of
things these systems can be used for
they may end up being racist or sexist
things like that which is another
potential issue there's also privacy
concerns people are worried about the
ability of AI systems to deduce private
information from the vast amounts of
public information available to them the
classic example of this is the young
woman who started receiving coupons for
baby food and other baby related stuff
from her supermarket and her father
stormed in there to complain but she
actually was pregnant and the
supermarket's product recommendation
algorithms had noticed that before her
own father I don't even know if that
story is true but it illustrates the
point AI may be able to discover things
about you that you didn't intend to make
public so all of those are problems that
can happen when AI systems are working
more or less as they were intended to
work then you have the stuff that's more
what I've been talking about in earlier
videos which is problems that happen
when AI systems have unexpected
unintended behavior which is harmful
either during development like in the
stop button problem where I'm talking
about this robot that wants to make you
a cup of tea and ends up running over a
child who's in the way that kind of
accident or problems that only happen
once the system is deployed and it
starts behaving in ways that were
unexpected and have negative
consequences this can already be a real
problem with existing machine learning
systems and as those systems get more
powerful and more influential and get
relied on more and more those problems
are going to become more and more
important and then you have the question
of general super intelligence
intelligent systems that dramatically
outperform human
elegance across a wide range of domains
that can be a maximally bad problem so
when people say oh yes I'm concerned
about possible problems with AI they're
really talking about a very wide range
of possible problems here I'm going to
go back to the nuclear analogy I've used
in the past imagine if some point around
the turn of the last century a load of
scientists got together and all signed a
letter saying we're concerned about the
risks of nuclear material and so
thousands of people signed this thing
and then it turns out that some of those
people are talking about like radiation
poisoning you know Marie Curie died
young as a consequence of the radiation
she was exposed to while doing her
research but other people are talking
more about things like their diamond
core this plutonium core which caused a
lot of problems during the Manhattan
Project where there were accidents that
resulted in sudden powerful bursts of
radiation which gave acute radiation
poisoning to the scientist conducting
the experiment and anyone who happened
to be standing nearby or you have other
people are more concerned with risks
associated with nuclear power you know
you have this nuclear waste generated
that needs to be disposed of that's a
problem or you have the other problem
with nuclear power which is the
possibility of meltdowns which can be
disastrous and then you have other
people saying well never mind all that
what about nuclear weapons right this is
the big problem what if people build
nuclear weapons that can kill millions
of people and then if they proliferate
then that can cause vast problems for
Humanity like global thermonuclear war
that's an issue and then you know beyond
that you also have concerns like okay
during the Manhattan Project there was
concern that the Trinity nuclear test
might ignite in the atmosphere the
principle of this is quite similar to
the way that a hydrogen bomb works you
have a standard atom bomb next to a
certain amount of hydrogen and then I'm
not a physicist but more or less the
energy from the fission reaction the
explosion of the atom bomb kicks off a
fusion reaction in the hydrogen the
hydrogen atoms fuse and give off a
tremendous amount of energy which is
another chain reaction it's the same
thing that's happening in the Sun right
turning hydrogen into helium but of
course in the Sun heavier elements are
also fused up to and including for
example nitrogen so there was some
concern when people were developing the
atom bomb what if we kickstart a fusion
chain reaction in nitrogen because the
atmosphere is like 78% nitrogen there's
a chance that we turned the entire
atmosphere in
to a thermonuclear bomb effectively from
the first time that this issue was
raised it seems pretty clear that it
wasn't going to happen in the sense that
the amount of energy given off by an
atom bomb or a hydrogen bomb is just not
big enough to do this their
understanding of physics at the time
pointed to it being nowhere near enough
energy but at the same time I don't
believe they'd ever actually fuse
nitrogen in the lab so they didn't know
for sure exactly what conditions caused
nitrogen to fuse and also they'd never
set off a nuclear bomb before so they
didn't know for sure exactly how much
energy they were going to get out of it
so there was a nonzero probability from
their perspective when they set off the
trinity explosion that it would end all
of humanity instantaneously more or less
right there and then so before they did
it they run the numbers again in a few
different ways and they looked at it
very carefully and made very very sure
ahead of time that they were very
confident this wasn't going to be an
issue but they weren't going to ignite
the atmosphere so people have had
various concerns about nuclear material
ranging from oh if you work with it you
might get radiation poisoning too if you
screw it up you may destroy all life on
the planet forever in a giant fire
explosion and so getting people to sign
a letter that says we're concerned about
nuclear material would cover a broad
range of possibilities so I like a
nuclear analogy here because it helps me
explain something about a paper that I'm
going to go through soon concrete
problems in AI safety because it's
concerned about accidents specifically
the paper is looking at unintended and
harmful behavior that may emerge from
poor design of real-world AI systems and
another way that this is similar to
nuclear research is that the type of
knowledge you need in order to prevent
something like the diamond core problem
of something going supercritical and
dumping out a huge amount of radiation
in your lab is the same kind of
understanding of radioactivity and
fulfill material in general that you
need in order to understand how nuclear
bombs work and make sure you don't set
off one of those by accident or to
understand what storage and
transportation technology is necessary
for nuclear waste or how to prevent
meltdown like a general good
understanding of nuclear physics will
help you with protecting yourself from
getting radiation poisoning and also
hopefully protect you from accidentally
igniting the atmosphere and I think it's
the same in AI and I think that's part
of what concrete problems in AI safety
is trying to do it's trying to bring
together the people who are concerned
about possibly igniting the atmosphere
the real epochal
superintelligence problems and the
people looking at the more
run-of-the-mill
what if my robot ignores my stock
partner type problems and it's trying to
point out areas of research that we
could look into that would actually
provide progress on both of those things
where there's overlap things that we can
study that would help us with current
existing AI systems but that may also
help avoiding these huge global scale
super intelligence related problems
problems which like igniting the
atmosphere may or may not at this point
actually turn out to be a real problem
but which are still definitely worth
looking into because these stakes are so
unbelievably high I want to end this
video with a quick thank you to my
excellent patreon supporters these
wonderful people around here I'm always
looking for ways to make good use of
your support to improve my videos and I
recently bought my own autocue which I
think works really well I've put up a
behind-the-scenes video on patreon of
how I did that you want to check that
out but in this video I especially want
to thank Stefan scales skills who
supports me for $20 a month thank you
I've actually just made a new $20 reward
level you can go check that out and let
me know what you think thanks again and
I'll see you next time


Status Report

hi welcome back sorry for the delay in
uploading
instead of our regularly scheduled video
today we're going to have a quick
comparison between this my laptop and if
a paper a4 notebook which one's lighter
and they feel about the same call that a
draw which one's thinner about the same
which one's better at editing videos
that are more than a few minutes long I
hang on let me check yeah neither of
them can do that at all I have the first
video all shot and worked out has been
trying to edit it for a long time now
but it's just not going to work the
laptop is not powerful enough so long
story short I am building a PC parts
around the way I'm going to delete this
video obviously when I get the actual
one up but just have patience and I'll
see you again soon
[Music]
quiet on set


How to Keep Improving When You're Better Than Any Teacher - Iterated Distillation and Amplification

hi today we're going to talk about
iterated distillation and amplification
so let's say we want to play go and we
want to be really good at it so we're
trying to create a function which if
we're given a go board in a particular
state we'll take that board state as
input and return a very high-quality
move that you can make from that
position what we're trying to create is
a policy a function which Maps States
onto actions suppose that what we have
though is something just slightly
different suppose what we have is our
intuition about moves which takes a
board state and it gives us for each of
the possible moves we could make some
sense of how good that move would be we
can think of this as an action value
function which assigns a real number to
each move which represents how good we
think that move is alternatively we can
think of it as outputting a distribution
over all possible moves so for a human
player this represents your intuition
the go player looks at the board state
and says it looks like maybe this move
might be good this move probably is a
bad idea
this one looks ok you could also have a
neural network which takes the board
state and a possible move as input and
outputs how good it thinks that move is
ok so how do you get the understanding
of the game that allows you to evaluate
moves well as a human you can study the
rules and watch some games played by
people who are better at go than you are
if you have a neural network then it's
also fairly straightforward you can
train the network with a large number of
high quality human played games until
its output gives a good prediction of
what a skilled human would do so
strictly speaking in that case the
network isn't evaluating how good a move
is it's evaluating how likely a good
player would be to make that move but
that can be used as a proxy for how good
the move is once we have this action
value function there's a pretty obvious
way to turn it into a policy which is
just Arg max you look at all of the
moves with your intuition or evaluate
them all with the network find the
best-looking move the move that's
highest rated and use that but if you
have more time to think or more
computational resources you can do
better rather than just going with your
first instinct about what you think is
good
you could play forward a few moves in
your head you might think okay from this
board state it looks like this move
would be good what does the board look
like if I play that and then
you can apply your action value function
again from the perspective of your
opponent often there'll be more than one
move that looks promising so you might
want to consider some of the
best-looking moves and then apply your
action value function again to think
about how you might respond to each of
them and so on exploring the tree so
what you're effectively doing here is
tree search right you have a game tree
of possible moves and you're searching
through it deciding which branches to
search down using your action value
function you can keep doing this for
however much time you have it might be
that you think far enough ahead that you
actually get to the end of the game and
you can see that some move is clearly
good because it wins you the game or
some other move is clearly bad because
it causes your opponent to win the game
well you might just look a little bit
ahead and try to evaluate where you are
you might look at the general quality of
the moves that you have available to get
a feel for whether this is a state you
want to be in or one you want to avoid
and after you've done all this thinking
you might have learned things that
contradict your initial intuition there
might be some move which seemed good to
you when you first thought of it but
then once you actually think through
what your opponent would do if you made
that move and what you would do in
response to that and so on that the move
actually doesn't look good at all so you
do all of this thinking ahead and then
you have some way of taking what you've
learned and getting a new set of ratings
for the moves you could make and this
can be more accurate than your original
action value function for a human this
is this kind of fuzzy process of
thinking about moves and their
consequences and in a program like
alphago or alpha zero this is done with
Monte Carlo tree search where there's a
structured way of extracting information
from this tree search process so there's
a sense in which this whole process of
using the action value function
repeatedly and searching the tree
represents something of the same type as
the original action value function it
takes a board state as input and it
gives you move evaluations it allows us
to take our original action value
function which on its own is a weak
player and by applying it lots of times
in this structured way we can amplify
that weak player to create a stronger
player so now our amplified action value
function is the same type of thing as
our unamplified one how do they compare
well the amplified one is much bigger so
it's more expensive
for a human it takes more thinking time
as a program it needs more computational
resources but it's also better than just
going with a single network or the
single human intuition it smooth
evaluations are more accurate so that's
pretty neat we can take a faster but not
very good player and amplify it to get a
more expensive but stronger player
there's something else we can do though
which is we can take what we've learned
as part of this process to improve our
original action value function we can
compare the outputs of the fast process
and the amplified version and say hmm
the quick process gives this move a high
rating but when we think it all through
with the amplified system it turns out
not to be a good move so where did the
quick system go wrong and how do we fix
it if you're a human you can maybe do
this explicitly perhaps you can spot the
mistake that you made that caused you to
think this was a good move and try to
keep it in mind next time you'll also
learn unconsciously your general pattern
matching ability will pick up some
information about the value of making
that kind of move from that kind of
position and with a neural network you
can just use the output of the amplified
process as training data for the network
as you keep doing this the small fast
system will come to reflect some of what
you've learned by exploring the game
tree so this process is kind of like
distilling down this big amplified
system into the quick cheap to run
system and the thing that makes this
really powerful is we can do the whole
thing again right now that we've got
slightly better intuitions or slightly
better weights for our network we can
then amplify that new action value
function and this will give us better
results firstly because obviously if
your movie valuations are more accurate
than before then the move evaluations at
the end of this process will be more
accurate than before better quality in
better quality out but secondly it also
allows you to search the tree more
efficiently if your intuitions about
move quality are better you can spend
more of your time looking at better
parts of the tree and less time
examining in detail the consequences of
bad moves that aren't going to get
played anyway so using the same extra
resources the new amplified system is
better than the previous amplified
system and that means that when it comes
to the distillation phase of learning
from the exploration there's more to
learn and your action value function can
improve again so it's a cycle with two
stages for
to amplify by using extra computational
resources to make the system more
powerful and then you distill by
training the fast system with the output
of the amplified system and then you
repeat so the system will keep on
improving so when does this process end
well it depends on your implementation
but eventually you'll reach a fixed
point where the fast system isn't able
to learn anything more from the
amplified system for simple problems
this might happen because the
unamplified system becomes so good that
there's nothing to be gained by the
amplification process if your action
value function always suggests the
optimal move then the amplified system
is always just going to agree and no
more learning happens for harder
problems though it's much more likely
that you'll reach the limits of your
action value function implementation you
hit a point where a neural network of
that size and architecture just isn't
able to learn how to be better than that
by being trained on amplified gameplay
as a human even if you could study go
for infinite time eventually you'll hit
the limits of what your brain can do the
point is that the strength of the end
result of this process isn't limited by
the strength of the initial action value
function the limit is determined by the
architecture it's a fixed point of the
amplification and distillation process a
version of alphago that starts out
trained on amateur level games might
take longer to train to a given level
than one that started out trained on
grandmaster level games but after enough
training they'd both end up around the
same strength and in fact alpha zero
ended up even stronger than alphago even
though it started from zero using no
human games at all so that's how you can
use amplification and distillation to
get better at go and why as a software
system you can keep getting better even
when you have no external source to
learn from even once you leave humans
behind and you're the best go player in
the universe so there's nobody who can
teach you you can still keep learning
because you can learn from the amplified
version of yourself ok so why is this
relevant fire-safety well we've just
talked about one example of iterated
distillation and amplification the idea
is actually much more general than that
it's not just for playing go and it's
not just for Monte Carlo tree search and
neural networks amplification might be
this kind of process of thinking ahead
if you're a human being it might be
Monte Carlo tree search or something
like it if you're a software system but
it might be something else if you are
for example an age
I it might involve spinning up lots of
copies of yourself to collaborate with
or delegate to so that the team of
copies can be better at solving the
problem then you would be on your own
for some types of problem it might just
involve running your mind at a faster
rate to work on the problem for a long
period of subjective time the core
characteristic is that amplification
uses the original process as a starting
point and applies more computational
resources to create a more powerful
agent in the same way distillation can
be any process whereby we compress this
more expensive amplified agent into
something that we can call cheaply just
as we call the original system for a
human playing go this can be the way
your intuition gets better as you play
for a neural network playing go we can
train the action value network to give
the same outputs as the tree search
process for an AGI it could involve the
AGI learning in whatever way it learns
how to predict and imitate the team of
copies of itself or the accelerator
version of itself or whatever the
amplified system is the core
characteristic is that the cheaper
faster agent learns to approximate the
behavior of the more expensive amplified
agent so these two processes together
define a way of training a stronger
agent from a weaker one the hope for
safety research is that we can find
designs for the amplify and distill
procedures which preserve alignment by
which I mean that if the agent we
amplify is aligned with our goals and
values then the amplified agent will be
aligned as well and if the amplified
agent is aligned then the agent we
distill it down to will be aligned as
well in the next video we'll talk about
some ideas for how this might be done
I want to end this video with a big
thank you to all of my wonderful patrons
that's all of these fantastic people
here who have been just so generous and
so patient with me thank you all so much
in this video I'm especially thanking
Sayed Polat who joined in December just
before the start of this gap in uploads
and the reason for that is I've recently
really had to focus on the road to AI
safety excellence the online course I've
been working on in fact the video you
just watched is the first lecture from
our module on AI da which hasn't been
released yet so I also want to thank
everyone at the Rays Project for their
work on the script and the research for
this video and really the whole raised
team I'm still making content just for
this channel as well and in fact I have
one that's nearly ready to go so look
out for that thanks again for watching
and I'll see you soon


AI? Just Sandbox it... - Computerphile

we see a lot of comments on your videos
about people who just say oh just simply
do this that will be the answer for all
of these problems yeah and I admire them
for getting stuck in and getting
involved but one thing that always
strikes me is people say just change
this bit of code or just change this
value and it strikes me that if we
invents a GRI or happen upon AGI the
reality is it's probably going to be in
your network they don't actually know
exactly how it's working anyway how are
you awesome that yeah I mean uh - yeah -
the people who think they solve it
either you're smarter than everyone else
who's thought about this problem so far
by a big margin or you're missing
something and maybe you should read some
more of what other people are thought
and you know learn more about the
subject because it's cool like I think
it's great that people come in and try
and solve it and try and find solutions
and we need that but the yeah the
problem is not a lack of ideas it's a
lack of good ideas this kind of coming
in from the outside and saying oh yeah
I've got it I figured out the solution
is obviously arrogance right but the
whole artificial general intelligence
thing is sort of an inherently arrogant
thing if it's quite cube ristic you know
I mean you talk about playing God making
God like this but also that that you oh
I've got it this is how you do it
sometimes that doesn't work that
approach yeah sometimes because you see
stuff that people have been too close to
the metal turning them I think that's
work close to the problem - right right
yeah sometimes you get too close to it
you can't see the picture you're inside
the frame whatever it's it's totally
possible that some random person is
going to come up with a real workable
solution and I would love I would love
that to happen I think that would be the
best because then everyone would have to
try and figure out how to cite a youtube
comment in a research paper I presume
there's already style advice for that
anyway but the problem is from the
outside view you get a million of these
right
and so you know a million minus one are
going to be not worth your time to read
which means even a good one isn't
actually worth your time to read our
balance because you can't you how you're
going to differentiate it so the only
thing you can do is get up-to-date on
the research read the papers read what
everybody else is doing make sure that
what you're doing is unique and then
actually write something down and send
it to a researcher you know write it
down properly and make it clear
immediately up front that you're
familiar with the existing work in the
field and then you know then maybe
you're in with a chance but what will
probably happen is in the process of all
about reading you realize your mistake
it's still worth doing you've learned
something you know this is part of why
AI safety is such a hard problem I think
in the sense that a problem can be quite
hard and you look at it and you can tell
it's quite hard a problem that's really
hard which is a problem you look at it
and then immediately think you've got a
solution and you don't because then you
don't even you're like the it's like if
you're like the sat-nav right you're
confidently with a wrong answer now
rather than at least being honestly
uncertain yeah like legibility in
machine learning systems is really low
right now get kind of black boxes right
they're not they're not legible and it
you can't easily tell what any given
part of it does or how it works and that
is a real problem for safety definitely
I think right now right now the stage
we're at with AI saving is we're trying
to specify any kind of safe agent which
is you know trying to build something
from the ground up so we'll be safe and
I think that's much easier than taking
some existing thing that works but isn't
safe and trying to make it safe I don't
think that approach to be honest is
likely to be fruitful I'll give a really
dodgy example of how this might kind of
be something who can get the grips with
which is the Star Wars scene where the
robots are given restraining bolts r2d2
says oh I can't do that unless you take
this restraining bolt off if you can
you might be able to say back in time
reporter we then promptly runs away and
so I guess you're too small to run away
I'm if I take this off yeah others like
retrofitting some kind of Australian
bulbs yeah I mean it so there's
different things right building an
unsafe AI and then trying to control it
against its will is idiotic I think
having some of those controls ways of
keeping the system you know limiting
what the system can do and stuff is
sensible but it's so much better to make
a system that doesn't want to do bad
things than to try and keep one in so
this news comes like the idea of old
karma just unbox it right yeah it's like
I mean constraining an AI necessarily
means outwitting it and so constraining
a superintelligence
means outwitting a super intelligence
which kind of just sort of by definition
is not a winning strategy you can't rely
on how waiting is it for intelligence
also it only has to get out once that's
the other thing if you have a super
intelligence and you've sort of put it
in a box so it can't do anything that's
cool maybe we could even build a box
that could successfully contain it but
now what we may as well just have a box
right there's no benefit to having a
super intelligence in a box if you can't
use it for anything it needs to be able
to do things AI properly properly
contained may as well just be a rock
right it doesn't do anything if you have
your AI you wanted to do something
meaningful so now you have a problem or
you've got something you don't know it's
been evident you don't know that what it
wants is what you want and you then need
to you presumably have some sort of
gatekeeper who it tries to says I'd like
to do this and you have to decide is
that something we want it to be doing
how the hell are we supposed to know I
mean how can we if we're outsmarted how
can we reliably differentiate actions we
want to allow it to take some actions we
don't and maybe the thing has a
long-term plan of doing a bunch of
things that we don't notice at the time
or a problem until it now then can get
out right
actually this speaks this speaks to a
more general thing which is there's
often a trade-off between safety and
effectiveness like with anything right
you're designing there's going to be
you're going to be trading off different
things against one another and often you
can trade in some effectiveness to get
some safety or vice-versa so some of the
things in this paper are like that where
the thing does become less powerful then
nai designed differently but it also
becomes safer you know that's always the
way it is it's just so where you put
your resources and suppose is not right
but it's it's kind of inherent to the to
the thing like I mean and this is true
of any tool right the more powerful the
tool is the more dangerous it is and if
you want to make a powerful tool less
dangerous one of the ways to do that is
going to involve making less powerful or
less flexible or less versatile or
something that's going to reduce the
overall effectiveness often as a tool in
exchange for more safety and it's the
same with AI and obviously going to be a
server for whatever product you're using
now anytime that Bob sends a little
message is going to go by this server by
definition because that's the thing that
relays the message to Alice it knows how
to communicate the valley you know it
knows what her phone number is it have
lists of your contacts and things you
know this is how it works this could be
a phone provider


Apply to Study AI Safety Now! #shorts

after an investigation of gpg4 AI
researchers at Microsoft suggested that
the language model along with others
like charge EBT and Google's Palm could
be considered an early form of
artificial general intelligence with GPT
4's ability to solve complex tasks
across domains such as mathematics
coding Vision medicine law and
psychology it may not be long until we
get fully fledged AGI and we are not
ready
in response to the growing need for AI
safety sarri mats an independent
research program is training the next
generation of AI safety researchers to
address existential threats from
Advanced AI at mats you'll dive into the
field of AI alignment through scientific
seminars and workshops get mentored by
experts and work amongst a network of
research peers there are multiple
research streams so Scholars can focus
on the alignment research agenda that
matches their interests and expertise
apply for the summer 2023 cohort by May
7th and to stay up to date on upcoming
AI safety courses and events check out
AI safety dot training where you can
subscribe for regular updates


121 - Artificial Stupidity

I was told you guys have access to it
beforehand so how many of you actually
read it I've written all details ok I
assume some of you didn't read it so
that is still a reason for me to present
about it I know my students never read
anything they don't even buy textbooks
so I assume if it's not in a
presentation you don't know about it if
you want to know more if you have
questions I'm happy if you follow me
email me elaborate but I'll try to kind
of do a self-contained presentation of
the topic ok so if you know history of
artificial intelligence you can trace a
lot of it to work of Alan Turing he
definitely had amazing contributions to
computer science but specifically an AI
is known for probably the most historic
test the so called imitation game and it
is probably the most popular way of
determining if a machine is intelligent
however what it really measures is how
human intelligent the machine is right
it's not helpful to be super intelligent
and make very intelligent answers which
quickly will be discovered to be
artificial so you have to you have to
make mistakes and typing you have to be
kind of slow and typing you have to not
be brilliant mathematically and Turing
himself talks about that in the paper
about the need for a machine to
recognize those limitations and to
integrate them into the answers but
surprisingly there's been very little
attention paid to this aspect so
everyone's trying to develop more
capable devices never looking at this
limit of human capability in any formal
way so what the paper does is kind of
formalize this notion and suggests that
it may be very useful to to look at it
some more so what is it like to be human
level intelligence but
not greater than that and how does that
apply to all sorts of different
subdomains mathematics psychology and so
not surprisingly there are fundamental
differences between capabilities of
people and machines we can concentrate
and few obvious ones but there are many
many others so for one computers a
graded computing not surprising you can
follow long chains of algorithms perform
very significant computations data-mine
people are not as great at that likewise
people have very limited memory
computers have almost perfect memory
whatever its long-term or short-term
they're much superior to us only have a
half amazing common sense they can
quickly figure out ambiguous sentences
and fuzzy visuals worst computers are
not that great at that with respect to
rationality we are capable of creating
very rational devices strictly
statistical analyzers whereas people are
usually not so rational and a number of
well-known biases so what is
specifically this concept of artificial
stupidity we want to study human
limitations and to formalize them in
different domains and to be able to
encode similar flaws similar properties
into a machine and purpose and of course
explain why it's useful but this is the
general idea so if you understand this
much unfortunately there is alternative
use of this term artificial stupidity is
kind of related to natural stupidity and
lots of funny jokes about that but we
explicitly mean this purification of
natural stupidity natural human limits
so why would you want to make your
computers dumber it seems like we
started at that point already
well there is quite a few useful
applications so obviously you needed to
pass the Turing test itself so if your
initial goal was what Turing hoped for
you need to understand what the limits
are in order to succeed there are also
quite a few applications in terms of
developing products so whatever you're
talking about domestic robots sex robots
whatever your flavor is it's wonderful
if they have good understanding of human
capabilities and can relate and can
interact in a kind of equal kind of way
if you designing games it's nice if you
actually have a reasonable chance of
competing in a game it makes it more fun
more interesting in general this
limiting of power differential between
people questions could be quite useful
for safety reasons and if you scale your
systems beyond what we have today you
trying to achieve super intelligence at
some point I think it would be necessary
for successful value alignment to
understand completely what it is you're
aligning to the model for for human
intelligence human sensory system and so
on so what we need to do this not so
much a lot of original work but maybe
collect relevant information from a
number of other fields we are interested
in research from psychology sociology
economics mathematics anything to do
with what are those limits in humans so
will we know this kind of limited
attention capacity limited recall
different interpretations of statistics
but there are also kind of interesting
observations we can make about human
visual system for example how many of
you can actually perceive the solution I
definitely see it moving ok so I shared
this on Twitter and I got something like
quarter million I wouldn't call them
likes but interactions of some kind
which gave
a grand total of one follower but people
seem to definitely understand this and
react in some interesting ways
whereas machine which would not have
this bargain their visual system would
not be able to relate and connect so
that's that's pretty much the example of
what we want we want full understanding
of what it's like to be biological here
there are some trivial examples that can
give you from different domains so the
famous one probably everyone heard who
took some psychology courses is this
magical number 7
speaking of short-term memory we usually
can remember plus or minus two seven
chunks of information so depending on
how you chunk it up it could be
character symbols word sentences but
usually that's the number and we see it
show up in many areas of life so for
example phone numbers typically have
seven digits because that's the best we
can do on average in terms of memorizing
them but such such constants are not
readily available if your programming in
AI and you would like to have access to
or what are the limits for different
human properties it's not something you
can quickly look up there is not an api
for it so that that seems to be the
limitation we want to address another
example which has been studied
extensively but again is not directly
available is a set of so called
cognitive biases there is quite a few of
them we try to not just list them but
explain how they may be useful in
specifically creating safer intelligent
systems what those limitations may imply
if you think about it AI started with a
lot of work and URIs --tx which were
exactly that shorthand limitations in
computation to improve efficiency
improve performance but then AI does it
we say okay it's a great heuristic
really efficient but then people do it
it's a horrible bias we need to fix it
but that's essentially we're talking
about the same same mathematical
approach to solving problems so what the
paper does it's not an expert
mental paper we didn't code anything
within running experiments but we
proposed this essentially direction for
doing additional work were from multiple
fields it's highly interdisciplinary we
want to understand physical cognitive
and cultural properties of people and
formalize them to make them easily
accessible to anyone programming AI so
here for example you have some extreme
properties of humans as physical systems
in the left bottom corner you can see an
example from recent experiments on moral
judgment about self-driving cars from
different people around the world and
there are some very difficult to predict
differences cultural differences who the
self-driving cars should sacrifice is it
young people old people men or women
poor or rich and things like that need
to be encoded for a system to to
understand what's going on and again I
don't think it's quite readily available
at this point so some examples of where
this has already been applied over we
will see it applied in the future games
is one example and pretty much all the
existing papers on this subject which is
like one or two on the main of games
where people quickly realize nobody
wants to play against God level AI it's
just not fun if you playing chess and it
destroys you every time it's boring so
you should be able to adjust level of
your non playing characters your
opponents and whatever it's from easy to
impossible or just make it sometimes
make mistakes it's definitely part of
modern game design where you have to
integrate such human limitations if it's
a shooter game you want the opponents to
miss you sometimes and so on another
example is specifically with chatbots
and we see it a lot for shared
competitions such as Lochner Prize
winning BOTS make a lot of mistakes they
pause the type of mistakes and I think
one of the best winning strategies is
just talkin I remain silent for a while
and seem very deep and interactive
interesting but it would be a bargain
code and people perceive it as quiet
high level performance so we saw this
exemplified with Chavez and more
recently we saw demonstrations of Google
duplex system and that's quite
impressive system very natural voice
allows you to make phone calls to
different businesses make appointments
but what may that sound very human is
all the kind of human like let me think
wait mmm sounds wish it made for no
reason other than to sound dumber than
it was so more human by extension so
that's I think something again we see
show up even an existing systems but
again without any formalization or what
are the optimal delays what we need to
say to sound even more human and so on
so that's some examples of application
the paper came out very recently already
started to gain some citations which is
nice to see it's not formally published
it's just print an archive but it
quickly went viral I don't know if it's
the catchy title over the official
stupidity of people actually think it's
a brilliant idea
but quite a few media outlets and a lot
of international coverage was given on
that and the comments for those
publications are called mine of
artificial and natural stupidity for
sure but quickly assemble the data set
from that so you kind of conclude what
we're trying to do is start this
direction of research will be collecting
necessary data from different domains
and human cognitive and physical
limitations different properties and
factors and the goal is to formalize it
make it available to researchers in
order to make more customized safe and
secure systems and with a long term plan
of assisting with value alignment of
PGI's super intelligent system
and better understanding humans in pasts
so with that I'm happy to switch to a
discussion mode and answer any questions
or recruit you guys to work on this
project as much as it works let's see if
I can switch stop sharing okay so I'd
like to say thank you to Romanian pulsky
for this presentation and then I'd like
to hear if there are any questions I
would like to ask how we also considered
artificial wisdom which would also kind
of apparent from artificial cleverness
because right now
what people are mostly working on is how
to make the AI clever and find any very
difficult baby complex algorithms that
solve some problems but maybe the
problems we have are actually stupid
problems and we have not solved them at
all so what about wisdom which is more
not about having some convoluted
solutions but instead changing core
viewpoint or rules I think I've missed
beginning of your question I got the end
of it but beginning was kind of cutoff
can you repeat that first so I was
asking have you talked about artificial
wisdom so we right now we were opposing
artificial cleverness with artificial
stupidity but I think there is third
component that and third I mentioned
which is artificial wisdom so we don't
have any convoluted algorithms instead
we have different viewpoints so we might
simply have very simple solutions or
very complex problems if we simply
change our viewpoint not sure I have a
brilliant idea here changing your
viewpoint as in changing our values to
make them easier to fulfill is that what
you have in mind
yes um but so what if we have some
artificial intelligence system that
helps us to change our viewpoint and
therefore if our problem solved without
huge side effects so figure something to
consider but it's also considered a
safety issue right if a system tries to
modify you in order to make its
processing more efficient there is a
danger of losing your utility function
you stopped being yourself right if you
are modified into someone who enjoys
paper clips and that's it
you can fail to preserve your identity
at least in the short run for sure yeah
I think there is some danger why I
thought that it would be useful is that
useful useful usually when people invent
some new technology they very soon
utilize it to the maximum extent over
the world and that's certainly dangerous
approach and then you have simply
keywords instead changing your viewpoint
you don't have to utilize new technology
with the maximum extent and I think for
example paper clipping will be still
utilizing the technology and what about
the cases where you only change the
viewpoint but do not utilize too much
technology in order to fulfill this you
knew you will point so in general I have
started thinking about values and in the
context of value alignment and just
because we call them values it doesn't
mean they are actually valuable right
they're pretty randomly assigned at
Birth based on your culture so there is
something to be investigated in terms of
can we be happier so as a foreigner in
the United States I'm quite happy to
enjoy unpopular sports and not care
about popular ones so I save on tickets
I don't have to go to Super Bowl it's
very good
I wonder if this can be streamlined
somehow but I haven't done any published
work on it I have a question then when I
looked at the list of biases
I thought of another candidate which is
that humans in general prefer to have
stories about what they do they don't of
course sometimes we act based on pure
intuition but we prefer to be able to
explain our actions to ourselves and to
others and this seems to be an important
issue in particular with the kind of AIS
we see neural networks that give
suggestions that are unexplainable
basically and we would prefer a safe
area to take actions to pass towards
actions that can explain just like
humans all right and that's another
subject I'm very interested in and
slowly collecting literature and
planning and doing things one thing to
observe so we have this expectation that
neural networks artificial deep neural
networks will be able to explain
themselves but that's not true of people
then we do experiments and split brain
patients for example will quickly
discover we mostly make up
excuses for why we do things after the
fact so it seems like we have a much
stronger requirement for eyes to be able
to explain things if they are truly
super intelligent and make decisions
based on let's say million factors with
the unique weights any explanation they
can give us would have to be dumbed down
 story or worse some sort of
mind hacking where they just make you
happy with the explanation so even that
part could be quiet unsafe and I don't
think you can understand the full reason
for something so so I think I really
liked about your paper more superior
than human intelligence but I still
think safety around even human level
intelligent AI humans in general can be
dangerous to me they're not safe I
wouldn't call them a safe intelligence
and even going one step down further
than that like what level of stupid AI
is like still safe
I can imagine of a lot of different
scenarios where the AI is sufficiently
stupid it's still dangerous to us so it
doesn't seem like those two vectors are
necessarily combined together what are
your thoughts on that space I agree
intro those stupid people acquired
dangerous I deal with them a lot and
they cause most of Constance in my life
I guess the difference is between
catastrophic damage because they can
outsmart everyone versus just kind of
screwing up locally and whatever you had
a car accident somebody died I think
there is a direct proportion between
intelligence and level of damage you can
cause as well as control ability level
of intelligence so for more intelligent
system is the more it is independent and
uncontrollable I don't know if you had a
chance to read my paper on differences
between AI safety and cybersecurity but
this safe human problem is something I
talk about explicitly and I'm trying to
reduce problem of making safe AI to the
problem of making a safe human which we
of course failed to do for millennia
with culture loss
prisons bribes you name it nothing
worked so far so it's really interesting
I haven't looked into this space and I
would definitely be open to
recommendations if anyone has it just
looking at definitions around safety I
think your point about humans not being
safe is kind of interesting I'm not sure
I you're pointing out I don't have a
good definition of what that means like
a very rigorous idea of what it means to
be safe so I'm definitely I would assume
this is the audience to difficut
recommendations in that space I don't
think that is a very formal definition
but just the idea of toss let's say we
want a very safe human to work with very
sensitive information so something like
an essay in Snowden right to have
polygraphs we had background checks
none of that work whatsoever so I have a
couple of related questions
so the first one would be if you were
able to implement artificial stupidity
into a into an AGI how would you keep it
stupid enough that it can't self modify
so that it's no longer artificially
stupid and yet intelligent enough that
it's still capable of doing real-world
work and the related question is if say
if we were able to successfully
implement artificials artificially
stupid AGI and yet we still had a
geopolitical system somewhat like our
current one what's to keep America from
making an AI with IQ 100 and then Russia
says we can out-compete America and
still be safe by making an AI with IQ
115 and then China says oh well we can
go up to 130 and still be fairly safe
and so on until we're just back to the
back to the paperclip maximizers right
so with self improvement then we talk
about artificial stupidity we are kind
of anchoring an average person so IQ of
100 that's actually quiet low if you
think about top machine learning
researchers people capable of modifying
code and improving it we're probably
looking at much higher levels on 3150
and so on so hopefully if it simply
would not be smart with those
restrictions to make those improvements
but we haven't gotten that far I mean at
this point I'm just happy if I find some
numbers to put in a dataset as far as
arm arms races between different
governments and governance of AI that's
a huge problem and unsolved one as far
as they can tell if problem of creating
a is huge like Manhattan Project you
need a government in in Google size
corporation maybe governments can agree
or sign something if a person can do it
in a garage and a laptop then it doesn't
really matter what
we on whatever standards we proposed
it's just going to be done immediately
by someone just you know get famous
though
alright financially so very very
difficult to say anything about control
from the governance point of view I have
a question it seems like many of the
axis that are currently pursue or
possibly in the future pursuing AGI
things like military intelligence
advertising giants like Google and
Facebook or researchers trying to the
Turing test and it seems like all these
these three kinds of projects have
deception of humans as a rather key
component you can if you want to do a
military intelligence project you need
to be able to explicitly model the AGI
needs to be able to model how to cheat
other people and advertising problem in
the same I guess and that seems very
problematic right huge so there is a few
directions I'm looking at with this and
one is artificial propaganda just kind
of optimizing effect on human psychology
from running lots of experiments and
seeing what works we kind of saw it with
influence on US election right you can
optimize its to trigger just a certain
set of responses and people from
unrelated data okay I like spicy food so
that means how they respond well to this
message another one I showed you this
visual illusion right so those are right
now designed by people for people but
what happens then machines are capable
of coming up with same type of illusions
but not just visual but in avid MA in
sexual allusions we had a paper actually
from the same group who's working on the
serial inputs for artificial neural
networks come up with some images for
people so they took a picture of the cat
flipped a few pixels now it looks like a
dog to most people so that's a proof of
concept but I think it's going to get to
the point where I show you a water
bottle and you go
for me Scott Alexander wrote a post for
Halloween about a fictional story about
some researchers finding a way to do
this
assistive statement I think they called
it which was a exceptional a short
statement that was exceptionally capable
of influencing humans it's a funny story
and you can post a link to that I would
definitely take a look I'm so behind in
my reading I need some super intelligent
health care to just some of those
examples of artificial stupidity that
you mentioned beginning there were
limitations built in and to the
performance of the machine and they were
essentially deceptive in nature weren't
they are things like you give example of
the duplex demonstration the hesitations
and the others and ours and so on which
are designed to to fool the other person
into thinking is a human they're dealing
with which you can justify as saying it
feels more natural but I hope that in
that in that one area that one limited
area I hope well except that a digital
assistant like that phony up to make an
appointment would announce that it was
digital and not human and in that case I
mean I don't really need it to stay home
and uh obviously I know but if it talks
it a hundred times
human rate it's got a talk at a rate
that I can understand and it's got to if
it presents some sort of logical
argument it's got to do it in steps but
uh that I can understand but from that I
don't mind
these artificial failings are rather
deceptive I think I hope that we don't
have too many of them
well they are by design kind of
artificial they don't have to be there
California just passed a law saying that
chatbots would have to self-identify as
bats so in a way there is some effort
but I think we need to do some studies
on psychology of human enjoyment of
those interactions it's quite possible
that we just relate better and enjoy
better conversing with other what we
think as humans and as the system's
become more easy more interactive more
humanoid bodies it will become probably
more so so right now we're talking about
just voice and the phone what if it's a
sex robot for example right you probably
want it to be pretty pretty good
imitation of a biological body to some
degree even if it didn't have to yes yes
I might prefer to for it to have a few
minutes again whatever could be just
energizer bunny yeah I have a quick
question in many of your proposals for
the artificial stupid ATI is not
completely clear to me whether you
expect it to have internet access
specifically in this paper in general in
this people to expect that I think most
about I don't think we explicitly
addressed it in my paper sorry I boxing
I strongly encourage no-one to connect
their AGI projects to Internet probably
ever but this can be decided maybe some
sort of limited North Korean internet
access
so how how promising do you think this
is for super intelligence because your
paper doesn't at the end you seem to
suggest that there's not really much of
a way to extend this but do you have any
sort of inkling for what you might do
and by the way I enjoy the paper it was
quite a lot I hadn't heard of thank you
so it's a contradiction in terms right
you can't be human level stupid and then
artificial is super intelligent so one
of the other but this is useful for the
new modeling people and you're trying to
create some sort of value alignment to
at least understand what are the values
why a values in that level and people
appreciate something in this range or
not so I think just having this
information would be a useful tool even
if it doesn't end up being some magical
bullet so it it seems to me that you're
talking you're really talking about two
distinct concepts here one is having
computers be able to model human flaws
for the purposes of understanding humans
or pretending to be humans or what have
you and then on the other hand you have
trying to actually limit the capability
of n AGI to human levels is there some
connection between those that are not
seeing or like well it's the same same
data set you would need same information
okay you could have multiple
applications for this data but you need
to collect some information on same same
entities would it be that common sense
is the intersection between artificial
stupidity and kind of wisdom in the
sense that the agents let's say humans
or robots don't too much too much of
their own thinking from down one hand so
if they are stupid in some sense but on
the other hand common sense even if it's
irrational and so on so it's stupid it's
somehow works so it has this was the
best of time so it is some kind of
wisdom it's practical
all right all this biases evolved
exactly because they work I mean we try
not to judge individuals based on
statistics for groups but it works if
you have no priors right that's why
those things exist hmm so I think being
artificially stupid and artificial
device is not opposite thinks it's the
same in some sense well if you look at
the human genius usually they brilliant
in one narrow area and really not that
capable and many others so it's possible
there is some buffer overflow from
genius to stupidity at some point but I
haven't looked into that I was more like
having in mind that every person who has
having common sense a very rich person
who is not too bright as this
conjunction of both stupidity and wisdom
they don't think too much but they do
things that work so we are very
resilient I will actually quite
impressed how terrible you can be at
making decisions and still kind of be ok
as a human right you still have a job
you're still probably not dead in most
cases so I see people make mistake after
mistake and yeah ok we still got a job
so there is a lot of tolerance built-in
for this I think society to our legal
system which assumes that we're going to
screw up like that I have a question do
you feel that a lot of the economic
potential from AGI comes from the fact
that they are indeed different from
humans in that they are perfectly
rational and
unbiassed that that might be why we want
to build them and in this case we want
the opposite of artificial stupidity in
practice I am lost connection


AI learns to Create  ̵K̵Z̵F̵ ̵V̵i̵d̵e̵o̵s̵ Cat Pictures: Papers in Two Minutes #1

dear fellow scholars this is papers in
two minutes with Robert Samuel
Konigsberg miles okay I'm making this
video for a few reasons firstly I've had
a lot of comments from people saying
they'd like me to do videos like Cara
Lee shown over here over at two minute
papers so this is that if you confused
link in the description check out that
channel it's amazing secondly this video
is a follow-up to a video of me that
recently went out on computer file about
generative adversarial networks
definitely check that out if you haven't
yet again link in the description this
video will make a lot more sense if
you've seen that one so on the computer
file video there were a fairly large
number of comments about there not being
enough pictures in that video not enough
sort of demonstrations or visualizations
of the actual images being produced by
these networks and that's largely my
fault I told Sean I would send him links
of the papers I was talking about and I
forgot to do that but we can talk about
them here so at the end of that video I
was talking about doing arithmetic on
the vectors in the latent space if you
take your men wearing sunglasses vector
subtract the man vector and add the
woman vector you get a point in your
space and if you run that through the
generator you get a woman wearing
sunglasses and people were asking if
that was a real thing or hypothetical
and if they could see pictures of and so
on so that came from this paper
unsupervised representation learning
with deep convolutional generative
adversarial networks by Radford and Metz
and I was talking specifically about
figure seven there's a link to this
paper in the description as well so you
can see here you have a bunch of images
of men wearing sunglasses and then the
average of all of those lake vectors is
this image of a man whose glasses then
we do the same thing for a man without
glasses and a woman without glasses and
then we can do arithmetic on those input
vectors and find that man with glasses
- man without glasses plus woman without
glasses gives us images of a woman with
glasses they've also got another one
here in this same figure that does the
same thing with smiling so you take a
smiling woman vector subtract the vector
for a woman with a neutral expression
and then add the vector for a man with a
neutral expression and you get a smiling
man which is pretty cool
so we can see that movements in the
latent space have meaning in human
understandable aspects of the image I
also mentioned that if you take that
point and smoothly move it around the
latent space you get a smoothly varying
picture of a cat now when I said that
I've never actually seen anyone do it I
just figured from the mathematics that
it was possible but just after that
video went live this paper was made
available which included as part of
their demo video exactly that smoothly
moving around the latent space to
produce smoothly varying cat pictures
and the results are terrifying actually
I like how the network decided that
black bordered white text in the impact
font is an important component of a cat
image or never happened but the core
point of this paper relates to something
else I said in the computer file video
they're fairly low resolution right now
protip whenever you mention some
limitation of a I always add right now
or yet because there's probably someone
out there at that very moment working on
something that'll prove you wrong anyway
this new paper uses a fascinating
technique of growing the neural network
as it's being trained so new layers of
neurons are added as the training
progresses to allow very large networks
without having to train such a large
number of neurons from the very
beginning this allows the system to
generate unprecedented ly high
resolution images I mean look at these
results it's just just beautiful it's
nice to be able to take a break from
being deeply concerned about the impact
of a eye on the future of humanity and
just be deeply concerned about the
output of this network what is that what
is that yeah anyway I'm sure you're now
wondering assuming I can get this video
out before everyone's already seen this
what it looks like to smoothly move
around the latent space for this
celebrity faces Network it looks like
this
I'm just gonna let this run I think it's
completely mesmerizing there's a link in
the description to the video that I got
this from which has a lot more examples
of the things that they can do with this
technique and it's really really
excellent there's also a link to the
paper you can read that as well
you
I want to thank my generous patrons
these people and in this video I'm
especially thanking Alexander Hartwig
Nielsen who supported the channel for a
really long time thank you so much I
want to apologize to two minute papers
and say thank you for watching and for
your generous support and I'll see you
next time


Reward Hacking Reloaded: Concrete Problems in AI Safety Part 3.5

hi in the previous video we introduced
the idea of reward hacking an AI system
that works by maximizing its reward like
a reinforcement learning agent will go
for whatever strategy it expects will
result in the highest reward and there's
a tendency for the strategies that have
the very highest rewards to be quite
different from the kinds of strategies
the AI designers were planning for for
example if you're using the score as the
reward in Super Mario World the highest
reward strategy might involve exploiting
a load of glitches to directly update
the score value rather than properly
playing the game we talked about some
ways that this can happen exploiting
bugs in the software like in Mario or
adversarial examples in neural networks
if you're confused right now check the
video description for the earlier videos
in this series but in this video we're
going to look at some more ways that
reward hacking can happen and how they
relate to one another so let's start by
drawing a diagram I can't believe I
already got this foreign to the subject
without drawing this diagram anyway
here's your agent here's the environment
the agent can take actions to affect the
environment and it can observe the
environment to get information about
what state the environments in there's
also a reward system which uses
information from the environment to
determine what reward to give the agent
so if the agent is pac-man the
environment is the maze and the reward
system is just looking at the score the
agent takes an action the action affects
the environment the change in the
environment creates new observations and
also provides a new information to the
reward system which decides what reward
to give the agent and the agent uses the
observation and the reward to decide
which action to take next and this kind
of goes in a cycle reward hacking is a
class of problems that can happen around
that reward system like in the previous
video we were talking about adversarial
examples and how they can be an issue
when your reward system relies on a
neural network but that's not the only
way this kind of problem can happen the
economist Charles Goodhart once said any
observed statistical regularity will
tend to collapse once pressure is placed
upon it for control purposes but despite
being true that was not very catchy so
it was changed to when a measure becomes
a target it ceases to be a good measure
much better isn't it that's good hearts
law and it shows up everywhere
like if you want to find out how much
students know about a subject you can
ask them questions
it's about it right you design a test
and if it's well designed it can be a
good measure of the students knowledge
but if you then use that measure as a
target by using it to decide which
students get to go to which universities
or which teachers are considered
successful then things will change
students will study exam technique
teachers will teach only what's on the
test so student a who has a good broad
knowledge of the subject might not do as
well as student B who studied just
exactly what's on the test and nothing
else so the test isn't such a good way
to measure the student's real knowledge
anymore the thing is student B only
decided to do that because the test is
being used to decide university places
you made your measure into a target and
now it's not a good measure anymore the
problem is the measure is pretty much
never a perfect representation of what
you care about and any differences can
cause problems this happens with people
it happens with AI systems it even
happens with animals and the Institute
for marine mammal studies the trainer's
wanted to keep the pools clear of
dropped litter
so they trained the Dolphins to do it
every time a dolphin came to a trainer
with a piece of litter they would get a
fish in return so of course the Dolphins
would hide pieces of waste paper and
then tear off little bits to trade for
fish tearing the paper up allowed the
Dolphins to get several fish for one
dropped item this is kind of good hearts
lure again if you count the number of
pieces of litter removed from the pool
that's a good measure for the thing you
care about the amount of litter
remaining in the pool but when you make
the measure a target the differences
between the measure and the thing you're
trying to change get amplified the fact
that there are a lot of pieces of litter
coming out of the pool no longer means
there's no litter in the pool so that's
good hearts law and you can see how that
kind of situation could result in reward
hacking your reward system needs to use
some kind of measure but that turns the
measure into a target so it will
probably stop being a good measure with
dolphins this can be cute with people it
can cause serious problems and with
advanced AI systems well let's just try
to keep that from happening
another way that reward hacking can
happen comes from partially observed
goals in our super mario world or pacman
examples the goal is fully observed the
reward is the score and the AI can just
read the score out of memory and it
knows it's reward but if we have an AI
system acting as
agent in the real world the reward
depends on the state of the environment
around it and the AI only has partial
knowledge of that through the robots
limited senses the goal is only
partially observed suppose we have a
cleaning robot with its mop and bucket
and we wanted to clean the office that
it's in so we can set it up so that it
gets more reward the less mess there is
like we subtract a bit of reward for
each bit of mess and the way it
determines the level of mass is to look
around the room with its cameras what
does this robot do well the answer is
obvious to anyone who's ever run for
Parliament in Maidenhead the old Skyrim
shoplifters trick if it covers up its
cameras say by putting its bucket on its
head it won't see any mess so it won't
lose any reward you've probably heard
about experiments on rats where
scientists implanted electrodes into the
rats brains allowing them to directly
stimulate their reward centers and if
the rats are able to press a button to
activate the electrode they never do
anything else people call that wire
heading and it's relevant here because
if we take our pac-man reinforcement
learning diagram and change it to the
cleaning robot it's not quite right is
it in pac-man the reward system is just
a little bit of code that runs
separately from the game program and
just reads the score out but for the
cleaning robot the reward system is a
real thing in the real world it's got
cameras sensors circuitry it physically
exists as an object in the office
so maybe the diagram should look more
like this because unlike in pac-man now
the reward system is part of the
environment which means it can be
affected by the actions of the agent the
agent isn't just limited to messing with
the environment to affect the
information going into the reward system
like putting a bucket on its head it can
mess with the reward system itself if
it's able to take the thing apart and
make it just returned maximum reward
regardless of what the environment is
like well that's an extremely high
reward strategy so some AI designs are
prone to deliberately tampering with
their reward systems to wire head
themselves but that's not the worst of
it there are a lot of AGI design
proposals out there where the reward is
determined by human smiling or being
happy
or saying certain things hitting a
certain button or whatever these designs
effectively make the human a component
in the reward system but whatever the
reward system is the agent is
incentivized to manipulate or modify it
to get the highest reward it can with
powerful general AI systems we don't
just have to worry about the AI wire
heading itself I want to end the video
with a quick thank you to my excellent
patreon supporters all of these people
in this video I especially want to thank
Robert Sanderson who's supported the
channel for a long time you know just
the other day my phone completely broke
and the phone is actually pretty
important because I use it to shoot all
of the behind-the-scenes stuff anything
random traveling that kind of thing so I
had to get a new one and I was able to
use patreon money to do that so I just
want to say thank you so much for your
support thanks again and I'll see you
next time