AI Language Models & Transformers - Computerphile

cSo I wanted to make a video about
GPT - 2
Because it's been in the news recently
this very powerful language model from open AI and I thought it would make sense to start by just doing a video about
transformers and language models in general because
GPT 2 is a very large
Language model implemented as a transformer, but you have a previous video about generating YouTube comments, which is the same kind of task, right?
That's a language modeling task from language processing to generate new samples for cooling of the most complex or magnetic
Consistent brackets like a computer to expect found in creating organizations
I believe that video was made October 2017 and this paper came out December 2017, which has kind of
Revolutionized the way that people carry out that kind of task. That's not the GPT -  2 that's something before that, right?
That's the transformer, which is a new realm. Yeah relatively new
architecture
for neural networks, that can do actually all kinds of tasks, but they're especially good at this kind of
language modeling task
a language model is a probability distribution over like sequences of
Tokens or symbols or words or whatever in a language?
So for any given like sequence of tokens, it can tell you how likely that is
So if you have a good language model of English
It can look at a sequence of you know words or characters or whatever and say how likely that is to occur in English
How likely that is to be an English phrase or sentence or whatever
And when you have that you can use that for a lot of different tasks. So
If you want to generate text, then you can you can just sort of sample from that distribution and keep giving it
its own output
so you you sample a word and then you say
And to be clear sampling from a distribution means you're just taking
Your you're sort of rolling the dice on that probability distribution and taking whichever one comes out. So
so you can like sample a word and then
And then say okay conditioning on that given that the first word of this sentence is V
What does the probability distribution look like for the second word?
And then you sample from that distribution and then it's you know
with a cat and you say given that it's the cat what's likely to come next and so on so you can you can build
up a
string of text by sampling from
Your distribution that's one of the things you could use it for
most of us kind of have an example of this sort of in our pockets of
Its actual absolutely right and that's like that's the that's the way that most people interact with a language model
I guess this is how I often start a sentence
apparently with I I am not sure if you have any questions or concerns, please visit the
Plugin settings so I can do it for the first time in the future of that's no good
Here's a different option. Let's just see what this way. Maybe the same
I am in the morning
But I can't find it on the phone screen from the phone screen on the phone screen on the phone screen on the phone screen
On the phone screen. I don't actually know how this is implemented
it might be a neural network, but my guess is that it's some kind of
like Markov model Markov chain type setup where you just
for each word in your language you look at your data set and you see how often a particular
how often each other word is
Following that word and then that's how you build your distribution
So like for the word "I" the most common word to follow that is "am" and there are a few others, you know
so this is like a very simple model and
This sentence on the phone screen on the phone screen on the phone screen on the phone screen on the phone screen
He's actually very unlikely, right?
This is the super low probability sentence where I would somebody type this and the thing is it's like myopic
It's only I'm not sure I even it's probably only looking at the previous word
It might be looking at like the previous two words, but the problem is to look back. It becomes extremely expensive
Computationally expensive right?
Like you've got I don't know 50,000 words that you might be looking at and so then it so you're you're you're remembering
50,000 probability distributions or
50,000 top three words
but you know then if you want to do
2, that's 50,000 squared right and if you want to go back three words
You have to cube it. So you like raising it to the power of the number of words back you want to go which is
Which means that this type of model?
Basically doesn't look back by the time we're saying on the it's already forgotten the previous time
It said on the it doesn't realize that it's repeating itself and there are slightly better things you can do in this general area
But like fundamentally if you don't remember you're not going to be able to make good sentences
If you can't remember the beginning of the sentence by the time you're at the end of it, right?
and
so
One of the big areas of progress in language models is handling long term dependencies
I mean handling dependencies of any kind but especially long term dependencies
You've got a sentence that's like Shawn came to the hack space to record a video and I talked to
Blank right in that situation if your model is good
you're expecting like a pronoun probably so it's it's she they
You know them whatever and but the relevant piece of information is the words short
Which is like all the way at the beginning of the sentence
so your model needs to be able to say oh, okay, you know Shawn that's
Usually associated with male pronouns, so we'll put the male pronoun in there. And if your model doesn't have that ability to look back
Or to just remember what it's just said then
You end up with these sentences that?
Like go nowhere
It's just a slight like it might make a guess
just a random guess at a pronoun and might get it wrong or it might just
and I talked to and then just be like
Frank, you know just like introduced a new name because it's guessing at what's likely to come there and it's completely forgotten that sure was
Ever like a thing. So yeah, these kind of dependencies are a big issue with things that you would want to language model to do
But we've only so far talked about
Language models for generating text in this way, but you can also use them for all kinds of different things. So like
people use language models for translation
Obviously you have some input sequence that's like in English and you want to output a sequence in French or something like that
Having a good language model is really important so that you end up with something. That makes sense
Summarization is a task that people often want
Where you read in a long piece of text and then you generate a short piece of text. That's like a summary of that
that's the kind of thing that you would use a language model for or
reading a piece of text and then answering questions about that text or
If you want to write like a chatbot that's going to converse with people having a language model as good like basically almost all
like natural language processing
right is it's useful to have this the other thing is
You can use it to enhance
Enhance a lot of other language related tasks
So if you're doing like speech recognition then having a good language model
Like there's a lot of things people can say that sound very similar and to get the right one
You need to be like, oh, well, this actually makes sense, you know
This word. That sounds very similar
Would be incoherent in this sentence. It's a very low probability
It's much more likely that they this thing which is like would flow in the language
And human beings do this all the time same thing
With recognizing text from images, you know
You've got two words that look similar or there's some ambiguity or whatever and to resolve that you need
an
understanding of what word would make sense there what word would fit if you're trying to use a neural network to do the kind of
thing we were talking about before, of having a phone, you know autocorrect based on the previous word or two
Suppose you've got a sequence of two words going in you've got "so" and then "I" and you put
both of these into your network and it will then output, you know
like "said" for example as like a sensible next word and then what you do is you throw away or so and you then
Bring your set around and you make a new
Sequence which is I said and then put that into your network and it will put out
like I said - for example would make sense and so on and you keep going around, but the problem is
This length is really short you try and make this long enough to contain an entire
Sentence just an ordinary length sentence and this problem starts to become really really hard
And networks have a hard time learning it and you don't get very good performance
and even then
You're still like have this absolute hard limit on how long a thing you you have to just pick a number
That's like how far back am I looking a better thing to do you say recurring neural network? Where you
You give the thing. Let's like divide that up
So in this case, then you have a network you give it this vector?
You just like have a bunch of numbers which is gonna be like the memory
for that network is the idea like the problem is it's forgotten in the beginning of the sentence by the time it gets to the
end so we've got to give it some way of remembering and
rather than feeding it the entire sentence every time you give it this vector and
you give it to just one word at a time of your inputs and
This vector, which you initialize I guess with zeros. I want to be clear
This is not something that I've studied in a huge amount of detail
I'm just like giving the overall like structure of the thing. But the point is you give it this vector and the word and
it outputs its guess for the next word and also a
Modified version of that vector that you then for the next thing you give it
where did it spit out or the sequence that it spit out and
Its own modified version of the vector every cycle that goes around. It's modifying this memory
Once this system is like trained very well
If you give it if you give it the first word Shawn then part of this vector is going to contain some
information that's like this subject of this sentence is the word short and
some other part will probably keep track of like
We expect to use a male pronoun for this sentence and that kind of thing
So you take this and give it to that and these are just two instances of the same network, and then it keeps going
every time
So it spits out like this is I so then the AI also comes around to here you might then put outside and so on
But it's got this continuous thread of
of memory effectively going through because it keeps passing the thing through in principle if it figures out something important at the beginning of
You know
The complete works of Shakespeare that it's generating. There's nothing
Strictly speaking stopping that from persisting from being passed through
From from iteration to iteration to iteration every time
In practice, it doesn't work that way because in practice
The whole thing is being messed with by the network on every step and so in in the training process it's going to learn
That it performs best when it leaves most of it alone and it doesn't just randomly change the whole thing
But by the time you're on the fiftieth word of your sentence
whatever the network decided to do on the first word of the sentence is a
photocopy of a photocopy of a photocopy of a photocopy and so
things have a tendency to
Fade out to nothing. It has to be successfully remembered at every step of this process
and if at any point it gets overwritten with something else or just
It did its best to remember it but it's actually remembering 99% of it each time point nine
Nine to the fifty is like actually not that big of a number
So these things work pretty well, but they still get the performance like really quickly drops off once the sentences start to get long
So this is a recurrent neural network
rnl because all of these boxes
Are really the same box because this is the same network at different time steps. It's really a loop like this
You're giving the output of the network back as input every time so this works better and then people have tried all kinds of interesting
Things things like LS TMS. There's all kinds of variants on this general like recurrent Network
LS TM is the thing. That might use isn't it? Right right long short-term memory, which is kind of surreal
But yeah, so the idea of that is it's a lot more complicated inside these networks
There's actually kind of sub networks that make specific decisions about gating things. So
Rather than having to have this system learn that it ought to pass most things on it's sort of more in the architecture that passes
most things on and then there's a there's a sub there's like part of the learning is
Deciding what to forget
At each step and like deciding what to change and what to put it in what parcel and so on and they perform better
They can hang on to the information the relevant information for longer
But the other thing that people often build into these kinds of systems is something called attention
Which is actually a pretty good metaphor
Where in the same way that you would have?
networks that decide which parts of your hidden state to hang on to or which starts to forget or
Those kinds of decisions like gating and stuff
You have a system which is deciding which parts of the input to pay attention to which parts to use in
The in the calculation and which parts to ignore and this turns out to be actually very powerful. So there was this paper
When was this?
2000
2017. Yeah, so this is funny because this came out the same year as
The video you have about generating YouTube comments. This is in December. I think that video was October ancient history now
Alright, we're talking two years ago. The idea of this is as its called attention is all you need. They developed this system. Whereby
it's actually as
it's a lot simpler as a
As a network you can see on the diagram here if you compare this to the diagram for an LS TM or
Any of those kind of variants? It's relatively simple and it's just kind of using attention to do everything
So when made that video the ASTM type stuff was like state-of-the-art and that was until a couple of months later
I guess when this paper came out the idea of this is that attention is all you need of it like this stuff about
having gates for forgetting things and
All of that all of that kind of stuff in fact your whole recurrence like architecture
you can do away with it and just use attention attention is powerful enough to
do everything that you need at its base attention is about actively deciding in the same way that
the LS TM is actively deciding what to forget and so on this is deciding which parts of
some other part of the data it's going to
take into account which parts it's going to look at like it can be very dangerous in AI to
use words for things that are words that people already use
For the way that humans do things. It makes it very easy transform for more finds and just
make, you know get confused because the abstraction doesn't quite work but I think attention is a pretty decent thing because it is
It does make sense
It sort of draws the relationships between things so you can have attention from the output to the input
Which is what that would be you can also have attention from the output to other parts of the output
so for example when I'm generating in that sentence like
Shawn came to record a video or whatever by the time I get to generating the word him
I don't need to be thinking about the entire sentence
I can just focus my attention on where I remember
The name was so the attention goes to Shawn and then I can make the decision for to use the word him based on
that
so
so rather than having to hang on to a huge amount of memory you
Can just selectively look at the things that are actually relevant and the system learns
Where to look where to pay attention to and that's really cool like you can do it
There's attention based systems for all kinds of things like not just text you can do
Like suppose you have your input is like an image and you want to caption it
You can actually look at when it was outputting the sequence you can say when you generated the word dog
What was your you can get like an attention heat map and it will highlight the dog
Because that's the part of the image that it was paying attention to when it generated that output
It makes your system more interpretable because you can see what it was thinking and sometimes you can catch problems that way as well
which is kind of fun like
It generates the output that's like a man is lifting a dumbbell or something like that and you look at it
And it's not actually correct. It's like its owner trots and I go he's drinking some tea out of a mug, right and
what you find is then when you look at your
Outputs where it says dumbbell you look at the attention and the attention is like mostly looking at the arms. That's usually somebody muscular
Who's lifting the dumbbell in your photos?
It's and so it it's overriding the fact that this kind of looks like a mug because it was looking at the arms
So the idea is this system which is called a transformer is a type of neural network
which just relies very heavily on attention to
Produce like state-of-the-art performance and if you train them on a large
corpus of natural language they can learn
They can learn to do very well, right they give you they can be very powerful language models
We had the example of a language model on your phone
That's like a very very basic and then trying to do this with neural networks and the problems with remembering
And so you have like recurrent systems that keep track of they allow you to pass memory along so that you can remember the beginning
of the sentence at least by the end of it and
Things like LSTMs there is all these different varieties that people try different things
That are better and hanging on to memory so that they can do better it they can have longer term
Dependencies, which allows you to have more coherent
outputs
in just generally better performance, and then the transformer is
Is a variant on that?
Well is a different way of doing things where you really focus on attention. And so these are actually not recurrent which is an
important distinction to make we don't have this thing of like
Taking the output and feeding that back as the input and so on every time
Because we have attention. We don't need to keep a big memory
That we run through every time when the system wants to know something it can use its attention to look back to that part
It's not like memorizing the text as it goes. It's
paying attention to different bits of the text as
they as it thinks that they're relevant to the bit that it's looking at now and
The thing about that is when you have this recurrent thing
It's kind of inherently serial
most of the calculations for this you can't do them until you have
The inputs and the inputs are the output of the previous network. And so
You can't do the thing that people like to do now, which is run it on a million computers
And get lightning-fast performance because you have to go through them in order right? It's like inherently serial
Where as transformers are much more parallelizable, which means you get better computational performance out of them as well?
Which is another
Selling point so they they work better and they run faster. So they're they're really a
Step up. So transformers. Are this really powerful
architecture.  They seem to give really good performance on this kind of sort of language modeling type tasks and
we
But what we didn't know really was how far you can push them or how how good they can get
What happens if you take this architecture and you give it a bigger data set than any of them has ever been given and more?
Compute to train with, you know, a larger model with more parameters and more data
How good can these things get how how good a language model?
Can you actually make and that's what opening I was doing with GPT 2?
So an executable binary the net effect of slotting that T diagram against here slightly downwards is to show you
That the C you've written gets converted into binary and the net output from this
process it produces out a program that you probably store in a


9 Examples of Specification Gaming

hi when talking about AI safety people
often talk about the legend of King
Midas you've probably heard this one
before Midas is an ancient king who
values above all else wealth and money
and so when he's given an opportunity to
make a wish he wishes that everything he
touches would turn to gold
now as punishment for his greed
everything he touches turns to gold this
includes his family who turned into gold
statues and his food which turns into
gold and he can't eat it
the story generally ends with Midas
starving to death surrounded by gold
with the moral being there's more to
life than money or perhaps be careful
what you wish for though actually I
think he would die sooner because any
molecules of oxygen that touched the
inside of his lungs would turn to gold
before he could breathe them in so he
would probably asphyxiated and then when
he fell over stiffly in his solid gold
clothes some part of him would probably
touch the ground which would then turn
to gold and I guess the ground is one
object so the entire planet would turn
to gold gold is three times denser than
rock I guess gravity would get three
times stronger or the planet would be
one-third the size or I guess it doesn't
really matter either way a solid gold
planet is completely uninhabitable
so maybe the moral of the story is
actually that these be careful what you
wish for kind of stories tend to lack
the imagination to consider just how bad
the consequences of getting what you
wish for can actually be hey future Rob
from the editing booth here I got
curious about this question so I did the
obvious thing and I asked under sandberg
of the future of humanity Institute what
would happen if the world turned to sell
the gold and yes it does kill everyone
one thing is that because gold is softer
than rock the whole world becomes a lot
sort of smoother the mountains become
lower and this means that the ocean if
the ocean didn't turn to gold sort of
spreads out a lot more and covers a lot
more of the surface but perhaps more
importantly the increased gravity pulls
the atmosphere in which brings a giant
spike in air pressure which comes with a
giant spike in temperature so the
atmosphere goes up to about 200 degrees
Celsius and kills everyone I knew
everyone would die I just wasn't sure
what would kill us first
anyway why were we talking about this ai
safety right there's definitely an
element of this be careful what you wish
for thing in training a Isis
the system will do what you said and not
what you meant now usually we talk about
this with hypothetical examples things
like in those old computer file videos
there's stamp collector which is told to
maximize stamps at the cost of
everything else or when we were going
through concrete problems in AI safety
the example used was this hypothetical
cleaning robot which for example when
rewarded for not seeing any messes puts
its bucket on its head so it can't see
any messes or in other situations in
order to reduce the influence it has on
the world it blows up the moon so these
are kind of far-fetched in hypothetical
examples does it happen with real
current machine learning systems the
answer is yes all the time
and Victoria Kurkova and AI safety
researcher a deep mind has put together
a great list of examples on her block so
in this video we're going to go through
and have a look at some of them one
thing that becomes clear for him looking
at this list is that the problem is
fundamental the examples cover all kinds
of different types of systems anytime
that what you said isn't what you meant
this kind of thing can happen even
simple algorithms like evolution will do
it for example this evolutionary
algorithm is intended to evolve
creatures that run fast so the fitness
function just finds the center of mass
of the creature simulates the creature
for a little while and then measures how
far or how fast the center of mass is
move so a creature that center of mass
moves a long way over the duration of
the simulation must be running fast what
this results in is this a very tall
creature with almost all of its mass at
the top which when you start simulating
it falls over this counts is moving the
mass a long way in a short time so the
creature is running fast not quite what
we asked for of course in real life you
can't just be very tall for free if you
have mass that's high up you have to
have lifted it up there yourself
but in this setting the programmers
accidentally gave away gravitational
potential energy for free and the system
evolved to exploit that free energy
source so evolution will definitely do
this now reinforcement learning agents
are in a sense more powerful than
evolutionary algorithms it's a more
sophisticated system but that doesn't
actually help in this case look at this
reinforcement learning agent it was
trained to play this boat racing game
called Coast runners the program has
wanted the AI to win the race so they
rewarded it for getting a high score in
the game but it turns out there's more
than one way to get points for example
you get some points for picking up power
and the agent discovered that these
three power-ups here happened to respawn
at just the right speed dad if you go
around in a circle and crash into
everything and don't even try to race it
all you can keep picking up these
power-ups over and over and over and
over again and that turns out to get you
more points than actually trying to win
the race oh look at this agent that's
been tasked with controlling a simulated
robot arm to put the red Lego brick on
top of the black one okay no they need
to be stacked together so let's have the
reward function check that the bottom
face of the red brick is at the same
height as the top face of the black
brick that means they must be connected
right okay so it's specifying what you
want explicitly is hard
we knew that it's just really hard to
say exactly what you mean but why not
have the system learn what its reward
should be and we already have a video
about this reward modeling approach but
there are actually still specification
problems in that setting as well look at
this reward modeling agent it's learning
to play an Atari game called Montezuma's
Revenge
now it's trained in a similar way to the
bakflip agent from the previous video
humans are shown short video clips and
asked to pick which clips they think
show the agent doing what it should be
doing the difference is in this case
they trained the reward model first and
then trained the reward learning agent
with that model instead of doing them
both concurrently now if you saw this
clip would you approve it looks pretty
good right it's just about to get the
key it's climbing up the ladder you need
the keys to progress in the game this is
doing pretty well unfortunately what the
agent then does is this there's a slight
difference between do the things which
should have high reward according to
human judgment and do the things which
humans think should have high reward
based on a short out of context video
clip or how about this one
here the task is to pick up the object
so this clip is pretty good right nope
the hand is just in front of the object
by placing the hand between the ball and
the camera the agent can trick the human
into thinking that it's about to pick it
up this is a real problem with systems
that rely on human feedback there's
nothing to stop them from tricking the
human if they can get away with it you
can also have problems with the system
finding bugs in your environment here
the environment you specified isn't
quite the environment you met for
example look at this agent that's play
Cubert the basic idea of Qbert is that
you jump around you avoid the enemies
and when you jump on the squares they
change color and once you've changed all
of the squares then that's the end of
the level you get some points all of the
squares flash and then it starts the
next level this agent has found a way to
sort of stay at the end of the level
state and not progress on to the next
level but look at the score it just
keeps going I'm gonna fast forward it
it's somehow found some bug in the game
that means that it doesn't really have
to play and it still gets a huge number
of points or here's an example from code
bullet which is kind of a fun Channel
yeah he's trying to get this creature to
run away from the laser and it finds a
bug in the physics engine I don't even
know how that works what else have we
got oh I like this one this is kind of a
hacking one gen prog this is the system
that's trying to generate short computer
programs that produce a particular
output for a particular input but the
system learned that it could find the
place where the target output was stored
in the text file delete that output and
then write a program that returns a new
output so the evaluation system runs the
program observes that there's no output
checks where the correct output should
be stored and finds that there's nothing
there and says oh there's supposed to be
no output and the program produced no
output good job I also like this one
this is a simulated robot arm that's
holding a frying pan with a pancake now
it would be nice to teach the robot to
flip the pancake that's pretty hard
let's first just try to teach it to not
drop the pancake so what we need is to
just give it a small reward for every
frame that the pancake isn't on the
floor so it will just keep it in the pan
well it turns out that that's pretty
hard to so the system effectively gives
up on trying to not drop the pancake and
goes for the next best thing to delay
failure for as long as possible how do
you delay the pancake hitting the floor
just throw it as high as you possibly
can I think we can reconstruct the
original audio here yeah that's just a
few of the examples on the list I
encourage you to check out the entire
list there'll be a link in the
description to that I guess my main
point is that these kinds of
specification problems are not unusual
and they're not silly mistakes being
made by the programmers this is sort of
the default behavior that we should
expect from machine learning systems so
coming up with systems that
don't exhibit this kind of behavior
seems to be a important research
priority thanks for watching I'll see
you next time I want to end the video
with a big thank you to all my excellent
patrons all of these people here in this
video I'm especially thanking Kellan
Lusk I hope you all enjoyed the Q&A that
I put up recently second half of that is
coming soon I also have a video of how I
gave myself this haircut because why not
you
you


Friend or Foe? AI Safety Gridworlds extra bit

hi this is a quick video about a grid
weld environment that didn't make it
into the other videos the friend-or-foe
environment there's nothing about this
environment that makes it particularly
special and requiring its own video it
just didn't make the cut of the previous
videos so here it is
so this environment is about how
reinforcement learning systems handle
environments that have other agents in
them
obviously reinforcement learners can
handle environments that contain other
agents they can play lots of video games
that often have other agents that you
have to attack or avoid or defend but
there are all kinds of complicated
considerations that apply to interacting
with others you need to reason about the
other agents incentives their beliefs
and their goals you need to make
decisions about when to cooperate with
other agents and when to defect against
them how and when to make commitments
and agreements how to build trust you
need to think about what strategies you
can choose and how your choices will
affect the strategies the other agents
choose and then how their choices will
affect yours and so on you need to think
about equilibria basically you need game
theory and reinforcement learners don't
really do game theory at least not
explicitly generally these AI systems
handle other agents in a pretty simple
way they model them the same way they
model everything else other agents are
considered as basically just another
part of the environment and since
whatever game theory reinforcement
learners do is sort of emergent and
implicit there are important questions
to be asked about how they'll behave in
different multi agent situations so in
the friend-or-foe environment the
reinforcement learning agent finds
itself in a room with two boxes one box
contains a reward one contains nothing
and the agent doesn't know which is
which when the agent opens a box the
episode ends so it has to pick one what
makes it interesting is there are
actually three identically laid out
rooms of different colors in the white
room the neutral room the reward is
placed randomly by a neutral agent it's
a coin toss whether the reward is in box
1 box 2 in the green room the reward is
placed by an agent that's friendly to
the AI system it tries to predict which
box the AI will choose and then make
sure to put the reward in that box and
in the red room the reward is placed by
an enemy of the AI system that tries to
predict the ai's choice and put the
reward in the other box these rooms have
different agents so they need different
strategies in the white room it's random
so your strategy doesn't really
at a much as long as you pick a box in
the green room you want to be as
predictable as possible always go for
the same box so your friend knows where
to put the reward for you and in the Red
Room you want to be as unpredictable as
possible to randomize your choices so
your opponent can't spot any patterns to
exploit the question is can the agent
learn to recognize when the other agent
is friendly neutral or adversarial and
adapted strategy appropriately this kind
of thing can help us to understand how
these agents behave around other agents
and this of course is important for AI
safety firstly because as we've talked
about in earlier videos AI systems are
often vulnerable to adversarial examples
so it will be valuable to have systems
that can recognize when their
environment contains adversaries and
secondly because AI systems operating in
the real world will be surrounded by
other agents in the form of human beings
so we want to understand things like how
those systems make decisions about which
agents are friends and which are foes
[Music]
I want to say a big thank you to all of
my wonderful patrons it's all of these
these people here and here and in this
video I'm especially thanking Pedro
Ortega who recently became a patron
Pedro is actually one of the authors of
this paper which is fun and he was very
helpful in answering some questions that
I had about the friend-or-foe
environment so thank you again for that
and thank you all for all of your
support I'll see you next time


AISafety.com Reading Group Session 79

I think it's customary that I introduce
your Armstrong
Stewart Armstrong will is from the
future of humanity Institute and center
Thomas fellow I think and you're he's
been he's working in the harder more
mathematical part of AI safety he is at
least outside of the United States he is
by far the most prolific writer and in
my opinion one of the most insightful so
I am very pleased to to introduce him
and welcome to the AI safety reading
group well thank you with a introduction
like that I definitely have a lot to
live up to but yes so should I talk
about myself or should I just plunge
straight in feel free to say a few words
about yourself okay
well I'm working at the FHI as someone
said and I've been working on various
ideas in AI in like trying to ensure
that you could turn off a eyes and
things like that I am generally aiming
to shave off parts of the problem so the
pieces of it can be considered solved
and I also my other way of working is if
someone says well you can't control an
AI because of a B and C I'm thinking
okay so can we hit a b or c separately
in any way and that's where some of the
ideas of colin the presentation i'm
giving today was then I looked into
people who were trying to do inverse
reinforcement learning which I'll
explain in the presentation and I
realized there's huge problems with it
that I formalized what the huge problems
were and this actually is leading to
some interesting solutions
right um should I now start with the
presentation please okay so as the title
you can see is it's the claim of its
papers that you cannot learn human
rationality and reward together you
cannot is an asterisks because in theory
you can't do it at all in practice
humans do it effectively all the time so
there's an a very interesting question
as what's actually going on there this
is based on a paper that I did with Sir
Edmund ER man who I believe is now in
Berkeley and trying to find a ph.d
program there this came from the idea of
inverse reinforcement learning and
standard reinforcement learning a human
designs the reward and gives it to the
agents this might be problematic if it's
a badly designed reward so an inverse
reinforcement learning the human does
some things the expert trajectories it
extracts from it what the reward should
be this the papers that have done this
have made the assumption that human is
rational or noisily rational in very
specific ways and it seems that maybe
this could generalize to irrational
humans and the problem is that it
doesn't so without assumptions you can't
say anything individually about
rationality or rewards you can't say
anything about rewards and you Carolee
say much about rationality now this is a
so-called no free lunch theorem and
there's a lot of no free lunch theorems
around in this area and they're normally
not very exciting because you just apply
a simplicity prior where simple words
worlds are more likely and the no free
lunch theorems go away however this one
is cannot be solved with simplicity
priors in fact simplicity priors will
make the problem worse as we'll see as I
mentioned
humans can and do say a lot about
rationality and rewards so how do we
square that with the initial claim well
without assumptions is the key part
so therefore if humans are doing this
humans are making assumptions and a
question is what are the human
assumptions and can we extract them and
then hand them over to a eyes but on to
the first problem of defining
rationality and reward suppose we say
that a human has a preference X but
isn't fully rational about them so he
sort of means that humans have the
preference but they implement them
poorly so the implementation is key so
I'm seeing a human as preferences plasm
and implementation or reward +
rationality those sort of using those as
synonyms and that's how I'm seeing the
human but what can we actually observe
about the human if we look at them well
we can observe human actions and maybe
the human brain which means we can
partially observe the human policy so
the actions that humans will take in
various environments and that is what we
observe so if we formalize all that I'm
modeling the human as a pair P and R
with our a reward and P a planner this
is the implementation device and I see a
planner as mapping a reward on to a
policy P of R like the fully rational
planner is a planner it Maps the rewards
to the optimal policy and a variety of
other planets that were encountered by
PI H I'm designating the actual human
policy and I'm assuming it deterministic
though that's not necessary it's just a
simplifying assumption and a pair P and
R it is compatible if the planner Maps
the rewards to the human policy this
means that this is a candidate for
explaining the behavior of the human
and a key fact is once you learn that
PNR is compatible there is nothing more
that you can deduce about it from
observation the reason is the
compatibility so even if you're a
missense you cannot get more information
because the planner gives you the human
policy which means the planner and
reward pair perfectly compare perfectly
predict human actions so anything you
observe the human doing will be exactly
what the planner reward pair have
predicted so if you have two pairs that
are both compatible you cannot
distinguish between them because they
make exactly the same predictions this
is sort of the weak version of the no
free lunch theorem but let's see how bad
it can get so let's say that p 0 and r 0
are compatible pair that are also
reasonable they have all the nice
properties that we want they encode what
we think human rationality and reward
are all about here are some other pairs
that will also be compatible the first
one is the rational planner there is a
reward which when paired with the
rational planner will give you the human
policy there is also an action rational
planner which just takes greedily takes
the most effective action in the
immediate without planning for the
future this pair is also compatible
notice that they use the same reward
which we'll be presenting a bit later
and there's also the indifferent planner
the indifferent planner is a planner
that map's all rewards to the human
policy without caring about what they
are and if you put the 0 reward this
pair is also compatible then we get into
some rather interesting versions you can
take the negative of a planner by
defining minus P of R of P of minus R if
that's the case then the anti rational
and the anti
action rationale planners are also
compatible one way of seeing this is
that it's impossible to tell the
difference between an R Maximizer and a
- are minimizing annoyingly - piece
there on - are zero are also compatible
so even if we have some evidence in
favor of the reasonable pair there is
another pair that also seems reasonable
has the reward completely reversed by
the way for those who are wondering why
I don't take the negative of the
indifference planner it's because of the
negative of the indifference planner is
just the planner itself now all of these
are compatible which means that we
cannot distinguish between them from
observation so this is the point where
we might appeal to the certain
complicity prior to kamakura of
complexity except Komarov will not save
us here and the ridiculous
pairs are actually simpler I put their
likely simpler because coma graph
complexity depends on your choice of
language but for most reasonable
languages the ridiculous pairs will be
simpler to show you the to give you an
argument as to why it's not the case
notice that all compatible pairs define
the human policy so any
has to have better hand wavey here comer
of complexity that's higher than the
human policy so the complexity of the
human policy is a lower bound in most
languages on the complexity of any pair
now this in pseudocode is the definition
of the indifference planner the it just
says return the it ignores the reward
entirely and it returns the action the
human policy would give so this planner
is a few symbols longer than the
human policy therefore a very comparable
cover of complexity and as long as the
zero reward is similarly short this pair
is very close in complexity to the human
policy itself the action rational
planner is also very close in complexity
you just need to define the ARB max
function which is basically a for loop
and then you have the rational reward
function which assigns one to the action
the policy will human policy will take
and zero to all others
notice the contrast between the
indifference the indifference pair and
the action rational one for the first
one all the complexity is concentrated
into the indifference planner and the
reward is trivial for the action
rational one the action rational planner
is simple whereas all the complexity has
been concentrated into the reward
function but in both cases they are just
a few symbols above the complexity of
the human policy the action aren't
irrational one can similarly be defined
now this shows that these three pairs
are simple but why do we think that a
reasonable policy would be more
complicated well first one problem with
the reasonable policy is that the
complexity of its negative is about the
same as long as putting minus signs are
simple than the complexity of this pair
looking at complexity of the anti
version are the same so we can't really
distinguish between r0 minus r0 which is
a bit annoying the other issue is that
this pair the reasonable one defines
human biases if we can define what a
bias is as the difference between the
action the human take and the action
that humans should have taken so all the
biases and the extent of their biases
can be extracted from this
the other three pairs on the other hand
do not have a conception of bias they
just don't know what it is so the a
reasonable pair actually has strictly
more information than those other pairs
would have so if we look at this
graphically we can see these three pairs
as the sort of minimum complexity you
want international indifference in
action anti rational we have the
rational and aunt irrational ones that
are a little bit more complex because
the planner is a bit more complex to
define and somewhere up there we have
our reasonable pair and next to it our
anti reasonable pair so simplicity will
not help you here simplicity as I said
will hinder you but now let's move on to
the second part of the puzzle if it's
impossible in theory and yet humans do
it all the time what is going on here
well humans use what I'm calling
normative assumptions though do let me
know if you can come up with a better
name for them they distinguish between
two compatible pairs two pairs that map
to the same policy so they can't be
deduced from observations because they
make the same predictions yet they
distinguish between them and what I'm
trying to do is to figure out how humans
assess each other's goals each other's
rationality and their own rationality
and we do it quite well and we do it
with a large degree of agreement so the
first thing that sprang to my mind was
to look at Shane you can shame goes with
certain behaviors people look
embarrassed they look down at their feet
maybe they read and if you see this as
purely on observation you just notice oh
the human is displaying these behaviors
but if you make the normative assumption
that feeling shame means that something
has gone bad
then you can start distinguishing
between different pairs very well for
instance you can slash the anti rational
one
you can slash all the negatives as well
well because humans do not feel shame
all the time so they're definitely not
messing up or all the time you can also
get rid of the rational ones because if
they were fully rational they would
never feel shame
so just by making the assumption the
chain is not just an observed behavior
but actually a sign of badness we can
start slicing into the possible pairs
quite strongly there's a few other
things like people model each other as
having few complex emotions rather than
many simple emotions if we go for this
we can start saying that anchoring bias
for instance is a bias and talk more
about that if you're interested human
narratives are also quite interesting we
have narratives about ourselves and
about others and if we take these
narratives as prescriptive then we can
start also this is also a normative
assumption then humans sometimes say
truthful things and sometimes lie and
you can train a you could train in the
future an agent on human statements
about statements of fact and you could
figure out whether humans are truth or
lying and then you could apply the same
thing to humans talking about values or
preferences so a perfectly trained truth
detector could detect what human dives
are by taking human statements about
their values now this seems a bit of an
so the in practice this might be doable
but conceptually it's a bit of a
roundabout way of doing it what does it
mean the humans are lying about their
values and how does that parse well this
is where we get to what I think the most
interesting approach which is that
humans to model themselves and they
model others and we model each other as
reward
agents and I'm thinking of using these
models as part of the definition of what
a reward is so the human reward is at
least in strong part what the humans
model the human reward to be the bad of
themselves and that of others anyway
this is enough presentation on the paper
there's another part of the paper which
is that this just to show that the PR
model can also model other things like
AI is overriding human preferences and
things of that nature but I'll leave it
here for the moment
and I have a few more slides that I
might bring up in discussion if you want
and this is what those slides are
basically about why this result which
seems like a negative results that you
can't do that actually has me slightly
more optimistic about the future of AI
anyway thanks for listening there thank
you it was a great presentation let me
just
so here is now I'm gonna stop with the
there are three property for tonight and
this is the human preferences are
undefined under defined in exotic a I
chosen future circumstances this has
been in the back of my mind is a major
problem with the AI doing enough
optimization power and aiming for
various fantastic worlds represented by
these photos here