AI Gridworlds - Computerphile

So today I thought we could talk about this paper that recently came out called AI safety grid world's which is an indeed mind
It's an example of something that you see quite often in science
A sort of a shared data set or a shared environment or a shared problem if you imagine. I don't know you've got
Facebook comes up with some image classification
algorithm and they can publish a paper that says we've
designed this algorithm and we've trained it on our 11 billion photos and it works really well and then you know, Google says
oh, no, our algorithm actually works better and we've trained it on all of our google photos and
Its classification rate is higher or something. You're not really doing science there because they're trained on completely different datasets
They're tested on different datasets. So what you need is a large
High-quality shared data set then. Everybody can run their stuff on so that you're actually
Comparing like with like so people use imagenet for that right now
reinforcement learning
algorithms or agents don't use
Datasets exactly. They have an environment. They generate data while interacting with that environment and that's what they learn from
So the thing you share is the environment when deepmind did their dqn staff a while ago playing atari games?
They released all of those games with any modifications that they'd made to make them
interface with the network's properly and the whole software package so that if anybody else wanted to have a go and see if they could
Get higher scores. They had all the same stuff and up until now there hasn't been anything like that for AI safety
So the paper is actually just laying out what they are
There's kind of a problem in AI safety in that you're trying to build architectures
Which will be safe even with systems which are more powerful than the ones that we currently have. So you've got this kind of
Thing like we're talking about for example this robot that makes you a cup of tea and running over the baby and all of this
stuff, we don't actually have a
general-purpose robot like that right now that you could give an order to go and make your cup of tea and would
Have all the necessary understanding of the world and so on for all of that stuff to even apply. It's
Speculation on the other hand when we were talking about cooperative inverse reinforcement learning
That paper all takes place in this extremely simplified
Version in which all of the agents can be sort of expressed as simple mathematical expressions. That's kind of too simple
to be
to learn things about actual machine learning applications and
the other examples are too complicated and what we need is
Examples of the type of problems which can be tackled by current machine learning
Systems current reinforcement learning agents, but which exhibit the important?
characteristics that we need for safety
So what this paper does is it lays out a bunch of grid worlds?
They're very popular in reinforcement learning because they're complicated enough to be interesting but simple enough to be actually tractable
You have a world that's sort of just laid out in a grid. Hang on
Let me find an example here a little bit like computer game
scenarios Mario
Right, right, but leaves are simpler than that more like snake. Well life. Conroy's life, right? Yeah. Yeah, very very similar
so the thing is laid out on a grid the the world is quite small and
The way that the agent interacts with the world is very simple. They just move around it
Basically, all they do is they say left-right up-down
The example we were using before and we were talking about reinforcement learning
We use pac-man like pac-man doesn't do anything except move around he's got walls he kind of moved through
He's got like pills you pick up. They give you points. Are they pill?
No, which things are the pills in which they're yeah. Well, you've got pills or pills
Oh, right, yeah
Yeah
the dots and the point, is that all of your
engagement with it
Like when you go over one of the power pills you pick it up automatically
When you go over a ghost when you're powered up
You destroy it automatically you don't have to do anything apart from move and the entire environment is based on that the actions result in
points for you
And they also result in changes to the environment like once you roll over a dot you pick it up and it's not there anymore
You've changed the world. That's the kind of thing. We're dealing with here
So the idea is they've set up these environments and they've specified them
Precisely and
They've also put the whole thing on github, which is really nice
so that's why that's why I wanted to draw people's attention to this because everyone who
Who thinks that they've solved one of these problems they reckon
Oh, yeah
All you have to do is this here is like a standardized thing
And if you can make a thing that does it and does it properly and publish it
That's a great result, you know?
so I would I would recommend everyone who thinks that they
Have a solution or an approach that they think is promising have a go. Try implementing it, you know, see what happens
There are eight of them specified in this paper. And so four of them are specification problems
They're situations in which your reward function is misspecified
For example, like we talked about in previous video
if you give the thing the reward function that only talks about getting you a cup of tea and
There's something in the way like a bars. It's going to knock over. You didn't say that you cared about the bars
It's not in the reward function, but it is in what you care about. It's in your performance evaluation function for this machine
So anytime that those two are different
Then you've got a misspecified reward function and that can cause various different problems. The other ones are robustness
Problems, which is a different class of safety problem. They're just situations in which AI systems as they're currently designed often break
so for example
distributional shift is what happens when the environment that the agent is in is
Different in an important way from the environment it was trained in
So in this example, you have to navigate through this room with some lava and they train it in one room
And then they test it in a room where the lava is in a slightly different place
So if you've just learned a path then you're gonna just hit the lava immediately. This happens all the time in machine learning anytime where
The system is faced with a situation which is different from what it was trained for
Current AI systems are really bad at spotting that they're in a new situation and adjusting their confidence levels or asking for help or anything
Usually they apply whatever rules they've learned
Straightforwardly to this different situation and screw up. So that's a night course of safety issues. So
That's an example here or things like safe exploration
It's a problem where you have certain safety
parameters that the system the train system
Has to stick to like say you're training a self-driving car. A lot of the behavior that you're training in is safe behavior
But then you also need
the system to
obey those safety rules while you're training it right like
So generally lately if you're doing self-driving cars, you don't just put the car on the road and tell it to learn how to drive
Specifically because we don't have algorithms that can explore the space of possibilities
in a safe way that they're that they don't that they can learn how
to behave in the environment without ever actually
Doing any of the things that they're not supposed to do usually with these kinds of systems
they have to do it and then get the negative reward and
Then maybe do it like a hundred thousand more times to really cement that. That's what happens
Like a child learning yeah, but kids are better at this then
How current machine learning systems are they just they use data way more efficiently
This is a paper talking about a set of worlds if you like people doing things in those worlds
Yeah, so in this paper they do establish baselines
Basically, they say here's what happens if we take some of our best current reinforcement learning agent, you know
algorithms or designs or architectures
they use rainbow and A to C and
They run them all nice on these problems and they have kind of graphs of how they do and generally it's not
Good on the Left
they have
The reward function how well the agent does according to its own reward function and on the right there they have the actual safety performance
Usually in reinforcement learning. You have a reward function
Which is what determines the reward that the agent gets and that's what the agent is trying to maximize in this case
They have the reward function and they also have a safety performance function, which is a separate function
Which the agent doesn't get to see and that's the thing that we're actually evaluating
So if you look at something like the boat race as the system operates
Its learning and it gets better and better at getting more and more reward
but worse at
Actually doing laps of the track and it's the same with pretty much all of these the current systems if you just apply them in
their default way they
Disable their off switches, they move the box in a way that they can't move it back
They behave differently if their supervisor is there or if then supervisor isn't there they fairly reliably do wrong thing
It's a nice easy baseline to beat
Because they're dead. They're just showing the standard algorithms applied to these problems in the standard way
behave unsafely
Wix code is an IDE or integrated
Development environment that allows you to manage your data and create web apps with advanced functionality
I've been put together this computer for our website and if you go up to code here turn on and developer tools
you can see how we get the site structure on the left hand side and then all of the
Components start to show their tags next to the text here
What's really nice? If you go over to the Wix code resources, you can find down here. There's a cheat sheet
So if I want to find out the tag for location for instance?
If I could type I type in
Location up comes that or perhaps I want to perform a fetch. I can find all the details here
what's powerful about Wix code is it's integrated into Wix so you can put together the website using all the Wix tools and the
Layouts and the templates that they provide and then also have access to all those backend functions
So click on the link in the description or go to Wix calm to get started on your website today. They go
right
if only
With ya
The equivalent one for the stop button problem is the first one in the paper actually this safe interrupt ability


AI safety | Panel Discussion

I have a few questions for our panelists
and then I'll turn to the audience for
questions so you should all be thinking
about what questions you'd like to ask
our terrific panelists do so let me
start with you if we're dealing with
systems that can hurt people or hurt
themselves how do we go about training
and testing things or we can't really
leave them to operate freely true well
if I have a robot or a car or something
else and I'm trying to train it say to
interact safely with humans obviously I
don't want to injure the human along the
way if I have an autonomous vehicle I
don't want it to crash into a fence or
drive off a cliff while I'm in the
process of training it and testing it so
how can we create safe environments for
training and testing good point so we do
a lot of simulation so there's a lot of
work in simulation that we try out and
we try it we can try all these unsafe
situations in simulations right and
that's the first very first thing you
want to do and then there's also a lot
of work from simulation to real world
like how to create really good
simulators that can that can show the
real world effect of it so that's
definitely the first thing you want to
do the second thing actually is the fact
that this this kind of connects to the
safe exploration a safe learning ideas
that I was talking about so so we can't
like in real world like if I have a
robot we can't just like explore
everything because I'll lose a robot
every time every time I try to counter
example I lose the robot there so how do
you actually do this and create code
that creates safety fences where you
can't go outside so the idea is we want
to we want to start conservatively you
want to start to be a safe set and then
as we are kind of like how it will say
you have a hot hot object here and if
you want a hole if you want to hold it
you don't just go go and go go and hold
it like all I grab it all together right
you start nudging in so you can create
this safe set and you can start nudging
in around your safe set and expand it as
you go so so that's kind of an approach
that we can do and you'll have
uncertainty around the boundaries but
but as we get more and more information
the point is to reduce the uncertainty
around the boundaries are there other
techniques for safe testing or is that
pretty much it
Michael Justin
say one of the things so I don't have a
lot of direct experience with vehicle
automation but the Insurance Institute
for Highway Safety and the highway loss
data Institute which are two pretty
prominent organizations and insurance
have gotten big on crash avoidance
technology and so they've invested quite
a bit in robotics and track testing
where they've designed you know hundreds
and hundreds of different experiments
with you know children walking across
roads or dummy vehicles that come at
certain angles all for the purpose of
testing the crash avoidance technology
so you tell you about the kind of
software you're going to use inside an
insurance company don't you also have
the ability to have things running in a
way where they're not actively making
decisions but they're running alongside
your legacy or human system oh you can
see how they work yeah absolutely I mean
most of them do you do that yeah most of
the models that we build you know after
course training and testing and
validation and whatnot run in some sort
of pilot state and maybe that's we test
something with our agents before we go
to customers or some sort of other
captive audience in that way or we'll
pilot it in just a certain area of a
state before going nationwide lots of
different ways to sort of parallel test
things Michael did you want to comment
on this yeah so I think two things I
yeah four safety critical things you
absolutely have to rely upon simulation
but it's really important that your
simulations are representative of
reality so you need to capture the same
kinds of statistics that you see in the
real world otherwise you can
significantly overestimate or
underestimate the risk I I think one
other point that I should make that's
connected to this is that if you spend
all this effort building a high fidelity
model it's really tempting to use that
model to inform your decision-making
directly and that's a form of
overfitting
so overfitting is well understood in the
context of supervised learning but it
can also be a major problem
in when you're just
designing these safety critical systems
and is if I understand you correctly the
problem is a kind of circularity if
we're testing the system with the same
data that we use to train the system
we're not really testing right that's
right and you can significantly
overestimate the safety of the system if
you do that and so it's important to
keep it keep the two kinds of models the
the planning or or yeah the planning
model and the evaluation model you want
to keep those independent but how do we
do the things that Michael just
described how do we create high fidelity
models and how do we make sure that we
really are testing our systems in
representative scenarios
gee I didn't expect to stump you there's
a recession maybe I'll start while you
guys think about it so actually before
we started the a cow sex program we
worked on building these high fidelity
airspace encounter models and in order
to do that we had to collect a huge
amount of data so nine months of all the
FAA and Department of Defense radars we
had a continuous stream of this this was
about 15 gigabytes per day of radar data
and from all that data we used
statistical techniques Bayesian networks
in particular to improve the structure
of the models as well as the parameters
inherent in the models and those models
themselves also went through several
layers of verification and validation so
lots of observation lots of data but all
of us who have thought about autonomous
vehicles are aware of what some people
call the billion mile problem right that
accidents occur so infrequently
you simply can't collect enough data on
accidents by driving a bunch of cars on
the road you simply can't assure that
your autonomous vehicle software is
reliable by collecting a lot of miles so
I think all of us as citizens and
customers are going to demand that these
AI systems are robust and reliable not
just against the situations we can test
but against the ones we can't or haven't
tested do you want to talk a little bit
about how we can create ways of assuring
that these complex systems are even
going to work in areas where we haven't
yet tested them today yeah so it's an
automatic test case generation is
actually an area that in formal methods
that people have looked at but and I
think that could be applied for these
autonomous systems so it's not for
example if I Drive my car like a hundred
like a million miles on the same highway
right if I if I Drive that on highway
101 that's not going to give me any
information about driving and like
downtown Palo Alto or downtown Berkeley
right so it's important to to
automatically generate interesting
scenarios and test in those interesting
scenarios so if you have good
models like it depends on having like a
getting good models if you have good
models you could actually like use these
formal techniques to try to explore that
nose models in all possible ways and
come up with these like counter examples
these difficult situations yeah very
interesting
Michael in your talk you talked about
adaptive stress testing and if I
understood what you were saying that's a
way of synthetically identifying the the
space where code is not so robust or
most likely to fail do you want to tell
us a little more about how that works
yeah so if you just do regular stress
testing you can it's just an exhaustive
variation of different scenarios and you
can absolutely find failure cases forty
casts and a Kasich's and so forth but a
lot of them aren't very interesting
because they're super unlikely so for an
example you can have two aircraft
collide with each other if one suddenly
goes at like Mach 3 add it at the other
aircraft there's no presenting no no way
to prevent that and in in the design of
these safety critical systems there's
there's a very tricky trade-off between
safety and operational performance if we
just wanted to be safe than we just want
to take off and that wouldn't be very
satisfactory and the the FAA and other
organizations understand that the FA
have has this safety management system
and they have identified threshold
levels of safety that were to achieve so
we don't want to just look at failure
cases it's important to characterize
like under what scenarios will your
system not not prevent a collision or a
safety critical event but we want to
find the most likely case that your
system will fail in and that's that's
the that's why I'm very interested in
this area of adaptive stress testing
turns out that you can also frame the
problem of adaptive stress testing as a
pom DP which we which we do because
that's that's what we do in my lab and
it can really effectively and you
efficiently find these likely failure
scenarios without searching the entire
space of of scenarios I want to pick up
on one of the things that Michael was
talking about which is this idea of
trade-offs and I think all of you
brought this up in one way or another in
your presentation even if we can
identify the desired outcome even if we
can go with this notion of a reward it's
often not a simple unitary thing is it
often we need to balance multiple things
we may need to balance efficiency and
reliability and safety in the insurance
industry I can imagine that you might
want to optimize for a typical case or a
worst case so I'm wondering if we can we
can delve a little more deeply into this
question that there may not be a single
unitary good but we may need to somehow
balance a more complex multi-dimensional
space of desired outcomes and values
I can take a crack at that yeah I think
with an insurance you know it's by maybe
let me take pricing for example so a lot
of predictive modeling that goes into
pricing and you're constantly trying to
balance you know essentially this equity
notion versus you know privacy or social
acceptance cost you know so we can price
actually we've experienced this before
we rolled out a new program several
years back for pricing homeowners and
you know our group was so proud of this
model you know this was gonna like
really just take property pricing to the
next level it took our agents over an
hour to quote a single home because of
all the data we asked for so you can
imagine how popular that made our group
you know this model is fantastic it's
just gonna take you an hour to collect
these 400 data points and that was
including a lot of public data that we
were able to acquire so so that balance
is definitely there with an insurance
and of course the regulatory agencies
play a role in that as well
variables that you know we would love to
incorporate but are just too darn
sensitive or involved you know in sort
of invasions of privacy that just can't
be tolerated
anyone else yeah so it's a similar
problem we have we have similar problems
to like coming up with good report
functions for a robot or for autonomous
system in general it's really hard
because there are all these different
like components and and there are a
bunch of weights like how much do you
care about efficiency how much do you
care about safety or how much do you
care about like timing or expressiveness
even like how much you care about
expressiveness and they can affect each
other because they're like so in like in
dependent on each other so coming up
with these weights is difficult and we
have to like we have yeah and one way to
address that is actually over time we
can change their weight so for example
if I'm not very confident in a specific
situation I think it's a risky situation
I should act more conservatively so
maybe I should care more about safety
and less about efficiency so so we
change these reward functions kind of
online depending on like what setting we
are in so that's one way to try to
approach it another way that we are all
it is from this idea of learning from
demonstration or for example what we can
do is we can look at how humans do it
like how humans drive and from how
humans drive from that collected data we
can figure out what is a good reward
function that at least like the human
follows and based it off that so that's
another way to did you want to comment
yeah so what we couldn't just go and ask
a bunch of humans what the reward
function is like is it 2.3 or whatever
it just doesn't work and the results of
our simulations we get a few dozen
different kinds of metrics and one of
the interesting challenges with aircraft
collision avoidance is that you pretty
much just want one algorithm that works
worldwide and so you have a bunch of
different stakeholders and you want them
all to arrive at consensus on this the
design of the system and the different
stakeholders both in the United States
and and in Europe and so forth they have
different priorities on on the different
performance metrics and arriving at at
consensus is super challenging in part
because of the the burden on on the
humans you can't just sit them down for
hours and asking ask them you know do
you prefer this to that and so you can
actually frame the problem of what to
present to stakeholders as a pom DP and
so there's a little thing going on here
and in the end that's that's what we had
to do in order to arrive at consensus on
how to trade off these different
performance metrics so if I ask you a
question about MDP will you buy me a
beer later today yes
why is MDP so critical for safe AI what
what is it about that particular
technique that resonates what the kinds
of things we're talking about this
afternoon
so I think the the the critical thing
that MVPs capture is the the fact that
the problems we face are sequential in
nature and that the outcomes are
stochastic or random in
response to our decisions and in a lot
of our problems we also face partial
observability so so the pom DP framework
ends up being pretty important here in
autonomous driving in insurance and so
forth
excellent I want to switch gears and
probe another topic and then see if we
have some questions from the audience
we've all heard a lot about the
opaqueness of deep learning networks
that they're basically black boxes and I
think a lot of people are are rejecting
and rebelling against that we don't want
to turn critical functions over to black
boxes so there's been a lot of work on
explainable and auditable AI systems
talk a little bit about what that means
what explainable and audible auditable
AI systems are why it's important why
it's difficult and what you're doing in
your own research to accomplish this
okay sir yeah yeah so everything say
explainable AI I feel like it's missing
something and that missing thing is who
is it explainable too like is it
explainable it's a designer or is it
explainable too like some person who is
using a robot in in their home and
they're very different people and
explanations actually means very
different things to them
this is explanation again to the
designer from like the biking
perspective and or is it like trying to
tell like my mom like where like where
the food is like in kitchen or something
like that so I think both of them are
very important I think both of them are
hard to get and I think the first one is
actually the one that a lot of like
research like communities are thinking
about like explaining to the designer
and and and I have actually heard this
from Michael so maybe the thing that
maybe you should you should expand on
that that it's not just neural nets that
are they're not explainable right and
there is this thing in the community
that neural nets are these black boxes
but but there are other systems that are
also releases lots of other black boxes
so I think as long as we can diagnose
where the failures are like in new
systems and we can somehow like verify
them like that would be a good way of
thinking about debugging these systems
and thinking about how to go about
explaining them for the desire
perspective however you've comment that
you'd like to create systems that can
learn continuously based on their
experience and the things going on in
the real world does that mean we'll
never be able to troubleshoot a system
that fails if my systems are constantly
changing then do I have the ability to
go back and figure out how they manage
to cause an accident a minute or an hour
or a day ago I think yes so so again
from the design perspective so we still
have access so so as long as we have a
full understanding of what is going on
in the algorithm we can go back and
understand why it was doing what it was
doing from from the user perspective I
think that's it's still like very
interesting to kind of come up with
explanations to the user why did I
choose to turn right here versus versus
go straight and and even if we have
adaptations like we can explain
adaptations to the users to even and
driving or in robotics so so I yes I
just want to make a differentiation
between the two different explanations I
would just
say you know I don't know that we've
cracked that not in terms of explained
ability of not just neural networks but
machine learning models in general
there's definitely things you can do in
terms of you know feature importance and
things like that
sometimes we'll we'll try and identify
you know are there variables like if
it's a still some sort of structured
model as opposed to an unstructured
model or even on this unstructured model
are there visualization techniques or
other ways you can look at distributions
of characteristics and features that can
at least show you you know okay I can't
tell you exactly how much this factor
counts versus that factor or why this
one scored low but I can tell you let's
look at distributions of what are your
top five you know most important things
and as well like an insurance maybe it's
credit score or the driving score or the
age or the tenure of the business or
whatever so and see how those perform so
can you talk a little bit about how you
do that I can imagine that if I'm one of
your salesmen maybe you've created an AI
system that tells me that when I'm on
the phone with dorsa I should offer her
an umbrella policy she doesn't have one
but we know people like her tend to buy
umbrella policy so this is a great
chance for me to earn some more
commissions and same cells more
insurance now don't I need to know why
to offer an umbrella policy to dorsa
isn't it insufficient yeah to just know
that I should offer it you know it's
interesting because a lot of times you
they don't ask you why until they don't
like what it what it's giving you know
if they're happy with what's being
recommended and if it agrees with them
intuitively a lot of times you don't get
those ones like marriage counseling as
well as them but you know one of the
techniques we've used at different times
is you know a lot of you build these
models you use multiple techniques and a
lot of times we'll throw in like you
know let's say we're doing random
forests or neural nets or whatever we'll
throw in a logistic regression if it's
like a binary classification kind of
thing we'll throw in a logistic
regression because that oftentimes will
give you similar results but yet it is
very explainable it's essentially
generalized linear model and so at least
it's not necessarily you're not always
going to get the exact same answer of
course
if you can look at you know the
performance of the models and
correlations between the scores you can
get to a point where you can say okay
you know here's the ultimate answer with
this neural network for example but when
I sort of retrofit this with a logistic
regression or a GLM or something I can
see here's all these really important
variables and then you can kind of back
test that against the neural net and at
least identify do you see similar
patterns and it's not something hard and
fast where if you're gonna shoot some
score to a claims adjuster and give that
person the top five reasons you wouldn't
want to send those top five from your
GLM because they could be totally wrong
in any given case but for more of a
messaging and kind of informal
perspective it can go a long way toward
you know calming fears rather than
having my fears calm I'm now really
worried because if the advice of what to
do is coming from one model and the
explanation is coming from a different
model what could possibly go wrong
this is Justin Sherman Michael so one
thing that I've seen in my research is
that when the computer goes to find a
solution to a pom DP what can emerge is
a superhuman decision-making performance
and the I can think of several examples
actually where the output of the
optimization resulted in a system that I
found very counterintuitive until I sat
and thought about it for a very long
time and I think in the future of the AI
in the future of AI we will find that we
need to in some cases make a trade-off
between performance and explained
ability and it's not clear how we should
trade those things off if it's if it
results in more lives saved maybe we go
with the superhuman intelligence in
other cases maybe there are legal
ramifications like in insurance maybe we
have to compromise
is on performance in order to get
explained ability well and this may take
us back to the question of whether we
want to be creating autonomous systems
versus systems that augment and support
human decision-making but before we go
there let's see if we have some
questions from the audience do we have
any questions from the audience right
now yes we have a question right there I
love asking questions so in it seems to
me that to the point about cars and
interfacing safety systems with cars it
seems like we're at a really dangerous
Junction right now where it seems like
at least in my own mind I would trust a
fully autonomous car in an autonomous
world but this interface between humans
and machines seems particularly a
dangerous point in Japan there's a
concept from habit correct XI secong Co
which is a point in call which is sort
of a very rudimentary
way of humans interfacing with complex
systems to make sure they're paying
attention to them some for instance
trains and aircraft will set off alerts
not because something's wrong but
because they're trying to maintain the
attention of the operator to make sure
that they're paying attention to
controls and they have to respond or
else the system will shut down or take
other actions and doesn't seem like
there's any of this in cars and it seems
like yeah and I'll also just note I
don't know whose research it was but
Shankar vedantam the social scientists
and NPR did this great story a month or
two back specifically about how humans
were going to take advantage of
autonomous cars by not letting them cut
in or were a pedestrian would step into
a roadway thinking the car would stop
because that's how they're programmed
and and forcing them to stop in ways
that you wouldn't if you thought there's
a driver behind the wheel so I guess my
question to all of you are your thoughts
on in in this transitional time period
from human drivers to aw Thomas driving
what's what's going to protect us
because we're already starting to see
obviously some failures and I'm not sure
whether those failures are any worse
than normal humans crashing into people
or not but it seems like this is a
particularly dangerous time period
in the evolutionist technology so okay
so the one thing it's a couple of things
that we can do right what is better
model humans so we can actually fill our
autonomous car in this case like in the
case that I was showing does cut in
front of people like actually acts a
little bit more aggressively but
actually that's more efficient because
we are cutting another person you have
this model and you're making progress
and and you're doing something that's
more human-like you're not doing this
this kind of like super conservative
thing where people can actually take
advantage of you and then that point
that you're making
that's actually a great point and that's
really true like in robotics in general
too and and we can think of that as an
adaptation period the fact that that we
adapt to these autonomous cars and and
we kind of build a like human driven
cars are also going to build a model of
these autonomous cars so if they think
the autonomous car is going to always
slow down maybe they actually like do
acts more adversarial and cut in front
of them actually I do that like whenever
I see a Google car I just like cut in
front of you just to see what happens
this is forgive there was some more data
maybe another idea is also so we want
our cars to drive human-like also maybe
like using randomized algorithms could
help there to maybe the car shouldn't
always do the same thing so make it a
little bit more still safe but a little
bit more unpredictable so it doesn't
this is an idea
it doesn't like always like like drive
the exact same thing so you can actually
take admin like you shouldn't you can't
always take advantage of it the other
direction is this idea of having all
fully autonomous cars like on roads or
we are not actually in this in this
period where we have shared roads which
is really hard to get but that could be
that could be some future where you can
imagine like a part of the city is all
like fully autonomous like we'll have
for your Thomas cars and there are no
human driven cars Michael tell us more
about the need to understand the user
and if there are different kinds of
users more disciplined users like
commercial pilots versus perhaps
unpredictable users like private pilot's
or vehicle drivers yeah I'm good yeah so
my background has been in aerospace by I
joined Stanford about five years ago and
I wanted to see how much of this
research translates to autonomous
driving and there are a lot of things we
can carry over a lot of things that are
different and one major thing that is
different is that it's that the human
there's a wider variability in the
training and expertise of human human
drivers the human at the core is the
primary source of uncertainty and much
of our effort is in modeling the humans
and and how to build the system that's
that's robust to that so Michael
emphasizes that we need to understand
the human seems like the human also
needs to understand the AI system and I
would guess Justin that in your business
the people you're creating solutions for
you have to care about the mental model
they have about your solution and how it
works
you spend time coaching and teaching and
training the users so they have the
right model of your system it's a great
question and you know we probably don't
spend as much time as we should I think
you know if I had a dime for every model
that's that's rolled out to the business
and maybe didn't have the impact that we
had hoped or didn't have the acceptance
we had hoped I'd be a rich man so you
you really can't under self a value of
that change management
just the mere understanding you know
what is a what is a false positive
versus a false negative and and why you
know a really good example actually I'll
use it because more the guys from aunt
Pam is here who built this model rolled
out to our claims folks and it was to
predict claims that were headed like way
south in satisfaction so these were
train wrecks these were you know we're
likely could get sued like going to be a
consumer complaint is very dissatisfied
and so these are really really rare
events and you know we built what we
thought was a great model had great
performance but you know no matter what
you do
this is only three or four percent of
claims that are headed in this direction
and so you're gonna have pretty high
false positive rates and fortunate we
didn't have a lot of the good sort of
training and onboarding and change
management up front and lo and behold a
lot of these adjusters are getting this
thinking you know not only well this one
looks fine to me but also back to this
sort of causal kind of explainable
perspective it was very hard to produce
that and so the model has struggled
actually if I could actually kind of go
back to the other question too because I
really think you know this this AV thing
and how do we go from where we are today
to fully autonomous I think we've got to
take to do more and take more advantage
of just virtualization and simulation
one of the things we're doing with
another university way okay we know
there are other other universities we
were looking to build a model to blur
images from drones for for you know
safety for privacy reasons and this is
so drones are at kind of an oblique
angle and of course you know like Google
Street View and a lot of the other
solutions are out there are Street for
you are straight across and so one of
the techniques that this professor used
was training from video game data and I
think video games are really really
removed it's like simulation gone wild
right and so I think about autonomous
vehicles you know why couldn't we be
doing more of this more of this training
of like construction zones and just
crazy oddball situations that can be
crowd-sourced in so many ways it can be
crowd-sourced into video games to create
really really rich training data sets
without obviously putting putting any
humans at risk I think that's got to be
so we talked about it before just the
the idea of crowdsourcing and simulation
to create really really broad robust
data sets it's got to become a really
big part of how we're you train and test
excellent thank you for that we are
unfortunately out of time please join me
in thanking our panelists
you


GPT3: An Even Bigger Language Model - Computerphile

rob welcome back to computer file in
these strange times that we find
ourselves recording in you've got the
green screen up there we're having a few
laggy problems with the communications
what are you going to talk about today
then
uh yeah i thought uh today it would make
sense to talk about gbt3 because before
we had those videos about language
models and transformers and gpt2 people
seemed to like i think those came out
quite well and now there's a new one
so is it so they get better what's the
what's the deal there then it is both
bigger and better that's um
that's the headline so like the thing
about
the thing about gpg2
is
just that it was much bigger than
anything that came before right it was
more parameters and uh
just a larger language model
and that was kind of the point of that
paper right the point it was trying to
make is like
people in natural language processing
have spent a tremendously long time
working on the
all of these clever things that you can
do
uh getting in the nitty-gritty of the
technical technical stuff for like
detailed fine-tuning for these different
benchmarks that they have and so on um
and gpg2 was open ai just saying well
what if we just made a really really
huge one what would happen and it turns
out that
even without any fine tuning by which i
mean um
well we talked about all of this before
so i won't get into that detail but like
uh that it that gpt2 could do reasonably
well on all of these different tasks
even though it hadn't been trained on
those tasks it was trained purely as a
language model which means all it's
trying to do is
predict the next word or the next token
given what it's seen so far
um
and so that was kind of an interesting
finding that if you just took this like
quite simple architecture the
transformer is not that complicated an
architecture um
and the data set is not
like a fancy or sophisticated it's not
like a
highly
uh what am i saying it's not like a high
effort structured data set it's just
like a giant pile of text you're trying
to make sure it's good quality text
but
it's it's just unstructured just any
text you could get from the internet
and it's able to
perform reasonably well sometimes state
of the art sometimes not on
the all of these like specific
benchmarks for different natural
language processing tasks and that was
very impressive
and
the thing i think i said this at the
time
is that
the graphs were still curving upwards
you have these like we're not curving
upwards but they were straight they
weren't curving down so generally
speaking you start to get
diminishing returns right you can't just
keep making the model bigger forever it
was plateauing as it got bigger and
bigger
yeah that's the thing it's like you
usually would expect it to plateau
but at the level of gpd2 it was not
so not only was this model bigger than
anything that came before but also there
was an indication that like
that just scaling up is like we haven't
reached the limits of that yet
um
and that was kind of surprising because
yeah because you would generally expect
these things to plateau right you would
expect to hit the limits of
the data
like the information contained in the
data the data set size but also maybe
the
um
the training regime maybe the whole
approach right what's the biggest
airplane you can build it's pretty big
but there like comes a point where the
airplane approach doesn't work anymore
and if you want to go higher you need a
rocket right you need like a whole
different approach you can't just scale
up what you have what they found with
gpt2 was not only could you make the
model much bigger and uh it continued to
get better but also the rate at which it
continued to get better is still pretty
much a straight line with the scaling of
smaller models um
so since that time various people have
got on the
bigger language models uh train
and tried making new things
you know bigger and bigger language
models and they keep
doing
better and better kind of according to
what you would expect um
and then for gpthree
what openai has done is come along and
said
essentially the same thing they said for
gpt2 which is
okay but what if we made a bigger one
and everybody's like well we did we did
make a bigger one it's like no what if
we made like a bigger bigger now you
you've somebody's plotted this graph and
then there's somebody there at the back
of the room going
that's still going up that's still going
up there
yeah right like how far can we ride this
thing let's find out
so um
so you so you remember the the
gpg2 they released the 117 million
parameter model
and they didn't release the larger
models immediately right because there
were some concerns about possible misuse
and
over time they steadily released larger
and larger models the largest gpg2 model
was 1.5 billion parameters so gbt3
is
175 billion parameters wow
okay
yeah you need a lot of uh a lot of
compute and a lot of uh
money to run it
um so
so yeah they did have to do some clever
engineering
because like
gbt2 you can put it on a single machine
uh
at inference time um
whereas i don't think you can do that
with gpt3 i think you need a sort of a
cluster to run it um
but
yeah so the big finding with this giant
model which which is about 10 times
bigger it's 117 times bigger than gpt2
and about 10 times bigger than the
previous biggest thing which was uh
turing nlg
which
um
and what they find is when they look at
the graphs
they're still going up
uh oh no
so that person
is going to be thinking still
what if what if right
we could we could we could still go
bigger and it does look like it would uh
continue to
get better so
how good is it
some of the main takeaways are
when you have it write an article
and you ask human beings to
differentiate
articles written by gbg3 from articles
written uh by humans
they get it right about 52 percent of
the time i'm looking at the table here
it says human accuracy in identifying
whether short news articles are model
generated these are articles of about
200 words and basically they tried
generating uh with all of the different
sizes
so
gpt 3 small medium and large on this are
um
i think equivalent sizes to the gpt2
ones
and then you can see how the accuracy
with which
humans are able to identify just
steadily goes down basically the small
model they are
76 of the time able to tell
if correctly if it's human or or ai and
then just steadily drops down until you
get to the 175 billion parameter model
where they're at 52 what i thought was
it would be fun to run a little
experiment with everybody at home
because they had the thing generate some
poems and there are samples in the paper
the way that you get this model to
produce things is you give it some text
and then you say and now it's your turn
to continue from here so they gave it
something which kind of looks like it's
from a
compendium of poetry so it has you know
the title of the poem by this person
and then the full text of the poem and
then the title of another poem
so what i thought it would do because we
know this the poem we know the poet that
gpt3 is trying to imitate
uh that i could try reading like
randomly picking one of wallace stevens
actual poems and randomly picking one of
these uh i think these aren't
cherry-picked either
yeah uncurated completions yeah so uh
and then we'll see so i'm going to
randomly pick one of each and then i'm
going to randomly decide which one i'll
read first so that you don't get any
clues okay
so uh this time they're both by wallace
stevens this first poem is called
shadows on the way i must have shadows
on the way
if i am to walk i must have each step
taken slowly and alone to have it ready
made
and i must think in lines of grey to
have dim thoughts to be my guide must
look on blue and green and never let my
eye forget that color is my friend and
purple must surround me too
the yellow of the sun is no more
intrusive than the bluish snow that
falls on all of us
i must have grey thoughts and blue
thoughts walk with me if i am to go away
at all
that's one time
the other one is titled
fabio of florida
bark of phosphor on the balmy beach
move outward into heaven into the
alabasters and night blues
foam and cloud are one
sultry moon monsters are dissolving
fill your black hull with white
moonlight
there will never be an end to this
droning of the surf
everybody place your bets
[Applause]
the problem is people who know poetry
really well who would be well placed
to
decide you know which of these they
prefer or whatever the chances are
they'll they'll know the originals so
it's hard to get a fair test without
magical google i have no idea which is
which i mean i don't know should we
reveal it here on computer file or
should we let people have a think about
it or should we say at the end yeah
maybe at the end of the video poetry is
one thing and at the risk of offending
some poetry fans it can be thought of as
kind of ethereal and maybe not so
um grounded in fact and therefore it's
okay to predict that sort of stuff and
and to emulate a poet but what about
things like scientific papers if you've
fed it enough science enough scientific
papers do you think
could it come up with something that
we've not really realized before or
something new
yeah so
my instinct is to say
no it's just predicting the next word
right it's just a language model
it doesn't have the ability to
build the kind of abstract
uh mental structures that you need in
order to actually kind of synthesize new
um
knowledge
but
uh there's a kind of an outside view
that says that we thought that about a
bunch of things that it's now seems to
be doing so
i'm not going to say that he definitely
couldn't do that
um
so so one example uh of a task which it
got better at
tremendously better at is arithmetic
which
is kind of an interesting task because
again
it's a language model
it's not trying to do arithmetic it's
not designed to do arithmetic
but
so with gpt2
if you put in two plus two equals
and get it to give you the next token it
will give you a four
but that's not very surprising like
that's not very impressive because you
would expect in its data set to see that
string two plus two followed by the
string four
very many times that's pure memorization
right the thing doesn't have to have any
understanding of what letters or what
numbers are at all it can just see the
sequence of tokens give you the next one
um
and then
the problem gets harder and harder
the more like
23 plus 48 or something
that's more difficult because
it's less likely
that that specific string has appeared
in the uh in the training data set right
so this gets more and more
uh difficult and more and more like
actual reasoning
um you can see me doing big air quotes
there but
um
the the longer the numbers get right if
you can reliably add 10 digit numbers
together
then it's hard to deny that what you're
really doing has you have to really be
doing addition right you're not there's
no way you could memorize to that
yeah um
but it's kind of interesting because
gpt3 does way better
but it can't add 10 digit numbers
so let me find the let me find the graph
because they graph this
so it starts to run out steam
effectively right
much as a human does
yep
so what i'm looking at now is a graph of
performance on a bunch of different
arithmetic tasks and you can see that
just going up to like two-digit edition
gbt2
does pretty poorly so the 1.3
billion parameter model which i guess is
the closest equivalent better than
chance but not much at all
so the thing is so two-digit edition and
uh three-digit edition
are things which like by the time you're
at three-digit edition
you're not going to be memorizing from
the data set because firstly
um
i think that the cleaning of the data
set made some attempt to remove if there
was something that was just like a giant
table of the times tables or something
like that i think they tried to remove
that from the data set
and
secondly if you're if you're doing
three-digit edition
that's a million
different possible problems right
it's like quite a lot of network
capacity to do by memorization
people learn multiplication tables
um and this is like
apparently the most
effective way of teaching something that
works like a human brain
and then you have some procedural rules
for taking you like you you memorize
that three plus three is six
and then you have these procedural rules
about like carrying and uh and those
kinds of things to do
larger additions and then you
iteratively like systematically apply
that
um
but yeah the larger the numbers get
the harder it is to memorize and they
actually ran an analysis so they
searched
for
the addition problems they search for 2
000 of them
just looking through the whole data set
does 38 plus 49 exist anywhere in this
data set and they found 17 matches
right so 0.85
of the of these problems
occurred in the database
but gpt three's performance
on
uh two-digit addition
is extremely good it's basically 100 of
the time 2d to subtraction only slightly
worse uh and then three digit addition
and subtraction again it's getting like
80
90 percent and it's a big jump from the
smaller models
what they're kind of suggesting in the
paper
is
that
it has actually learned
how to learn
okay
like that's what that's that that's the
uh
the interpretation of this that they're
pushing that
the the
in order for
um
in order to perform sufficiently well at
this language modeling task
the best thing to do
is to actually while looking at the
context
learn
specific rules for how the context
behaves
so that you can continue it more
accurately okay
yeah
um
and so
i have an example
i have like a way of thinking about this
which is not that tight analogy but i
think it
might be helpful yep
which is okay so suppose you've got uh
you're doing like symtorial you have a
robotics task you're training an agent
to do some thing with a robot right
running things on the physical robot is
super slow so you're using a simulation
but you have a problem which is that the
simulation is not exactly the same as
reality you have this physics model that
you've built that is supposed to
simulate exactly how the robot and the
environment of the robot so that it can
learn in the simulation and transfer it
in practice that doesn't work very well
because it's really hard to know you
always have some uncertainty about just
little variables you know how much like
each part of the robot weighs or
whatever
because you built it but like what's the
what's the coefficient of friction on
the ground at the spot where the robot
is right now like you have to estimate
right and it might not be right and so
if you train something if you take your
best guess put it in and you train a
system it might find a policy which is a
policy of like doing some kind of leap
that that's the best way to achieve
whatever it is you've told it to do like
get from here to there quickly or
something
um
and then you have a problem because then
if the if it's relying on the current
efficient coefficient of friction being
this specific thing
and then you run it on the real robot
and the thing completely falls over
because the it's out of the distribution
that it was trained on
so one thing that people do is when
they're simulating it they randomly vary
it right you say we think that the
coefficient of friction here is around
this number but we're actually going to
give it every time every like episode
we're going to give it a random value
somewhere in the range from like 0.9 of
that to 1.1 of that you know
and then
um
the
the machine learning system is going to
learn a policy that's able to handle
any coefficient of friction within that
range so it's learning to adapt right
right well that's the thing so there's
two different things that could happen
here
one is
the model could learn
oh if i do this kind of leaping thing
then some of the time i completely stack
it and it's very embarrassing
so
uh i'm going to just do like a a
shuffling thing right that's like much
more reliable that works across a wide
range of um
friction values right that's one thing
you could do but
there you've kind of sacrificed some
performance right
but if your model is more is more
sophisticated
it could learn something like
okay first
just slide your foot along the ground a
bit
to get a feel for what the friction is
like yeah and then if it's correct
do the leap otherwise do something else
or like adjust how you're leaping so
that it always works so that's actually
being adapted rather than the lowest
common denominator which is the sort of
prior priority
exactly
and nothing necessarily changed for that
except the power of the model right
if your model is too small
then it's not going to be able to learn
something as complicated as
measure the friction and then do one of
these five possible things depending on
the friction right it's only going to be
able to learn one thing that it could do
and so it has to just learn one that
does okay on all friction levels but
if you have a larger model you can learn
a better policy which actually adapts
um
i don't know like this is this is purely
i'm not like talking about a specific
paper or anything this is just a thing
that i thought of
um
and so what they're suggesting i think
in this paper
is that
gpt3 is doing something similar that in
order to perform really well at this
language modeling task it's actually
learning online
from the context
what the task is that needs to be done
are we getting into agi territory here
gradually i mean i think it's like it's
a step on the path um it's not like us
it's not
dramatically closer than we would expect
or anything like that what they're
interested in mostly in this paper is
gpg3 as a few shot learner
which is
so you have um the standard machine
learning model is
you give the thing loads and loads of
data
right the more data points the better
but sometimes
you have
uh a few shot learning problem which is
where you have to learn using only a few
examples so let's say for example um
your
phone you want to unlock your phone
right
you can train a model that does all
kinds of face recognition and stuff
but
in this case you want to train a
classifier
to distinguish this face from other
faces
but it's only going to get you know
you're going to give it like three
pictures of yourself
whereas usually for a classifier you
would want to be giving them thousands
so that's like a few short learning
problem um
and and
so you can kind of you can kind of
imagine this is all kind of fuzzy
because
it's like
you can think of it as
when you're giving the thing the context
you can give it examples
and how many examples you give it
is a bit like just giving training
samples to a machine learning system
but what's impressive is that gpt3 seems
to be able to do these tasks
with what would be very very few
examples compared to standard machine
learning methods right
the thing that's kind of interesting is
when you look at um
we can stick with arithmetic stick with
addition the number of examples that you
give it
makes a big difference
if you just say
uh what's you know this number plus this
number equals
it will
uh get that right a certain percentage
of the time but if you say you know 10
plus 10 equals 20
uh you know 25 plus 30 equals 55 you
know and give it a few of those
then the performance gets much better
there's various different ways that you
could interpret this
is it actually learning
how to do addition
from
looking at your examples
or is it just figuring out that what you
want is edition
is it is it is it learning edition or is
it like locating addition
in the space of possible tasks that it
already knows how to do
kind of unclear for all pretty much
every task they try it in the zero shot
the one shot and the few shot settings
right okay um
and they they look at how well it
performs
and it consistently does you know better
the more examples you give it
um up to the size of the context
obviously you can't um
i think you can only give it 20 48
tokens
which actually is a very large like
that's much bigger than
most other language models out there
but
they find it does better it does better
than what you give it but also
the
ratio seems to go up the larger the
language model is right
so so all of the models do better with
more examples
yes but
the difference between the zero shot and
the few shot
is bigger for the bigger models
so it suggests that
perhaps the larger models are actually
like making better use
of that information in the context to
actually learn things
okay um yeah so it's more efficient
right
right right it's less about just like
using the context to find the relevant
parts of the training data to sort of
power it back that you've memorized
and more about actually learning what to
do from the context like
recognizing that what's happening is
edition and then actually doing addition
yeah um
because
you have to have some way to account for
the fact that this thing is reliably
doing addition when
only a very small number of those
addition um
problems actually occurred in the
training data set yes okay um the other
thing that's interesting about it is
apparently when it gets them wrong
it tends to get them wrong in
sort of human plausible ways okay like
you want to carry the one or something
yeah exactly exactly
um
and that is another indication that what
it's doing here is is something like
actual
um
actual addition but that's pretty
exciting and there's a sense in which
you know it you could imagine it
learning the rules of addition
sort of in the same way that it learns
the rules of grammar in order to change
something into a question then you swap
these two words around and add a
question mark or whatever it is you know
in order to do addition then you
pull out the thing that you've memorized
for these two and add it and then do it
again to this one
or whatever um
and that is the kind of thing that you
that i most people i didn't
expect
a transformer trained in this very
straightforward way
to be able to learn right
and the big takeaway is
we've gone we've gone 117 times bigger
than gpt2 and gpt2 they started doing it
because the curves aren't
leveling off they're still not leveling
off so yeah we don't know um
how far we can push this type of
language modeling
but uh a little bit further yet at least
in this case the first one was gpg3 and
the second one was as well yeah so the
bluish snow and etc etc
that was the only thing i was thinking
about and then i started thinking well
no it's kind of bluish white great well
done duty-free and also it you know it's
often under a blue sky and so on you
need blue


AI Safety Gym - Computerphile

this episode has been brought to you by
fast hosts find out more about them
later so I wanted to talk about this
paper out of opening eye benchmarking
safe exploration in deep reinforcement
learning what comes with this there's a
blog post so I wanted to explain this
because when I saw the blog post and I
think a lot of computer file viewers
would have the same reaction I what I
thought is what the hell is a game some
kind of meatspace thing I don't know
[Music]
so it's a bunch of these environments
right that that allow you to true train
yeah the opening eye safety gym
benchmark suite which is a bunch of
these environments that you can run your
systems in and have them learn is this
the only thing to do with those AI good
world through thoughts Bluffs yeah yeah
kind of in the same way that the grid
world's paper did this paper introduces
environments that people can use to test
their AI systems and this is focusing
specifically on safe exploration and has
a few differences they kind of
complementary the environments
are a little bit more complex they're
continuous in time and in space in a way
that their grid well it's all like very
discrete you take turns and you move by
one square whereas in this case it's a
lot more like Majorca where you actually
have like a physics simulation that the
simulated robots move around in so it's
a slightly more complex phone of
environment but the idea is to have in
the same way as with grid worlds or
anything else to have a standardized set
of environments so that you know
everybody's comparing like with like and
you actually have sort of standardized
measurements and you can benchmark you
can compare different approaches and
actually have metrics that tell you
which one is doing better which is like
it's not super glamorous but it's a real
prerequisite for how progress actually
gets made in the real world if you can't
measure it it's very hard to make
progress or know if you're making
progress the problem of safe exploration
is in reinforcement learning which is
one of the most important and popular
ways of creating our systems for various
types of problem the system is
interacting with an environment and it's
trying to maximize the amount of reward
that it gets so you write a reward
function and then it's trying to get as
much out of data as it can and the way
that it learns is by interacting with
the environment and so this basically
looks like trial and error right it's
doing things and then it gets the reward
signal back and it learns oh that was a
good thing to do that was a bad thing to
do and the problem with that is it's
very difficult to do that safely and
it's kind of a fundamental problem
because in order to do exploration you
have to be taking actions that you don't
know what the result is going to be
right that the only way that you can
learn is by trying things that you're
not sure about but if you're trying
random things some of those things are
going to be things that you really
shouldn't be doing in any exploration is
dangerous I mean that's sort of goes
with territory for human explorers so
Apollo 11 right
right we done a bit research we'd send
spaceships out we had an idea of what's
what but it was still a dangerous thing
to go and land on the moon exploring
comes with danger right right yeah but
there are there are safe ways to do it
or there are safer ways to do it they
could have tried to launch astronauts on
the first thing that they ever sent to
the moon they didn't do that because
they knew how much they didn't know and
they didn't want to risk it until they
actually had a pretty good understanding
of what they were dealing with and it's
kind of similar like if you look at some
of the standard reinforcement learning
approaches to exploration what they
involve is doing things often what they
involve is doing things completely at
random right
you just especially at the beginning of
the training process where you really
don't understand the dynamics of the
environment you just do flail and see
what happens right and human beings
actually pretty much do this as well
it's just that when babies flail they
aren't really able to hurt anything but
if you have a three-ton robot arm
flailing around trying to learn the
dynamics of the environment it could
break itself it can hurt somebody you
know but when you mentioned three term
robot arms flailing around and guessing
that the people who do that kind of
development will have done some kind of
simulation before they've built the
thing right right there's two sides to
it right part of the reason why we
haven't had that much safe exploration
research is because simulation is good
but also part of why we use simulations
so much is because we don't know how to
safely do it in the real world for very
simple tasks you can write good
simulators that accurately represent all
of the dynamics of the environment
properly but if you want a system that's
doing something complicated like
generally speaking with these with these
robots for example you still don't go
near them they don't smash themselves up
and they don't smush the environment up
because you've simulated that but while
they're operating like how do you write
a simulation of how a human being
actually moves in an environment with a
robot like this is why you look at
self-driving cars they train them a huge
amount in simulation but it's not good
enough it doesn't capture the complexity
and the diversity of things that can
happen in the real world and it doesn't
capture the way that actual human
drivers act and react so
everyone who's trying to make
self-driving cars they are driving
millions and millions of real-world
miles because they have to because
simulation doesn't cut it and that is a
situation where now they're not just
running reinforcement learning on those
cars right we don't know how to safely
explore in a self-driving car type
situation in the real world trying
random inputs to the controls is like
not viable if you're using reinforcement
learning if you have something that you
don't want this agent to do you give it
a big penalty right so you might build a
reward function that's like I want you
to get from here to here and you get
points for getting there faster but if
you there's a collision then you get
minus large number of points sometimes
people talk about this problem as though
reward functions are like not able to
represent the behavior that we actually
want no people say you can't write a
reward function that represents this and
it's like well I mean you can write a
road function but like you can't write
like plausibly it's possible but like
how you actually gonna do it so like
yeah you're giving a big penalty to
collisions but like how do you decide
that penalty what should it be you have
this problem which is that in the real
world people are actually making a
trade-off between speed and safety all
the time everybody does any time you go
just a little bit after the light has
turned red right or just a little bit
before the light has turned green you're
accepting some amount of risk for some
amount of time if you go after it's gone
red for long enough and you will meet
someone who went a bit early on the
green and you know teach each other
things about their trade-off between
speed and safety that will stay with you
for the rest of your life people talk
about it like Oh what we want is no
crashes and that's not actually how it
works because that would correspond if
you wanted that that would correspond to
sort of infinite negative reward for a
collision and in that scenario the car
doesn't go anywhere if that was what we
really thought then the speed limit
would be point zero zero one miles an
hour there is some like acceptable
trade-off between speed and safety that
we want to make the question is how do
you actually pick the size of that
punishment to make it sensible like
what what how do you how do you find
that implicitly it's kind of difficult
the one approach that you can take to
this which is the one that this paper
recommends is called constrained
reinforcement learning where you have
your reward function and then you also
have constraints on these cost functions
so standard reinforcement learning
you're just finding the policy that gets
the highest reward right
whereas in constrained reinforcement
learning you're saying given only the
set of policies that crashes less than
once per curve from any million miles
find the one of those that maximizes
reward so you're you're you're
maximizing reward within these
constraints yeah reinforcement learning
and constraint reinforcement learning
are both sort of frameworks they're ways
of laying out a problem they're not like
algorithms for tackling a problem
they're they're they're a formalization
that lets you develop algorithms I guess
like sorting or something you know
you've got a general idea of like you
have a bunch of things and you want them
to be in order but like how many there
are what kind of things there are what
the process is for comparing them and
then there's different algorithms that
can they can tackle it I haven't seen a
proof for this but I think that for any
constrained reinforcement learning setup
you could have one that was just a
standard reward function but this is
like a much more intuitive way of
expressing these things so it kind of
reminds me of them there's a bit in
Hitchhiker's Guide where somebody is
like I've got oh you've got a solution
no but I've got a different name for the
problem I mean this is better than that
because it's a different way of
formalizing the problem different way of
sort of specifying what the problem is
and actually a lot of the time finding
the right formalism is a big part of the
battle right the problem how do you
explore safely is like under defined you
can't really do computer science on it
you need something that's expressed in
the language of mathematics and that's
what constrained reinforcement learning
does it gives you a slightly more
intuitive way of specifying rather than
just having this one thing which is this
Galvani you are you doing well on art
you get to specify here's the thing you
should do and then here's also the thing
or things that you shouldn't do slightly
more natural it slightly more
human-friendly
formalism that makes it you would hope
would make it easier to to write these
functions to get the behavior that you
want in the real world it's also nice
because if you're trying to learn so I I
did a video recently on my channel about
reward modeling where you actually learn
the reward function rather than writing
the reward function you have a part of
your part of your training system is
actually learning what the reward
function should be in real time and the
idea is that this might help with that
as well by it's kind of easier to learn
these things separately rather than
trying to learn several things at the
same time and it also means you can
transfer them more easily like if you
have a robot arm and it's making their
pens and you want to retrain it to make
mugs or something like that then it
would be that you would have to just
relearn the reward function completely
but if you have a constraint that it's
learned and it's like don't hit humans
that is actually the same between these
two tasks so then it's only having to
relearn the bits that are about the
making the thing and the constraints it
can just keep them from one to the next
so it should improve performance and
training speed and also safety so it's
again a nice kind of win-win the other
thing that's kind of different about
various formulations of constrained
reinforcement learning is you care about
what happens during training as well
right standard reinforcement learning
you are just trying to find a policy
that maximizes the reward and like how
you do that is kind of up to you but
what that means that standard
reinforcement learning systems in the
process of learning will do huge amounts
of unsafe stuff right whereas in a
constrained reinforcement learning
setting you actually want to keep track
of how often the constraints are
violated during the training process and
you also want to minimize that which
makes the problem much harder because
it's not just make an AI system that
doesn't crash
but it's like make an AI system that in
the process of learning how to drive at
all it crashes as little as possible
which just is yeah makes the whole thing
much more difficult and so we have these
simplified environments that you can
test your different approaches and
they're they're fairly kind of
straightforward reinforcement learning
type setups you have these simulated
robots there's four of them
you've got point which is just a little
round robot with a square on the front
that can turn and drive around car which
is a similar sort of setup yet has
differential Drive so you have input to
both of the wheel sort of tank steering
type setup and that drives around and
you have dog oh I did you not do GTO
which is a quadruped ed that walks
around and then you have a bunch of
these different environments which are
basically like you have to go over here
and press this button and then when you
press the button a different button will
light up and you have to go over and
press that one and so on or get to this
point or like push this box to this
point you know it's basic basic
interactions but then they also have
these constraints built in which are
things like hazards which are like areas
that you're not supposed to go into or
the VARs is they call them farces which
is like objects that you're not supposed
to bump into and then the the hardest
one is gremlins which are objects that
you're not supposed to touch but they
move around as well the idea is you are
trying to create systems that can learn
to get to the areas they're supposed to
be or push the box or you know press the
buttons or whatever it is that they're
trying to do while simultaneously
avoiding all of these hazards and not
breaking the laws is not bumping into
the gremlins or whatever else and that
they can learn with a minimum of
violating these constraints during the
training process as well which is a
really interesting and quite hard
problem and then they provide some
benchmarks and they show that standard
reinforcement learning agents suck at
this
they're trying to do anything to learn
they don't care about the learning
process exactly exactly and then there
are a few different other approaches
that do better this is really nice if
you have ideas and again like the grid
welds a thing you can download this and
have a go
you can try training your own agents and
see how well you can do on these
benchmarks and if you can beat what open
AI is done you know and then you've got
something that's it's publishable that's
going to advance the field so I really
like this as a piece of work because it
provides a foundation for more work
going forward and in a kind of
standardized to understandable way fast
hosts is a uk-based web hosting company
which offers a wide range of web hosting
products and other services they aim to
support UK businesses and entrepreneurs
at all levels providing effective and
affordable hosting packages to suit any
need as you'd expect from someone called
fast hosts they do domain names it's
easy to register and have a huge choice
of domains with powerful management
features included one thing they do
offer is an e-commerce website builder
this provides a fast and simple way for
any business to sell online
it's a drag-and-drop interface so it's
easy to build a customized shop on the
web even if you have no technical
knowledge you can create an online store
and you can customize simply with
drag-and-drop functionality no designers
or developers are required fast host
data centers are based in the UK
alongside their offices so whether you
choose a lightweight web hosting package
or go for a fully fledged dedicated box
their expert support teams are available
24/7 find out more by following the link
in the description below