How I think students should orient to AI safety | Buck Shlegeris | EA Student Summit 2020

hello everyone and welcome to this
session
with buck sluggarus on ai safety
i'm sophia davis fogel and i'll be the
emcee for this session
we'll be starting with a 15-minute talk
by buck then we'll move on to a live q a
session where he will respond to some of
your questions
you can submit questions using the box
to the right hand side of this video
you can also vote for your favorite
questions to push them higher up the
list
buck slegeris does independent research
on topics related to long termism
he worked at the machine intelligence
research institute from
2017 to 2020 doing a mixture of research
and
outreach without further ado here's buck
hello there ladies and gentlemen uh my
name is buck i'm super glad to be here
i want to tell you how i think college
students should orient to ai safety
uh for bonus content on this talk
including the google doc where i wrote
it
uh you can look at the link that i've
provided there and
read all the places that my friends
think that various points i've made are
subtly wrong
uh all right so my background i ran
across les from when i was in high
school in 2010
started reading about ai safety and wild
animal suffering and give well and all
these interesting things
uh uh i was in university
from 2012 to 2014 the australian
national university where i
wore colorful clothes and played music
sometimes i
was a software engineer earning to give
from 2015 to 2017
2017 to earlier this year i was working
at miri doing a mixture of recruiting
and movement building
and ai safety research stuff and i'm now
working on various independent projects
so i want to talk about how to orient to
ai safety
uh so by this i mean maybe a level of
abstraction higher than i think you
should
learn about this particular topic or i
think you should study this particular
thing
i want to give you my sense of how you
should be prioritizing
what things to do my assumption about
you is that you're a college student who
thinks ai safety is plausibly important
i sometimes assume you're a person who
wants to do ai safety technical research
and
knows computer science and math uh i
sometimes assume that you're someone who
does ai safety
who wants to do asap technical research
in as much as something isn't aimed at
you you will
figure it out for context then it'll be
good uh all right
so my key claim is that
you should try to engage with the
content of ai safety while you're in
school
and by engaging with the content i mean
something like you should be trying to
read things that are written by ai
safety researchers for ai safety
researchers
and you should be trying to come to your
own opinion about the answers to the
questions of the big
uh the answers to the questions that are
most important to the field
so i don't think you should do this
because i think that you'll come up with
lots of great stuff on your own
necessarily
but i do think that doing this kind of
thing
uh makes it more likely that you end up
having
good judgment and doing useful research
eventually so i don't think this is just
trivial advice for instance i wouldn't
give this advice
uh to someone who wanted to be a
theoretical physicist or mathematician
because i think that if you try to read
cutting-edge theoretical physics
or cutting-edge math while you're in
school you will basically fail to
understand it and waste your time and
you should be focusing on learning the
basics
i think this is roughly speaking not
true of ai safety
i'll come back to talk about that in a
bit more detail later
the reason why i think this is important
is that i think we really need people
who've tried to think through
the whole thing by that i mean something
like
people whose ai safety research is
guided by a
specific model they have of why the
things they're doing are going to reduce
the probability of ai existential risk
not everyone thinks that this is
obviously true the reason why i think
it's important
for ai safety to have people who try to
think through the whole thing
uh is that in my experience ai safety
researchers
who have clear pictures of the goal of
their research are more likely to do
research which seems to me
likely to actually reduce existential
risk um
another reason for this is some of the
most valuable work that i've seen on ai
safety
research from the last few years in my
opinion falls into the category of
you know trying to think through the
whole thing so for instance i thought
ben garfinkel's stuff and richard knows
stuff from recently were both pretty
cool
uh though of course we'd expect the
proportion of research and ai safety
that's kind of high level to full over
time
as we know more about what we're doing
uh i also think that asaf people having
thought through ai safety more makes the
field a bit healthier and makes it seem
smarter to outsiders
i think that if you don't have an
opinion
on the route by which your work leads to
a reduction in existential risk
you're gambling on the person you're
working for having a good sense of this
which i think is reasonable in a lot of
cases i think it's nice to preserve the
option of not having to make that gamble
uh and that's why i think it's valuable
to try and think through the whole thing
um yeah so
all right suppose you're sold on wanting
to think through the whole thing what
are the things you should think about
exactly
i want to break it down into two
categories first category here is
high level forecasting questions the
most interesting forecasting question
about ai risk is what's the probability
that we all die
due to agi um i think that
thinking through what you think this
number is is valuable not just because
the number is pretty interesting
but also because i think that once
you've thought it through
you will have built for yourself a
concrete model
of what leads to ai risk being higher or
lower so if one of the things that was
part of your estimate was
you know how hard will various companies
try to
do various ai alignment things and then
you'll realize that anything which
increases the probability of that
is going to reduce your estimate of
existential risk i think this is a real
effect
uh another high level question which is
pretty interesting is what will the
world look like in the five years before
and after agi
uh i this is related to questions about
takeoff speeds and obviously it
plays into the question of what's the
probability of extinction quite a lot
seems very important to me
the other type of question is technical
questions about ai safety of which
i think the key one is something like
what are the alignment strategies that
are scalably safe and competitive by
which i mean
what are the alignment strategies that
keep being aligned when the system is
really smart
and aren't hopelessly incompetent
compared to ai systems with a
similar amount of resources being thrown
into them
those are the questions i think the most
important i want to go back to
my claim about how ai safety compares to
other fields where i was saying
that it is in some sense shallower than
math and physics because it doesn't have
deep chains of prerequisites
um it's not that part of the cutting
edge it's another way of saying that
i think that there's probably something
like one to two undergraduate courses of
ai safety content
and four to eight undergraduate courses
of prerequisites
where by ai safety content i mean like
the basic explanation of things such
that afterwards
you can read the alignment newsletter
and understand
basically the context for all the new
research which is happening
i think that ai safety has kind of a
similar amount of prerequisites to
advanced data structures and a similar
depth to some parts of this field
so in stanford and mit uh there are
these courses called advanced data
structures i think the one from stanford
has particularly good lecture notes
uh and these courses have final projects
where you have to come up with some data
structure stuff
and the people who do these courses
sometimes end up publishing their final
projects
i think ai safety is something like as
deep as this where it's not unreasonable
to think that after your second course
uh in a particular in a particular path
you're
saying things that are new and
interesting some other ways ai safety
compares to other fields is i think it's
kind of a wild west kind of situation
uh one way in which this is true is
there isn't really a textbook for ai
safety
so a lot of the time when you want to
learn things you're having to scrounge
around random blog posts
various people are trying to make this
better in various ways of course um
i think there are a reasonable number of
ai safety arguments that are folklore in
the sense that
they haven't really been written down
but they're just a thing that ai safety
researchers have said to each other over
the course of however many years
uh i think it's really great when people
do write these down
another fun fact about ai safety is i
think there are a lot of hidden
prerequisites
by which i mean other subjects like
machine learning or algorithmic
information theory
where uh a bunch of like
we've imported a bunch of jargon and
ideas from these fields
often not in a super deep way such that
you'd be fine if you just read the
wikipedia page rather than having read
the
um you know done the relevant
undergraduate course
but that still uh makes ai safety a
little bit harder to get into than a lot
of things and you should be very willing
to open up wikipedia when there's
something you don't get
so i want to talk about two mistakes
that i see students making sometimes
about ai safety so one of them is taking
the
ideas in the field insufficiently
seriously where they
don't spend as much time as i think they
ought to reading things that the ai
safety research community is coming
is coming out with thinking that they
should instead spend
all of their time learning more computer
science or machine learning or math
uh and planning to just learn the bits
of ai safety that they need later
i think that this has the tendency to
leave people kind of bad at steering
themselves towards the right problems in
ai safety
like according to me a lot of the value
to be created
comes from having a deep understanding
of the specific things people have
talked about
inside the field of ai safety it's not
like there is a huge number of shovel
ready tasks
where we just need people who are very
good at this particular technical thing
to come in and solve these nice open
problems we have
though that is a little bit true
sometimes i think there's kind of an
inverse mistake that people make as well
where they take the field too seriously
and think that it's hopelessly
inaccessible because people have surely
built
giant uh layers of abstractions on top
of each other such that they'll never be
able to understand the cutting edge and
never be able to contribute to the
discussions
until they are far more uh
far more experienced than an
undergraduate i don't think that this is
the right call either i think it's
better to
start reading this stuff and arguing
about this stuff
uh when you're an undergraduate i think
that it's very normal for undergraduates
to have very reasonable understandings
of what's happening in ai safety
so a question that i want to talk about
specifically
uh which is kind of off the theme of
uh thinking that you should get really
deep into the actual content of ai
safety
is uh whether you should get really good
at machine learning
i think the answer is you should
probably get reasonably good at machine
learning
so if you get really good at machine
learning uh this means that you're able
to do certain kinds of ai safety
research that i think are really
exciting
so open ai needs some people to help
them align gpt3
i think this is just quite an exciting
thing that they're working on over there
and i really hope they succeeded it and
the only way
to produce value there is being really
good at machine learning or programming
um so i think that if you're the kind of
person who would plausibly get really
good at machine learning
it's probably worth your time to try and
do that
on the other hand i think there are many
parts of ai safety research that don't
rely basically at all on being good at
actually doing machine learning
uh for instance uh writing
richard ner's piece on uh
uh ai safety from first principles or
lots of other a lot of other types of
research
um i think that basically everyone who
wants to do technical ai safety research
should probably know basic machine
learning things like how to train neural
net how convolutional neural nets
work how deep queue learning works uh
similar things to this you can see a
longer list in my google doc
for this post um and most people
involved in air safety technical
research should probably know
substantially more than that
uh this is because some extent by the
argument
that knowing more about machine learning
just means you have a better sense of
what's going on and you're less likely
to do research that is just
predicated on some really dumb
assumptions about what machine learning
is like
um different researchers differ on how
far that goes
so there are some people who think that
if you don't know really a lot of
machine learning
you're just gonna have really dumb
opinions about ai safety uh and you're
just gonna
your your intuitions are gonna be
substantially off i personally
am not persuaded by this i think there
is like one
really good example of someone having a
really good ai safety idea
that i had completely missed because i
didn't know any machine learning and i
just definitely would not have thought
of that because i did not know some
particular machine learning facts
um but i don't think this is massively
ubiquitous
uh the main thing the main thing in ai
safety where it seems really crucial to
know lots and lots of machine learning
is if you want to in fact do good via
[Music]
training models which i think is a
pretty reasonable way of doing
ai safety stuff so my overall take here
is i think that if you want to do ai
safety research and you're planning to
get a stem major
you should probably spend like 20 to 80
percent of your effort that you're
spending on
ai safety uh on learning machine
learning and you should decide between
these extremes by thinking about how
good you seem to be at machine learning
compared to other things
and how enthusiastic you are about
spending more time on it
so here are some concrete suggestions
about what things you should maybe do
uh i think you should go to evan and
osseo's talks
uh evan's talk is about the
technical question i was describing
earlier uh what are strategies for
building scalable and competitive
aligned ai systems
it's based on a post of his which i
think is one of the best posts to try to
read to orient yourself to the technical
questions of ai safety
um osseo's talk is about uh
what some interesting research according
to her
uh about ai safety uh based on her
experience writing the ai alignment
newsletter
and reading a bunch of these reading a
bunch of ailments stuff
uh so along those lines my reading
recommendation is the ai alignment
newsletter
i think the ai alignment newsletter is
great it covers all different parts of
ai alignment
and more general machine learning
results
i think that there's a whole lot of
stuff i think it's just
very well worth your time to read those
summaries uh i have a longer list of
recommendations on my website as as
mentioned
uh sometimes i run workshops the ai risk
for computer scientists workshops or
other workshops uh i'd love to have you
apply to those
if we ever run them again after this
terrible crevid times
uh and yeah i would love to get emails
from you all and answer questions you
have
uh evan cubinger is also down for
answering questions about air safety
stuff over email
that's my talk
thank you for that talk buck uh i'd love
to dive right into some questions
one of your suggestions is for students
who are interested in ai safety
to engage with the content of ai safety
by reading things that are written
by ai safety researchers for ai safety
researchers
later in the talk you mentioned the
alignment newsletter which anyone can
sign up for
but aside from things like that people
may be wondering how they can find or
access this type of content can you give
some specific suggestions
in my bonus content doc i have a list of
some of my favorite resources that i
think are good along
along these lines i think also the ai
alignment newsletter is just
a really excellent resource because it
has uh relatively accessible
explanations of a wide variety of
safety things so i think that would be
my first recommendation um
but in my dock i recommend the things
that i think are uh the best
things along the category i mentioned
fantastic okay that's great thank you
um you also said that we need people in
ai safety who
tried to think through the whole thing
by which you mean
people whose ai safety research is
guided by a specific model they have
of why the things they're doing are
going to reduce the probability of aix
risk
i i think this is an excellent insight
and that the reasons you give are quite
compelling
um but one fear i have is that this
might
sound quite intimidating for people who
are just starting out thinking about
this field
and wondering how to dip their toes in
uh so i think you did a great job of
explaining why it's worth it but i'm
wondering if you have any comments you
can add to help people understand why
it might not be as daunting as it sounds
yeah so i guess i want one thing i want
to say is uh just because you have an
opinion on the answer to this question
doesn't mean you have to rely on it
being correct
i kind of think of these as uh as
separate exercises
one where you're you're trying to answer
these questions and the other one where
you're deciding what to do
uh and i think that for example when
you're trying to decide what to actually
do
one thing you should be really excited
for is finding someone who's a more
experienced ai safety researcher than
you
to say that they really need this
particular thing done
uh just because i think that's a better
learning experience um
i think that's the first part uh i think
that
i i agree that the question of these
questions of you know
what's probably of a extras and what
needs to be done to reduce it are quite
hard and daunting
it doesn't seem like there's anyone
who's particularly qualified to answer
them there
so it's not like you're entering a field
with lots of qualified experts
uh it's more like everyone is trying to
figure out things to mention their
abilities in a field where
everyone is kind of an amateur you can't
get degrees in this uh answer for that
reason you might expect to do
less badly compared to the best
qualified people you otherwise would
yeah that's that's great i think that
makes a lot of sense and i think that's
quite helpful on this point
uh the first major question that you
named is
what is the probability that we all die
due to agi and
you recommend that people spend some
time developing a model that
will inform their view here um so i'm
just wondering if you can kind of talk
about what are some good reasons for
people to
really grapple with this themselves
instead of just deferring to current
experts and i recognize that
um part of your your recent answer kind
of addresses this but
i'd love to kind of just draw that out a
little more
yeah so an example of something specific
which i once upon a time didn't realize
uh an example of a place where i think i
always think about air safety research
wrong
because i haven't thought sufficiently
about the roush to impact of the aicht
research
is uh i think one of the great things
that paul cristiano has pushed over the
past few years
is claiming that ai safety research
should be aiming to make it as cheap as
possible
to implement whatever alignment
strategies are
um you know are going to be safe so this
is the competitiveness concern
um and i think that
this is really obvious in hindsight or
something like you can kind of think of
it as uh
there's two goals of ai safety you want
to make it so people are willing to
spend as much as
it takes on uh building align systems
instead of on the line systems and you
also want to make it as cheap as
possible
to build align systems instead of
unaligned systems uh and so in as much
as you can come up with an alignment
strategy that is only 10
slower than an online thing versus uh
50 slower uh this makes you a lot better
off
and this is an example of an insight
that i hadn't thought of on my own
and i think that the kind of strategy
that would have led me to think of this
on my own
is something like uh trying to think
about uh
you know if i'm trying to think like
what's the probability that uh
bad things happen i'd probably want to
say something like well i have the
following distribution over how much
tolerance people will have for ai
systems being
more expensive in return for them being
aligned and i just have some
distribution here and we're okay as long
as we are to the right
uh of the minimum cost required for
alignment
uh that's an example it feels compelling
to me
yeah great okay um so one thing that
somebody from the audience asked and i
think is a really great question here
is um how dependent are these thoughts
and i gather they mean just sort of all
the thoughts that you've mentioned in
your talk so far
um how dependent are these thoughts on
your ai progress timelines
um i guess it seems to me that in worlds
where
agi is really soon like in five or ten
years
uh we probably
need to be spending more of our relative
effort on aligning current
systems and so that's going to mean that
good strategies
look a lot more like becoming really
good at machine learning
and then um trying to find some of the
people who are
working on aligning current systems um
i don't know i guess it seems like they
seem relatively robust
to different timelines i could imagine i
think that if i imagine
thinking agi happens in 15 years versus
40 years
i don't think that changes very much of
my guest here
um if agi happened is going to happen in
more than 50 years
then i stopped thinking that it's a good
idea to work on it in the same way that
i'm currently thinking because i think
it's just very hard to do useful
technical research for something which
is happening 50 years in the future
i say a lot more about this in my uh
stanford talk on
personal crisis for working on asap
great
great yeah that makes perfect sense okay
um
so you talked about what you refer to as
the shallowness
of the field of ai safety um and i'm
wondering do you expect that this
shallowness is due to the relative
newness of the field or are there
intrinsic characteristics
of ai safety that make it this way and i
i ask because
you know again similar point to to in
your last answer but i expect that an
assumption that the field will
deepen relatively quickly might serve as
an incentive for
for people to get involved sooner rather
than later so this it feels like it
could be decision relevant for people in
the audience here
yeah perhaps i mean i think that part of
it is that the field is kind of young
um you know ai safety's been around for
20 years or something
um computer science also feels like a
controller field so for instance it's a
lot easier to do
cutting edge computer science research
while you're an undergraduate than to do
cutting edge
um you know whatever kind of like
particle physics research
um i think part of this is that
ai safety uh so i expect this to some
extent to just continue being true
into the long term um machine learning
also feels like a really shallow field
and then i think there's like aren't
three semesters of good
content on machine learning that are
well established or something you know
um i have the deep learning textbook uh
is from 2015 and already feels like
pretty out of date
so i think that's evidence against the
hypothesis that having a whole bunch of
people working on the field for another
10 years is gonna make it feel a lot
deeper
um i imagine there'll be a lot more
stuff and each of the little subfields
will have a lot more interesting little
ideas in them
but it wouldn't surprise me if compared
to theoretical physics
uh it just it just like never gets petty
right right so a follow-up question
um you know and this is also submitted
from the audience if somebody's
interested in whether you think it's too
soon for an ai safety textbook and you
know to some extent i think
yeah i mean you could think that you
know the tendency for a textbook to go
out of date very quickly is maybe a
reason not to write it but i'm not
entirely sure that that that's that's a
given so i'm curious what you think
about that
i'd love for there to be a text book
than any safety uh
the main problem is that all of the
people who i think would do a good job
of writing this textbook are already
doing really good ai safety research
um and that seems like a shame i
seriously considered putting some of my
time
into writing a really like the shittier
version of a textbook
is like a list of posts
it's like taking the list of texts that
you would make into lately you would
rewrite
love yourself in order to make a
textbook i've seriously considered doing
this um
i would love for this thing to exist it
wouldn't surprise me if one of these
does exist in the next five years and
that's great
do you imagine that that's something
that uh somebody at maybe undergraduate
or master's level could help flesh out
like if somebody like you or somebody
else more experienced in the field
were to make sort of that slimmed down
version or just the bare bullet points
uh does that seem like a project that
somebody could take up and kind of flesh
out into like a more textbook textbook
um so i think this this kind of two
parts of making good explanations in my
opinion
one of them is clearly writing down
ideas
that have already been established and
the other one
is really deeply understanding the ideas
and taking them apart and putting them
together again in a new way such that
the understanding is
just easier um and i think this kind of
conceptual labor where you
try to break down the ideas and like
spot connections and
figure out the best order of explaining
them i think this kind of does rely on
having a very deep understanding of the
field so for instance i wouldn't trust
myself to do this very well compared to
people who have a much better sense of
the field
um i think that it's not crazy to
imagine undergrads being able to
help out with this uh but because of the
fact that i expect most of the effort
goes into arranging the ideas to start
with i don't think particularly
it doesn't seem like the thing where i
imagine undergrads can spend a lot of
time on
uh-huh okay great um so i think we we
have time for maybe
just one more quick question and
hopefully you can answer this quickly
um but somebody asked industries
implementing ai safety measures means
taking a hit in revenue
how successful have ai safety
researchers been in making industries
implement their suggestions so far
uh so the goal of ai safety is to make
it so that implementing ai safety
measures doesn't mean taking a hit in
revenue
i would say um like in as much
uh my answer is actually kind of subtle
uh that's my main part of it seems like
the goal is to make it not take a
hit in revenue i think ai safety
organizations have approximately no
success in making industries implement
their suggestions
except for open ai where they just
really want to make gp3 not save racist
things or whatever because that's
important for their goal of making an
enormous amount of money um
check out focus general's recent ea
forum where it's about trying to hire
for this if you want to know more about
that
okay great thank you um that's about all
we have time for
i really appreciate it buck this is a
great talk um and before we end i just
wanted to
remind you all to take part in the ea
student summit forecasting tournament on
illicit
uh you can find this at the swap card
home page
the center for effective altruism will
donate 500
on behalf of the most accurate
forecaster across all seven questions
the winner can direct the donation to
any of the funds or organizations listed
on ea funds
and they will also be offered a
30-minute call with jonas ballmer out of
ea funds
to provide advice on their donation
decision so thank you everyone for
watching and before you leave the
session please give us your feedback in
the polls
bye


AI Alignment in AI Trustworthy

in this short video i will be talking
about how the ai trustworthy problem
is an ai alignment problem and
demonstrate three aspects
that ai alignment must adopt to make
trustworthy ai
possible now before talking about ai
let's first do a thought experiment on
the trustworthiness between a fellow
human being
and an alien generally speaking if
presented with these two
options without further context we will
trust a human more
than the alien why is that because
human beings share a common value system
and are thus
predictable and understandable to one
another the alien on the other hand
most likely has a different value system
and is thus less trustworthy at first
sight
now think one step further we are
defining
trustworthiness not based on any
objective
parameter that can be measured but based
on
some subjective comprehension of the
shared experience
of human history to put it in another
way
this is to say that we will likely not
trust the alien
even if it has passed some standard
tests
for measuring trustworthiness
now let's swap alien with ai the
situation remains the same
a.i is like an alien to us even though
it is man-made
however the very fact that we are
discussing ai trustworthiness
shows that by default we do not trust ai
and just like aliens we cannot simply
conjure some tests
and label an ai trustworthy just because
it has passed the test
the only way to make ai trustworthy is
to incorporate into it
the same value system as human being
and this incorporation process is termed
ai alignment more precisely ai alignment
describes that any action by an
ai must align with the common value of a
human being
there are three fundamental aspects that
ai alignment must adopt
the first of which is the hidden social
contract hidden social contract is the
unspoken rules
that humans have developed over time we
usually
call this common sense yet ai does not
have common sense
and therefore this is the foundation on
which ai must align
now here is an example to show how
hidden social contract works suppose an
ai and a human are tasked to move a box
from one side of the room to the other
in the shortest path possible both can
accomplish this task with ease
now suppose we place a vas in the
shortest path
without any additional rules the ai will
still take the shortest path
and knocking over and even breaking the
vows
a human however will most likely walk
around the vas
and take a slightly longer path the
difference
is that the hidden social contract
has dictated that if a person breaks
something
and he will bear the cost of the item as
a penalty
now you might argue okay so how about we
add this rule directly to the ai
and then the ai will also walk around
the vas and not break it
well this seems to work for this
scenario but this solution is not
scalable
consider this scenario where the box
needs to be delivered
to save a patient's life with a vas
still
in the shortest path a human will be
more than willing
to break the vast to save the life yet
an ai
programmed with the rule we just added
will still walk around the vast
and waste precious time to save a life
it is pretty clear now that it is
not possible to write every single rule
for every single scenario in real life
therefore the only solution is to align
ai
with the hidden social contract and
train it
to identify the best strategy under
different scenarios
based on the common sense
another aspect that ai alignment must
adopt is silly rules
city rules are those that do not
directly generate material benefits
to the betterment of the society for
instance
the cultural norms of what close to
where at what occasions
or the superstition of numbers although
these silly rules do not have a direct
contribution
study has shown that they are very
important
to the stability and well-being of
society because they serve
as indicators of the integrity of the
social fabric
to be more precise since silly rules are
low in cost people can afford to break
them
often in order to gauge how consistently
such violations are punished
if they are consistently punished then
it is a good indicator
that important rules rules that do
contribute
directly to the betterment of society
will be observed as well
this allows to people to be more risk
taking and
engage in important rules more often
which in turn
can generate more welfare for the
society
as a whole unfortunately despite
the importance of city rules ai by
default
will not pick them up because they do
not contribute directly to the final
goal
now this is where ai alignment have to
come in
to incorporate city rules as part of the
decision making
of an ai finally
the last piece of ai alignment is to
allow
gold adaptability in other words ai must
be designed
to be rewarded when a changing goal
happens
otherwise an ai stuck on its original
goal
will very likely refuse to be turned off
because being turned off means the ai
cannot achieve its original goal
similarly
if ai is not designed to favor gold
change
it will very likely refuse to update its
original goal
even if the original goal has been found
to be wrong
later in conclusion
we discussed in this video why the ai
trustworthy problem is essentially an ai
alignment problem
and we listed three aspects where ai
alignment must adopt
to make ai trustworthy possible they are
the hidden social contract
the city rules and the goal adaptability
and here are the references that i use
for this video
thank you very much for watching


AI in Short: The Value Alignment Problem

meet theodore and his assistant robbie
an artificial intelligence in the shell
of a robot
theodore has a problem he loves to write
letters
but he's missing writing utensils
luckily
robbie's goal is to make him happy and
it is able to produce a limited amount
of items so theodore start thinking
about what he needs
and whether he prefers writing letters
with a pen or a pencil
ultimately he feels like he likes pens a
little bit better
he can't really quantify it but we know
that his true value is the sum of pens
and pencils
each multiplied by the values he gains
from them which is basically
an exchange rate now for theodore this
is just a feeling that can't be directly
observed
or communicated consequently
theodore tries to teach robbie his
preferences by providing rewards
and penalties whenever robbie produces
an item
this training exercise goes well but in
the following night
robbie calculates that it would be more
efficient to directly reprogram
theodore's brain
to simply be happy all the time theodore
becomes very angry with robbie he says
you're my tool not my mother
we need some boundaries and i need some
time for myself
taking a break from each other robbie
turns to his recordings from last month
and sees theodore picking two pens over
other combinations
since robbie can only predict the true
value based on its data
it falsely assumes that theodore places
much more value on pens
than on pencils faced with a choice to
produce a larger amount of items
it predicts that nine pens would
maximize theodore's happiness
over the choice of five pens and five
pencils and yet
because theodore only very slightly
prefers pens over pencils
he would rather have gotten a bundle of
10 mixed items
processing the failed interactions
theodore start thinking about
how to align his own preferences with
the goals that robbie is pursuing
and finally he comes up with an idea
he wants to interact more openly and
frequently with robbie
so it can make proper adjustments he
also reflects his own values a bit
better
and tries to intentionally exhibit
technically sub-optimal choices
after he noticed that robbie's
prediction of his preferences for pence
was inaccurate what he learned is
that underspecified reinforcement
learning may have unintended
consequences he learned that
he needs to know what and how much he
values things himself first
and then clearly communicate his values
to robbie
who he must understand as an optimizing
algorithm
instead of a sentient being
[Music]


What Are You Optimizing For? Aligning Recommender Systems to Human Values

hello and welcome to our paper on
aligning recommender systems to human
values
recommender systems are the largest ai
systems and deeply entrenched in both
markets and democracy
because of this the problems of
recommenders are problems for society
conversely there are huge opportunities
to promote positive outcomes
these are all questions of values in
practice
this paper has three parts first we'll
survey how recommenders are being built
to serve human values today
then why this isn't good enough followed
by what we can do better
in short we want to bring in theoretical
ideas from the field of ai value
alignment
and apply them in practice
we'll start by reviewing what people do
today
the engineering around many issues has
followed a common pattern
initially systems optimize for a simple
objective like clicks
this results in side effects or perverse
incentives like click bait
the designers go through increasingly
complex metrics culminating in a machine
learning classifier
hand engineering metrics to represent
our values has become quite a
sophisticated art
spotify uses a diversity metric based on
popularity
to encourage musical exploration and
promote underrepresented artists
on the other hand concepts such as
harassment cannot be captured using
simple metrics
the perspective api uses a natural
language classifier to rate the toxicity
of comments
it's usually not used alone but as part
of a triage system for human moderators
the most sophisticated recommender
systems use deep learning to optimize
for multiple objectives simultaneously
today youtube uses both engagement data
like clicks
and user satisfaction data which comes
from surveys
there's a pattern here which we call the
standard approach to recommender
alignment
first designers identify a desired
outcome at the conceptual level
this must be operationalized in the form
of a metric or classifier
this signal can then be used to adjust
the recommendations
let's take facebook's meaningful social
interactions work as an example
the company first articulated this
concept related to well-being
in late 2017.
this concept of meaningful social
interactions was operationalized through
user surveys
and these survey results were used to
build a predictive model that fed
directly into the recommendation system
here's what the overall system looked
like notice how the survey results were
used to build a model
which then shaped the objective function
for the news feed ranking
also notice that the system designers
are ultimately making all the decisions
about what to value
how it's measured and how it trades off
against other goals
so that's what everyone is doing now and
it works to a point
it's a reactive slow process and it
really just reproduces the designers
values
these may not be the user's values we
need faster more transparent
multi-stakeholder approaches
we can do better by taking some ideas
from ai alignment
informally speaking ai alignment is the
problem of making ai systems do what we
want
recommender alignment is similar except
the wii includes more and different
types of people
we see four promising research
directions
first although there is no single
definition of values
there are already many useful domain
specific metrics that could be used for
test data sets and evaluation protocols
for example the ieee recently published
a standard collection of well-being
metrics
originally developed for public policy
applications that are applicable to ai
systems
second there are emerging techniques for
scalable participatory design ranking
systems
the we build ai framework uses
interactive techniques to build a model
of each person's preferences
which are then combined through a voting
method this allows the interests of
different types of stakeholders to be
directly represented
in the algorithmic results
third recommender systems need to be
able to learn from users on the fly
not just when engineers code a new
objective one promising approach is to
design recommendation systems that
explicitly interact
with users to illicit preferences
finally informed judgment especially
retrospective judgment
can help us learn what users value over
the long term and avoid traps like
addiction
many recommenders already use feedback
from surveys
what happens if we ask over longer
periods of time
ultimately we want a world where anyone
can just tell recommender what to do
we believe today's value engineering
methods are insufficient and recommender
engineering will need to borrow
techniques from ai alignment to achieve
this vision
thanks so much for watching and to our
collaborators on this paper
if you're building recommenders or
working on related problems please get
in touch
thanks


The Value Alignment Problem in Artificial Intelligence

school and this gets summarized in this
nifty little equation up here where
we're maximizing over a vector of
actions over a sequence of actions the
sum over all time of the reward you get
four states and this is related to
models of rationality where we say
people are trying to maximize reward or
accomplish tasks in the world this
particular model is called a Markov
decision process there are a bunch of
similar things in most applications of
AI to give you a sense of what this
looks like for a robotics application
you might have this arm which is a Jayco
seven degree of freedom arm in a world
with this simulated person and this face
on the table we would describe that by
the position of the person the position
of the robot in the position of the base
the different actions or motions the
robot can take with the joints and the
reward in this case the way we tell the
robot what we want is going to be a sum
of distances distances from the robots
hand to different objects in the world
so you might talk about distances from
the robots hand to the person's face it
turns out people don't like it when you
move your hand right like close to their
head it's bad you also want to stay away
from the person's torso but it's kind of
a different object to avoid you don't
want to run into the table so you want
to stay away from that and you don't
want to run into the base and so we
would include these as objects to stay
away from and then the last thing that
we need is to set these weights we have
to say how bad is it to be close to
these different objects and this is the
encoding of our goal for the robot this
is what determines what's what it's
going to do and it turns out tuning
these weights is really hard and it's
something that I started to spend a lot
of my time doing over the course of grad
school so here's an example of what that
might look like we've got here a slider
each controlling one of those weights so
this is telling you how important it is
to avoid running into the head for
assist to avoid running into the base
and you go back and forth between tuning
them and saying and changing the values
and
keeping the trajectories and then you
measure your performance by how well it
does in a bunch of environments so the
robot arm is kind of moving we have an
idea of how we'd like it to move in each
setting and you go through this process
of iteratively tuning these weights in
order to get the robot to do what you
want and what this pointed out to me was
that that story I told you guys before
about how robots make choices is
actually a bit of a fiction it's kind of
how we pretend the robots make decisions
in practice sitting outside of that
planning algorithm or whatever
optimization you have you have a system
designer and that system designer is
responsible for figuring out the reward
function that the robot actually
optimizes in this programming process is
really one of the it's the new type of
programming that we're doing when we put
AI systems into the world and
furthermore that process of setting or
word functions which is not like slow
and tedious and hard it's also
incredibly brittle so you find you
finally work hard get something that
works in a bunch of different situations
you're feeling good and then you go to a
new environment right then it does this
and just moves its hand right through
the vase kind of smacks it out of the
way
um and the thing is I can't tell you
what's different about this situation
just that whatever weights we had in
that previous setting didn't work here
so not only is it that we have to think
about this process of communicating our
goals but we also have to be thinking
about the fact that any goal we write
down is probably wrong in at least some
ways and there's in this case you know a
representation of what you really want
your true intent it's something that we
are trying to communicate and we're
doing that and we can't always do it
right because at the end of the day
we're only human so this can go wrong in
a whole bunch of ways not just for
robotics this is an example of a deep
reinforcement learning system that was
learning how to do boat racing so play a
bow racing video game it's really
clearly not doing that right
but in this case the thing is this is
not due to a bug in the code this robot
is actually doing exactly what you asked
what they asked it to do what happened
was the system designers looked at this
problem they said well we like the book
to win the race which involves going
around the track and going faster than
everyone else but we don't know how to
write that down but luckily enough
there's this really handy score button
the score function down in the bottom
and you get points for playing with
racing game and we can just tell the
robot to get as high of a score as to
can and it just happens to be that these
green balloons which the robot was
spinning in circles and collecting our
worth points and so the robot actually
found an ingenious strategy to get
essentially an infinitely high score
because by spinning in circles the boat
boat race wouldn't end your score again
as high as it could and you would just
continue collecting points forever and
ever you just like discovered like a
money tree or something for those of you
who are Animal Crossing so the question
is how what happened of course is we had
this score as an objective but what we
really wanted was to have systems that
could win and we didn't know how to
describe that appropriately and this was
something I was beginning to pivot my
research towards and I was thinking
about well you know we've got
reinforcement learning and we've got
robotics and we want to be able to
describe what we want to these systems
really well and then 2016 came along
which was it tumult Ruis year for many
of us and one of the interesting things
that grew out of that
was there started to be a lot of media
articles that looked something like this
so here's a representative one by a
political scientist from the University
of North Carolina named zeyneb chief
ecchi and what she said was to keep
users watching YouTube utilizes a
recommendation system powered by
artificial intelligence indeed after
Google brain took over YouTube's
recommendations in 2015 there a
laudatory article
and how it had significantly increased
engagement engagement is a measure of
how much time people spend on the site
how much they click on different things
how many comments they write and so on
YouTube the algorithms will push
whatever they deem engaging and it
appears they have figured out that wild
claims as well as hate speech and
outrage peddling can be particularly so
and so for me as someone who is thinking
about the wrong objectives and
incentives for robots this was really
interesting because what's happening in
those content recommendation systems
like YouTube and Facebook is you have a
bunch of different pieces of content
articles and videos that they could show
you there is in effect a robot choosing
which of those to show to someone and
how are they choosing that well they're
choosing that in order to maximize the
engagement that people have with that
system and so in this case though
they'll select maybe that long piece of
content and this seems pretty innocuous
pretty reasonable um but to me I mean
remembered like think about that score
function in that boat racing game and
what seemed like a pretty reasonable
goal or objective actually led to
counterintuitive and surprising behavior
and it turns out the same thing does
happen with engagement objectives so
particular there's one piece of
information which for us we know about
because of the Journal of political
psychology in 1994 and what happened is
they went and surveyed people to find
out properties of people's belief in
conspiracy theories and the relevant
part is this highlighted section right
here which says that people who believed
in one conspiracy were more likely to
believe in others and if you're trying
to optimize engage with sets of videos
this is really really useful because
there are tons of videos about
conspiracy theories and there's a bunch
of different branches of them and so if
you found someone who's interacted with
one a lot
that's a predictor that they're going to
interact with other types of conspiracy
videos and furthermore these videos tend
to be very engaging once people buy into
them they spend tons of times watching
they engage very highly and so I've
actually found is
as those articles pointed out systems
have developed a bias for this engaging
content in order to recommend and
disseminate conspiracy videos broadly
and for me what I noticed is this is the
same problem that we get with that boat
spinning in circles sitting outside of
that engagement optimization is the
company as the system designer that's
putting in a goal of engagement to the
system and in their head there's
something separate which is is their
true intent their real goal which could
be engagement that's certainly important
now but it includes a lot of other
properties it includes the stock price I
think these companies don't want to be
exposed to regulatory risk from bad PR
around this the companies maybe don't
care what the people within the
companies care about the fairness of
these algorithms and how they interact
with people um there's also sides of
this like creator loyalty the fact that
people who are producing these videos
and content care about who it gets
served to and so on and so forth this is
really just scratching the surface of
the types of concerns you might have
and so really this is a story about how
AI makes decisions and how artificial
intelligence is programmed with its
goals and to me what this points us
towards is a statement about what the
field of artificial intelligence is
trying to accomplish I think many
practitioners would describe the goal of
artificial intelligence as a goal to
design and build a machine that can
effectively optimize for any specified
objective you write down a task you
write down a goal and you have a system
that can optimize for that effectively
and accomplish it and what my work was
about and what I'm continuing to work on
and what I think is crucial for us to
adopt both as a research field and
society is a kind of small of a
consequential change here which is that
the goal of artificial intelligence has
to be to build and design a machine that
can effectively optimize for the
intended objective that people have in
their heads
with that I'm gonna stop the talk thanks
very much guys this was great that's
great
are there any questions that I can take
yeah I want to ask a question can you go
back to the last slide kind of
broadening the the horizons here when I
saw this first quote Ellis or I also
wanted to change that same section but I
wanted to say something like the goal of
artificial intelligence is to design and
build a machine that can effectively
optimize for an arbitrary objective what
do you think of broadening the horizons
and saying hey we built an AI that can
not only play this boat racing game but
it can also do laundry or an arbitrary
objective that maybe the robot comes up
with on its own or maybe we can you know
on the fly give it a new target what are
your thoughts on the future of that and
existing research I mean I think it's
certainly related here I know to me and
I actually see it as being quite related
to optimize for any specified objective
right you can also think about changing
your specification or updating it and
the specifications here can be arbitrary
the point that I'm trying to make and
what I think the arbitrary objective
phrasing still doesn't quite capture is
that there's a process of expressing
your goal and of communicating that to
the system and if the onus is on us
entirely to get that right that could
actually be quite bad and in practice
what we find is that tuning objectives
and setting objectives is one of the
hardest part of AI practice hardest
parts of AI practice so for me I think
the purpose of this is not to say that
entry is not about really increasing the
scope of what systems can accomplish and
certainly that's most of what AI
research is about what this is saying is
that in order for AI to be successful
and sort of broadly beneficial in
society we need to be working on a
separate
related but not but but also different
problem which is helping to elicit the
correct objective and dealing with
uncertainty in incorrect specifications
of the goal absolutely yeah
it even takes it further have a
follow-up question what do you think
when the goal is well here this might be
you know shortening the scope what do
you think when the goal is clear but the
method to which you obtain that goal is
not clear at all for example I play some
chess and the goal of an ARS system is
to win the game but that's not obvious
how you would do so one of the
heuristics they have or one of the like
checkpoints they'll say is controlling
the center and capturing pieces so in
that case our objective our intermediate
objective becomes capturing pieces for
the long-term goal of perhaps winning
the game what's your perspective on
using midpoint objectives to perhaps
proxy our long term objective I think
communicating proxy objectives is a lot
of what we do in practice and in fact if
I were to continue this talk and there
is there's more content to go on and I
think I will I will say I'm going to be
doing a another talk later this week
that's a half hour long going into some
of the details of this research program
so I know who wants to join for the fad
I'm gonna ask Jay to send out a little
ad but we basically proxy objectives or
something I think about a lot and the
thing you have to consider with proxy
objectives is the way that they're
standing is for a long term goal and you
can still fit into this framework of an
intent an intended objective but still
try to optimize for proxies so I guess
that that got a little bit technical
yeah I think the main point is I think
sort of sub goals are really valuable
and they're one
the more useful types of information
about our goals we can have and the
trick is sometimes you want to
accomplish the sub goal in a limited
fashion right so controlling the center
is a good objective up until the point
where it's not mm-hmm which point you
need to shift to moving to checkmate and
things like that and so it makes it soft
goals is the active area of research and
something I'm actually planning to work
on in the future mm-hmm that seems to
make the this concept even more
difficult where you're aware of these
you know variables that could
distractors put potential distractors
anyway and you have to you know ride
them where they're useful and ignore
them when they're not so it sounds like
this on the other hand it also creates
opportunities because if you're
optimizing for sub goals you actually
don't have to plan as hard so it
actually plays into a lot of the things
that a I systems are actually really
good at which is rapid thinking over
like very short timescales
right now


A Formal Methods Perspective to AI Safety: Promises and Challenges (Wenchao Li, FSL Workshop)

[Music]
okay thank you for the introduction so
instead of making a blanket statement on
the role of formal methods in the AI
safety as the agenda suggests I'm going
to focus on a specific problem which is
improving or even proving safety of
neural network control systems so again
I'm going to have from bu this is John
work with my student John L'Enfant and
also my collaborators at Northwestern
University and also University of Dayton
so let me begin by telling you a little
bit about neural network controllers so
really spurred by the exciting
developments in deep learning in recent
years there's been a growing interest in
Ewing in using neural networks in place
of traditional controllers or even
humans controllers so this nature papers
you know talks about how to train a
agent to achieve human-level performance
in a set of Atari games of course these
performance driven wheel of the problems
sometimes can actually end up with
unintended consequences so for example
if your safety requirement is not
properly encoded in the reward function
then you can actually end up with the
agent that does really well in terms of
performance but also crashes a lot and
is in simulation so another popular
learning paradigm is imitation learning
so what that does is you're trying to
transfer knowledge of an expert of
oftentimes a human expert to to the
learning agent and in this case your
people have tried to do this even
end-to-end for robotic tasks or maybe
even in a simulated autonomous driving
environment
so a common observation here is that a
lot of these successes are actually in
stability environments so the natural
questions we ask is why have we haven't
seen new and network controllers in
actions in real life interestingly
there's actually some recent work in in
implementing or in using the network
controllers and these are motivated
Speight practical constraints such as
computational resources or maybe timing
and the reason is because inference is
typically quite fast for neural networks
and you actually want to use neural
networks to approximate an optimal model
predictive controller okay so in this
case you actually have a good idea of
what the model weighs so let me just
quickly go through you know what the
setup is so this is a very typical
cyber-physical systems you have a
controller controlling some physical
plant let's for simplicity assume that
we have perfect sensors and actuators
which is never the case in real life and
in this case you know the physical model
which means that you know like for
example the differential equations that
governs the evolution of the physical
process and just to four notations you
know X represent the state of the
physical plant and can you represent the
controls that the controller produces so
let's consider a sample data or
time-triggered controllers which means
that the controller operates at some
period Delta if we look at the sixth
system executions first of all you know
you would sense a stay of the physical
plants and then after Delta you would
actually apply the controlled output
okay so it looks like this starting from
some initial state at time zero right
the controller would drive the system to
a new state okay again you applied a new
input a control input at time Delta
right and then you drive the system to
another state
so what are soft neural network control
systems simply now we are just replacing
this controller typically designed using
traditional control laws by a neural
network okay so what are the properties
that we might want to care about in such
a system so let me show you a similar
robotic manipulation task over here we
are trying to train a robotic arm to
reach some random randomly generated
code stays in the workspace without
hitting this box obstacle in the middle
okay so you can do this in various ways
you might want to train this using
reinforcement learning you might want to
do it using imitation learning okay now
the question is does it actually work in
real deployment right well we actually
transfer the Lund policy to an actual
robot so this is an experiment that sort
we did in the lab so this is a sawyer
robot and again you're randomly
generating code states and then you have
a small initial set that the any factor
can start in and you want to know if you
can always reach the go state without
actually hitting or knocking over the
cut on this box okay so more formally
the reachability problem of neural
network control systems looks like this
your system start in some initial set X
0 there's some static a voice at X a
that you're given and your goal is to it
was to check if your controller or your
neural network controller would actually
drive the systems to some target set
okay so pictorially
you see that the state of the system
would involve this is a trajectory of
assistant and then you want to doubt
whether it's you know you would end up
in the target set after some finite and
finite time okay so more generally so if
you're just given a single initial set
you can just simulate it right and then
we'll figure this out but suppose you
now have a initial set of
State and the rich avoid problem is to
ask if we start from any state in this
initial set candy an NCS actually
reached a state in the target set at
some specific time horizon without
hitting the obstacle right or without
reaching the advice set okay so this is
the mathematical or the decision problem
that we cleared out it turns out that
these problems actually undecidable and
there's a very simple reason for the
undecidability it's because n NCS are as
at least as expressive as nonlinear
continuous systems so for example your
dynamical plan could actually be a non
non linear continuous systems so the
exact reachability problem is not
decidable okay so what do we do one way
to soft circumvent undecidability is to
use over approximations so here the stay
of the Arts
you can try to over approximate the
reachable sets okay I'll show use of
what it means in the next slide but here
are soft really what's out there that
can do this job okay and these are all
papers this year so data all really
proposes to use piecewise linear
polynomials to approximate the
input/output mapping of the neural
network and and also give aid on error
down by solving a mixed integer linear
program
darris sake is an interesting approach
where instead of directly approximating
the input/output mapping you actually
transform the neural network controllers
into an equivalent hybrid automaton for
special classes of new networks and then
you just throw the resulting combined
high predominant in the reachability
analysis - okay which I wish I can show
you in the next slide so our work is you
know a bit later than these two and we
show that we are more advantageous in
terms of the classes of neural networks
they will hand over and also in terms of
the tightness of the reachable set okay
and our approach is called reach and
then okay so the common thing across all
three approaches is that we relied on
what is called flow pipe constructions
and flow pipes are essentially our
approximation of the reachable set and
in this particular case we are looking
at what are called Taylor model flow
flow pipes what that means is your
original set is actually represented by
a finite set of Taylor or higher modeled
higher ordered Taylor models okay so
there are tools to do this and what the
diagram is trying to illustrate is you
are trying to use these flow pipes to
over approximate the trajectory is that
that the system can actually generate
and then you want to check if this over
approximation ever intersect with the
alloy set and at some target time T you
want to check if the reachable that is
contained in the target set okay if
that's the case then we have a positive
answer to the reach ability problem okay
so naturally you might sort you know
already as an idea how to have
approached this problem and you might
want to essentially over approximate the
states that you can reach on each data
find T equals to K Delta and then you
would feed this new you would treat this
new reachable set as my now my new
initial set and then put it into this to
go flow start for constructing the flow
pipes okay so we did a simple experiment
on the very simple physical plant given
by those differential equations and we
also can just consider a simple neural
network with two hidden layers right and
the rate of activation actually with two
types of activation functions so what
happens here is that for the initial set
given in the top right box the red lines
represent
actual trajectories if you were to run
the system from a single or a specific
initial state the the green boxes are
really just big pictorial
representations of these flow pipes okay
so what this means is if you do this
interval approximation over
approximation the reachable set explodes
very quickly and the reasons for that
explosion is interval approximation
fails to capture the dependencies of
state variables so essentially it does
not directly over proximate the input
output mapping of the neural network it
just tries to over proximate the set of
reachable States at each time step okay
so our idea is to use
Bernstein polynomials to actually over
proximate and your network controllers
and that's a specific you know that's
how they personally in polynomial look
like we don't need the details here but
the idea is that it's also as nice
properties for for a universal
approximation and in fact our approach
can work with n the ellipsis continuous
function or network at most new networks
are Lipschitz continuous so this means
that it can be applied to your necks
when networks with activation radio
activation functions times X activation
functions a combination of those
activations and so on okay so if the
black lines I showed a neural network
input output function then you actually
want to construct a burst in polynomial
approximations in the sense that if you
can you can you can construct the error
downs on the over pass emissions
okay so FX is the neural network
functions and bf DS represents the
Bernstein polynomial up to degree D for
that for that F okay
so so the the trick here is that we want
to keep epsilon small so the over
approximation is small okay so there's a
known result for an ellipsis continuous
functions and in this case here the
error is given by this particular
formula okay but this simple Lipschitz
constant Bayes error is actually very
pessimistic you notice that it's
proportional to the Lipschitz constant
that there is this additional term here
right is actually the degree of the
pristine polynomial and M is actually
the dimension of the input okay so what
we beat is we actually use a sampling
base approach is a very simple idea and
the key observation here is that the
Bernstein polynomial approximation is
also leaches concerned continuous with
the same Lipschitz constant al so what
that means is now if I partition my
state space into some you know K
partitions and then I can analyze the
approximation errors in each of these
partitions by essentially doing a simple
triangular inequality okay and then
ultimately the actual overpour summation
error would be the max errors across all
of these partitions okay so the details
you can actually check out the paper if
you're interested but let me just
quickly show you the comparisons with
the interval over approximation that I
show you earlier
so the LEF is the same picture and on
the right it's the same neural network
control systems but now you can see the
green boxes which are really
representing these flow pipes or the
reachable set a lot tighter right and we
can actually get a positive answer to
the reachability questions that we set
out to investigate because here there's
no avoid set but this green box at this
particular time instants are actually
completely contained in the in the blue
box which is my targets
okay so long we know for sure or we have
a proof that is near a network control
system we've reached the target set okay
we also compare with other state of the
Arts
there are not that many Fournier and
network control systems essentially you
know very sick and Charlotte which is
the first paper that I talked about so
we look at different benchmarks
essentially different dynamical systems
and different neural networks with
different activation functions across
the board you can see that the greens
pipes which are really the flow pipes
that we get using which an end are a lot
tighter than very sick which is doing
this equivalent hyper automaton
transformation and we can also handle
more varieties of neural networks
so this Sherlock that's a very good job
in actually bounding the over
approximation but you can only handle
regular networks and it relies on
solving a fairly expensive mi LP to
actually construct the estimate that
error bound okay okay so after that we
make an interesting observations which
is if we consider different neural
networks for the same task but with
different Lipschitz constant then we get
this behavior so if we have a large if
the neural network have a large leaps is
constant then we are not able to do this
reachable set computation for very long
time because of at some point it would
explode okay and there's an intuitive
reason for that because essentially
lives as constants tells you how quickly
your your controller output can change
with respect to smart changes in the
input so this would explodes so if we
are able to keep that Lipschitz constant
small
then we are actually able to run which
ability analysis for a longer time
periods and this is actually a
consistent observation across different
tools - not just for our tool so now the
question becomes can we somehow code
design the neural network controllers
for the purpose of doing safety
verifications right so the idea that we
had was to use knowledge distillation
right to transfer the knowledge of a
trainer and our controller onto an apps
easier to verified neural network
controller any knowledge distillation
the model is just a simple teacher and
student models where the teacher is the
trainer and network controller and the
student is the controller that I'm
trying to learn which is hopefully
easier to to verify ok so the approach
that we actually developed with your
peers later this year
is that we recognize we have two
objectives here right if you are trying
to retrain the neural network or trying
to transfer the knowledge of one into
another we better keep the a regression
very small okay so the we want the new
network to have a similar performance as
the originally trained network which you
might have trained using you know
different or very complicated strategies
and now we have an additional objective
of keeping that Lipsius constant small
so you might have actually some target
which is constant and you want to
minimize the square lost of that of that
okay so now you can actually end up with
two gradients one is the reversal
respect to the original loss function
and the other is with respect to this
new loss function now lip okay and the
way to do it is you can actually try to
compute the angular bisector of these
two gradients and if if you see that if
you are going in the direction that will
improve both both loss functions then
you just simply take the angular
bisector right in the case where G lost
agility is actually greater than zero
but suppose if these two gradients and
telling you to go to a space where in
one you might reduce the loss regression
loss but the other you might actually
increase the Lipschitz constant then you
would actually first prioritize on
performance which is keeping the
original loss functions small which
means that you would take this gradient
G final which is a projections of G loss
bonds on onto a hyper plane that is
perpendicular to G look okay
and if at some point your G loss is
small enough let's say is more than some
epsilon that your design then you can
actually do some repair ties Asians
which means that now maybe you can have
an opportunity to further reduce the
Lipchitz constant okay so essentially
you're just trying to do these two
objective gradient descent to
simultaneously hopefully write reduce J
loss and jelly okay so let me show you
some perhaps visualization of what for a
case study that we beat so this is a car
starting from a position in the top left
so Tangis means that this is a
neural network controller that has both
reloj activation functions and times
actually a tangential activation
function in this case we know that I'm
Michael model of the car and we train
the neural network actually using a two
that confuse the robust optimal
controller and that's very
computationally expensive so when you
actually want to deploy it you might
actually want to have eight or neural
network controller okay
okay so the again the red lines indicate
trajectories of these of the car and
there's actually sets of the
trajectories and we end up with a neural
network with ellipses constant around
two hundred forty four forty four for
just this simple loss function what I'm
not happening is the reachable self
explodes after maybe 20 steps if he set
the targets Lipsius constant of the
neural network to be hundred then you
know in simulation we still can see that
the system achieved a task in everett
collides with these the obstacle which
is described by this yellow box and also
the small yellow box here but if we do
reachability analysis then at some point
because of again the explosion of the
reachable set or the very conservative
approximation of the whichever said then
intersects with this obstacle right so
we don't know whether you can reach the
target for sure and using our
essentially two gradient to objective
gradient descent then we are able to say
for sure that for any state that you
start in the initial initial set the
system we eventually reach the target
set which is this blue box and also
avoids the obstacles okay so one thing
why I want to know here is that these
are essentially different neural network
controllers they are not the same
controllers in terms of performance you
can see that the one with the smaller
Lipsius constant actually makes a slower
turn okay but for the purpose of these
control tasks it doesn't matter right
because the goal is to reach the target
and avoid the avoid set so what are the
open challenges there are actually quite
a few so all the state-of-the-art
methods for verifying or answering which
ability of neuron our controllers they
cannot scale to large number of the
inputs which means that
we cannot verify it you know end-to-end
learning or image input for now and the
other problem is verification requires
knowing the dynamical model in practice
you might have a good idea of what that
model is and you might know some
perturbation bounds on that models but
you need a model right to answer these
types of verification questions but the
caveat is actually you you you may be
able to improve safety okay
without knowing the model okay I mean
may not be able to prove reach ability
or safety but they might still be able
to improve it by quite a lot and we
study this in the setting of model free
reinforcement learning and in
particularly interested in safe
exploration okay so meaning that you
know you you want you want the agent
when it's interacting with a real
environment doesn't go into a lot of
safety hazards there are statistical
ways to guarantee safety in even in the
model free setting in particular people
have considered using Gaussian processes
and so on just to name a few of this
work the problem with Gaussian processes
is you know you cannot adjust address
high dimensional systems so what we did
is instead of using GP to approximate
the dynamics we use GP to approximate a
function that captures only the safety
part of the systems right and
particularly captures the evolution of
trajectory based safety and we borrow
ideas from control theory using control
the up enough functions for asymptotic
stability so what that gives us is that
you want some convergence to safe
behaviors so if we continue to train
your system in the real environment and
ultimately you want this agent to be
safe which also means that the agent
should be able to recover from unsafe
behaviors so I have about 4 min
slap so I won't bore you with all the
details but our setup is to use Gaussian
process but do it in an online fashion
and to steer the pollicis policy search
in reinforcement learning so the
difference is that you actually have two
additional components for safety
estimations one is this G network which
is again a neural network the other is
this G PI which is a Gaussian process
model so a key observation that we make
is that these G which is function
approximator would be a candidate
control the Uppada functions for the
discrete control problem if it satisfies
these properties and the small G is the
Gaussian process models for the safety
evolution for a for a for the current
policy part so in some sense you
actually want to influence the policy
search in a way that it satisfies that
inequality where G is greater than zero
okay so so basically the outputs of this
function approximated G we treat it as
is the observations of the GP model
again G PI does not model the the food
dynamics of the system but only models
the evolutions of safety safety part and
now somebody you want to solve a
constrained optimization problems where
L is actually the lower bound or in the
confidence interval when you are doing
the GPS teammate and one thing is that
you cannot solve these constraint
optimization problem directly because
you actually have to compute out based
on your Gaussian process estimates so
what we did eventually is we would
softly impose this constraint okay so
under some assumptions which are DDD
thousand
papers you actually get statistical
guarantees okay
so here's an experiment that we did on a
relatively high dimensioned system which
is a half cheetah with unknown dynamics
so the cheetah kind of looks like this
is a half cheetah so you know he has two
legs and we can see that catastrophes or
failures when the chilla false tongue
and cannot actually recover okay so
there are some interesting of weird ways
that you know this system can fall so we
are comparing these two the
state-of-the-art DD PG method for doing
DRL we also consider two different types
of safety cost functions so the desired
size because function is actually quite
tricky so what we did is we just use
simple safety cost functions so one is
just related to the body rotation of the
cheetah the other is related to the
height of the body okay so these are
proxies to whether to cheat on me for or
not we don't know the exact trestles for
those safety cost functions so so for
the experiments so the in terms of
performance the red lines and the yellow
lines are our methods for these two
different safety cost functions with
online GP estimation and one thing to
notice these can actually accelerate
learning because for this particular
task now following means you're the
cheetah can move so okay that so there's
a correlation between not failing and
actually getting performance in terms of
safety violations so we are looking at
you know cumulative catastrophes then
both of our approach would have a very
small number if no failures no force at
all varies for for DD PG it takes a long
time to train and also the agent force a
lot okay
and we also compare with what time
we are doing this additional procedure
of doing calcium process estimations and
even with that overhead taking into
account or a method is able to learn
much faster than the DPG okay so the
green line is the what you get with PD
PG and these two lines are what we get
with our methods so just to show you
quickly how that looks so in iteration
ones so this is DD PG so it's
deterministic the policy the gradient
and what we did is we are trying to use
Gaussian process to estimate the safety
part of the system and very quickly we
can achieve a high return there is DVP
she would still fluctuate you know in
terms of reward and also you get into
this strange scenario where it's
actually moving a little bit but with
the head of the cheetah okay all right
so just to conclude we have an approach
to answer the reachability question of
neural network control systems and the
idea is to combine polynomial or
Bernstein polynomial regression with
some sample base and neurosis and then
additionally we want to control the the
licious constant of the neural network I
think that's an important part because
verification typically is an
afterthought and you might be too late
you know to get to that point after all
the training is done and there's an
interesting question to think about
safety for model free learning because
in this case here you don't have the
modeled but your agent actually has to
interact with a new really environment
so think about the number of crashes
that your car eventually learns to drive
in the real world and I want to live
with this remark that there's a big gap
between the scale of problems that
verification can tackle and at least the
scale of
past that learning has demonstrated okay
but not necessarily in deployment okay
so and hopefully we can close this gap
you know in the short term so thank you
this is the end of my talk yeah
yes
yeah sure sure so I can't talk about
maybe the first part of your question
first which is AI square and so on so
they're actually looking at local
robustness so even though your input
dimension might be let's say 100 by 100
they actually look in a specific input
okay looking at specific input and then
you're looking about looking at the
robustness other's adversarial business
question so input dimension is kind of
not an issue for them you're you know
they use abstract interpretation to
essentially over approximate the values
across the neural network but it's for a
specific input so it's not for sets of
inputs so that's why it might look like
you can hide a high dimensional systems
but really is for you know individual
inputs so it's local robustness so for
for these problems or I can say that
this point is I don't think any of
polynomial function approximation
approach would work for high dimensional
systems just because you know the
dimensionality issue doesn't go away so
there might be waste if you would take
the design part of the design part into
into the analysis and so you might be
able to actively reduce the dimensions
when you designed the neural network but
for in terms of verification you know
it's an inherent complexity I don't
think it would go away yeah it's also
why this problem is really hard and
that's why it I guess we shouldn't trust
and to end earning at least at this
point the other questions yeah so I'm
happy to take questions offline I'll be
around but maybe there's one more sure
yeah so I mean I this is relatively high
dimension from the dynamical systems
point of view so is 17 dimensions I
wouldn't call it like really high
dimension as in an image so so there is
a lot I think details in how to make
this tractable in the sense that we had
to limit the size up so data so here you
know as you get more data points right
as team a becomes better but because of
the computational complexity right this
is a cubic complexity for doing the GP
estimate we have to keep that data set
to about five thousand so which means
that we have to find a way to kick out a
data point and take the new data point
so that we have D thousand in the paper
as well
so so so even fraud I think the takeaway
message for this problem is that we four
model freezes not learning we shouldn't
attempt to model the entire system but
we should only model the specification
relevant parts of the model in this case
with the safety or as asymptotic
recovery from unsafe States yeah okay
thank you so I'll be around
[Applause]