Can Deep Learning Models Be Trusted? (Luca Daniel, Foundations of Safe Learning Workshop)

[Music]
[Applause]
[Music]
thank you for the kind introduction so
first and most importantly I would like
to acknowledge the very crucial
contributions of my senior grad student
Sui
Lilly Lilly could not be here she's
traveling internationally I really
wanted her to be able to present her
work I would also like to acknowledge
the precious contribution of my
collaborators at IBM CGI is one of them
P knew a lot of other collaborators
students colleagues funding from IBM AI
lab and MIT quest for artificial
intelligence I have to apologize I could
not attend the beginning of this session
what I have to say I'm guessing is not
going to be that different from what
some talks had to say at the very
beginning I have seen prefer my
colleague Alexandra magneri giving
fantastic talks before I have to say
that I agree on everything that he says
and you probably recognize a lot of the
things that that is that in my talk
that is to say that you know it's the
last session it's the last talk I will
not be offended if you decided to start
your weekend early I've also decided to
target a different audience at least a
different audience that would have seen
done in other talks at this conference a
non-technical audience because as you
can see my title is a question
and I think that when it comes to this
topic the questions are more important
than the answers and that is because I
myself my group and probably a lot of
people in the community if not all of us
still don't have very clear answers so
I'm going to keep it high level it's
also good because right before lunch a
lot of math might be problematic so
here's our good friend the neural
network I collect from neurons with
weights and activation functions that
you can use to do pretty much anything
these days but for example they can be
used to do image classification we know
that performance gets better and better
than more training data you have we've
heard this a million times here's one
more time for me
but how many times do you stop thinking
about what it means performance what
people actually measure a lot of my talk
is about what should we measure so what
typically is measured is the number of
errors or the percentage of errors
especially when compared to what a human
could do and maybe 20 years ago things
were not going very well for newer
networks yes they've been around for 20
years for those of you that were not
born at that time
maybe a human could do 95% accuracy
meaning number of correct guesses or
classifications out of 100 and a human
network and a neural network might be
able to do only 30 40 percent 50 other
known something like that things kept
getting better and better
eventually they saturated I apologize
for the contrast here not very visible
but again fortunately if you're an
optimistic maybe not so fortunately if
you're like me more of a skeptical
person start things started getting
better again so much better meaning
again number of errors or percentage
error that we started getting better
than humans so out of a hundred a very
highly performant deep neural network
could for example get 99 things right
just make one mistake a human still and
five percent so five mistakes so what do
you choose well of course we choose the
deep neural network for any of our
decision-making from now on so they are
quickly becoming what I would call our
here the neural network is coming to do
everything for us as I was mentioning
before from ejek image classification to
object detection to natural language
processing so we're already most of us
are dreaming to having them drive our
cars hire our people invest in the stock
market for us so on and so forth
but I would like to bring back the
question that I mentioned before what
does it mean to be better and what I
mentioned before is something that will
highlight a dark side of our here
potentially at least let's have some
questions about this so what we count is
the number of errors but do we ever stop
and look at those errors I would like to
stop and look at those errors so for
instance yes it's true that this very
highly performant deep neural network
make very few errors but well I don't
know that one error that they make maybe
that's a very difficult case and no
human would ever make a good
classification on that specific input ok
or maybe there are very few humans that
would be able to guess it right on those
specific cases that a deep neural
network get wrong but how would you
react right now if I were to give you
examples of errors that the deep neural
network makes a very highly performant
and no human on earth would ever make an
error on that specific input specific
input is the key word here specific
input so we should actually try to look
at those and also how about ok maybe we
just stick with errors that both the
deep neural network and the human make
can we
at how big those errors are so what if a
human yes makes an error but it's not
that big while a deep neural network
makes a huge mistake on that specific
case how would you react in that case
what would you prefer these are
questions we should ask and they go
beyond math in my opinion all right so
let me start digging in and show you
some images you probably if you're in
the community I've seen all of these
images and you've seen where I'm going
if you weren't a Madras task talk today
you know where I'm going but don't need
to answer just answer to yourself what
do you think this is I'm guessing that a
lot of you might conclude this is an
ostrich good for you the first time I
saw it I wasn't really sure maybe
because I'm not a native speaker it's
about the language as well but
definitely I was able to conclude it's
some kind of bird and no question in my
mind and most likely no question in
anybody's mind in this audience this is
at least an animal well good for us the
highly performant deep neural network
agrees completely with us and extremely
confident tells us it's an ostrich what
about this one well if you ask me I will
tell you well now that I learned that
that's an ostrich this is an ostrich too
actually it seems like it's the same
ostrich to me how would you react if I
told you that that same highly
performant deep neural network that gets
fewer errors than humans tells me it's a
unicycle particularly concerned about
the fact that it's not even the mistake
this neural network makes this
superhuman human
deep neural network makes a mistake and
it makes a mistake that is not even
another kind of bird it's not even an
animal
it's a unicycle it's not even an animal
so I'm upset about this I'm also upset
about the second thing if you look at
the last neuron the last layer that is
responsible for this assertion
corresponds to an ostrich it's happily
firing extremely
confident just as comfortably as before
that that's an ostrich and to me there
is nothing worse when you're trying to
build trust than someone who says
something wrong extremely confidently
and thinking you're right that's very
upsetting or at least doesn't make me
want to trust that thing or person or
whatever that is all right I said a lot
of negative things about the pure
Network it's about time I say something
positive so let me try to defend this
well this is the very very clever work
of some extremely gifted hacker that had
managed to get information inside
information on that specific deep neural
network and knows the structure knows
the number of layers knows the
coefficients knows everything about that
neural network and is able to exploit
very cleverly all of those things this
is called an attack all right so it's
not something that happens every day you
need hacker that has all of this
information this means that the people
that started worrying about this are the
security people because there is an
attacker there's a hacker and they
started thinking about what can happen
for example if your self-driving car
gets hacked and it doesn't recognize
stop signs anymore is someone trying to
kill you or something well this could be
one way unlikely but possible what if
you're Syria doesn't recognize what
you're saying because a hacker against
gets in and has access to everything
inside the deep neural network of your
serial phone or whatever you're using
people started designing classes that
you can wear and the network will think
that this man here is actually this
woman over here just because they were
in that special classes students here at
MIT were able to take a living animal
like this turtle and just paint
something over it and mix it and make
people not to think that that's
confidently a rifle so it's not just an
image problem where you change a few
pixels
all right so this feels like it's a
serious security issue yes let me bust a
few myths for you point number one that
hacker that I mentioned before needs to
have complete access to the inside
information that you in that quote
actually does not need to have complete
information on the neural network does
not need to know anything at all about
that newer network and some of my
collaborators showed that you can just
look at that neural network as a deep
lesson as a black box and just be able
to ask questions and get answers is
enough to synthesize a picture like this
to me looks like there are two guys
happily skiing up in the mountains
there's no question to me that that's
what this picture represents but then I
look at the deep neural network and
there is a neuron there corresponding to
dog that says ninety one ninety one
percent higher than all the others they
had no information about how that neural
network was trained what data set was
used what the coefficients were what the
structure was just ask questions all
right now I'm getting a little more
concerned because it's not enough to
protect the inside information on my
network let me bust the second myth for
you I don't even need the hacker I don't
even need that very skilled person there
are very few in the world and they're
all in this community of trying to hack
neural networks and they're very smart
much smarter than I am and they are
capable of doing this I don't need them
things are a lot more serious than that
because people have looked inside
imagenet the database of existing images
and you can find images that have not
been hacked by any hacker and gifted
person with this insane ability of doing
these things and they were able to find
pictures that this highly performant
deep neural network the one that has
superhuman performance the one that gets
99% of the answers correctly more than a
human would do that's the same your neck
would have tells me that this thing here
number three for example it's a pretzel
yeah I see a mushroom I might not know
what kind of mushroom but I do see a
mushroom this is not a butterfly
apparently this is not a squirrel yes I
don't know that it's a fox squirrel but
it's a squirrel it's not a sea lion so
hopefully you're starting getting
concerned this is why I said that
sometimes asking the question are
actually having new ask the questions
it's more important because what I want
to do is have more people that ask these
questions because all of us can
hopefully find better answers for the
community rather than focusing on me
giving answers to some of these
questions so this is particularly
concerning to me because it moves
outside of the realm of just security
there's no hacker anymore there's no
malicious attacker there's no getting
possession of inside information of your
network so if you are considering
planning or dreaming about handing over
your decision-making to deep neural
network you better start being concerned
about safety in fairness for example
issues especially and and here I have to
make a distinction of course because if
you're asking your deep neural network
to make decisions about things that have
no consequences you're trying to do
specialized individual pricing and ask
me a different price for priority
boarding and privilege of getting my
seat before others who cares if you miss
classify me what's the worst thing that
can happen I'm sitting in the back of
the plane in the middle seat I can live
with that it changes if there are really
substantial consequences to the mistakes
of this network and I don't care if the
number of mistakes is smaller I get
particularly concerned if it makes a
mistake on something that no other human
on earth would make a mistake on that's
what I got concerned about and the
mistakes they make is much much larger
than any human to do topics actions
things like autonomous driving insurance
influence medical treatment selection
employment hiring decisions prison
probation approval pricing for large
equity mortgage
life-changing decisions let's be careful
as we move into handing over
decision-making to deep neural network
when we're talking about very counts big
consequences on errors all right so what
can we do about it
this is topic for research for a whole
community in my opinion and my goal here
is just to get more and more people
interested in doing research here
because we need a whole village of
people working on this I'm gonna start
small I'm gonna mention a few things
that we are doing it's just part of a
big village and the whole session today
showed all the really valuable and very
important contribution our people have
been doing to me for example if I need
to start building trust on someone let's
forget deep in your network even a
person that talks to me the very first
necessary condition not sufficient just
necessary but starting point is that
well this person Brno and better be able
to say what they don't know if they want
to start building any chance of trust
from me so for me raising awareness is
particularly important especially when
you are planning about using deep neural
network for decision-making and one of
the things that we started doing or a
whole community of people started doing
is start defining what it could mean to
have a lack of business define and maybe
develop tools that measure it and
quantify it so the goal the dream here
is to have a tool that runs parallel to
the deep neural network in it's about
the same time we'd be able to tell me
also something about how confident that
answer is and I'm not just talking about
reading the value of the nast newer on
the SAS 91% because we decided that's
really not a value of confidence
something else if what if I start
perturbing my input like many people
said today in this session all of a
sudden that the answer changes if
if I see that that happens maybe that's
not transported how large of a
perturbation do I need to make before
the answer changes well if I need to
make a large perturbation before the
answer changes completely
maybe I start that's just a starting
point for building the trust it doesn't
mean that I'm gonna trust so that's the
goal hopefully this tool will be able to
raise a flag produce some warning for a
small number of cases so that hopefully
in that case the human is still there
being called up to look at that specific
case they are not many fortunately
because our highly performant deep
neural network are so good that they're
so they get it right on so many times
great so let's look at those mistakes
and let's really have something else
involved in those mistakes hopefully so
let me proceed now in this direction of
trying to develop ways to measure and
quantify this amount of perturbation and
I'm gonna give you first a graphical
representation then I'm gonna give you a
little bit of math but really I'm trying
to stay away from math because the more
math I give you the more you lose track
of the key point which is the questions
so if this is an ostrich label and maybe
that's associated with a specific vector
which has maybe all the amplitudes of
the pixels on this picture this happens
to be a very large dimensional space
which I cannot represent on this slide
so it's actually a point in a 2d space I
apologize for that there might be a
decision boundary and across that
decision boundary you might have another
label maybe it's a chicken label and
there's many decision variables this is
actually not to scale imagine there's
maybe a lot of a lot more space and
things are closer or further away we're
also in a thousand if not million
dimensional space so there's millions of
this decision boundaries around it's not
so easy as it looks in this picture so
if I start modifying this picture I
might end up somewhere close to the
boundary maybe just a cross with
something crazy looking like that I'm
sure that a lot of you that are again
learn to appreciate the beauty of these
crossed animals
or things this is actually not I don't
know if it was generated by again or
whatever I just searched on Google
ostrich chicken and look at whatever
came out that looked strange I have no
problem with this this happens all the
time somewhere in between there you're
gonna have with something that even a
human cannot tell and actually I'm no
problem with this because a lot of times
deep neural network also have a problem
with that and they tell you that you
have a problem in this case maybe it's
firing 55% that's not my issue my issue
is the existence of other situations and
here I'm just showing the amount of
perturbation on an axis so we can keep
track of it a smaller perturbation so
that I end up with something looking
like exactly the same picture as before
so tiny perturbation that I cannot tell
the difference
now it gets classified really really
really confidently as a vacuum cleaner
that's what I have a problem with on
these boundaries there are situations
like that they're not very common but
they can exist and people have shown the
existence they are called attacked like
I mentioned before so let's put that
distance here on this plot and now let's
look for this ball a lot of people
mentioned balls today and Here I am
showing this other representation of
exactly the same thing how large of a
perturbation do you need to get to get
to the closest boundary and if we can
measure that that's possibly an
indication for how much robustness you
might have in the perturbation of your
specific this is specific its input
specific there's not a single ball it
works for all the inputs every every
single input has its own ball because it
might be closer to the to a nearby
boundary that's very important to keep
in mind so it turns out that this can
actually be measured but it turns out
it's extremely expensive to do it
exactly people have done it there's this
raloo Plex algorithm that does it for
you on a very small network three
hundred neurons what can you do think
about what you can do with 300 neurons
not image that it takes three hours so
this is far from my goal of running the
deep neural network assessment and then
running also this tool in parallel and
getting a level of confidence a
decision-maker
might not want to wait these three hours
so if you start thinking about what you
should do as a research I mean improving
the speed for the evaluation of these
whatever measures or scores for
robustness is a great thing to do so
Lele it's my graduate student who
started looking at these issues and one
of the first thing that she did is try
to find some theoretical result that
linked this radius of this ball minimum
perturbation and p-norm
to the closest boundary and she was able
to relate it to the Lipschitz constant
without going too much in the details I
promise no math and Here I am showing a
math I couldn't resist
but if basically you start thinking very
very intuitively please apologize for
the people are technical in the field
but something about steepness of the
activation functions and high steepness
very very steep activation function
seems to be a bad thing from what we've
seen and I have developed some kind of
intuition for why might that that may be
the case maybe I don't want to say it
yet on a video recording but you can ask
me later
so once she has established this theorem
the next question is how do I actually
compute this value so the first thing
that she did is all let's just use some
sampling for example let's sample around
the ball and try to get an idea for how
large that is let me emphasize and I put
it in red that no I didn't put it in red
I'd put it right on the next slide this
is an estimation of that ball estimation
means that you accept the risk that it
might be incorrect so let me show you
how well it works and let me show you
also why sometimes it doesn't because if
I want to build trust I need to show you
what I don't know and when things don't
work so these are different data bases
with different examples there are
different attacks this is the amount of
perturbation that these specific attacks
needed to impose in order to get a miss
classification that look very much like
the same image to a human because these
are very small numbers of very small
perturbations
this core here the clever score is
Lily's score in terms of using sampling
and extreme value theory to try to get
an estimate for that number and they're
similar
so I'm we're happy in terms of that they
are supposed to be smaller because
you're trying to estimate the minimum
perturbation before you get a Miss
classification and they are always
smaller similar except for at least this
case that we were able to find and we
looked for it on purpose because I knew
it was going to happen and there it is
where in this case is slightly larger so
that shows you that it is an estimation
so it gives an indication it's something
that can use it to raise a flag warning
warning it doesn't mean anything else
but just be careful and look closely
with other tools to that specific input
so the next thing that she did and a lot
of people did in the literature and I'm
really happy that this is moving in this
direction is what if we give up some of
our desire to estimate the exact number
and we try to look for a more
conservative number maybe even two or
three times smaller radius but we
guarantee it it's a lower bound that's
also an interesting interaction for
research that I encourage people to take
and Lily also moved in that direction so
that was the estimation and trying to
progress some history here on things and
start looking into this robustness
certification which is basically trying
to find this lower bound guarantee and
because this robustness seems to have to
do with steepness and slope of these
activation functions then she started
with very simple activation function
that had only two slopes flat and
another slope to your value and in that
case started bounding locally each
activation function with two linear
bounds the same up and down and then
propagating them through and making the
apps from the perturbation larger and
larger until you get an overlap don't
want to go too much in the details there
are many things that I want to mention
but the bottom line I think looking at
how well it works is probably more
interesting I'm showing here small
examples these are small examples two
layers twenty
for different norms different ways of
measure that are measuring that ball as
a reference what do we use of course we
use read duplex it can be done because
these are small examples
remember it's expensive maybe it takes
three hours just for running one of them
it can be done only for infinity-norm
not for the other P norms so for those
we had to look for other references for
example this linear programming approach
which doesn't give you the exact value
but it gives you a lower bound and a
decent one maybe it's far off a factor
of two from the actual value and her
technique that I just described before
about producing linear bounds on
activation functions and propagating
them that's pretty well compared to that
pretty much for a lower bound that gives
up a factor of two to four meaning that
two to four is a smaller radius of the
perturbation compared to the exact
number she was able to complete the
completed computation in time that is
twelve thousand times faster than real
duplex and maybe for a similar number in
the bound as an LP she was able to do it
two orders of magnitude faster for three
layers things get better
six million faster than relics a
thousand times faster than LP speed is
important right remember the goal we
need to provide something that you can
run in parallel to your neural network
so that for each point you run you also
get a really better indication if you
can't trust it or not
so in summary well it does good quality
of the bound close to LP result but
faster what happens if you start scaling
go even higher well we were able to go
up to seven layers a thousand euros for
each layer which is larger than that
larger and but definitely not at the
scale of imagenet and this is where a
lot of research in my opinion needs to
happen because right now we are about
maybe 10 to 15 seconds which is the time
it takes to certify a specific answer
and give you a very good indication a
lower bound certified for the amount of
perturbation you can afford before you
have a Miss klegg with potential miss
classification
so it depends on what is my goal here
what is what I'm making a decision on if
I'm making a decision on boarding
priority I couldn't care less I mean
really but if I'm making a decision
about diagnosis I'm making a decision
about something more substantial I'm
definitely willing to wait 10 seconds to
verify even more than 10 seconds in my
opinion so she chose to move in a
different direction instead of improving
speed which I think needs to happen
she started expanding the technique with
other collaborators in the group to
other kind of activation functions more
generic activation functions instead of
using the same linear bound for below
and above she started propagating two
different bounds that seems to work and
then you can handle different kind of
activation functions then a really
brilliant in undergrad student under
grad student I'm so happy and the grad
students are contributing to research in
this field in a way that I've never seen
in my career before been in many fields
and undergraduate all it takes it to be
smart to be in this field that's a good
thing for the field so let me introduce
you to a client here
aquiline was able to extend these
results producing bound certified bound
to convolutional neural network and
instead of propagating this linear
bounds since we have a convolutional
neural network he tried to use
convolutional type of form for the
bounds I want to keep it short so I'm
gonna move very quickly to final remarks
first of all I'm really happy that the
community is finally beginning to
understand the issue that there is a
question and start looking into
measuring robustness it's a lot of
people that have been producing very
interesting measures I'm also very happy
that the people outside of our community
are beginning to understand this because
our community has to reach outside it's
extremely important because in
decision-makers they are considering
using deep neural network to make their
decisions are not experts of machine
learning and this is my problem when
they don't know anything about all of
this things that are going on inside and
they want to trust them completely
that's where I get concerned so I'm
really happy
people outside the community are
beginning to appreciate the fact that
there are situations where things might
not be as nice as we would like them to
be and there is more research that needs
to be done to improve that I'm also very
happy that the community is beginning to
develop tools because this is absolutely
needed in my opinion that can help these
users not only get an answer but also
get an assessment for how much they can
trust that answer and iBM has been doing
a great job it's been a fantastic
collaborator in terms of doing this
because they've been developing tools in
parallel with us with these papers that
are being publishing and deploying these
tools for people to use where you can
quickly assess and get an idea and get a
flag a warning is this an answer I can
trust or do I need to think harder about
this specific input all right so let me
summarize for me very important to raise
awareness that there are some potential
issues the definition of performance is
something that I hope you understood
it's not a single way of measuring that
we need to look at there are money many
more ways of measuring we started
defining robustness for example looking
at the size of perturbation I welcome
many other definitions please everyone
that is involved or wants to get
involved there's a lot of space here
very important that you also think about
computing it fast whatever you do
because otherwise users will not use it
and then you defeat the purpose and most
importantly I think I would like there's
one thing that deep neural network even
the highest performing deep neural
network haven't learned to do yet and
they really need my opinion to learn to
admit what they don't know or what they
might not be sure about that's my final
message
[Applause]
do I have an idea for how these there's
a whole community of attackers that have
pretty good ideas on how they actually
do this and it's a variety of reasons
some of them for example have to do with
the fact yes I've said before that I
wouldn't say it publicly maybe I said
well when people do training for example
what do they do they typically use
activation functions with very small
slopes because it's easier to Train
because you always know which way to go
but those produce network that don't
give very clear answers they give you 55
57 60 % firing users love to see 91
percent so what they do is once they're
done almost done training they increase
up those slopes of the activation
functions and they make them much
steeper so that means that all of the
sudden something that was maybe firing a
60% is now firing and 91% is this so
that means that sometimes we're doing it
to ourselves now we have a network that
answers more confidently but that means
we expose it to also making more
mistakes on things that shouldn't be
answering so confidently there are other
mechanism there are mechanisms where for
example the attackers exploit part of
the image that we as humans are not
sensible to so it could be that you look
at the spectrum the frequency components
that we are not used to are not capable
of looking at very easily you can hide
these variations in different places
there are situations where deep neural
network may be recognized that one
specific animal might have a specific
feature and only that animal in all
those millions of pictures only that
animal has that specific feature if it
recognizes that is going to look for
that specific feature only and if he
finds it it says that's that animal now
if that features for example I don't
know
a triangular ear and there's only one
animal in the world that has that what
happens if there is another animal that
gets in a fight and someone is biting
off a piece of the ear and then it ends
up being triangular the neural network
will see the triangular year will say oh
it's an ostrich
no it's not a human would not say that
it would say it's a dog with a year that
has been bit enough so these are there's
so many failure mechanism the more I
look into it the more I discover more
it's like making a list of all this
failure mechanism is not the way to go
in my opinion because there would be
always more and more that people can
find or people can invent this is why I
decided to not even get into that fight
I'm just got into the let's measure the
robustness try to provide an assessment
[Music]
yes there's different ways to plot you
can just look at the noise that was
applied you can try to look at which
neurons are more firing more or less you
can do maps of which neurons are
responsible and what they're looking you
can see what the neural network is
looking at in the picture and do a heat
map to understand what made it or where
at least the problem might be I am
trying to be vague on purpose because I
don't have answered myself and I don't
think that all of these people are
looking at these answers have already
answers these are tools that they're
developing to try to get two answers and
understand but that itself to me is
concerning that we don't know these
things
[Applause]


Secure Learning in Adversarial Environments (Bo Li, Foundations of Safe Learning Workshop)

our next speaker is Paulie her boy is
assistant professor in the computer
science department at the UI you see she
got a PhD from one debate in 2016
her research focus on machine learning
security privacy and the game Syria
she's particularly interesting exploring
vulnerabilities of machine learning
systems to various adversely attacks and
developing robust approaches for
learning system
please welcome Bob it's a great pleasure
to share my research about adverse or
machine learning here with you all so
first of all we know that machine deep
learning machine learning AI has been
ubiquitous in the world in every domain
including on homes driving healthcare
and smart city however in my research I
mainly focus on even such very powerful
machine learning and AI technologies has
brought us a lot of security and privacy
concerns for example in 2016 in rural
example that the Associated Press
Twitter account has spreading a rumor
saying that the White House has been
attacked which has Swiper about 136
billion dollars within seconds because
of the Ottomans trading BOTS just dumped
a large of amount of stocks within
seconds based on the fake news and
similarly in the hospital data sets like
in one of our previous work we find out
that even with just as the discharge
workflows segmentations
we can identify a lot of privacy medical
treatments about each patient so this
just saying that within this big data
area we can actually have a lot of
security and privacy concerns even
though they have brought us a lot of
convenience so that conclude as that
even currently with a lot of convenient
platforms for them for patterns and a
flaw as a previous speaker mentioned and
we are still living on adverse
environment that's why typically us
your machine and I still have a very
hard life to make our machine learning
platform robust and that's why here we
want to get together and understand also
add virtual model and why the machine
learning models are so vulnerable
currently and therefore we can find
certain solutions hopefully to improve
the robustness of our current machine
any platforms and this is just showing
that this is an important problem and
both government and different funding
agencies recognize that adverse
immersion learning as an important topic
in current scenarios
so after this back one I think the first
question people may ask is about what
exactly as advice for machine learning
and why such vulnerability happens so
there has been a lot of phenomenon
described by for example a professor
Mendes research from the image
perspective and fundamentally I will say
there is one like question or problem
that causes adverse or behaviors which
is that in traditional machine learning
techniques and people always assume or
increasingly assumed that training and
testing data from the similar even the
same distribution and that's why we can
always make the training and one and
make inference and the other even they
are an seen however in the considering
adverse or behaviors is that the
training or the testing data
distribution can be manipulated in
certain way so for example when the
training data distribution was
manipulated it is caused poisoning
attack or the back door attacks
mentioned by the first speaker and when
the testing data distribution is
manipulated it's called evasion attacks
where when the model is trained and then
you can still generate so-called adverse
to attacks or there was two examples to
fool the existing well-trained machine
any models so in this talk I will mainly
focus on evasion attacks things a lot of
times we can protect our model and to
now release it so evasion seems kind of
very severe problems and I will give
some examples of physical attacks
against the different types of
real-world sensors and then hopefully
give to potential defence principles
from both
games theoretical and properties of
machine any models so first let's look
at the physical examples of other words
or text so even like given a typical
learning algorithm for example if your
network a simple example is that we can
have a loss function and based on the
training data x and y pairs given the
example of supervised learning we can
minimize this loss function and use for
example SGD you very efficiently and
find some good very good local optimal
however given an adversary behavior what
they can do is that they can add a very
small perturbation on top of a given
example for example X and then the goal
is to maximize loss function here which
is a positive with the learners
objective so in this way we can see that
by adding a small magnitude of
perturbation is possible to generate the
search adversity havior x' against
existing trained machine learning models
and here's a probation if shown here is
actually easy to relatively easy to find
for example we can use local search
command toriel optimization and also use
certain convex relaxation to find the
optimal minimize the perturbation
magnitude and this is all the in digital
domain from the also from professor
memories examples and then the people
always ask a question so how such add
virtual examples add like realistic in
real world take an example of autumns
drivings
so how realistic that was so examples
could be for example in terms of
attacking existing atoms driving systems
which is one of the biggest use case in
for machine learning and technologies so
here we know that incomes driving we
have a lot of different sensors for car
for example perception and lidar and
radar all kinds of sensor fusion
together however in real world we can
see give an example of stop site this is
a real stop sign taken near ones
vancouver wines berkeley and we can see
that always some random pad
on the side so people may question like
what if those patterns actually
adversarial but actually they do not
cause any attention of humans so you not
recognized that would not replace the
sign in reality and there are several
challenges if you want to do that which
is you want to generate a kind of
so-called physical force or perturbation
on a stop sign so in rewards compared
with the digit award additional
challenges raise up for example the
first is different physical conditions
for example different angles and
distance and landing conditions will
make this law as we mentioned the low
magnitude of perturbation vulnerable so
for example if you have a like like the
dog and the cat example if you change
the image a little bit with different
angles and distances they may fail to
perform such attacks so how to solve
this question and make a generalize that
so called the robust physical so
perturbation that's one challenge in
real world and the other challenge is
that as we mentioned the perturbation in
gradua with hopefully that is of low
magnitude and in imperceptible but in
real world for example considering atoms
driving car driving by and show you hope
our camera can capture the perturbation
itself in real world so how can you make
the perturbation visible and
unnoticeable to human that's another
challenge and also other challenges
reserved for example in reward you
always have some fabrication error so
when you want to print out something
like red but it turns out the color
printer will give you something like
print pink or orange and also when we
generate the perturbation giving an
image in the digital domain we can
general prohibition everywhere including
the background but here if you want to
have a physical motivation it has to be
uncertain objects so this is another
different challenges so next I will just
quickly go through how we can leverage
and resolve these challenges with
different techniques and makes them
coherent together and eventually
generate a so-called physical
perturbation against the real world
systems
so first is the challenge about the
different physical conditions for
example recall that the first loss
function is what we used to generate and
with the perturbations in digital world
for example the whole objective function
we want is to minimize the doubter which
is the magnitude of perturbation plus
the loss function of that was to example
which is expressed Delta is generated
add virtual instance and then hopefully
that instance will go to the target
which means the y star here which is
different with the ground truth why and
considering different physical
conditions here what you can generate is
instead of the original loss function
like the digital loss function here we
can generate a large loss function by
sampling our different transformations
from real robot distributions for
example here is an example of different
angles of a stop sign in real world and
you can draw a sample of the
distribution and add to this big loss
function and in addition to this
distribution of different angles you can
change the colors and the physical
conditions of lightnings etcetera so
make this can loss function to consider
as much conditions as possible and
improve the generalization of this
attack so by doing this we hopefully can
improve the robustness of the generated
adversity or perturbation and second is
about the spatial constraints so as we
mentioned we hope with this perturbation
can be onto some objects itself and
still of some meaningful shape so that
it will be less realizable by human
which is called from hidden in the human
psyche so basically what you can get is
that you can add a mask to this
perturbation region so that the
generally the perturbation will be only
limited to certain spatial regions and
therefore this if the shape itself is a
meaningful like for example is some like
some gravities or some patterns then you
will not raise humans attention too much
and this way there are some tricks to
find those vulnerable regions so as we
make
there are some regions are correlated
atul robust features and some regions
are less robust eight features and the
trick here is that for example you can
use some sparse
regular risers for example our one to
generate those parts perturbations and
then find those vulnerable regions and
specifically to focus on those regions
and use for example our two or our t v--
loss to smooth out those regions and
generate more realistic perturbations
and the last challenge is the
fabrication error so basically there is
a more general loss here that you can
use the cps law mps loss to minimize the
distance between the principal vector p
prime here and the desired vector p hat
so that by minimizing - you will be able
to make sure the generated color or
generated the perturbation will be as
close as possible with what we can get
in real world so by solving this several
challenges together we will see what we
can get in a static experiment so
basically what you can see here maybe
someone has saw some of such stop signs
in reward is that each column shows a
type of predation and for example for
the first row you can see that the first
two rows are kind of subtle perturbation
as you can see this perturbation looks
like fading out but actually all the
stop signs in these lives was miss
recognized as a speed limit of 45 by a
real word camera and under different
conditions and each row here shows
different physical conditions and under
later story Collins
some of them show some random letters
and the later two occurrence so just the
white and black rectangles so this is
somewhat surprising that several like
fall the fall rectangles enough to for
the stop sign in real world and make it
targeted to be miss recognized as
certain adverse or targets so this is
some examples of static test
and there are some physical tests that
we can see what it happens if a real car
driving by towards such physical
perturbed stop signs so the left-hand
side is control experiment of a sort
stop sign the right hand side is a
control experiment and when you pay
attention to the captions you will see
the result of the classification so you
can see on the red hand left hand side
is consistently miss recognized as speed
limits at 45 for most of the time even
the car is driving with certain speed
and this is another example of different
types of perturbation however on the
right hand side as a control experiment
has always been recognized as a stop
sign so this just showed that in waster
car driving in certain speed under
different conditions in real world this
physical stop signs can always miss lead
the real motion any models and the
vision systems encrypted in a car and
actually another this is for attacking
classification models and just now one
audience asked about the object
detections and that's exactly one
question we get before so people ask
what if it is object detections
because there are some automatic
cropping mechanisms here with the
proposed with an automatic a proposal
generation approach if like whether this
kind of physical predation effective or
not so this is an example that's
attacking your law so basically you can
see the real the physical stop sign here
most of the time actually all the time
it cannot be recognized as a stop sign
but other objects can be recognized
without any problem so this shows that
under different physical conditions the
physical perturbation can affect Li
attack the objective hector's as well
unless you go to very very close to the
sign so they recognize the letters here
because we are purposed to not block the
letters so the ones recognize the
letters they can find it's a stop sign
however the argument is that it will be
too close and it will be too close for
you to stop a car with this speed
and other questions may raise about the
white box and black box attack because
in this one is a specific white box and
you get so gradient directly from the
model and you can optimize your
perturbation and actually we can
leverage the same perturbation use a
phenomenon called transfer abilities
that use the same probationer model a
you can test it on totally different
model and perform the so-called black
box attack and what you can see here is
that this is the same probation test on
the fastest again and in a very
different physical environment and you
can see even like this and this is a
less complicated environment the stop
sign is always miss recognized until you
get very close so this stop sign
increased a lot of interest intentions
and actually this physical stop sign was
displayed in the Museum of London
currently and people get quite excited
about why the real world attack examples
can be generated and attack the reward
perception systems of car
so after this physical attack examples
people ask us a question about okay if
the perception system is vulnerable with
no images are vulnerable there are some
overfitting information so the models
have over over capacities so they always
have some room to generate such
perturbation and attacks so people think
for example some autumn driving startups
companies think that if we have sense of
fusion for example if we equipped it
with lidar together with perception
sensors this may be more robust and here
I just want to quickly go through we
will not go to the details because we
also want to go to some examples of
defense and here just the one example
want to show that actually lidar system
itself in real world is not as robust
either
for example this is testing an Apollo
open source lidar system from Baidu and
in reality is a lot of additional
challenges in terms of attacking a real
word lidar system for example the rural
idyll system is not an end-to-end model
you cannot directly get a gradient as in
the classic
image model and also they have a lot of
different components connecting together
which will reduce the aggregated
adversity or perturbation effects
however even with with such real watch
constraints and challenges and we can
still be West you able to achieve
certain and what your goals so one goal
is that what if we can generate a
real-world objects and put this object
on the road so that even though it's
obviously large objects it still cannot
be recognized by the lidar system and
the other goal is that what if we put
her object onto the car or near the car
so that your car in front of homes
driving car will not be recognized by
that car behind you which is obviously
quite dangerous and I want to actually
again show some videos and this is chess
it and reward like highways and the
experiment here set up is that on the
top there is some object on most objects
showing there and the left-hand side is
a lighter detection system so basically
you can see if it's optimized add
virtual object put on the highway when
items driving car driving by Celina
actually fail to detect this object
itself and if we put a box with regular
with regular size and with regular shape
and is a similar size with adverse the
object we generated and you can see it
clearly as a control experiment clearly
detect this object so here on the top
you can see this box is always almost a
Miss recognized even though there is
clearly a box here which is like a 3d
printed object which is like relatively
this large as a box and if we put a box
with similar size in the middle we can
see that most of the time let's see most
of the time actually Texans are add
virtual objects with a green box so
another example so on the top you can
see that most of the time
the green box show up which means this
this real box here was detected but if
this like the object with a slightly
weird shape here this is really printed
and optimized in a way to attack the
lighter and this lighter system always
cannot find the green box here which
means it fails to detect the objects so
yeah so this two example just want to
show that in reward actually no matter
is vision or the light of perception
systems different physical adverse your
attacks can happen in real world even
though the real world scenarios can be
much more complicated compared with the
digital environment so the next based on
all these attacks and I don't think I
need to emphasize the severe
consequences of the digital was to
example since there are like hundreds of
work for its so far already so next I
want to introduce two principles in
terms of defense from both schemes
theoretical perspective and based on the
properties of machine any models so
first let's look at games or perspective
so so far we know adverse so machine
learning as we mentioned is quite
important and there are many many work
in terms of defense and detection has
been proposed however one significant or
important message is that those defense
or detections are mostly broken by
adaptive or more clever intelligent
sophisticated attackers again so this is
a bad news and this forms like certain
cat-and-mouse game like we have better
attack and then again we have better
defense and then it can be attacked
again so how can we end this game and
here I want to introduce another game
formulation which is dagger per game but
first let's look at a simple example
from the NLP example because in the NLP
domain the features are more improved
interpretable for example the words
compared with the pixels of the in
so I want to use this in for example to
show how can we build a good game model
to improve the robustness of our current
learning platforms so first this is a
very simple example of spam filter and
what you can see here is that if you
have a spam and you can train for
example bag-of-words
classifiers and by assigning some ways
to the words and then what you can get
is some scores and by combining
comparing with some threshold you can
find out this is a spam for example and
this is what we called a spam filter
version one very simple and then with
attacker comes in what attacker can do
is add for example add some good words
so by adding these good words even those
adverse all words are still here but
then you can lower the weight and
compare with the static swash hold you
will see it clearly beam is recognized
and this intuition can go on so the
spammer can put it back so the defender
can put it back into the model to return
it kind of like adversary training and
you can improve the efficiency
again but this can go on and on and form
as we said a repeated game but how we
can end this game so I would propose
actually proposed a new leverages a new
game stackable game to fall to model the
behavior of the attacker and defender in
learning perspective for example in the
stackable game typically there are two
players one is a leader once a follower
so basically the leader will commit some
strategy and the follower will follow
the strategy and make the final decision
so this is not a repeated game this
YouTube player and one step game and the
good thing for this is that if for
example the leader here is formulated as
a learner so the learner can commit some
strategy which kind of for example the
classifier here and you can see the key
point here is the Oracle which we have
an adverse model so as long as we can
somewhat model that was to behavior in
from us for example Oracle we can
consider this attacker behaviors
in choices and also strategy so that the
defender can always generate his robust
learning strategies based on the
attackers consideration and this show
such with this commit strategy we have
two goals one is to minimize attackers
attack available successful rate and the
other is even though we cannot defend
100% of the attack we can always
increase the cost of the attacker to
attack the system which is actually a
very important principle in security to
make that harder and harder and there
are several ways to model attacker
behaviors here for example we can model
it from the mathematical perspective or
we can use human subject to understand
the real-world attacker behavior so in
this work I will mainly focus on the
mathematical modeling here to show how
we can leverage the attackers behavior
to improve the robustness of the
learning process so we can see here from
the previous simple example there are
several three important components one
is for example if we want to model that
was the behavior that was a goal is to
minimize the cost function between the
acts the original instance and the X a
which is adverse to a generated instance
such that the generated instance will
always be recognized as B'nai by
comparing with certain threshold Delta
here so we can see either the cost
function is important because we need to
consider the future substitution all the
future selection is important because as
we mentioned we have robust and non
robust features and the dynamic
operational decision is important here
because if we have a static optional
threshold here the if the attacker knows
it we can always perform such an
adaptive attack so how to model this
dynamic optional decision with a certain
function to automatically learn age is
actually another important research so I
will mainly focus on this part and give
you some very interesting results to get
a sense of how
can we make our learning algorithms more
dynamic in real world so that you by
adding those dynamics that hacker will
not be able to evenly get the algorithm
architecture itself
they will not always be able to get the
whole framework and instead of attack it
efficiently for example here still with
original very simple like optimization
function for the attacker so instead of
a static threshold outta here we want to
use a function like Q to learn this
threshold Delta dynamically and one
drawback you can see here is that this
function can be the input of this
function can be of very high dimension
because it takes the X prime which is a
generated input and the predicted input
as the input as the input of this
function but the dimension of this input
can be of thousand by thousand as for
example image so here the nice thing is
or it's a virginal generic technique is
you can use a finance set of basis
function to reduce the dimension of this
learn function so that you can always
perform of like optimize as final set of
basis function and optimize the
coefficient of it for each of the basis
function so that based on this learning
dynamic learnings worth threshold you
can actually prove it will be
np-complete for attacker to attack the
system and eventually you can optimize
this whole optimization function as this
process for example here this is a
traditional learning algorithm that you
want to minimize a loss by optimizing
the weights given the training data for
example x and y pairs and here you add
another loss term which is you consider
the animals or behaviors for example for
all the y equals to 1 which is adverse
to instance you can generate her box
here that to model like model what's the
cost of the adverse race and what's a
stretch like budget of the adverse way
to in to consider what they can do in
the worst case
Arielle and add these two laws together
together with the Iowan laws to make the
trade-off between the future reduction
and the robustness and based on this
optimization you can see as long as we
can solve it efficiently you will be
able to have a robust learning systems
because you have considered both benign
accuracy and the adversative behavior
however this is not convexity and this
is hard to solve and but we can use the
mixed integer linear program to directly
formulate the original optimization into
this linear program and use existing
solver to solve it but we don't need to
go to detail about this but we can see
this is definitely solvable efficiently
and by solving this with existing
solvers we can see that the approach
actually compare with different state of
art adverse though behaviors including
other also training this can achieve
much better robustness and this is based
on the game of stagger Burgamy
which is essentially can formulated as a
min/max game and however people may ask
this may not be quite scalable in
certain sense because we have to use
these tricks like finite differences
basis functions to improve the
efficiency and can we directly leverage
the properties of current machine
learning models to improve the
robustness
so our introduced one work in five
minutes and I think this is actually
quite interesting and although it could
be a very simple solution but I will
show that it's quite big it could be
very effective because for example we
have discussed a lot about segmentation
as our classification and object
detections but those actually have some
information concentrated in the model
itself but for example for segmentation
itself it has some properties for this
algorithm itself for example for
segmentation like here if you have image
you can attack it to different targets
for example you can have a Hello Kitty
or like just appear colors to attack the
segmentation model but we know that for
segmentation
which is different with classification
is that you always have some spatial
consistency among different features or
like the pixels so if you if you general
into some heat map for the cross entropy
around the pixels of from from the
surrounding pixel prediction for one
pixel you will see that if it's a B'nai
you can clearly see the edges of a
street but if it's also like when you
have prohibition added into the images
you can see also boundaries a very
blurry is kind of just a mess up which
shows that when you add a perturbation
even though it's not perceptible to
human eyes systems actually it breaks
the spatial consistency intrinsically so
such property cannot be removed easily
by attacker so if there is a way to
identify such property you will be able
to improve the robustness of the models
at least you can detect such be adverse
of behaviors so how to detect it
actually very simple so you can as we
mentioned the spatial consistency was
broken by the adverse occupation right
so what you can do is for example
leverage in spatial consistency PI give
an image you can generate the two crops
with overlaps and then you can throw
this two overlapped crops into a
segmentation model and get the
segmentation from out from this model so
you can see if it is a B'nai the overlap
the region will have very consistent
results in terms of for example mr you
because if you are doing the
segmentation task and if it's adverse
rate because the spatial consistency is
broken so the behavior is a prediction
are very different from this figure and
based on this we even do not need to
learn any function we can directly
generate especially because you can see
this separation is very clear it's not
mixed up together so these types of
interesting properties are very hard to
avoid by attackers
so by some connotative results you can
see under different types of Italian
Houdini are different types of attacks
against the segmentation and you can see
the detection rate is almost 100% and do
one nice thing is that if you consider
the
adaptive attacker as I mentioned even
though the attacker knows you use this
detection method to detect my other
words or examples still the detection
rate is almost 100% the reason is that
when you generate those random crops is
it's very random so for the attacker
what they need to do is to make sure
every crop is consistent with our
original like the segmentation results
that's very hard because first they do
not to now know which patch was used by
the defender and second is if you
generate a lot of patches the probation
will be add up and it will be very high
so that is a trade-off for attacker so
they cannot do that so eventually it
shows that attacker fails so by doing
this we can see with this specific
learning properties we can leverage it
and improve the robustness and detect
adverse or behaviors so based on this
well I think I'm out of time but I will
say that actually those attacks and
defenses our work has been reported by
different media's and actually it shows
that different attacks in real world has
risked a lot of attentions and defense
is very important and the so far we have
for example leveraged game theory which
is very effective but it may not be so
scalable and we can leverage different
properties of learning models to improve
the robustness however we have to
identify different specific properties
for different models which is another
hard open question can we optimize early
identify what properties will be useful
to improve the robustness of machinery
models so that could be a very
interesting direction to go frauds yeah
thank you
[Applause]
[Music]
classification or
which are
yeah I will say attacks are always easy
any model is easy no matter how you
order any model digital world but
defense is hard by the sudden defense
you can elaborate for example
segmentation now we have solved it I
would think oh it's maybe much easier
than classification because
classification has you have to actually
leverage some semantic additional
information to improve the robustness
but segmentation only based on spatial
consistency we can achieve almost a 100%
detection that I would be very happy and
some I don't show the example here are
some model for example videos you can
leverage the temporal consistency and
audios you can use temporal consistent
we also achieve almost a 100% detection
rate for audio and videos so those like
you can imagine how some specific
properties would help to improve the
robustness but some models are very
generic for some of classification I
would say it's a little bit harder yeah
thank you
personal passions he set up critical
yeah for like AI system you have to have
an object right if you put want to
generate some object onto the like
spoofing on to the car or certain things
it will be possible but for like that we
always need an object so that the
reflection can be captured by the lidar
systems yeah
[Applause]