Alan Fern – Toward Recognizing and Explaining Uncertainty – CSRBAI 2016

so welcome back so I'm now please send
introduce Professor Alan fern who's
going to be presenting his joint work
with Professor Tom do direct Alan ferns
the professor and the associate head of
research at the School of Engineering
computer science at Oregon State
University
his research is very much on automated
planning and feedback in reinforcement
learning systems Tom Dirac is the is the
distinguished professor and the director
of intelligent systems at the School of
Engineering computer science at Oregon
State University and just completed his
term as the president of the Association
for the Advancement of artificial
intelligence and his research is at the
intersection of machine learning data
science and big data so I'm very pleased
to like introduce their their joint work
anomaly detection and competence in
modeling the environment for our
intelligence so please join me in
welcoming professor Oliver so uh yeah so
originally we had thought of us talking
about our FLI project that's actually a
little more aligned with robustness than
transparency and but we figured out that
we can try to address the this uh topic
of transparency here and there's gonna
be two parts to this basically we're
interested in recognizing uncertainty
getting systems that can recognize their
uncertainty and then explain the
uncertainty so it's a goal one we'd like
uncertainty aware ml systems now this is
it was a nice paper in ICML 2008 they
introduced a framework of knows what it
knows and they wanted a sort of a PAC
learning framework where the system had
to make confident predictions it was
allowed to abstain and they were trying
to minimize the the amount of
staining while guaranteeing that the
predictions were confident so I thought
that was a neat framework and that's
along the lines that we're thinking here
as well so we're gonna allow ambiguous
responses from the system as long as
they are correct in some some way that
will specify and we're particularly
interested in studying this both in
closed worlds so this is sometimes
called the known unknowns situation
where you sort of quantify the
possibilities and there's uncertainty
over those possibilities and then we're
also we're really interested in the open
worlds case the unknown unknowns case so
we we don't even know all of the
possibilities so we're we want to
explore uncertainty aware systems in
both of these cases and why I think
there's obvious reasons
we'd like safe and Trust for the AI I
don't have to really talk about that
much I think end-user acceptability is
another major reason if the system is
making lots of crazy predictions the end
user just going to get very unclear
about whether they should trust the
system in any particular situation and
and my own research this might not be an
obvious reason but computational
efficiency if you think about trying to
get more and more accurate systems that
have confidence you might have to
develop very complicated models but if
you can get smaller models that actually
understand their uncertainty then when
they are certain you can just go with
those and otherwise you back off to more
complicated models I study learning and
planning so I think of this in the
context of what we'd really like to a
lot of obviously there's a lot of
obvious bad things to do and we can be
confident that they're bad and we'd like
the planning systems and sort of search
planning or reasoning systems to search
the rest of this
so that's another reason that
uncertainty of where AI is is
interesting so goal two is transparent
uncertainty in ML systems so we'd like
the systems to explain their uncertainty
not just abstain for making a prediction
but give us some insight into why
they're uncertain and this this really
hasn't been studied a whole lot again
we'd like to do this in closed worlds
and open worlds and and why well a lot
of times I think it could be a basis for
feedback to a machine learning system
you know I can't tell whether this is a
cat or a dog because why and maybe you
could give a piece of advice
or give a new feature or say well this
is actually a really ambiguous situation
you're right to be uncertain here so
that might be one reason you would like
to explain uncertainty at the the second
part of this talk I'll say I'll talk
about anomaly detection and how how
often explaining uncertainty can be
important to the end user to to give
insight into the system itself you know
this is I think it even outside of
anomaly detection getting some insight
into why a system is uncertain might
help you debug it or improve it in some
way and then I think it's also a
mechanism for building trust if you get
explanations for when a system is
uncertain and is generally uncertain in
reasonable ways the reasons are
reasonable I think think you would tend
to trust a system more that has that
property as opposed to explanations are
kind of crazy and you can clearly see it
doesn't understand what's going on by
its explanations of uncertainty so those
are the two goals that I want to address
and the talk today I'm gonna have two
parts part one is going to address the
uncertainty awareness aspect and we're
going to talk about it's part tutorial
and a little bit of work we're just
getting started on this but so we're
going to talk about conformal prediction
as a way of getting uncertainty
awareness
at least in the classification setting
and we'll talk about open worlds and
closed worlds some experiments there and
at the end of the day what we're gonna
see is that when it comes to open worlds
empirically and the theory says nothing
conformal prediction theory says nothing
about open worlds but empirically we see
that it's not conformal prediction is
not adequate and it suggests integrating
with anomaly detection so the second
part is going to be some work we've done
more of a case study in explanations for
anomaly detection this is in a security
setting and that case study forced us to
try to give some initial answers in that
context to these questions what is an
explanation for an anomaly how do you
compute an explanation and then how do
you evaluate explanations we had to try
to think about those questions in that
context so that would be a little case
study focused on those questions so
first you know you're often though so I
these are examples or a little
cartoonish my student picked these out I
guess it's lilo and stitch I'm just
gonna go with them but the
classification we've got our known
classes you know this is if we're in a
closed world there are humans animals
and hybrids and a standard predictor
will output one of those choices this is
a human and this is actually a hybrid
but the system says human we don't get
much information from a pure predictor
like this that would let us gauge the
the certainty some systems do give you
numbers that you know a larger number
means maybe that it's more certain
smaller number is less certain I find a
lot of times those numbers are not very
reliable but the uncertainty has not
made explicit and there there's really
no guarantee from a standard predictor
now what conformal predictors are going
to do and this is a Boff
actually it was an ICML paper in 1999
that kind of set the stage for this work
he has a nice book on this type of
material came out in 2005 but what what
conformal prediction is going to allow
is to output sets of classes so I'm
going to focus on just the
classification problem you can also do
this for regression and then you would
have confidence intervals so we could
output animal or hybrid and we're gonna
say that this is a correct response if
the true label is in the set okay so
it's fairly straightforward here the
system was very confident and human and
it output this set as sort of the ideal
thing we'd like from a predictor there's
no ambiguity in its output here
apparently the system was very uncertain
and really couldn't tell us anything
about this object that it's looking at
so basically conformal prediction lets
us output sets as opposed to point
predictions and the correctness is
judged by whether the true label is in
the set now you could say this is
already a type of explanation of
uncertainty in the sense that if you go
back to the knows what it knows
framework that I that I pointed out
earlier by Lee Hong Lee Michel Whitman's
group they they were allowed to abstain
or give a prediction and abstaining
doesn't tell you much other than the
system doesn't really have confidence
right now this at least gives you some
indication of the level of confidence in
the system and you know here here it's
kind of a reasonable you can see these
are reasonable sets in terms of
understanding the the uncertainty now so
so how do we evaluate a conformal
predictor well one way is accuracy right
and so so conformal predictors will take
as input an accuracy constraint where
95% would mean that 95% of these sets
that you output
have to be have to have the correct
label in them okay so that's all we mean
by accuracy and the the obvious thing
here though is that well you can always
get a hundred percent accuracy by just
returning all of the labels right so so
just maximizing the the set size is a
perfect solution if we only care about
accuracy now obviously we we care about
more than just accuracy in this setting
what we'd like to do is a remove as much
ambiguity in the predictions as possible
yeah and and there wasn't really a clear
measure of ambiguity in the conformal
prediction literature they would sort of
talk about the number of labels that's a
hard thing to think about when you have
data sets with different numbers of
labels to just look at graphs so uh so
we've introduced you know for our own
purposes a notion of ambiguity that this
is the formula that's 0 when it outputs
one label so it's 0 and big.you a-- t
and if it outputs all the labels it's 1
okay and so we see this prediction has
an ambiguity of 0 this prediction has an
ambiguity of 1 this is in the middle all
right so uh so we're gonna be interested
in minimizing the expected ambiguity
given a constraint on the accuracy okay
so you'll specify a 95% accuracy
constraint and you'd like the
predictions to be as unambiguous as
possible right and that's really what
conformal prediction is is trying to get
at so so let's go inside
I don't know how many people have come
across conformal prediction it's not
sort of a widely widely known framework
so I'll give you a little insight into
what what these systems do so a
conformal prediction system is really a
wrapper around a standard learner you
know you can pick your favorite learner
random forest neural network nearest
neighbor
you know your favorite predictor here
and you've got some training data and
what we're going to expect from the
system is we have to be able to compute
nonconformity scores from its output and
I'll show you some examples of what
nonconformity scores will look like but
this is just showing you you know a six
means that it does not think this is a
human because we're calling it a
nonconformity scores larger values mean
that it's less consistent with this
human label in this case so we're going
to assume that we can get these
nonconformity scores and and actually
these can be arbitrary scores or the
conformal predictor will still achieve
the accuracy constraint we're gonna feed
those into the conformal prediction
framework with an accuracy constraint
and it will output some labels okay now
the the main thing that's going to
control efficiency so the conformal
prediction framework under some
assumptions the data has to be
exchangeable and there has to be some
assumptions about the nonconformity
score some pretty weak ones it will
always maintain this accuracy constraint
no matter what you could just have a
pretty dumb that you could have a
constant predictor here and I will
always get a it will always maintain the
accuracy constraint I'll show you you
know how how it does that in a moment
what really influences the ambiguity is
going to be the quality of this
predictor and the type of nonconformity
score that you define from it okay so so
the stronger the predictor is you're
really going to influence the ambiguity
the accuracy is just taken care of by
the framework itself okay you could have
a horrible predictor and it will still
be accurate it might output highly
ambiguous labels so let's look at some
examples before I show you how it uses
these nonconformity scores let's look at
some examples of nonconformity scores so
say you're using a nearest neighbor
classifier here's your data
a typical nonconformity score we want so
if you
have an input X and you have to consider
each of the possible labels the
nonconformity score for for human would
be what's the distance to the closest
human so if you're very close to a human
you're gonna be close to zero so you're
gonna sort of have a low nonconformity
score divided by the distance to the
closest non-human yeah so that's a
fairly intuitive nonconformity score if
this is not a human and the distance to
the closest team in might be rather
large and you would have a larger
nonconformity score so that's nearest
neighbor another example would be random
for us so this is a random forest it's a
bunch of trees and here are the
predictions that these trees give in the
random forest so a natural pretty basic
nonconformity score would be the number
of trees not predicting the label that
we're considering right so how many
trees did not predict human divided by
the number of trees and that would be
larger when when a label doesn't conform
as well to to the true label all right
and finally I'm showing you the the
random forest and neural networks as
we'll use those in our experiments for a
neural network it could be a big
convolutional network the outputs are
usually scores often between 0 and 1 and
a 1 nonconformity measure that's used is
the max output for the node that's not
human in this case if you're considering
the human label divided by the output
for human yeah so these are just fairly
natural things and and you could play
around with different nonconformity
measures and it would not influence the
correctness or accuracy of the conformal
predictor but it could influence the
efficiency all right but these are the
ones that we'll be using
so so given a nonconformity score and
your favorite predictor how do we
actually do the conformal prediction
we're
we're gonna use a framework called
inductive conformal prediction the the
the original work was in a transductive
setting but it's just computationally
implausible because you have to retrain
your model for every test example so
thus obviously not possible if you're
using a convolutional neural network on
image images so so what we do is we have
some calibration data that we're going
to hold out and from that you can
basically get a histogram of
nonconformity scores okay and you know
and basically out here these are the the
this is sort of the tail of a
nonconformity score part of the curve
and if we get a particular instance we
can ask for what's the nonconformity
score for animal hybrid here that's one
of the classes and human right they land
at different points on this histogram
and we can just compute a essentially a
p-value right so so you're basically
asking the question for a 95% guarantee
or at least 5% of the calibration scores
weirder than the example that then this
example with this label right so so for
animal there's a large percentage of
things that are weirder than then this
particular nonconformity score and so we
would we would consider keeping animal
in the set right and you know we would
rule out human because it's sort of at
the tail here right there's there's less
than 5% we'll say so so essentially
we're just doing p-value is just a
p-value and you're you're ruling out
labels that basically fail a hypothesis
test all right so that's conformal
prediction fairly fairly basic idea but
but you can actually get some guarantees
under the assumptions of exchange
ability and some basic assumptions about
the the nonconformity score but so so so
when we
it was kind of compelling framework for
for quantifying uncertainty but there
were really very few experimental
evaluations and especially they rarely
looked at ambiguity other than did the
system output a single label or more
than one label not many data sets were
used most of the experiments were with
nearest neighbor which we found doesn't
work very well we have much better
classifiers than the nearest neighbor
and so so we we just started doing a
little empirical evaluation I'll show
you I'll give you a glimpse of that
today with with better classifiers and
nearest neighbor random forests for non
image data and then convolutional
networks for four images or a lot more
perceptual data all right and we're also
interested in what happens when you take
a conformal predictor to an open world
so we'll be looking at that well so what
we'll be looking at both closed world
and open world performance and now I'm
going to go through these relatively
quickly there's nothing terribly
surprising but if we look at random
forest results we picked out some UCI
data sets and these vary in different
ways we won't get into details here but
uh for this is what a learning curve
looks like so so the number of training
examples and over here we have accuracy
so this is this is a number of training
examples versus accuracy we're using a
95% guarantee and you see early on it's
not actually hitting the guarantee and
my hypothesis here is that the theory is
based on some basic assumptions and
those assumptions are more clearly
violated you know there's a higher
chance that they're sort of not in the
data set for small sample sizes as you
get larger sample sizes though that sort
of washes out but generally we find that
with larger with enough examples you
will hit the guarantee and that's sort
of these are p-values so so it's not too
surprising
also we generally find that for these
data sets at least you get zero
ambiguity with enough training examples
early on you have quite a bit of
ambiguity but remember zero ambiguity
basically means that is predicting one
label confidently and I've got this
green curve here and a blue curve the
blue curve is just showing remember the
predictor can be wrong 5% of the time so
the green curve is the examples where
it's wrong what's the ambiguity when
it's wrong and here you see that it's a
little more ambiguous when it's wrong
but but really it gets pretty confident
when it's wrong as well so so here you
couldn't tell the difference between
when it's wrong or right and you know
just showing you one more this is more
extreme it just goes to zero ambiguity
very quickly and the other data sets are
qualitatively similar so basically these
learning curves we've never seen
learning curve analysis or empirical
evaluations for this I think it
motivates trying to study a little of
the finite sample complexity of
conformal prediction to understand why
these converters so quickly we also
looked at well varying the accuracy
constraint and how does that influence
the the ambiguity right you'd think that
if you're going to have a more stringent
accuracy constraint I want 99% accuracy
the ambiguity would probably have to
increase and we see that for one data
set here that's actually the case so the
ambiguity goes up a lot when you
increase accuracy for this one you know
the predictor just sort of nails it I
guess and we see that you have zero
ambiguity even for high accuracies yeah
similar you know you see a little
increase in ambiguity for these these
data sets so we were also interested in
what would a convolutional neural
network do these are these are very
popular now
state-of-the-art in computer vision and
we're looking at one of the smaller data
sets CFR 10 there's a you know big
network that people have trained for
this that were we're using as the
predictor so so we we didn't have the
resources to retrain this network to
produce learning curves so they're
they're only they're only ten classes in
this data set but we thought it was a
good starting point so here we're just
curious well what what would a you know
what would a conformal predictor do with
the network is it going to have some
ambiguity or not and we see that
basically we're varying the number of
calibration examples right this is what
the conformal predictor is p-value is
based on we're not retraining the
network for these learning curves the
results would look different if you were
but we want to been able to produce
those in time but basically we see that
the network is very very confident the
ambiguities low for all these cases when
you increase the accuracy constraint you
see that the network does become a
little more ambiguous so so so there is
ambiguity once you really ratchet up the
accuracy constraint so there's room
potentially to improve it's interesting
to consider how would you train a neural
network if you are training it to be a
good conformant to work well with a
conformal predictor right to minimize
ambiguity would you be able to do better
with a slightly different loss function
for example so basically there's nothing
surprising in the closed world results
we see that they the the conformal
predictor basically behaves ideally and
you generally producing zero ambiguity
so so when we open the world up to new
object so a monster might come in we
don't we don't have a current label for
monster that's that's outside of our
closed world and so you know what what
response would we expect to conformal
predictor to
for an object that's not in our current
model I mean what would you want to
conform a predictor to give any ideas
the empty set exactly yeah so smart
audience here so the only really
reasonable thing for a conformal
predictor to do is to return the empty
set now the theory of conformal
prediction does not apply at all the to
the open world setting but we were still
curious about what conformal predictors
would do because it actually is
reasonable to expect that maybe it will
output maybe these systems will output
the empty set in a lot of these cases so
what we did is we set up some open world
experiments and so for the random forest
experiments with the UCI data sets we
basically held out a label or two and
and and train the system on the
remaining labels right so so that the
held-out label is effectively a new a
new type of entity that we can then give
to the system and see how it behaves all
right and for the convolutional net work
we I'll show you the the images that we
fed it we just edit images that had
nothing to do with a C far of 10 dataset
and to see what it would do so for the
random forest what I'm showing you here
is ambiguity but I'm only showing you
the ambiguity averaged over the novel
class okay so so this is basically how
confident how confident the predictor is
and outputting a label for the novel
class that that it should not be
confident about right is predicting one
of the labels that knows about but it
should not it should be outputting the
empty set we see that basically it's
very very confident about a single label
so most of the time it's outputting a
single label from its set and it's a not
outputting the empty set it's not
seeming to be confused by outputting by
having more ambiguity
so that's what we see there this data
set was now that one thing I did point
out negative ambiguity basically means
it's outputting in the empty set if you
go back and look at the equation and
here this one data set at least random
forest knows what it knows and so for
these novel classes it's outputting the
empty set most of the time right and but
but for the other data sets basically
it's usually outputting a single label
and so so it does not know what it knows
it's confident in that single label we
would have no idea that this new object
was sort of in town alright so that was
random for us yes well so you're
building this set up and you're gonna
put something in it if you can't sort of
confidently rule out that it should be
out of the set you could think of it
like that the different labels and so
you would like it to confidently rule
out that all the labels I know about
should not be in the set but the theory
says nothing about this this is just our
investigation into what happens when you
use sort of state-of-the-art predictors
and conformal prediction yeah so is that
clear yeah all right this one here these
are just different data sets but
qualitatively know I probably just
mislabeled it or something yeah so yeah
that must have been a that was it that
was iris yeah that was the iris data set
here no so for the convolutional network
remember this was trained on I'll show
you the types of images this was trained
on these images here right
sort of natural images of these ten
classes what we did is we we found a
bunch of sprites and we some game
nethack and it was just convenient that
they happen to have the same image
dimensions and everything so we said
well let's go with those so those
clearly have little to do in most cases
with the the data set that it was
trained on this one we find here is that
the the convolutional neural network is
again usually confident about a single
label that it knows about so it's seeing
a sprite in saying it's a horse or
something and it's not giving us the
empty set it's not reflecting that it's
uncertain in any way we'd have no way of
knowing that we're seeing a sprite as
opposed to a horse so so that that
eleazar experiment with a convolutional
neural network which which we found
interesting I think this has a lot to do
you know there's probably ways to to
improve this to some degree but these
networks have a softmax function at the
final layer and they're really trying to
amplify something that a label that they
know about that's what the softmax is
roughly doing so you know in all but one
case there was practically no abstention
right though it was never outputting the
empty set except in one data set where
it was able to do that it's usually
confident about a single label but you
know the theory doesn't apply here so
you know we can't really complain the
conformal prediction is not behaving as
we would like it to but there there was
sort of reason to believe that maybe it
would have and it appears that standard
conformal prediction on its own isn't
going to be sufficient for open worlds
right and so one thing that you might do
is you could say well let maybe we can
have new ways of training predictors
that will work better we can formal
prediction in open worlds so you can
imagine getting predictors that tend to
appear more confused when a new object
comes in right that's what you would
like you know the neural network output
layer would sort of be flat when a new
object comes in and effectively what
you'd be doing is you'd be trying to
Train it to be an anomaly detector of
some sort so that's one way to go at it
and we've got some ideas of how you
might try to do that another way and
this is what this is another thing that
we're going to be exploring using
anomaly detection as sort of a first
stage of this whole process so when
something is detected to be an anomaly
or you've got an anomaly handler which
you know that's very problem dependent
otherwise you're gonna go through your
normal process and so the anomaly
detector is really going to look at sort
of the distribution of the data or some
a lot of times these predictors are just
trying to discriminate and they sort of
ignore the fact that there's a really
weird aspect to this input because it
didn't need it for discrimination so uh
yeah there there are some issues here
this is what we'll be working on how do
you select the threshold in a rational
way and can you provide any sort of
guarantees in an open world right so so
that's where we're going with this and
when we think about explain ability
basically you know there's a lot of
things to explain in this system right
we haven't talked about doing any more
sophisticated explanations of the output
right if the predictor outputs a set as
opposed to a singleton it's interesting
to consider well what would an
explanation for why it's outputting more
than one thing be what would an
explanation be in that context we
haven't studied that yet but that's
something where we're interested in but
the other component of this system is
the anomaly detector
we'd like to explain well if an anomaly
detector is saying this thing's
anomalous why is it anomalous that's
that's a question that we're going to
about now so I'm gonna go through right
so how much more time do we have I know
people are hungry yes I don't want to
keep people too long 12 tens No
okay so I'll try to go for 10 minutes
and we'll be able to you've lunch so so
anomaly detection hopefully a lot of you
have are familiar with the concept but
these red dots are anomalies and there's
lots of ways to try to define anomalies
we generally think of anomalies as
they're generated by a different process
from the normal points and that you know
that's not a not a very concrete
definition but this case study that
we're looking at an anomaly is basically
a threat this is a security application
so a true anomaly is something that a
human would look at and say this is
actually a threat so you've got behave
behavioral data of people in a system
for example then you want to know is
this person a threat or not and most
anomaly detectors instead of actually
you know working with a semantic notion
of anomaly they're looking at
statistical outliers so they're trying
to compute statistical outliers and and
we're going to consider in this talk
density based anomaly detectors which
this is your data it will fit a density
model to it we use you know ensembles of
mixture models Gaussian mixture models
and then you can look at the points we
have low density values right and you
would call those anomalies now and if
you had an analyst that's going to look
at these anomalies you would just order
them by the density values but of course
not all statistical outliers are true
anomalies and this is just the
difference between statistics and
semantics right so some of these are
things that analysts might care about
anomaly and some are just just rare rare
normal things right so if we I think to
try to understand why we might want
explanations it's worth thinking about
this particular case study where our
data corresponds to sort of sort of
security application and some data
points or threats and some are not
threats and we want to find the threats
that's the whole point of this this case
study and so uh you can apply an anomaly
detector you're hoping that it will
return all the threats but it's not it's
gonna you know here's some missed
threats when it outputs the what what it
thinks are non outliers but you have
outliers here and it contains threats
and false positives
these are we're gonna call type oneness
threats and the only way to really
improve these this this miss rate is uh
by improving the anomaly detector so
after the anomaly detector you give
whatever the anomaly detector says is
suspicious to a human analyst and the
human analysts will will mark some of
them as alarms and you know feed them to
whoever they feed alarms to so they're
gonna look at the data and some of those
threats they're gonna get and though
they might have some false positives as
well but the human analysts might also
miss threats - for example we say this
person looks suspicious there's a huge
database of information about this
person they may not have come across a
suspicious aspect and so we're
interested in reducing the missed
threats of type 2 and the way we want to
do this is by giving explanations to the
analyst of why why the system thought a
particular point was anomalous to kind
of narrow down the data that the analyst
has to look at and we we you know this
brings up the question of what is an
explanation you know so so what is going
to be an explanation and we'd like that
to be tied to
sort of thinking about the application
you know an analyst is going to use this
explanation to try to quickly identify
whether something's a threat or an odd
threat right and so the type of
explanation we're going to give is
something called a sequential feature
explanation the explanation is basically
going to be an ordering of the features
you could say it's also an ordering of
views of the database perhaps where
what's going to happen is the analyst is
going to first start with the first
feature in the ordering and look at it
and say well do I think this person is
suspicious or a threat based on that
feature and if they can't make a
determination will show them the next
feature so they'll be able to look at
the two bits of information and so we're
going to grow this information over time
until the analyst can determine that
this individual is a threat or they just
run out of time basically and then the
individual would be termed not a threat
all right so that's the protocol and
it's very you know that the type of
explanation was motivated by our
application goal and I think that's a
general theme when it comes to
transparency because there's many ways
you could make a system transparent but
why are you doing it you have to tie tie
those things together this is just
showing you the the analyst belief in so
we show it the first feature the analyst
is still pretty sure it's a normal data
point we show it the second feature you
know it's looking at two features and
then you show more and more information
and eventually the analyst the
probability that they think it's normal
goop drops low enough and then this is
where you'd have the alarm right so if
you think of the the work of the analyst
you know how do we evaluate whether an
explanation is good or bad
we'd like explanations to reduce the
workload of the analysts which means
that the analyst should only have to
look at the small smallest number of
features possible right so we'd like to
minimize the number of features the
analyst has to look at in order to make
a determination that a threat is a
threat in this case and we call this the
minimum feature pre
fix and that's what we want to minimize
so there's there's a lot to this I'm not
going to I'm not going to go into detail
but basically we have some methods that
compute sequential feature explanations
so I'm going to go to a part of the talk
that I think might be more relevant so
these sequential feature explanations
I'll just show you one of the methods so
in general this is a hard problem we
have a branch and bound algorithm with
nice pruning rules but but generally we
find that that this method works as well
as as the optimal method as well and
it's a lot faster branching down can
take a long time so so just to give you
some insight we're basically going to
pretend that the analysts is is thinking
according to our our density function
that we've learned over the data so
there notion of normality is like ours
and so what we're gonna do is we're
first going to choose the feature that
minimizes this is the density that we've
learned over the data that minimizes the
marginal density just looking at one
feature so which feature looks strangest
for this example if you just consider
that one feature and then the second
feature will be the one that minimizes
this joint marginal density all right so
if I look at two features together which
one if I look if I look at all the
features that I could combine with the
first one which one will look most
suspicious so you can keep on doing that
and you know the it's a fairly intuitive
greedy algorithm we have some other
things that were motivated by previous
work but the one I just showed you was
the best so so the question now is how
do you evaluate an explanation and I
think this is a question that we
struggled with you can look at it and
say hey that looks good so so anytime
you're going to be doing work with
explanations you have to figure out how
am I going to evaluate it does it feel
good you'd like a quantitative way when
you can get get get one
so what we're going to do is basically
the problem is we don't have a real
analyst right so we can't measure how
much work they do and so you have to
have some way of doing quantitative
evaluations and so we created a
synthetic analysts and this is based on
some work that we've done in the past
and there's actually a longer journal
version of this on creating anomaly
detection benchmarks from standard
machine learning data sets but you can
imagine you have a UCI data set you pick
out some classes that corresponding to
call normal some classes that you core
that you call anomaly classes and that
gives you an anomaly detection benchmark
and you can vary certain properties of
it now we could learn an analyst using
supervised learning that that the
analyst will learn the properties that
distinguish these anomaly points from
normal points and we can use this as a
simulated analyst so so we can sort of
simulate the probability that the
analysts think something is normal given
an example all right so now we've got
our simulated analyst we can do
experiments we can measure the the
amount of work the analyst would have to
do to identify threats so that's uh
that's all I'll say about that
and we have some if you look at
producing random explanations this is
what you get this is average this is a
bunch of data sets but what this means
here is that if you give random
explanations the simulated analysts
would need to look at roughly 3.5
features to determine that a threat is a
threat okay and this is more we produce
optimal explanations and you see there's
a gap between random and optimal and I'm
gonna jump to the last bar here this is
what I showed you this is our best
method generally and then we were
generally closer to optimal than than
random there so we won't go into the
distinctions between the
methods now but there is there is quite
a gap between the optimal and our best
method in some cases yeah one thing that
you know I'll skip through this but one
thing that was interesting a standard
data set in anomaly detection this is a
KD D cup it's a security data set we can
see that the explanations we produce are
this kind of shows you the the
simplicity of that data set and why
maybe you shouldn't use it for anomaly
detection work but our our best method
is able to produce basically
explanations that are generally one
feature long and can be confidently
classified as a threat so there's
usually one feature in any of these
examples that sort of indicates there
piece of threats so those kind of a kind
of a comment on the data set as well so
so we'll just say that you know all the
methods basically beat random and you
know we have one method that we like the
best you can look at the paper if you're
if you're interested in that and and
basically you know I think we're gonna
continue to do this do some work as we
go go ahead but I think the main points
are we had to think hard about what is
an explanation
there's many choices that we could have
made and that required us to think about
the application that you are using the
explanations for right so that was the
first point whenever you're dealing with
transparency or explanations and then we
had to deal with the issue of how do you
evaluate your explanation algorithm and
that's very tricky
you'll get papers rejected because
people want you to use domain experts to
to run your algorithms but you can't get
any sort of quantitative results doing
that so so the approach that we took was
well figure out how to get some sort of
plausible simulated analyst in this case
to sort of feed our explanations to but
you need to figure that out for other
applications as well and I think that's
that's one of the major challenges in
doing work on transparency and
explanations I think evaluation so I
will stop and it's almost exactly yes so
so this is sort of one of the directions
that you could explore how do you train
neural network or any other model to
have better properties in open worlds
and effectively you're trying to train
it to be confused when it should be
confused
and that requires there's a lot of
things that go into that right so you
have to give it some data maybe that you
want to train it on where it should be
confused that may bias it you know it
you know so so is that actually going to
be reflective of the real open world or
or not but but doing doing things like
that could definitely definitely one
direction that we're thinking about it
also seems that ensembles or one way to
to go about this as well if you have
many models you would hope that they
would be you know that they wouldn't be
consistent on things from the open world
with random forests we you know that
happened in one data set but the other
datasets that wasn't the case
but there might be ways of training
these ensembles and they have that
property so that's a very good point and
there's been some work also in so in
chibok jose laughing and india paper
i've seen of us year maybe two years ago
sitting i suppose i have access to a
bunch of unsupervised unless the data I
can figure out you know between my
classes that I apply today for what
parts of the space are populated by
other stuff and I should maybe you know
because the classic discriminative
classifiers is a partition of the world
into K that's right and so he doesn't
say is discover oh there's empty space
out here that is not empty community my
classifier thinks it's empty when it's
not ok so let me pull back my scissor
boundaries and create a new space so
that I know there's stuff out there that
I haven't seen but that one question is
what statistical assumptions are you
entitled to make about stuff you haven't
seen yet like these are all of those
things also iid samples from some
distribution which is by the way so
means making or is it maybe there's an
adversary so any other questions yes you
know that what you want is only if it
does have this constraint of that's got
to be right 95% of the time and right
just means you know containing the
yeah Petula answer it could just be the
light it's constrained it's what is once
walking into throwing things and
actually like yeah so so the conformal
predictor framework does assume a closed
world right so there's nothing about the
theory that suggests it should meet any
of the accuracy guarantees in an open
world some intuition that maybe it would
just because of the way it works it's
doing hypotheses tests is this a label
for is this a feasible label for this
example and you would hope it could
reject those but but it doesn't and I
think largely it's because these are
discriminative systems they sort of hone
in on just the things that you need to
look at to discriminate and they ignore
the fact that well maybe this object is
purple and you know that wasn't useful
for discrimination but you know that the
color would be useful for recognizing
something that's completely different
and that's that's why we feel it anomaly
detection as a preprocessor is
potentially the right way to go his
anomaly detection that it will look at
sort of all of the properties less
certainly you know there's certainly a
lot of things to think about there to
get that to work


Win $50k for Solving a Single AI Problem? #Shorts

say you've got a huge diamond you want
to protect so you put it in a cool
sci-fi vault with all sorts of sensors
and actuators you have an ai system to
run the fault and the plans it comes up
with might be too complex for you to
understand but it also predicts the
final view from the camera after the
plan happens so before you okay a plan
you can check that the diamond is still
there at the end
but imagine a plan that allows a thief
to come in and set up a screen in front
of the camera showing a diamond the
predicted outcome looks good so you okay
the plan and the diamond is stolen but
this should be avoidable right in order
to predict the right fake image the ai
has to know that the diamond's been
stolen but how do you get that
information out solving this problem in
its hardest form is extremely difficult
so difficult in fact that the alignment
research center is offering prizes of
five to fifty thousand dollars for good
ideas so if you think you've got a
solution based on the description i've
just given you don't read the full
technical report it's 105 pages of
reasons why your idea won't work but if
you've carefully gone through all of
that and still think you've got
something send it in link below the
deadline is february 15th i think i'll
have a go myself


Avoiding Positive Side Effects: Concrete Problems in AI Safety part 1.5

hi just a quick one today because I've
got a lot of other stuff going on right
now more on that later hopefully this is
a follow-up to the previous video about
avoiding negative side effects making
the dooblydoo so watch that first if you
haven't yet I wanted to talk about
another possible problem you might have
with low impact or impact regularizing
agents one I forgot to mention in the
last video some people definitely
mention some things close to this in the
comments if you've got this good job
feel free to brag about it the problem I
want to talk about is avoiding positive
side effects so before we talked about
how most side effects are negative
rather than having to figure out how to
avoid negative side effects maybe it's a
more tractable problem to just avoid all
side effects but if you look at the side
effects of for example getting me a cup
of tea that includes effects like me
being happy or me not being thirsty or
me feeling more awake because I've had
some caffeine in other words every
reason I wanted a cup of tea in the
first place if the robot can think of a
way of getting me a cup of tea that
still results in me being thirsty and
tired just as though I hadn't had a cup
of tea it will prefer that option now I
can't off the top of my head think of
any way to do that assuming we've
defined what a cup of tea as well enough
so maybe the robot will conclude that
these positive side effects are
unavoidable just like how using up a tea
bag is an unavoidable side effect but
it's not great that it's looking for
ways to negate the benefits of its work
and again the more intelligent assistant
becomes the more it's going to be able
to figure out ways to do that
or how about this we set up our system
to try to keep their outcomes close to
what it predicts would happen if the
robot just sat there and did nothing at
all right what actually would happen if
it just sat there and did nothing
instead of getting me a cup of tea one
thing is I would probably become
confused and maybe annoyed and I would
try to debug it and figure out why it
wasn't working so our robot may want to
try and find a course of action such
that it does get me a cup of tea but
still leaves me confused and annoyed and
try to debug it and figure out why it's
not working because that would have been
the outcome of the safe policy how do we
deal with that issue and all of this is
assuming that we can come up with an
impact metric or a distance metric
between world states that properly
captures our intuitions there are a
whole bunch of difficulties there but
that
look for another video so that's it for
now next time I think we'll keep talking
about the paper concrete problems in AI
safety and look at some other approaches
to avoiding negative side effects so be
sure to click on the bell if you want to
be notified when that comes out oh and
if you think this stuff is interesting
and you're at a place in your life where
you're thinking about your career the
careers advice organization $80,000 has
just put up a really good guide to
careers and AI safety I'll put a link to
that in the description as well highly
recommended checking that out especially
if you don't think your technical enough
to directly work on the research there
are a lot of different ways you might
want to get involved oh and let me know
in the comments if you'd like me to make
some videos about AI safety careers you
know if that's something you'd want to
see and to end the video quick thank you
to my patreon supporters that's all of
these excellent people right here I
especially want to thank Yonatan are
represented here by this sock his choice
not mine
anyway thank you so much for your
support I hope to have some more behind
the scenes stuff going up on patreon
this weekend if I can I hope you like it