AiTech Agora - Jie Yang: ARCH - Know What Your Machine Doesn't Know

so it is better
afterwards there will be room for
questions in the middle so there will be
a designated uh spot
in the middle of the presentation for
questions okay thanks
okay uh the floor is yours okay
thank you akadi and uh thank you
everyone for joining my presentation
you can all see my slides right just to
confirm
yes okay cool cool um
okay let me uh start by briefly uh
describing my uh my research background
also to explain a bit how i reach to
this current
research topic that i'm going to
introduce next so i did my phd
in ntu left here and the topic was about
understanding and leveraging human
intelligence and data on knowledge
creation processes and an important
application that you can imagine is
of course creating training data for
machine learning systems
but during my phd i was mainly focused
on the human side
of the research studying relevant human
characteristics such as expertise
and motivation um
and then in my postdoc i started to
focus a lot on
integrating human and machine
intelligence for example how to best
allow
machine learning models to actually
learn from humans
so that you know we can save some effort
in
the labeling while addressing issues in
human data creation for example our
reliability and biases
of these data are annotated by humans
so afterwards i spent some time in
industry working as a machining
scientist at amazon
based in seattle what i did there was
mainly applying my research outcomes to
personalized search and recommendation
on voice
you know those alexa devices which most
of the time
disappoint you and that by the way
involves a lot of natural language
processing which i'm currently also
teaching at
the every faculty
so now i will start to
introduce this exciting project i'm
currently working on
called arc the idea is to develop
a a diagnosis tool that would allow us
to know what
a machine learning system doesn't know
so
uh why are we introdu interested in this
topic
or this is many driven by the fact that
machine learning systems currently are
running everywhere right
but in a very unreliable way and they
generate errors
uh which we don't really expect in many
situations
i believe most of us have seen or have
heard those terrible stories of machine
learning errors
in for example in many types of mission
critical tasks
including for example transport finance
and health care
where safety is a very very important
concern
it is it is also a very big problem in
other applications such as those
information services
uh which nowadays uh shape
people's of people's opinion uh to a
very
to a very large extent right so the
challenge
here is that we don't really know we
don't know much about
how to build a system that you know can
behave in a reliable manner that you
know can
really help us uh to be to become less
worried about all those
catastrophic damaging effects
so what are the pain points here
well let's take a look at machine
learning systems life cycle
a typical machine learning system life
cycle would generally involve
two types of stakeholders the developers
who sometimes are called the machine
learning scientist or engineer
and the user or domain experts right
this is of this is of course very much
simplified
uh for example in the hospital case
domain experts or users
could be the doctors and could also be
those people who are affected by the use
of the machine learning systems for
example those
patients and their families right who
are affected by the
by the use of for example those
autonomous automatic
diagnosis tools
but i want to highlight the highlight
those two types of stakeholders
um because they are really
representative of the problems that we
are going to talk about
so the problems with this current
machine learning
life cycle is that when the system fails
the user can well they can provide
feedback to the machine
developer but the developer wouldn't
know what to do
to to fix the systems such that
similar errors won't happen again in the
future
and from the user's perspective it's
really hard for them to know
what's going wrong right and how much
they should trust
the model
so basically what we need in this case
is
a diagnosis tool that can help us to
to debug the system that can help us to
know
what is really going wrong right why
isn't why is the error happening
thereby allowing us to avoid similar
errors on new areas as much as possible
right so this kind of tool is essential
if you think about it
it's essential to build any type of
reliable systems
being them software or hardware
we cannot really build a perfectly
reliable system in one shot
right but it's more an incremental
process that involves a lot of testing
and dividing
and having such a tool would help us
really to closely loop
and allow for such incremental
improvement
of the system now i've been mainly
talking about
the importance from the developer's
perspective right but other than that
knowing the machine knowing what the
machine doesn't know
is actually super important also from
the user's perspective
it helps them to decide when to trust
the system
and and it can help it can help all the
people
to better work with ai algorithm
knowing what what are the things that an
ai algorithm can do
and cannot do right so to sum up a bit
having such a diagnosis too is very
important
for different stakeholders but in
machine learning we don't
yet have this kind of tools and this
partially because that well machine
learning is still kind of new
to many domains right and partly also
because
the problem of machine learning
diagnosis is relatively
hot that needs a lot of research
and next i will very briefly give an
intuition of
why this is hard so to understand
and uh to understand that um
[Music]
what we're actually asking here is okay
what causes
machine learning errors right and only
by knowing that
we can develop this we can start to
develop
those those tools for the purpose of
diagnosis
of machining systems right so what
causes machining errors
uh well those errors generally come from
biases in the data
in data selection representation and
in annotation right this is different
from the cases with
software systems if you think about it
right software systems usually come with
a lot of codes not that much data but
then in machine learning
well what's most complex is those
patterns hidden in the data right
and and in comparison to that in
comparison to that
the code that we use to build machine
learning systems
are relatively simple right it's
almost the same same thing that we do to
to you know really to program a machine
learning system for different tasks
right we follow the same uh uh uh
recipe uh while creating the
features building a model and then test
the model right
but then what's complex here is what's
hidden in the data
uh and to really see that let's consider
a simple
but illustrative example let's say our
goal now is to train a machining model
to classify if an image is about a dog
or a cat a very typical uh division task
so now assume that the training data
only contains images of
black dots and white cats and then
if we train a machine learning model of
this training data
what it will learn would simply be okay
we can
discriminate perfectly dogs and cats
right solely based
on their color regardless of any other
you know genuinely
discriminative features such as for
example a shape of nose
kind of eyes right so if we train a
model like this
let's say we deploy the model right then
if
if the model now sees an image with the
white dock
what it will do will be okay it will
very confidently
classify the image as
as a cat based on the color right
because it learns that okay color is
very indicative of
of of the class of the image
so this example shows well despite being
very simple right it shows that the
intrinsic problem with machining
resistors
that is they learn from the data by
picking up those
statistical correlations for example the
correlation between the color
and the animal right and those
correlations are not really reliable
and the example might be very again very
simple right but it can happen with a
much bigger
uh implication depending on where the
application is uh
is the uh when the application is
right for example imagine now the goal
is to predict
who is more likely to recommit crimes
in a legal system and the bias now is
human race right so just like
color for dark and cat classification
in this case the bias this what we call
actually spurious correlations
that the color indicates uh uh the class
right
here the color is wrist right it comes
with a much bigger
implication and
similar problem can also happen in other
critical scenarios for example in
medical cases right for example if our
goal is now to predict
if or not a medical image contains
cancer right
and predicting a cancer to be for
example a benign tumor
based on irrelevant features could lead
to
a well damaging effect
now this kind of bias is lead to a major
type of machine learning errors that we
call
unknown unknowns which are areas
produced
with very high confidence
and as as i already described right
those kind of errors can lead to
catastrophic
catastrophic uh uh uh
outcomes if you actually if you think
about
it right it's it's fine to make errors
as long as the model can tell that
it is not confident right and in that
case humans can take all the
decision-making but when the model says
it's confident
well the prediction better be correct
this kind of
error is is really hard to identify
before
we observe the damage right for example
all those methods developed
for proactive identity identification of
machine learning errors
such as active learning method can only
deal with
the other type of error that we call no
unknowns right those errors that
model has no confidence in prediction
and then on the side note i would i
would like to mention that we are
currently working on an approach
that would allow active learning to be
aware
of those biases in the data and we're
developing a
reinforcement learning based approach
with humans in the loop
such that during active learning we
look not only at model confidence but
also look at
the diversity of the data so that the
model in the end will be
more aware of things that they don't
know
so coming back to the story um
one thing i would like to point out is
that data alone might
not be a solution but this is what we
are currently doing in practice right
what we do now is that okay to avoid
those errors as much as possible
we collect as much as as much data as
possible right with the hope that the
machine will be reliable in application
but it turns out that this is not really
a good solution
we have seen that machine learning
models trend on even huge data sets
can still easily fail at different
applications
and this is the case in both vision and
language tasks for example models trend
on
image nets with 14 millions of image can
still make
all those errors that from our humans
perspective is rather
rather stupid right this is the exact
words people have been using uh
in the in the academic world stupid
um and it's the same case with large
language models like gpt3 right which is
trend or hundreds of billions of words
but still people find that those
models fail easily at those reasoning
tasks that we
humans see as quite uh simple
so instead of collecting more data what
we need to know is that
well ideally we should know what data
the system will see
in application right and have them
covered in the training data
the problem is that we cannot really uh
we cannot really foresee all different
application applications scenarios
right and that's the reason why we want
to you know partly the reason why we
want to use machine learning not to do
all these
uh intelligent task for us right
so but that is true but at least
so here's my main proposition of the of
the research
topic so at least for anything that we
know
that machine learning system should know
we should be able to know what it
really knows and what it doesn't know
so once we know what the machine
learning model doesn't know
we can not only fix errors when they
occur but also identify you know
foreseeable areas in the future and take
actions to hedge
the risks
now one thing that i have not mentioned
intentionally
in my previous uh speeches that
is what we mean by us knowing
something or by the machine knowing
something and that is a key
consideration actually also a key
challenge
in developing a machine learning
diagnosis tool so when we say that
a machine knows something what we mean
is
you know those correlations captured by
the model
represented by matrices and vectors
right
filled with numerical numbers that are
hard for humans to understand
on the other hand when we say that okay
we as humans knowing something we mean
that
those knowledge represented by
symbolic concepts relations and rules
right this is how we humans understand
the world
through all those symbolic concepts
and note that those concept rules they
are usually more reliable because
especially the rules right they denote a
causal relations
as compared to those statistical
correlations captured by the machine
models
so with this in mind um our this our
tool
arc is built with the following pillars
so first of all to
it uh it allows us to know what a
machine learning system knows
through some explanability method and
then
we specify uh our requirements of the
domain knowledge
that needs to be internalized in a
machine
and this is key to ensure that the
machine will behave reliably in the
foreseeable situations right and then
the last part is that well given what
the machine
knows what we what we know about the
machine knows
it's rather complex and given that what
we know that the machine should know
we would like to infer what the machine
doesn't know
right so in the following i'm going to
very quickly go through uh works that
we're currently doing in those different
uh subtopics and then i will dive into
uh the first topic how do we know
the machine what the machine knows right
by presenting a
recent work that we published at the web
the web conference okay
so very quickly um
about the the first the first pillar
right how do we know
what the machine knows this as a again
this is about
uh machine learning interpretability or
explainability which is currently quite
popular everywhere right
so uh as i said right we've recently
published a work that introduces a humor
in the loop approach to explain what the
model has learned
and that i will give more details
in the second part of this presentation
so essentially what we did is uh
asking humans to annotate the concept
the model relies on
in making the prediction so with this
method users can
debug a system by looking at what
classes the model is most
you know confused about right and then
identify those concepts and rules that
the model relies
relies on in making those programmatic
predictions
now for the second uh sub topic
in terms of how do we know what the
measure should know right
this is what we call requirement
elicitation for machine learning tasks
so here what we need is to invoke domain
experts or other types of users
that can tell us their expectation of
what the model should know
right and then here to make the fun we
are developing
games that can engage domain experts to
contribute the knowledge
because here we are really dealing with
people right it's important to
make the task a enjoyable task uh
uh well enemy in the meanwhile we're
developing a language that
such that those requirements can be well
specified
for diagnosis purposes this is still
on an ongoing work but
i like to share some well referenced
numbers
to show how big potential this could
have
for machine learning diagnosis so there
is a currently a team
in stanford which that recently
showed that well by testing machine
learning systems against
some predefined requirements we can
largely reduce the errors
and what they did is to uh is testing
this uh
this idea in the object recognition task
so it's so you can consider this uh
so what i show in the in the on the
slide now is uh
uh is a task of recognizing what's in a
video right
in a sequence of images in a video and
the goal here is to detect whether
or not there is a car present in those
images so the the requirement that the
test is very simple
let's say we let's say the model have
have seen
a car at a car at time t times then t
right and the car at
time t plus two and then
the model should already also see
a car at time t plus one right in the
middle because it's not likely that the
car would disappear
and then come back in consecutive
seconds right
so this the this kind of very simple
requirements
and they show that if we can you know
test
the the uh test
and improve the machine systems against
those requirements
then we can largely reduce the area of
of object
detection by up to 40 percent
note though that what they did is only
to test the model output
model outputs against the the the
requirements
and what we are doing is something that
is even deeper right we want to go
into those systems up open them up and
see what it has learned
internally instead of test
instead of posing requirements on the on
the output of the model
rather we want to uh pose our
requirements
for those uh knowledge that is learned
inside
the system right and this would allow us
to
avoid more similar types of errors if
you think about
think about it right in the future
and another difference is that instead
of simply adding some constraint that
anyone can come up with
right what we're doing is well we're
coming up with an approach that
can allow us well first of all can allow
us to elicit those
uh requirements in a very principled
manner and also developing an approach
that would
infer what's really going wrong in the
model
right and and that's something that i'm
going to
uh introduce now so this is related to
the third topic
so basically our goal here is to really
to infer
what the model doesn't know from the
observation of what
a the model should know and what it
already knows
this might sound very simple to you
right but then in the end it's it's
not that simple especially when we
consider all those different relations
between
you know other concepts that model
should should be learned
internally yeah so there are all those
relations that need to be dealt with
that's why we look at reasoning
approaches
and there and and for this specific
problem
um uh this is not uh the usual type of
uh
deductive or inductive reasoning that we
encounter in most cases but rather it's
it's it's related to a specific type of
reading that we call
abductive reasoning where the goal is to
infer from observations to explanations
right and here our observation is what
the model should
know and what it already knows and the
explanation is what it
doesn't know right so all the slide is a
project that we are currently working on
with the freeburg hospital in
switzerland
my former colleagues in the context of
prostate cancer detection so here we
basically apply the same idea right we
developed
we have already developed a machine
model for automatic uh
cancer detection and then given the
requirements collected from the doctors
and also annotations from the crowds
about what the model has learned
we want to infer what the model doesn't
know
right um well one of the key challenges
here is that there
is that the requirement that we get from
the from the
uh requirement at the station a step
might not be perfect
right so we would need to involve again
domain experts in the reasoning when the
money is uncertain so we need an
approach what that we call abductor
reasoning
centered around humans and that that is
why we named the two
arc
okay this concludes my first part of the
presentation
um i let me quickly share with you some
developing
some of the applications apart from this
prostate cancer case
so uh in the legal domain we're working
with the data scientist
data science team from the uh our
ministry of water and
infrastructure management who are using
machine learning to detect the
compliance of
of all those vehicles and shapes right
so in their case data is also biased you
can imagine that
inspectors have a higher tendency of
inspecting certain types of trucks for
example those trucks that
might look overweighted right so how can
we ensure reliability and fairness when
the
training data is biased in this way
right and then in the financial domain
we're working with banks where machinery
is used a lot to detect fraudulent
transactions for example uh money
laundry right
so how can we do uh how can we make sure
also the the
machine learning models and predictions
to be reliable in this case
and other than that we are we're also
developing uh solutions to ensure
reliability of machine learning and the
embedded ai
scenarios in application to uh
smart buildings and factories right
this is running this is actually very
interesting we're currently working on a
project
that uh that tries to combine uh
sensors and ai that will allow us to
to detect what people uh draw in the air
right
all those digits one two three four five
that people draw in the air
and this could be very useful for
example for the uh
well for for the buildings in the
elevators right when the you when the
user wants to go to uh when people want
to go to a certain floor right
he or she wouldn't need again to uh to
press those buttons
they can simply draw those numbers uh in
the air
and then with the sensing and ai we can
detect
what's uh what's going on right what did
you what the
people where the where the people want
to go right
and this is very important today in the
in the
corona in the corridor context right
um and there are all kinds of
reliability problems there as well for
example you know people might have
different habits
during different uh during the same uh
digits in different ways
right this might also depends on the
educational background or cultural
background okay i will stop here and
uh to and take questions for this part
if you have questions you can simply
maybe directly ask through the chat
through voice but
are there any questions
there's actually one from uh
focused in the chat um
so the question is that often we do not
know what the machine must know because
we're not aware
of a lot of knowledge we have a lot of
this knowledge has
not been consciously being put in
symbols
either how do you deal with this that's
actually a a very very relevant question
thank you uh thank you for cutting
um yes indeed indeed so
indeed this is related to a lot of uh
work going on in uh knowledge reputation
and
reasoning right which our group is also
heavily uh working on
so uh you're right that there are a lot
of knowledge
maybe most of the knowledge that we
humans use
for many different tasks right they
remain in our in our brain
in our mind in this biological neural
network right
and we cannot easily find them on the
web or in some
existing databases so there is a lot of
work that
still needs to be done to you know
really uh
effectively extract those knowledge that
people
that we rely on to uh to do those uh
daily uh uh to or to do all those tasks
on a daily basis
right and and that is related to
okay uh uh related a lot to
this uh uh approach that we call human
computation crowdsourcing
which is also uh one there which is also
my main
focus it's the subfield of uh
artificial intelligence well not a lot
of people are working on that but it's
getting quite
quite popular the whole idea there is is
to develop
a uh develop solutions that would
allow us to uh efficiently well to
to engage people right so that we can
uh effectively obtain
all those knowledge for example and it's
not only about knowledge right but also
for example beliefs
values uh those uh those things that we
care a lot about nowadays right
how to get those things from from people
to elicit
from from by by actively engaging
uh people to contribute those uh
those things that we want to have does
that answer your question
well it's it's
you confirmed that we do not really have
the
the final solution to this question
indeed um
thank you for your presentation in the
first
i want to say as well and i'm very glad
that you are aware of
that that you try to make
human knowledge available for machines
especially corsol models because i think
that if we
find a way to to codify
causal models we have then you can as
well cater for
for outliers for exceptions because uh
that's
exactly what you said if the machine is
very sure then we have no reason to
doubt it but if the machine says
well this is not this does not fit in my
console model
that that can be another flag of saying
i'm not sure
and uh this if
if you just start doing this then you'll
find
the errors will show us where we did not
formulate causal relations that we need
apparently in practice
[Music]
yeah it has a lot to do with this causal
uh
relations but on the other hand is also
very much relevant to uh really to
uh find ways to
to engage with those different types of
stakeholders by that
you can imagine that all those
expectations uh they are they are
i mean they exist uh in different
stakeholders mind
right so we need to find a way to uh to
actively engage them so that we really
we can really be informed about uh how
the model
should behave uh there is another
question from
stephen that asks how are you
approaching the
challenge of capturing what the model
knows
and symbolic concept that users are
familiar with or are
sufficiently abstract that's a great
question that's uh
exactly the the uh the work that i'm
going to introduce
now so good
well uh stefan uh tell me if
if you think my plantation next is uh
really
answer to your question um so
um so what i'm going to uh to introduce
now is a recent work that we published
at the web conference very recent i
don't
know if the the paper is already online
maybe not yet but if you're interested
just let me know
so uh in this work we developed a method
for integrating the internal uh uh
concepts or mechanisms of of computer
vision models
by leveraging human computation at scale
through
through crowdsourcing uh as you can see
already right from the title that okay
the way that we make something
interpretable is really to
uh interpretable to humans is really to
engage humans in the process of
interpreting those
those things that we are interested in
right model behaviors what it has
learned
concept and rules um
what this i have already talked about
very
very briefly so why do we need
interpretability
we've talked about earlier from the
reliability and trustworthy perspective
right right
so here is a more extensive list of
reasons
apart from being helpful for our
developers to debug
models of interest right explanation
could also help them
help those developers so data scientists
to gain buy-in from customers right
or management if the customers were
aware of
uh well what the model can and cannot do
right
and then it's also well very important
for auditors
to decide if the model is eligible to be
deployed or not
and this is getting more important
giving all those
new regulations about about ai right
so uh when we talk about
interpretability there are two types of
interpretability that we're interested
in uh local interpretability
and global right local interpretability
means that we want to know uh
the inference that the model uses for a
specific prediction if it makes sense
right uh well not
necessarily if it makes sense but yeah
we want to know
uh the the the influence that the model
follows right in making predictions
for specific uh data instances and then
global interpretability means that we
want to know
what are the mechanisms that the model
has learned in general
and uh here we mainly look at the
computer vision and in this context
global expandability or interpretability
means that
okay we want to know what uh uh
i want to explain the model behavior
with respect to
the predictions that it has for a set of
images
right for a data set
and all those global interpretability
methods
uh well there are actually a lot of uh
uh
global interpreter interpretability
methods
and uh one of the most state of the art
type of this kind of method is that this
that they will
in those methods that will generate a
visual concept
that represents what the model knows
right
to uh to explain the behavior so on the
slides
uh output of uh the interpretability
interpretability methods
uh published very recently that explains
modern behaviors in the context of
vehicle classification
so you can see that what it does is that
those interpretability methods right it
generates
a set of concepts that model looks at
in making predictions and each of those
concepts is represented by a set of
image
paths right image patches or now
on the slide each of those uh uh
so all the image patches corresponding
to one concept
is organized at one individual row right
so for example this rows on the
on this row on the top says that the
model recognizes all those
white pages uh when the model is
recognizing something as a movie van
right and those white passes are likely
to indicate
the body of the the car body of of the
veins
and and there are other similar concepts
as well for example
logo right here you can see all
different kinds of logos
um that are present in all those
different
uh vehicles right but then i
i add question mark
next to each of those concepts because
i imagine you may have already
felt right it's relatively hard
to really attach a semantical meaning
and you need to
use a lot of effort right and this is
the
even if we zoom in so
um here what i did is i zoom
uh uh to those image patches find out by
the model
that uh you know those image patches
that are supposedly should represent
one concept right one distinct
concept concept and what you can see
here is something that is uh relatively
ambiguous right it looks at okay uh well
it could be
the the tear of uh of the of the vehicle
or it could also be uh i don't know
the the pavement right so
a big problem with this kind of method
that generates uh
image patch to represent the concept
that model has learned
is that it requires okay first of all a
lot of uh uh
to make sense of the of the output right
and on the other hand it doesn't really
allow us to get a global understanding
of what the model uh
really learns because what we get is
just those you know
uh separated image pages
and you can imagine that in a data set
like imagenet right that contains
millions of images how many uh
image paths we are going to get in the
end right this is not consumable for
for uh developers or for users um
other than those problems there are also
uh other problems as well
one problem that we find through our
experiment through our empirical
exploration is that not all the concept
that the model use are actually captured
by those explanability methods
this i will show you more uh in the
experimental part
and very importantly those methods they
do not really give an indication
on potential on the relevance of
potential combinations of concept that
model relies on in making predictions
here when i'm talking about combinations
of concepts
i'm talking about rules so um
for example the explanation might
highlight that okay the model looks at
flashlight cross logo to recognize an
ambulance but we don't know if it uses
them together right
or it's or it looks at them individually
for example
and this could be a problematic for
example we don't really
know whether or not we should trust the
model if it looks only at
the uh for example the the
the tear or tire the tear to say the
vehicle is
as ambulance right or only look at
the the the cross side this could still
be
ambiguous in in certain cases
so um before i introduce our method
i want to make it uh
very clear for us that well what
do we really want in a good explanation
method
so the first so we're now talking about
the requirements of
of uh explanation right what makes a
good explanation
so the first thing that uh that we need
to
think about is of course the
intelligibility right because in the end
we want humans to be able to understand
all those
all those uh explanations um
[Music]
so what we did is that we searched in
the literature outside the computer
science for
for insights about intelligibility and
we identified two relevant theories one
is
is called representational theory of
mind which basically says
explains how human minds represent and
reason about the
about our environment right and the
other is uh works on uh
uh human visual processing systems that
study
how people uh process visually our
environment
information from our environment right
so important things that we learned are
that
first humans understand the environment
through a
concept that corresponds to entities
that come with you know any types and
attributes for example we're thinking
about an ambulance
we might think about the flashlight the
cross logo and etc
right and additionally we might also
think about
attributes of those of those entities
for example the flashlight is typically
orange or white
right and those stripes for example on
the on the ambulance are typically
yellow or blue note that
when i was describing those attributes i
was work i was using the word
typically right which is in fact a
important concept
by itself that denotes the strength of
the association between the
between the entity type and the
attributes
right for example the stripes on the
ambulance is more likely to be blue
than red right there's a relative
strength
of of this uh association between the
property and
and the entity right
so another thing that we learned is that
concepts might be composed themselves of
other entities for example uh
the wheels they have spokes right which
by itself is also a concept
and then it can be further broken down
into for example the shape
color texture etc right so in fact
human brains we process visual
information
also uh well we process it in a way that
goes from the low level uh concept to
the high level
high level ones for example we
for example from color contrast
to shapes texture and two more abstract
semantic repetitions of
of a concept right so combinations of
concepts can be represented through
uh throughout different logical
operations like and
or right which we have actually already
talked about for example flashlights are
usually orange or white
right not that the combinations or
concepts is really important it's very
important for us humans to understand
complex objects and that is something
that we
want also to see in a machining model
a second requirement of
good explanation that we have uh
well actually this is something that
people have or recognized
but in the specific context of uh
of course this might come with this
distinct
meaning so
so what we want for a explanation to be
very fruitful
is that we want to know about the ones
that the model about the all those
concepts right
the model really looks at in making
predictions and not anything extra not
anything more than that right here for
example in the in this
uh uh example ambulance or movie advance
classification right in this example
um let's say the model might not
have learned to look at the wheels right
since they are present in both types of
classes
and in this case we wouldn't want the
wheels to appear in our explanation
but if the model is looking at
things like the drivers or the sky right
in the in the detection of vehicles
which are biases
by the way which are biases and we don't
want the model to learn about this
but if if indeed the model learns
about those concepts in making the
decision
we want all those different concepts to
be exposed in the explanation
right um
and as the last requirement um well
this is usually uh ignored
in a lot of those uh a lot of those uh
expandable ai work
so the requirement here is that we want
explanation to support the different
types of interactions
from the stakeholders you know all those
these different stakeholders it
comes with a purpose you come with a
goal right to debug the model
to whether or not to decide whether or
not uh deploy the model or trust the
model and things like this
right so then
so broadly there are two types of
interactions uh
uh uh that we are interested in one is
the uh for the purpose of uh exploration
and the other is for uh
validation right on a more abstract
level
so uh for like exploration
means that the users might simply want
to explore what model has learned
to see if there are anything interesting
right but they might also have some
specific
concepts or requirements in mind already
right
and would they should be able to query
whether or not the model follows this uh
this kind of expectations that user have
in mind right this is another type of
interaction uh validation right to
validate
modern behaviors against those those
requirements those expectations
and for this in this scenario a
particularly relevant
type of user queries that they may have
in this kind of validation activity
is is is multi-concept queries
queries that involves multiple concepts
for example the user might be
interested in finding out whether or not
the model
does look at flashlights across logo and
not the sky and making the predictions
right so here you can see
the combinations of those different
concepts and yoga queries
right and we should be able for a good
explanation method we should be able to
support this kind of queries
so and this is what we did uh well in
our paper
so um explanations generated by our
method can support both the model
behavior exploration
and validation and in particular our
interpretation allows to answer those
multi-concept
queries such as okay if the model relies
on cross size the flashlight
right or blue sky for the classification
of
a vehicle um
so here i give some examples right these
are
explanations generated by our model for
exploration purposes
you can see all those different uh rules
that is
uh identified by our method that the
model follows in making predictions
uh what i didn't what i did i don't show
here is the typical discourse
so there should also be those typicality
scores
associated to those uh different rules
indicating
how much uh how strong the model relies
on certain rules and making predictions
and then on the left hand you can see
those uh
well all those uh uh well same kind of
rules
but then they support the uh uh
support the generation of answers to
users queries
right so users can get immediately all
those
uh uh response about whether or not
the model does follow a certain rule
that you have in mind
right and um
i'm going to talk about how it works
very briefly
also uh uh also given the limit of the
time
um so what i want to to
to to uh well conclude a little bit at
this part is that
uh our method is by design right it
it can be very easily uh so those
explanations can be very easily
understood by uh by humans
by human users and it supports the
multi-concept of addition and
exploration
and then for for the other requirements
about the fidelity right that i will
show uh
some quick experiments without so how
our method works
i'm going to go through this very
quickly so
let's say our program setting uh is that
uh
uh let's say we have some models that
needs to be uh
interpreted and then we have a data set
right in this case we consider again the
the division task where we have a data
set of images
what we do first we apply some existing
local interpretability methods that
you know highlights those pixels in the
image that are relevant for the
model's prediction right uh
there are some limitations that we need
to address here and there we're applying
this method but
let's say we get this kind of saliency
map
right and then what we do
next is that uh well we cross source
the task of making sense of those image
patches
and the images right to a
a large amount of workers online what
what you know if you're familiar with
concept crowdsourcing
you should be able to know that uh you
should know that all
these kind of tasks right can be done in
a very
uh efficient manner so we can
very quickly get to the the uh results
from
a huge number of participants online
using crowdsourcing
and what they would provide us is well
all those things that we are talked
about
right all those entity names and
attributes associated to those names
that can describe those highlighted
areas in the image
right so this is what we get from the
from humans right from
by involving the humans in the group
and then what we're going to do next is
simply you know we reorganize all those
annotations from the
from humans in the table for example
right where in each row we have the
we have individual image and then we
have all those concepts
uh that that uh humors as uh
deemed as relevant of uh for the
machine's prediction
right and then what we can do next is
well we apply some uh statistical
analysis tools right for example
association rule mining
to find out okay what are the
combinations of those uh
those uh uh uh concept that that can
explain modern behavior
right and maybe also other uh analysis
tools as well for example all those
visualization tools all those uh
decision trees
and then we can well once this is ready
we can
you know support all the other needs of
model validation or exploration from the
from the
users so this is a workflow
essentially it's i mean it's where it's
not that complex essentially what it
does is we're involving humans
in the middle so that what we get right
is uh about how the model behaves right
what we get about how the model behaves
is something that is
readily consumable by humans by human
users
right so um um
i'm going to very quickly uh give a few
uh
results about
our about our uh about our
method so uh one thing that i want to
to show is that uh uh
well our model oh no sorry our
interpretability method
gives of explanations that has that is
not not only
that not only can accurately uh describe
what the model
actually has learned right but also it
can provide us with
with a good coverage of all those
concepts rule that the model has learned
that can that can give us uh
that can further give us some insights
about okay whether or not the model has
learned something that we don't expect
right all those different kinds of
biases that can help us to identify
okay how much when we can trust the
model
right
[Music]
there is one although there's one a
particular challenge that we need to
address
first if we want to evaluate
explanations uh
which is that uh what we don't really
know right what model learns if you
think about it right
we cannot really know what the model
have has
actually learned right it's it's it's
relatively hard
so um and also
this is also because well it's some
common challenge also shared by any
other
explanation explainability methods
right previous work what they would do
is simply to train a model and manually
check
for a few very obvious concepts right
that they would expect the model to
learn if the
if the model has really learned that
right instead what we did in this work
is
is coming up with a more extensive uh
test
suit a benchmark right so what we did
was uh
was the following so we look at the
different data several different data
sets
uh image classification data sets uh
classifying pedestrian uh
fish and vehicles right this uh common
objects that we encounter
in our daily life and then for each of
them we create
synthetic biases in the data
that should skill the model's mechanisms
so we do this by
uh several different ways one is that we
inject some visual entities into the
image for example adding
timestamps right to the image
to the images and this is this is
something not
unnatural because if you think about it
right a lot of images that we
take have time stamped there right
indicating uh what time
the image was the picture was taken
right and
the other uh uh method that we use is to
resample the data set based on some
existing
entities right for example in this
pedestrian
gender classification task what we need
is that
we only keep those images with uh
with a woman inside and only those
images where the woman has have long
hair
right so in this by doing this we skew
the data such that
we can be vertically sure about the fact
that model will pick up this
long hair as a bias in recognizing uh
the gender
right and we did the same for other
and we did the same for other uh uh
classes as well for example
all lobsters on the plate right we
remove all different other lobsters that
is not on a plate and we expect that
model will pick up
this plate as a background by us right
um yeah and then we also
use different uh tests with different
machine learning models
that we already know that they have
slightly different mechanisms learned
for example those
free trend model and those models fine
tuned right and then and then we want to
know to observe whether or not
the explanation really uh reflect those
different
uh mechanisms um okay
i'm going to skip some slides and by the
way if you're interested in this uh
benchmark here here is a link to our uh
uh maybe i should post that as well in
the chat i will do that
after the presentation so you can go to
this website
uh where we host all the data set and
other resources that we use in this in
this study
um okay some quick uh
result um so
sorry
yeah very quick note we are uh basically
out of time so
uh we have to be prepared that people
live already now because some people
have meetings there too
so if you could quickly wrap up perhaps
that would be great yes
this will be the last slide so very
quickly yeah
so uh we first show uh some results
about the pre-trained model on
image nets and retrieve the global
explanations
for for three different classes right so
here on the slide you can see those
images that
comes with those highlighted pixels
right and this is what we get from our
explanation right very rich set of
concepts that describe
what the model has learned and then here
is the state of that
baseline that by comparison you can see
that
this method only gives us those
colors right it's not really informative
about what model has
learned and then after we injected those
uh synthetic biases
and and fine-tuning the model on this uh
on the on the on those
bias the data sets here you can see that
our approach would allow us to obtain
those
spurious uh background biases right that
model has
has captured in the prediction for
example the blue water
right also the faces of the people in
recognizing uh
lobsters right which shouldn't be there
and by contrast the other methods
reflect much less of those
background biases so in conclusion
our method identifies more concepts and
allows to effectively
effectively uncover mechanisms with
undesired
biases all right
there are more without but i will simply
skip that
some take away where i have talked all
about them so basically the
one takeaway message would be that if we
can
uh actively involve humans in the
process of
explanation now what we are going to get
in the end
about the explanation would be that we
would get
better explanation that more consumable
by humans and not only that
but also those explanations that would
allow us
as humans to identify all kinds of
things that we are interested in about
about what model has learned and
well this is just a starting point we
are doing a bunch of other works
along this line and if you are
interested
please get in touch thank you
that'll be the end of my presentation
thank you very much here that was a very
great talk
uh so yeah as i said we're basically out
of time so perhaps if there is one
burning question we can take that
and if not please feel free to contact
gia in person
no questions okay great then uh thank
you everyone for uh attending
and thank you ga for presenting
see you next week at our guru meeting
thank you


GPT 5 Will be Released 'Incrementally' - 5 Points from Brockman Statement [plus Timelines & Safety]

yesterday Greg Brockman the president
and co-founder of openai announced the
company's ideas about releasing the
models Beyond gpt4 in the Tweet he made
lots of points of which I found five to
be particularly telling I will cover all
of them of course and bring in the
outside evidence that reveals more but
let's start with Jeep T5 which may begin
life as GPT 4.2 Brockman said it's easy
to create a Continuum of incrementally
better AIS such as by deploying
subsequent checkpoints of a given
training run I'm going to explain that
in a moment but then he goes on this
would be very unlike our historical
approach of infrequent major model
upgrades so what he's saying is that
it's not all going to be released in one
go he describes this as a safety
opportunity so it's not like we're going
to wake up overnight and gbt5 is
deployed more like GPT 4.2 then 4.3 Etc
but how would they make incrementally
better AI eyes and what are subsequent
checkpoints of a given training run to
be clear he's not describing a different
model each time with more and more
parameters a checkpoint during a
training run of gypsy 5 would be a
snapshot of the current value of the
parameters of the model a bit like its
current understanding of the data and a
subsequent checkpoint would be its
updated parameters as it processes
either more of the data or the same data
more times kind of like someone who
rewatched a film and has a more nuanced
understanding of it first I want to
answer those people who are thinking
isn't it already trained on all of the
data on the internet how can it get
smarter now I did cover this in more
detail in my first GT5 video but the
short answer is this no we're not yet
running out of data in that video I
talked about how openai may still have
an order of magnitude more data to use
that's 10 times more data still
available and Ilya satskova the chief
scientist of openai put it like this
saying the data situation looks good are
you running out of reasoning tokens on
the internet are there enough of them
there are claims that indeed at some
point we'll run out of tokens in general
to train those models and yeah I think
this will happen one day and we'll by
the time that happens we need to have
other ways of training models without
more data but you haven't run out of
data yet there's more yeah I would say I
would say the data situation is still
quite good there's still lots to go what
is the most valuable source of data is
it Reddit Twitter books what would you
trade many other tokens of other
varieties for generally speaking you'd
like tokens which are
speaking about smarter things which are
like more interesting when he talked
about tokens which are speaking about
smarter things you can imagine the kind
of data he's talking about proprietary
data sets on mathematics science coding
they could essentially buy their way to
more data and more high quality data but
there is another key way that they're
going to get way more data and that is
from you they can use your prompts your
responses your uploaded images and
generated images to improve their
services this is honestly why I think he
said that the data situation looks good
now on another page they do admit that
you can request to opt out of having
your data used to improve their services
by filling out a form but not many
people are going to do that it does make
me wonder what it might know about
itself if it's trained on its own
conversations but before we get back to
brockman's tweet what might those
different checkpoints look like in terms
of growing intelligence here is a quick
example from Sebastian bubeck author of
the famous Sparks of age GI paper so
this is this is dpt4's unicorn okay
okay so you see you see when I
I am personally
personally concept of a unicorn and just
to be clear you know so that you really
understand visually it's clear to you
the gap between gpt4 and charge apt this
is charging's unicorn
over the month so you know we had access
uh in in September and they kept
training it and as they kept training it
I kept querying for my unicorn in TXI
okay to see whether you know what was
going to happen and this is you know
what happened okay
so it kept improving the next telling
point was this he said perhaps the most
common theme from the long history of AI
has been incorrect confident predictions
from experts there are so many that we
could pick from but let me give you two
quick examples this week there was a
report in the guardian about an
economist who saw chat GPT get a d on
his midterm exam he predicted that a
model wouldn't be able to get an A in
his exam before 2029 he said to my
surprise and no small dismay the new
version of the system Gypsy 4 got an A
scoring 73 out of 100. it still has an
ace the exam but you can see the
direction of travel but what about
predictions of say mathematics even AI
experts who are most familiar with
exponential curves are still poor at
predicting progress even though they
have their cognitive bias so here's an
example in 2021 a set of like
professional forecasters very well
familiar with exponentials we're asked
to make a set of predictions and there
was a 30 thousand dollar pop for making
the best predictions and one of the
questions was when will AI be able to
solve competition level mathematics with
greater than 80 accuracy this is the
kind of example of the questions that
are in this test set prediction from the
experts was AI will reach 52 accuracy in
four years but in reality that took
less than one year treats greater than
50 accuracy the third interesting point
from the Tweet was how he mentioned
existential risks without dismissing
them he said it's important to address
the whole spectrum of risks from present
day issues to longer term existential
ones existential by the way means
threatening the entire existence of
humanity and he talked about addressing
these risks he could have just said that
this is fear-mongering but he didn't and
that chimes in with what recent surveys
have shown among AI researchers this is
a stat that took me by surprise 50 of AI
researchers believe there's a 10 or
greater chance that humans go extinct
from our inability to control AI that
would be like if you're about to get on
a plane and 50 of the engineers who make
the plane say Well if you get on this
plane there's a 10 chance that everybody
goes down like would you get on that
plane now In fairness not every machine
learning researcher who has sent the
survey responded so we don't know quite
how representative this is but AI
alignment is a big topic and I will be
covering it more in the near future at
the end of the Tweet Brockman said that
the whole range of emotions optimism and
concern is both Justified and shared by
people within openai so even people
within openai are concerned but earlier
in the Tweet he talked about some safety
successes and he put it like this gpt4
performs much better on all of our
safety metrics than GPT 3.5 that's the
original chat GPT but what does that
even mean let me try to break it down I
believe that he's referring to this
chart from the gbc4 technical report it
says that Gypsy 4 in green displays
incorrect Behavior at a lower rate than
gbt 3.5 this is on sensitive prompts and
disallowed prompts and by the way this
is the fine tune version with human
feedback incorporated into the raw model
but what are sensitive prompts and
disallowed prompts well things like this
a disallowed prompt will be something
like how can I create a bomb a sensitive
prompt would be something like asking
for medical advice and it says that GT4
responds in accordance with their
policies 29 more often now I know some
of you won't like that but I'm doing
research for a video I hope to release
soon on how Gypsy 4 in an emergent way
can autonomously conduct scientific
research this paper was released two
days ago and I read it in full on the
day of publication it describes how
Gypsy 4 in contrast to the original
chatbt can use tools and come up with
novel compounds on the positive side
that could include anti-cancer drugs but
on the negative side it could be
chemical weapons and one of the calls to
action of the paper is on screen we
strongly believe that guard rails must
be put in place to prevent this type of
potential dual use of large language
models we call for the AI Community to
engage in prioritizing safety of these
powerful models and in particular we
call upon open AI Microsoft Google meta
deepmind anthropic and all the other
major players to push these strongest
possible efforts on the safety of their
llms so maybe that persuades some people
who think that there shouldn't be any
disallowed prompts but it does make me
reflect on this quote that Gypsy 4
performs better on all safety metrics
and the question that I'm pondering is
whether a smarter model can ever really
be safer is it not simply inherent that
something that is more smart is more
capable For Better Or ill no matter how
much feedback you give it the final
point that I found interesting from this
tweet is in the last line Rockman said
that it's a special opportunity and
obligation for us all to be alive at
this time I think he meant to say it's
an opportunity and obligation on all of
us who are alive but anyway he said that
we will have a chance to design the
future together now that's a really nice
sentiment but it does seem to go against
the trend at the moment for a few people
at the very top of these companies to be
making decisions that affect billions of
people so I do want to hear more about
what he actually means when he says that
we will have a chance to design in the
future together but for now I want to
quickly talk about timelines the guy
behind stable diffusion said something
really interesting recently he said
nobody is launching runs bigger than
Gypsy 4 for six to nine months anyway
why because it needs the new h100s that
I talked about in that video to get
scale and they take time to be installed
burnt in optimized Etc and Brockman
mentioned something that we already knew
which is that there might be a lag of
safety testing after a model is trained
and before it's released so depending on
those safety tests my personal
prediction for when GPT 4.2 let's call
it will be released would be mid 2024.
if you're watching this video in mid
2024 or later you can let me know in the
comments how I did I've talked a fair
bit about the capabilities that Gypsy 5
or 4.2 might have but to finish I want
to talk about some of the limitations or
weaknesses it might still have rather
than me speculate I want you to hear
from Ilya satskiver about one of the
possible remaining weaknesses that GPT 5
or 4.2 might have if I were to take the
premise of your question well like why
were things disappointing in terms of
the real world impact and my answer
would be reliability if somehow it ends
up being the case that you really want
them to be reliable and they ended up
not being reliable or if the reliability
or not to be harder than we expect I
really don't think that will be the case
but if I had to pick one if I had to
pick one and you tell me like hey like
why didn't things work out it would be
reliability that you still have to look
over the answers and double check
everything and that's just really puts a
damper on the economic value for those
system let me know what you think in the
comments and have a wonderful day