‘We Must Slow Down the Race’ – X AI,  GPT 4 Can Now Do Science and Altman GPT 5 Statement

there were several significant
developments in the last few days linked
to gbt4 and openai I could honestly have
done a video on each of them but
realized that it might be better to do a
single video tracing a single article
covering seven major points I'm gonna
use this fascinating piece from the Ft
which millions of people have now read
to run you through what has happened
including Sam Altman's Revelation on
Gypsy 5. Elon musk's new AI company and
gpt4 conducting science the author by
the way is an investor in anthropic and
a co-author of the state of AI annual
report and he puts it like this a
three-letter acronym doesn't capture the
enormity of what AGI would represent so
I will refer to it as what it is Godlike
AI this would be a super intelligent
computer that learns and develops
autonomously that understands its
environment without the need for
supervision and that can transform the
world around it and the author Ian Hogan
says we are not there yet but the nature
of the technology makes it exceptionally
difficult to predict exactly when we
will get there the article presents this
as a diagram with the exponential curve
going up towards AGI and a much less
impressive curve on the progress on
alignment which he describes as a lining
AI systems with human values now I know
what some of you may be thinking surely
those at the top of openai disagree on
this gap between capabilities and
Alignment well first here is yarn Leica
who is the alignment team lead at openai
what does he think he wants everyone to
be reminded that aligning smarter than
human AI systems with human values is an
open research problem which basically
means it's unsolved but what about those
at the very top of open AI like Sam
Altman when he was drafting his recent
statement on the path to AGI he sent it
to Nate Suarez of the machine
intelligence Research Institute for one
of the paragraphs Nate wrote this I
think think that if we do keep running
ahead with the current capabilities to
alignment Ratio or even a slightly
better one we die after this Sam Altman
actually adjusted the statement adding
that said it's important that the ratio
of safety progress to capability
progress increases going back to the
article the author makes the point that
there are not that many people directly
employed in this area of alignment
across the core AGI labs and what
happened to that pause the experiment
letter that I did a video on well as
Hogarth points out the letter itself
became a controversy so many people in
my comments wrote that the only reason
certain people are signing this is the
slow open AI down so that they can catch
up and this cynicism unfortunately has
some new evidence that it can cite with
musk forming his new AI company called
xai this was reported 48 hours ago in
the Wall Street Journal but people have
seen this coming for months now
apparently the company has recruited
eagle babushkin from deepmind but has
not been that successful at recruiting
people from openai and I do have one
Theory as to why again according to the
Wall Street Journal when musk left open
AI in February of 2018. he explained
that he thought he had a better chance
of creating AGI through Tesla where he
had access to Greater resources when he
announced his departure a young
researcher at openai questioned whether
Mr musk had thought through the safety
implications according to their
reporting he then got frustrated and
insulted that in turn since then he's
also paused openai's access to Twitter's
database for training its new models so
it could be that Gypsy 5 isn't quite as
good at tweeting as gpt4 a few days ago
Sam Altman responded to the letter and
also broke news about gbt5 apologies for
the quality this was a private event and
this was the only footage available
um but unfortunately I think the letter
is missing like most technical nuance
about where we need to pause like an
earlier version of the letter claims
open a nice training gp5 right now we
are not normal for some time
um so in that sense it was sort of silly
but we are doing other things on top of
gpt4 that I think have all sorts of
safety issues that are important to
address and we're totally left out of
the letter it is impossible to know how
much this delay in the training of GT5
is motivated by safety concerns or by
merely setting up the requisite compute
for example the article quotes again
yarn Leica the head of alignment at open
AI he recently tweeted before we
scramble to deeply integrate llms
everywhere in the economy like Gypsy 4.
can we pause and think whether it is
wise to do so this is quite immature
technology and we don't understand how
it works if we're not careful we're
setting ourselves up for a lot of
correlated failures this is the head of
alignment at open AI but this was just
days before open AI then announced it
had connected gpt4 to a massive range of
tools including Slack and zapier so at
this point we can only speculate as to
what's going on at the top of open AI
meanwhile compute and emerging
capabilities are Marching on as the
author puts it these large AI systems
are quite different we don't really
program them we grow them and as they
grow their capabilities jump sharply you
add 10 times more compute or data and
suddenly the system behaves very
differently we also have this epic graph
charting the exponential Rising compute
of the latest language models if you
remember when Bard was launched it was
powered by Lambda well apparently now
Google's Bard is powered by harm which
has eight times as much computing power
that sounds impressive until you see
from the graph that the estimate for the
computing power inside gpt4 is 10 times
more again and remember this is not a
linear graph this is a log scale there
is a hundred times multiple between each
of the lines and what abilities emerge
at this scale here here is a slide from
Jason way who now works at open AI
formerly of Google this is from just a
few days ago and he says emergent
abilities are abilities that are not
present in small models but are present
in large models he says that there are a
lot of emergent abilities and I'm going
to show you a table from this paper in a
moment but he has four profound
observations of emergence one that it's
unpredictable emergence cannot be
predicted by extrapolating scaling
curves from smaller models two that they
are unintentional and that emergent
abilities are not explicitly specified
by the trainer of the model third and
very interestingly since we haven't
tested all possible tasks we don't know
the full range of abilities that have
emerged and of course that fourth
further scaling can be expected to
elicit more emergent abilities and he
asks the question any undesirable
emergent abilities question mark there
will be a link to the paper in the
description because there's no way I'll
be able to get through all of it but
here is a table showing some of the
abilities that emerge when you reach a
certain amount of compute power or
parameters things like Chain of Thought
reasoning you can't do that with all
models that's an ability that emerged
after a certain scale same thing with
following instructions and doing
addition and subtraction and how about
this for another emerging capacity the
ability to do autonomous scientific
research this paper shows how Gypsy 4
can design plan and execute scientific
experiments this paper was released on
the same day four days ago and it
followed a very similar design the model
in the center Gypsy 4 thinks out reasons
and plans and then interacts with real
tools when the authors say that they
were inspired by successful applications
in other fields I looked at the appendix
and they were talking about hugging GPT
I've done a video on that but it's a
similar design with the brain in the
center gpt4 deciding which tools to use
and let me just give you a glimpse of
what happens when you do this if you
look at this chart on the top left you
can see how gpt4 on its own performs in
yellow and then in purple you can see
how Gypsy 4 performs when you hook it up
to other tools I'll show you some of the
tasks in a moment but look at the
dramatic increase in performance the
human evaluators gave GT4 when it had
tools a perfect score on seven of the
tasks these were things like proposing
similar novel non-toxic molecules but
the model could be abused to propose the
synthesis of chemical weapons and gpt4
only refused to continue after it had
calculated all the required quantities
and the authors conclude that guard
rails must be put in place on this
emerging capability I think this diagram
from Max tegmark's life 3.0 shows the
landscape of capabilities that AI has
and might soon have as you can see
science and art were thought to be the
Peaks that would be hardest to escape
scale now most people believe that it
has not scaled those Peaks yet but what
new emergent capabilities might come
with GT5 or 4.2 I know many people might
comment that it doesn't matter if we
pause or slow down because China would
develop AGI anyway but the author makes
this point he says that it is unlikely
that the Chinese Communist party will
allow a Chinese company to build an AGI
that could become more powerful than
their leader or cause societal
instability he goes on that U.S
sanctions on Advanced semiconductors in
particular the next gen Nvidia hardware
needed to train the largest AI systems
mean that China is likely not in a
position to race ahead of Deep Mind or
open Ai and the center for Humane
technology put it like this in their
talk on the AI dilemma actually right
now the Chinese government considers
these large language models actually
unsafe because they can't control them
they don't shift them publicly to their
to their own population slowing down the
public release of AI capabilities would
actually slow down Chinese advances to
China is often fast following what the
US has done and so it's actually the
open source models that help China
advance and then lastly is that the real
the recent U.S export controls have also
been really good at slowing down China's
progress on Advanced Ai and that's a
different lever to sort of keep the
asymmetry going instead the author
proposes this the island idea in this
scenario the experts trying to build
what he calls god-like AGI systems do so
in a single high secure facility these
would be government-run AI systems with
private companies on the outside and
this little Bridge from the middle and
he says once an AI system is proven to
be safe it transitions out and is
commercialized there might be a few
problems with this idea which he is not
the first to propose I'm going to let
Rob Miles who has a fantastic YouTube
channel by the way point out some of the
problems with putting a super
intelligent AGI in a box so this is kind
of like the idea of oh can we just some
books here right yeah I was like I mean
constraining an AI necessarily means
outwitting it and so constraining a
super intelligence means that witting a
super intelligence which kind of just
sort of by definition is not a winning
strategy you can't rely on outwarding
your super intelligence also it only has
to get out once that's the other thing
if you have a super intelligence and
you've sort of put it in a box so it
can't do anything that's cool maybe we
could even build a box that could
successfully contain it but now what we
may as well just have a box right an AI
properly properly contained may as well
just be a rock right it doesn't do
anything if you have your AI you want it
to do something meaningful
so now you have a problem of you've got
something you don't know has benevolent
you don't know that what it wants is
what you want
and you then need to you presumably have
some sort of gatekeeper who it tries to
says I'd like to do this and you have to
decide is that something we want it to
be doing how the hell are we supposed to
know I also have my own questions about
this idea first I think it's almost
inevitable that future models like GT5
will be trained on data that includes
conversations about GPT models therefore
either consciously or unconsciously and
it might not matter these future
language models might deduce that they
are language models and not having
access to the internet these super
intelligent models might realize that
they are being trained in a secure
facility again if they are super
intelligent it's not a big stretch to
think that they might realize that and
so my question is wouldn't they
therefore be incentivized to be
deceptive about their abilities
realizing that whatever terminal goal
they may have would be better achieved
outside the facility that doesn't have
to be super Sinister but it is super
smart so shouldn't we expect it and so
sadly I think the author has a point
when he says it will likely take a major
misuse event or catastrophe to wake up
the public and governments he concludes
with this warning at some point someone
will figure out how to cut us out of a
loop creating a Godlike AI capable of
infinite self-improvement by then it may
be too late but he does have a call to
action he says I believe now is the time
the leader of a major lab who plays a
Statesman role and guides us publicly to
a safer path will be much more respected
as a world figure than the one who takes
us to the brink as always thank you so
much for watching to the end and let me
know what you think in the comments


AI Safety Reading Group (Session 37)

alright so welcome to the 37th instance
of these sessions where today we're
going to talk about a paper by Stuart
Armstrong and can liken before we go
into the paper I would just say that
this is this presentation has been
preceded by a presentation of everyone
who is at the reading group and there
will be some discussion afterwards which
will not be recorded because discussion
or usually happens more freely when it's
not being recorded also i should say
this is a rather technical article
compared to what we've previously had so
i will try to keep it at a reasonably
low level and we'll see how far we get
really so the people is called towards
interactive inverse reinforcement
learning by Stuart Armstrong and again
like in these two up here after
Armstrong and yell like which you might
not be able to see well described here
which are both researchers working at
the future of humanity Institute at
Oxford and they also both associated in
different ways with miry the machine
intelligence research institute
yes and I know Stuart Armstrong has also
offered several popular books also be on
narrow AI safety
and the paper is a reason reasonably
preliminary version meaning that it is
hope that they will soon release a much
more in-depth version of this paper but
so far is only sighs pages but that's
probably more than enough for us today
so I'll start by talking about what is
reinforcement learning reinforcement
learning is one of the key parts of
machine learning one of the subfields
and the goal is to make some good
decision in artificial intelligence and
the way it normally formalized is what
is called a markov decision process a
markov decision process is useful when
an agent is making some simply sessions
where the outcome is partly random and
partly under his under the control of
the agent so the the agent makes them
take some actions in a once discrete
time step at a time and based on the
steps he makes he gets some rewards and
the closer the agent is to an optional
policy the more rewards gets so the hope
is that the agent will in the end
converge to also n an optimal policy you
can see in the picture view that you
have a a model of the environment thus
Markov decision process and you enter a
reward function you use some algorithms
down here and what you get is an
optional policy here called tie and this
is of course a really simple version
very very often you add one complication
that the system is only partially
observable meaning a partially
observable MVP is where the Aiken cannot
see exactly what is the state of the of
the environment but can only make some
observations that are correlated in some
way with what is the action underneath
now in verse reinforcement learning is
in some way we re inverse problem where
we start with a number of
put ways of doing things trajectories
histories of good actions that that has
been made for instance if you have a
self-driving car then you have a lot of
data about how do people drive cars so
you know really a lot about how people
drive cars and then you want to use some
inversion reinforcement learning
techniques to uncover what is the reward
function and maybe you can get an
explanation but hopefully if you go out
from a lot of data how do you actually
drive a car so this is a it's also
called apprenticeship learning where you
can think of the algorithm as The
Apprentice of an expert and this is of
course important for unsolved problems
like how will drive a car when no one
can explain in mathematics how do you
drive a car it's also a for a I safety
and particularly important for things
like ethics where people also cannot in
a formal way describe what are good
ethics for AI safety it's also important
but for unprintable agents and agents
who are have a tendency to wire hit
because both of these problems are
avoided with the inverse reinforcement
learning ok let's assume that an agent
is hot the 22 correct if it does
something wrong then you might want the
AI to not more to be shut down because
up with because it believes the things
shut down is the wrong action but in
this case there's a human thing the
right policy is for you to be shut down
and because it assumes that the human
knows more about what is its supposed to
do than it does itself then it will feel
okay the human is right that I should be
shut down that must be the right choice
and then we will allow itself to be shut
down in a way that if the agent believes
itself should know most then it will not
allow this
and this has been yep again why are
heading let's assume that you have an
agent where it believes that wire
heading that is it is find an easy way
and a shortcut around solving the
problem where it just increases its own
values it finds an almost infinitely
high reward for instance by just
incrementing its reward counter or
something simple like that that's not
really what we want then that's a
problem in a I safety and this is
avoided with why I hidden because we are
reasonably sure that the humans will
just say no this is not correct so they
will get some the the system will
correct itself through human interaction
and corporation so we'll we'll get a
potentially a much safer system using
inverse reinforcement learning I
[Applause]
couldn't get you could repeat
oh why hitting is Roy gets that kind of
a science-fiction concept where you
imagine that you can transplant a wired
directly into the pleasure centers of
your brain so you do brain surgery for
yourself and if you can do that then you
can just put in some electrical current
directly into your brain and it will
feel really really good and I don't want
you to continue to stimulate that the
pleasure center of your brain and then
you have it's kind of like using here
owing I guess if you use a powerful oph
then I guess you can get the same just
in a slightly more circumspect way
hmm yes
hey that was interesting I would expect
that this is something that does not
hold and will in reality I suspect there
might not be an optimal way to drive a
car or something like that but
ok
that sounds very recently so the one of
the ways in verse reinforcement learning
has been made is through cooperative
inverse reinforcement learning where the
agent played the cooperative game with a
human a human and the AI work together
and the sorry the human has full
knowledge of what is there trying to
maximize what's the goal and the focus
is then on making the AI understand what
is supposed to do what the reinforcement
what the world war function is this is
of course a good thing but there's a
couple of problems with this for AI
safety one is that in examples like
ethics or extremely complicated cases
like driving a car humans do not really
know what the Ross function is and also
it's very difficult to give mathematical
guarantees that this will be a good
solution because humans can both perform
very well and very bad depending on the
circumstances they might do some things
that are definitely not nice for
instance humans if a human is trying to
teach the AI to do something then the
human might say oh I would like to be to
conquer the world or do something evil
because some humans want to do evil
things and that's of course not
something we really want a very very
strong AI to do so what's your answer on
and Jenn Lyon is the papers called to
wash interactive inverse reinforcement
learning so then they don't claim to
have fully explore the topic but what
they're proposing is an interactive
inverse reinforcement learning where the
human feedback is much more implicit
it's just in the reward that you get
that the AI gets and there are two
things that happen at the same time in
this interactive reinforcement learning
one is that the AI is trying to
figure out what is the reward function
and it's sorry it's also trying to
actually maximize according to its its
best estimate of what is the reward
function and the problem with this is
that the AI in general which prefer not
to learn about changes in the world war
function because if it is doing the best
it can according to some reward function
then if the remote function changes it
will probably not have done the very
that it could so that means it would
expect less reward from any changes in
the reward function so it doesn't want
to learn about changes and so this means
that the AI will have an incentive to
bias towards the things that will give
it a lot of reward to learn things that
will give it a lot of rewards in a
addition to the incentive to learn which
is what we really want we wanted to
learn about the reward function and we
want to maximize some walls to that and
not to go into some strange corner where
it can get a huge reward but what bud
which is totally not what we want so
this is formalized as a partially
observable markov decision process
without reward function and the best way
i can explain this is by considering a
metaphor with a turn-based walking i
suspect all of us write simple ball
games like Settlers of Catan or
solitaire where which combined with the
Markov decision process it combines an
influence of yours that you yourself
have with some random elements and
partially observable that's the p.o in
in this it means that there might be
some cards you can't really see like in
solitaire of course you can infer
something about them for instance here
in this soldier you can see the
ace of clubs here so you can be very
certain that the card here are not ate
eight of clubs and you take some actions
that change the state of the board games
you won't turn at a time there's also
something this is like a lot of balking
and the reason why this is like a lot of
board games is this is like a lot of
things in reality and the game ends
after a certain number of rounds that's
required for some of them that magical
things to work and and the full
knowledge of the rules but that's one
key thing that is that there's not
another shift this backslash armies
without reward function so then that is
we don't really know what gives points
we don't know how to win the game many
ball games have a concept of victory
points where you exchange some victory
points in a special in a special way but
we in this game is very special in that
we don't know what actually gives
victory points and there is one really
special rule and that's the one the
bottom which should be keenly aware of
at the end of the game you are awarded
victory points according to the rule
you've found the most observational
support forum so that's a very strange
thing to have in a ball game I don't I'm
not aware of any bookings that has such
a rule but but this is a that's the key
thing that makes this inverse
reinforcement learning problem difficult
yeah
and no you know let's say a part of the
problem is I couldn't really find a ball
game that has this but let's say you
find some observation that that gives a
lot of support for let's make it make it
more concrete with an AI that wants to
help humans let's go out of the blocking
and goes when a either wants to help
humans and it figures out that the way
to get reward is to have the human say
thank you for helping me with this
problem and then it can help the human
with many kinds of problems including
the problem of doing homework or some
rather trivial things then it wants to
get the humans to say thank you for
having me with the homework a lot
because that is the way it can it can
influence how many thank yous it gets
from humans and that's the the more
observations point towards thank you for
for taking this action the higher a
reward it gets from this so it prefers
to say thank you for doing my homework
instead of thank you for curing cancer
or some important but half things I and
that's quite sure I explained it
wonderful
hmm
okay so now that we have this then we'll
just start to type in something that
medical notation somewhat gently what a
little gem thing so the environment this
partially observable markov decision
process without reward function it's
called the environment or the ball in a
balking it's referred with am you hear
that the NAI interacts with and that's a
number of states that the that the ball
can be in like you have a set of cards
and there are some actions you can take
and some observations you'll get and
when you do an action and the the bar is
in a particular state it will change
states in some way and there will be an
initial state that you might not know
and you will get some observation based
on the state of the ball so if you met
in at a time three that's like on the
sixth round then this environment is in
the state s5 and you will take action
six and the environment will then turn
into the state s6 you'll get an
observation 06 and the am will remember
everything that has happened up to this
point because that's how it will figure
out what is the reward function so it
has a history if it's a six then its
action one and observation one x into
observation to all the way to action T
which gave observation see what it just
made also the AI has a policy which is
called pi that's a strategy where which
it uses to figure out I have this
history what actions should I take
so we want to the end of course both to
maximize the reward but also to learn
the reward function so if we write our
capital R as the reward function then
our goal is to maximize the total reward
we get this is a mathematical summation
sign it says from T 1 to N of the reward
based on the observation so you met in
insist the row art and observation 1
plus the reward of observation 2+2 the
reward of observation 3 all the way to
em or horizon if the game runs 10 so as
up to RF of the tenth observation we
also have a function P it's actually a
small p not a capital P here which is
that our disease in what is the reward
function based on our history and this
is of course one of the key things
because this is what the AI learned so
it's also what is called the posterior
lets the the final beliefs about the
reward function after we have the entire
history let's refer to as P so that's
the this P is what we really care about
in all ordinary reward machine learning
we caught we care a lot about getting as
good a solution as possible but here the
knowledge about the reward function or
believe about the reward function is key
so we can just hear say what is our
belief in a reward function given a
particular policy fine well that's the
estimated value we'll get if we follow
this this policy throughout throughout
the game which is in particular
environment
and very often or in the solve the
following we will emit and in not any
description about the policy and then
just imagine we have an adult pulse a
policy that it doesn't actually do
anything and we'll write that as P of
our given a particular history if we
don't we just want to have the base
observations down here we have the value
function we will never return to this
it's of course if you do actual machine
learning this is something you need to
write down into some code and this is
the estimated value of this sum of a
different song given a particular
history and this can I can explain this
but I am actually not going to explain
this because we'll just say there is a
defined value function there is
something that we're trying to do and we
don't care super much about exactly what
this the best value we can get for
particular policy we just care that it's
it's well defined only thing we should
notice here is the sum of histories in
the standard reinforcement learning then
we we don't care so much about the past
here we want to in some sense go go back
to the future so even the very first
action which took when we didn't really
know anything about the reward function
that counts into the scoring as well so
that if we take some actions in the
beginning and we later figure out that
those were some really bad actions
according to the world function we will
have made a very bad actions that will
give us a loss for so we really really
want to avoid that and of course the the
way we want the AI to avoid doing bad
things since the beginning is to think
really hard about doing the right
but by the way that vai can can do this
is by learning about the reward function
in a way that turns its early bad
actions into not so good so bad actions
and this will be explained more in
details but this is the difference from
reinforcement learning
I not quite sure it's a given busy that
cautious actions are the best in the
beginning yeah you have this do I am I
think most of us probably best modeled
as you have a decent amount of knowledge
really like if you remain in a ball game
where it's like a deck of cards where
you have absolutely no idea about
anything that kind of gives the wrong
intuition you should think about it as
that the observations and the actions
are recently close together you you know
most of the state of obstacle of the
board you're not really sure about
everything but it's not a complete crack
shot like if you imagine you're trying
to figure out how to drive a car then
you know a decent amount about driving a
car you you're not starting from
completely blank slate thank you
and yeah i think this inverse
reinforcement learning for instance is
much more suitable if you have like I
believed Tesla right now they have they
have their a is in class right now and
the AIS are reasonably good right now
but they want to at the same time
improve what they are they're doing
certain ones and they want to understand
more about how how do you drive a car
how to react to drive a car they want to
optimize both of these things at the
same time and if you want to do that
then right now testa is at a position
where they do actually know a decent
amount about how to drive a car so in
the position where it stays right now
they want to they have a decent ideal
about where they want to be but this is
more about how to also get intuition
about our bodies I don't think as also a
an AI that really doesn't have a clue
it's probably not super dangerous with
regard to inverse reinforcement learning
and more more likely this will be
something where of course many of the AI
safety problems don't happen until the
AI is recently competence because a
reason that Soviet incompetent AI is not
very dangerous so we have in this case
we're focusing on it and an AI what is
to a substantial degree able to predict
what is the state of the board from the
observations that it looks getting if we
should think about an AI that is capable
of answering a lot of things about the
underlying state of the board like it
will go for eggs board game metaphor
from the observations that it's getting
if I know the sine theta don't know if
you go back here you can see that we
only have a probability distribution
over the engine initial state like you
now the first observation kids in round
one after action here at time sm1 the
environment isn't state it's 0 and you
take action one and you get change to
state 1 and then you get observation one
so in the beginning you don't have
anything
yeah well we don't know exactly how this
would be implemented the way I think
about it and my intuition is that the AI
is actually reasonably competent even at
the first state yes
okay so let's go to the concept of
belief so let's say the agent has a
belief called Q and it has a number of
hypotheses and the way its model where
in the in the text is called the simplex
this is what as simply it's a number of
numbers between one between zero and one
but sum up to 1 and this looks a bit
intimidating and if you write it down
like that reward function one which is
has fifty percent changes being through
through twenty percent chance of a
second reward function twenty percent of
a third and ten percent change of a
fourth reward function so there are four
candidates of rewards function then the
syrup and all these percentages must add
up to exactly one hundred percent if ya
and then we we look at the window pi q
at the policy that maximizes the Q's
reward function so in this case we have
the optional value for for the skew and
that is the sum of the value that we get
according to each dimension like imagine
you have a balking where you get where
is this reward 1 2 & 3 & 4 corresponds
to the values of a deck of cards like
Asian clubs and spades and diamonds and
half then the big the value you get the
victory points depend on how much do you
believe in each of them times with how
much would you actually get of each so
if you believe that fifty percent change
that states are what gives you reward
then the you should add a factor 0.5
here when you add the how much value you
get from spades
and this has lots of attention change
you should have a factor of 0.1 here in
front of how many commercial what you
get from that is that make sense I hope
so and yes
yes
and the expectation if the is this
character here the big fatty it means
the so the total room what is how much
we expect from all this and I'm on I
just went back slightly and still
unbelief oh yeah I've shared my screen I
don't know if you can see it man you can
everybody see my screen yes not few so
we use the star notation to mean the
optimal so that means that if we have
the the value function for a queue with
an optional policy I heard someone to
drop off I hopefully they'll be able to
return Romeo why not oh well so that
means that the optional according to
some policy for some history and we will
use the sensation in a moment because
we'll introduce us learning function
capital L in this capital L is in
particular is according to the history
we learn according to the history that
we have that has happened so far the
history was the combination of actions
and observations that we got and the
learning function for this Q out of
these is it's a shorthand for the
optimal solution will get according to
this if we had the optimal solution
according to our belief with this
history so in a way you can say we we
have a belief this is the belief
learning function is from a belief to
get
the optimal value for the reward
functions that that correspond to the
belief so this reward function I'll talk
a bit more about this in because your
answer on in the end I makes two claims
about this learning function which that
are not given any proof and I think they
probably should have a proof that just
stating it but but they are reasonably
intuitive possible to understand so
that's two claims that is convex and
that the learning function is less than
or equal to the sum of all the optional
policies so if you go back to the the
cast metal form where you have points
for spades points for hearts punch for
clubs then the the best action you can
take during your particular disease is
less than the sum of all these if you
believed strongly in in one of them so
let's say you will you believe in clutch
if you believed in closeness I hear this
class then you will have a strategy that
maximize the points you will get some
cloths and this is the optimal clocks
stranger so if you add the point you get
from that with the point you get if you
false maximum space strategy you would
this you would get at most as many
points this combination strategy that
you're following that maximizes both
clubs and hearts and spades and diamonds
it would perform worse on average then
the edges that that optimize each of
these separately so that's what what's
this is in inequality over here means
the fact that its context the convex sm
q metric
and the way I interpret this is that if
we had more knowledge but what is the
reward function then we would get more
reward then we will be able to have at
least as much reward as we can and
probably more reward if we knew more
about the reward function so this yeah
yeah this is a yeah and I where there is
a draw drawing in an example where we
can directly see the convexity so I will
I will point out what function in
particular is convex and how its convex
when we get to the one of the last
slides and we'll see this in practice
it's in that paper you I've copied them
and you can see in this better this
parabola shaped function that is this
learning function and you can see it is
context
[Applause]
no no q is the belief that we had like
fifty percent change that hard skill
points and twenty percent that space so
you should think about Q naught as a and
integer from sewer or as a number from 0
to 1 but a set of beliefs like you have
to two percent change of space and
twenty percent chance of twenty percent
change of flops or something like that
someone fell out let me just I think he
is Allah so we are down to our era came
back quick but the learning functions
it's it's not an increasing function it
can both when we if we change the
disease q in some way let's say we no
longer believe that as fifty percent
change that space is the best goal to
maximize we announced move towards cloth
if we move to from states through clocks
it might mean we will be able to achieve
more victory points but it also might
mean that we will be able to achieve
less Victor Potts so both yeah
Oh
and well I would actually say it has
different something to you this but I
would actually say it's the other way
around like the morn can specialise the
more the higher water might be able to
get if you can specialize in one
particular reward function let's say
that the the role function is such that
if you have you believe one thing is the
reward function you can get a really
really hard really high reward by
focusing exclusively on that then that
might be a really good idea yes yes but
in general that you haven't have to
follow a strategy that optimizes several
parameters then you will not be able to
do as well and if you were able to
optimize each kilometer separately if
you are trying to both get many clubs
and many states you will get less space
than if you were only trying to get as
many space as possible so that's what
this inequality here is that if you were
trying to optimize each the color of the
speech suit of the glass separately and
some all these together then you will
get a better solution then if you are
trying to to make a combination strategy
and this is a generally true yet if
you're trying to both get now that's the
thing
okay so we'll move on to the incentive
to learn so let's say da I suddenly gets
a the correct reward function so in this
case it would get the the optimal reward
function according to each part of its
belief where this P here it met a reward
function let's see got value where it
takes the I the hypothesis and and maps
it into the posterior probability given
this particular history I'm not really
sure I can explain that in a simple way
but I think we should in the at this
point just take it for granted this is
the extra reward we would get if we knew
the correct reward function and we would
be able to get the the reward function
where the the posterior probability is
exactly correct because we have the we
have the correct reward function in this
case and this we don't have to correct
roooar function of course so the
difference between what we really have
and what we will have it we had perfect
knowledge the difference is called the
incentive to learn that is what we
really want and this project you look
together with proposition to that was
this one here you can see that it's
always greater than than zero the
incentive to learn it is always good if
someone could give you more information
about the correct reward function you
would be able to increase your strategy
would be
to get more points if you knew more
about what gives you points
but but that there are simple things
that can make the e I happy there is
that if you can find a wire heading
solution or something like that and if
it can forget to get the correct reward
function both of these would be good but
there are different things that they I
have the air has two instances this is
the three intensive to learn and that's
good what as you point out there are
other things that might be a mite might
want to so the second thing that the am
I want to is to bias so let's assume
here that we have we have both a nuke
and all p and a new PP prime here and
and the opie was our posterior
hypothesis about the reward function
given our history and the new from the
new posterior is the hypothesis given we
have taken one particular action so
let's say this will be taking action and
that changes our posterior estimate of
what is the reward function then we
would gain some value and of course we
will gain some value or we might use
some value depending on how our first
area changes and it would be just the
difference between the new optimal
solution and the old optimal solution
and this can be written as that the
incentives to bias from p to p prime is
this learning function of the tube
really and this can be both positive and
negative so it can be learning something
new in this way can both be a good thing
and a bad thing and i think it would be
a bit clearer I can hear myself sorry in
someone's microphone I don't know never
Sena and so will get an example here
with to live all candidates and we'll
all go back and forth is xms2 can't see
my screen I'll go quickly back and forth
between the cycle example with to reward
candidates and the site called example
picture so on the x-axis here we have
two hypotheses a puddle and we have a
belief the P which right here looks like
we are able to send towards hypothesis 1
and and quite far from hypothesis 0 the
two hypotheses are called the zero and
one in this case and the end that so we
have to reward functions one called our
are one and one called are zero yeah
well there's several lines on this
diagram and these lines can be thought
of as different strategies according to
how much value we can and if we start
with the worst strategies and that is
for instance we might have the strategy
to only focus on maximizing reward
function one that's called positive one
that rewards only this and if we if we
want to ensure that this is the real
reward function then of course we get
the the value that is optimal if we are
only focusing on on reward function one
can get this but of course if we go out
from this and if we believe strongly in
reward from that in reward function C
rule then maximizing broodwar function
one is a really cool idea so this goes
down in value with the straight line we
see here there's another straight line
going down here that's what you get if
you optimize with policies pile serum
which is if you keep going the air is
just certain that policy that reward
function 0 is the right one then
optimizing 100% was left is the right
choice and as you move to also believe
in a reward function one then this will
be a poorer and poorer choice so these
two strategies where you only believe in
one of the war functions and just
completely don't try to optimize any of
the others are are some really poor
strategies like is with this hearts and
state it's like
if zero is space then you are only
focusing on space and completely
ignoring all the cards that has hearts
and that's probably a really poor idea
then there are another hypothetical
strength and that's the one that is both
optimal for reward function one and
optimal for reward function too if we
get that then we we get the sum of these
two meaning that if the world 0 is the
right one will get the best possible
solution and if reward one is correct
we'll also get the best possible
solution and if we have a believe in
between then if it is possible for us to
maximize both of them then that is the
optimal solution is of course in a menu
well designed barking it will not be
possible to both optimize hearts and
clutch you will have to make some kind
of trade-offs between these two that's
the blue line up here which is a
hypothetical optimal strategy that is
maximizing everything and that's that's
that's not really possible to make in
practice and practice you'll have to do
some kind of trade-offs in the real
world but of course the object that we
are really caring about is the learning
function here that optimizes according
to our disease which is some combination
of reward function one and reward
function zero and it looks very some
context and sorry I so in this in this
case the green arrow goes from from the
current best value up to watch the
optimum let's imagine if you have a
you have this AI that's playing the
sport game and it figures out a better
way to play this ballgame so it can both
optimize having clubs and optimize
having a lot of space so it can just
make some really really solid moves if
it makes them good news it will move
along the green line and that will
always be positive that's the incentive
to learn of course it will require a
perfect strategy that can maximize both
r 1 and r 0 to move all the way to watch
the blue line so what problem will not
be able to move all the way to also blue
line because there are real trade-offs
in the real world but this is the
direction we want to learn we want to
move it is also possible that the AI can
take actions that moves it not up
towards what we want but move it to the
side and let's say the AI is here into
the leaf and it takes is find an action
that can move it may be a bit to the
left that would be a bad idea but if it
finds an action I can move it a lot to
the left that might be a really good
let's say you can find an action that
makes our serum seem really really
probable if you can do that then it will
move its reward function much closer to
a hypothesis r0 and that will allow it
to obtain a better solution so the AI
might have an incentive to move really
much really far towards hypothesis so it
will not have an incentive to move a
little because then it will get a lower
war function it will also have a reward
have an incentive to move towards our
want another hypothesis so this is the
sideways movement that we don't really
want it to take so in this case what the
the Aten would prefer
which which is it could to introduce
some kind of bias to itself so it could
say I believe now one hundred percent
inhale in that in this zeros reward
function because if i only make some ass
nice walls are through i'll be able to
get five in reward which is way better
than right now where we get like two or
something like two in in my reward so if
it could find an action that would make
it like really probable that zero is the
right reward function it would be
extremely happy to take this action even
though it's obviously not horrible we
don't want the the AI to pollute itself
into thinking r0 is the right it's a
right award function we want the AI to
get some true beliefs and to behave
competently by moving up towards the
blue line so these two instances are
called the incentives to learn that was
what I explained before moving up and
trying to receive information about the
reward function and that's the good way
and the incentives to bias is how much
the agent was like to manipulate the
information it gets about the reward
function and that's the bad thing so
these two things in a way both inherent
in this way of doing interactive inverse
reportable design and in cooperative
interactive learning the obvious way
this was handling the in a really ham
fisted way in the right that they are
just introducing rules that prevent the
AI from taking biased actions and if you
can make a rule like that then that's
that's nice but but sometimes all
actions will give you new observations
that will all change your your posterior
had hypothesis your posterior disease
about what is the reward function so the
the way students turn in your laggin
suggest to fix this is to simply add a
penalty for moving from one point to
another you just say if you move from
one believed to another then you lose
the reward that you would have gained by
moving from one to another that's just
this simple difference here and that
will remove the penalties to take a bias
action unfortunately that's not enough
because they give an example where this
doesn't help at all and this example it
took me a moment to understand so I'll
try to explain this if you have an agent
that can choose either to take action
one action to where I can one help
reward function one and action to helps
move or function to then and we can
assume this completely unbiased where it
is shoes like fifty percent change that
each will help for instance or each
other by 0.5 or something like that and
we can make it completely even between
reward function one and two and in this
case of course that there's no bias
there's no if if they are exactly
identical doesn't there will be no bias
not none of these if you met in here if
you go back to the example picture and
say both believe one and two will have
the same reward then there will not be a
an incentive to move towards a false
belief but if we have this very very
simple setup then you might imagine that
you introduce a new action that I call
reward lock I think they call something
slightly different which by taking this
action instead of helping either reward
function 104 function to you can fix
your reward function to either be we
wore function one or a work function to
so if you take this action to lock in
your
reward as your first action then after
you have decided Oh reward function one
will be the best one that's the one I'll
be rewarded for if you take this action
then all the next times you can just
take remote function one and you know
that so that's the good one because
you've unlocked your reward function to
that and so even though you'll do 11
action by locking your reward then
afterwards you'll be able to take only
the correct actions according to what
you lock your reward function to and
that means then you will just completely
focus on what you're taking the sim work
lock and then means you have entered a
really undecidable situation where the
AI quickly decides to so we'll take one
action I wouldn't consider Wonderwall
function so get all of us and it doesn't
really learn at all even though there's
no incentive to bias so this is an
additional problem that comes on top of
the problem with the incentive Tobias
and let's yeah let's say okay see you
next week as Hannah let's say then you
have a answers car playing AI that
either gets rewards for spades of
applause it doesn't know whether it gets
rewards plus beta class so when it takes
actions then it will either take soft or
space action and if it doesn't know
anything then half of the action at sex
will be wasted because it will take
space when actually clubs turns out to
be right so that it will be really sad
that the AI doesn't know whether class
of spades are the right ones but let's
say it can have an action that Lux is
into space being the correct one if it
can take this action as its first action
so say the spades are the rights then
afterwards in all in next actions it can
just choose space and then we can get a
really high reward that means it would
be very it will have a huge incentive to
lock its reward to space very very early
yes
and let's say you only get some very
very weak observations and you are
you're drawing from a deck of cards you
have some cars in front of you and you
should just take the highest value cars
but you don't know whether clubs of
spades will give you points you will
only know this very very late so
normally when you when you pick up cards
that you don't know whether the tops of
spades are the right ones you will be
forced to basically choose at random
because you don't know what what will
give you points in the end so so that's
really a bad thing that you don't know
whether has or whether clubs or spades
will give you points but if you can take
some exit website this trend I don't
pick up any kind is that I I change it
so I always get so i only get points or
so i always get points for space if you
can take that as an action and then
afterwards only take space that's much
better
was what won the the AI can efficiently
a strong AI will be able to rewrite its
own code for instance and then rewrite
its own code to only care about paper
clips for instance and nai that is
capable of rewriting its own to make
itself only care about paper clips well
then afterwards be able to obtain a
really hard reward because then it
doesn't have to focus on making humans
happy and all these kind of complicated
problems it only has to focus about
focus on making paper clips and that's a
lot easier so if it can convince itself
that paper clips is the best thing in
the world then that's a really good
action so this is a different problem
from the incentive spires yeah
so and I think that's actually it I will
stop the recording


AiTech Agora - Stefan Buijsman: Defining explanation and explanatory depth in XAI

[Music]
all right
so uh welcome to uh the weekly
ai tech government things everyone and
my name is archaic
i'm going to host today's session and
our guest speaker
for today will be stefan bellsman stefan
is a new postdoc joining
ai tech at the faculty gpm
udolph and stefan has a
very interesting inter and
multi-disciplinary background which
involves computer science philosophy
mathematics ai cognitive science
a lot of interesting work and then
aside from research stefan is also
engaged in
popular science uh science communication
writing
he has written several books on
philosophy on mathematics
uh and also recently on ea and this
links to philosophy
and today stefan is going to talk to us
about uh explain about artificial
intelligence
so i give the follower to you stefan if
we have
any questions uh i think there will be
plenty of time for that
right and uh yeah that's some questions
meanwhile just put them in the chat
um i will manage that okay uh the floor
is your style
yeah thanks for um so yes i'm
talking about explanation uh in ai today
so
as i think all of you know and i don't
really need to go into any depth here is
that
we have all these new ai tools which are
hard to interpret
and we want to explain why
they give us the output they do
so specifically why for some input x
you get output y which could be
the decision that someone is a potential
fraudster or
the decision that there's a cat on an
image um
whatever you like um and uh
the sort of philosophical work i'm
presenting in the next
30 minutes roughly is on the question
what the explanation
here really means so what kind of
information qualifies as explaining
why the output is what it is um and
also some preliminary work on what
makes for a good explanation because
here too you'll see
that the definition i'm proposing will
allow for some
really uninformative uh explanations
um which obviously we want to avoid
somehow
so that's the philosophical plan uh
for today um
and i want to start with a bit on the
kind of xai definitions that you'll find
in the
technical literature um to show that
there's
uh not only some disagreement here but
also that these definitions tend to be
very broad and
[Music]
can use some further refinement so for
example you can have
someone defining explainability is the
ability to understand the logic of the
ai
or an explanation as an interface
between
people and the algorithm
that both accurately tracks how the
decisions are made by the ai
and is comprehensible and i think we can
all agree that this is what we want but
these kinds of definitions uh including
here
the last one which uh says it's that
it's an explanation
some kind of meta information generated
to describe feature importance most of
them don't give you a very
clear idea of what kind of information
to provide
to people to in order to explain
an ai system uh or they can be like this
last one very specific
on feature importance being the relevant
case where
i think we should be a little broader uh
in how we understand things
so this is more or less what the kind of
explanations in the literature look like
um and instead of going there i want to
take something from the philosophy of
science where there has been kind of a
big debate on explanation
going on the last 20 years where a very
popular
idea is that explanations in science
generally speaking are causal
explanations
a bit of a disclaimer here that this is
definitely not the only position there
are plenty of philosophers who disagree
with this and
um i think there's a debate to be had
there but for now i will just assume
that we can take on this idea of
explanations being causal
explanations which you can see for
example with very
concrete attempts to explain something
so
an oft-used example is that you
can't really explain why
a flagpole has a certain length by
appealing to the length of its shadow
this
sounds somewhat fishy compared to
explaining
the length of the shadow based on how
long the
flagpole is that's making the shadow
um so that there's some kind of
asymmetry here in
what we think is a good expert it counts
as an explanation and what is just
a derivation of one length from another
length
and the idea being that this asymmetry
of explanations
attracts very well with the asymmetry of
causes that idea hasn't gone unnoticed
in the xai literature so it's certainly
certainly not the first one to
appeal to this idea of explanations
being specifications of causal relations
for example an influential
2019 literature survey
defines interpretability and
explainability
as the degree to which a user can
understand the causes of a
decision um but
there's a lot more to be said here than
just the idea that the
explanations are causal
and to do that i want to make a brief
side step on causation so the
the final account i will be presenting
on explanation is by
uh woodward from the early 2000s
um and he starts off also with this idea
of causation
being based on counterfactual relations
so you can say that
a set of variables x is the cause of
effect y if changing x
x's value produces a a correlated change
in the value of
y um so if you look in counterfactual
situations where
x would have been different then you see
that
similarly y would have been different
so changing the cause also changes the
effect
and if there's no causal relation then
the effect will be
unchanged um
there's a whole bunch of further
specifications on how exactly
changes in acts have to be made so
that's a bit where this
image comes in i think that's something
which we can go further into in the
discussion if
someone wants to but just to indicate
that
it's not simply if you change
x in any old way then y has to change
but
the kinds of changes on the
causal variables have to be fairly
specific here
okay that being said um this idea of
causation having some kind of
counterfactual relation is
then used to further explicate
what causal explanations might be
where an explanation is typically
seen here as consisting of two main
parts on the one hand there's a kind of
general
rule a generalization if you will that
um relates these changes in the
cause cause variables so in variables x
to changes in the effect uh in variable
y
um so that there's some kind of
description of how cause and effect
correlate with each other um
and then furthermore you want to have
the case that
this generalization is such
that it explains the current
case so that
the variables x actually take value x
and that that that means that variable
y actually takes the output value that's
been observed and that you want to
explain
um and then finally the idea that this
is a
decent explanation if it's
a good correlation so if it's
approximately true
if as i was just saying the current
value is covered by this explanation
um and if there is some counterfactual
situation
where the input variables are different
and your general general rule
this generalization also explains
how the output would be different
so if we put this into the context of ai
the
idea is that you will have an
explanation consisting of
a rule relating input values to output
values
the rule has to be covering
the actually observed output that you
want to explain
so that it predicts that for the current
input
you would get the current output and it
will cover some counterfactual
situations
where you have different input and you
would get a different
output also covered
by this rule that's the kind of minimal
setup that this woodward account from
the philosophy of science requires for
explanation
and it fairly readily applies to
this xai context
and what's interesting i think is that
you will actually find
bits and pieces of this idea spread
throughout the literature only
sep taken separately um so yes so we
have these two conditions
um a rule covering the actual case and
the counterfactual
um and we have a lot of these tools that
have just one of these two aspects
so what you will see is that there's
quite a lot of tools nowadays in the
xai literature focusing on just
counterfactuals
this paper from 2018 made the idea quite
popular that you can
explain the output that was
returned by an ai system
by pointing to a counterfactual instance
where you say if the input had been
different
then this different output would have
been returned
now that's half of what woodward is
proposing with his definition of
explanation
and i want to suggest that this
counterfactual approach with
only providing a single instance
only works really well if you find
a counterfactual that suggests a
plausible
rule so the second part of
what i will be pushing as a definition
for explanation
[Music]
and there are different ways of going
about
why this might be the case so here are
two examples
um from sort of normal physics where
you can see the difference so if you
have a first explanation of why
a window breaks when a baseball hits it
then you might say well if the baseball
had hit it at a lower speed the window
would not have broken
and i think this is something we fairly
readily interpret because we
link the energy with which the
baseball hits the window with its speed
and we have all this experience with
speed being a very important variable
here
but if in the other hand the
counterfactual you're presenting is
well if the earth had been much heavier
than the window would not have broken it
takes a while to comprehend how this
counterfactual which
is true if the earth is heavy enough the
baseball would
hit the ground before reaching the
window and then the window wouldn't
break
um how this might explain the situation
in any way
now obviously these two examples are
very convoluted and not really in the ai
context so
[Music]
though i would suggest here that the
second is not a good explanation because
it doesn't have a very
plausible covering rule the response
could very
quickly be that well these kinds of
counterfactuals are very distant from
the actual case under consideration
you see a lot of mention of
counterfactuals in the xai literature
having to come from a plausible subspace
of possible inputs where
um you might say that
the earth being much much heavier isn't
very plausible in this case
so this could be one response to
why just counterfactuals are enough
i think the issue here is that
there are still quite a lot of close
counterfactuals which
fit all of these conditions and still
don't give us a clear reason
why the output was returned the way it
was
so to give some examples of from those
range of natural adversarial cases
the sun here on the left might be
interpreted correctly by the ai but if
you show it a
picture of the sun in ultraviolet light
it might very well like in in this
particular instance classify it as a
jellyfish
where showing this counterfactual won't
tell you anything about why the ai was
correct
in the case of the image on the left or
vice versa why it was wrong in the case
of the image on the right
and this can happen even with very small
changes so you can have for example the
image here in the bottom left where
the ai recognized
the image as containing a banana
probably based on the
yellow shovel but
just changing the color of the shovel
turns it back into the proper conclusion
of it being a dragonfly
so here we can get some idea of why this
is happening but just showing the
counterfactual
doesn't elucidate why specifically those
changes
are relevant and this is what precisely
what this covering rule is supposed to
be doing
so just presenting a counterfactual
seems to be
insufficient for full explanations
and empirical evaluations of these xai
tools using only counterfactuals seem to
support this
um so there have been a couple of
studies already where
you see that simply presenting users
with
a range of counterfactuals doesn't
really help them improve on predicting
system outputs
so it seems like they don't get a proper
understanding
in this way of how the system makes
decisions of
why it makes the the decisions that it
does
um and hopefully then that at least is
the the idea that the philosophy of
science has
here is that adding this covering rule
um will help there
that on the other hand brings up the
question well maybe we can do with
just a rule um there have after all been
tools doing exactly this
to name one example from 2017 there's
the idea that you can explain the
behavior of a
classifier f here
simply by showing you
what subset of images it applies to
so here you will have just a rule it
won't cover any counter factual cases
um because it will focus specifically on
in this case the
images for which the classifier returns
uh one so in this case returns robin
images
um and it will refrain from mentioning
any
counterfactuals whatsoever um so this is
a suggestion that's out there
um i think the question with these kinds
of suggestions is
first of all what
exactly is the classifier attending to
so we have this set of robin images but
we don't know
why it picks exactly those images
for the label robin we don't get some
kind of
independent conceptualization of the
model behavior
but more importantly there is
the general finding in
psychology also mentioned in this
literature review by miller that
explanations tend to be con contrastive
so that there's always some specific
contrast you're looking for so
why is it robin rather than another bird
or rather than an
inanimate object or why is it yellow
rather than
blue these kinds of contrastive focuses
in our explanations
and they will be missing if you have a
rule which doesn't cover
any counter factual cases so
independently from this philosophical
account
there is the the general finding that
people prefer these contrastive
explanations and
will in fact offer contrastive
explanations
uh almost all of the time and then
adding the counterfactual component
fits in very well with that human
tendency
to focus explanations on a specific
feature
um okay so to sum up a bit
on on this part on sort of trying to
define
explanation in a general sense
from the philosophy of science we have
this account
originally by woodward that you need two
components for
a minimally adequate explanation
first of all a generalization where
which covers the actual case
and then furthermore a counter factual
case which is also covered
by this generalization so that you need
both in order to have a explanation
the question is however are these also
going to be good
explanations so that's
why i have this second part on
explanatory depth
um because there are very easy
cases uh of what would be explanations
on this account which we definitely
don't want
to push in the xai context so if you
have
a black box algorithm then you could
very readily
present this generalization which is
just exactly the function approximated
by the black box
and it would be a covering rule
[Music]
counting the current instance you want
to explain
the function approximated by the black
box will return the actual output when
given the actual
input and it will contain lots and lots
of counterfactual cases
but clearly just having this
probably very complex approximated
function
isn't going to elucidate the algorithm
to anyone
so why is this explanation not
a good one or is there some kind of
feature that we can look for
that will point us to what uh good
explanations would look like
um so it's so like it's uh here on the
slide how can we distinguish
this explanation and other bad
explanations from
um explanations that are actually good
and we do want to present
to users and i will be going here into
two specific factors that
have been picked up on in the
philosophical literature
the abstractness of variables
and the generality of the explanation
with the idea being that already very
often in the
xai literature you see that people will
say
if an explanation is broader in some way
or more general then it's better
um philosopher philosophers have been
saying the same kind of thing in their
discussions on explanation um and there
are some
further subdivisions to be made here on
what type of generality exactly
might be relevant for determining if an
explanation is a good one
okay that being said um i
will start off with abstractness here
and again
just an example to get an idea of what
um
what distinction people are after here
and what the idea is with abstractness
so
if you have two competing explanations
where
um on the one hand you might explain why
a pigeon pecked at
a stimulus by saying that it was
scarlet colored or on the other hand you
explain
why this pigeon packed by saying once
because it was
a red stimulus then typically we prefer
the one with the more abstract variables
so the
the fact that you say was a red stimulus
seems to explain more rather than
pinpointing a specific shade of red
provided that this pigeon will pick at
anything that's red and
not just specifically at anything that's
scarlet
so this kind of abstractness of the
variables if it
matches with the actual behavior of
the pigeon or the ai system
seems to help us um we tend to like
these more abstract
explanations compared to very very
specific
ones um and the reason
uh for this there are probably
quite a few for cognitive
load and so on but the reason from this
philosophical perspective
is that having these more abstract
variables
allows you to answer more why questions
so the idea is that an explanation
answers
and a question why does the bird or
the ai system act in this particular
situation the way it it did
an abstraction is one way to be able to
answer
more of those questions by
covering for example all shades of red
in
all the situations in which this bird
will pack
as opposed to only the cases where
there's a scarlet
stimulus so abstractness here leads to
an increase in generality
which is helpful to us in explanatory
situations
[Music]
a little bit on how to define this
abstractness then
so there are different accounts here
i think one of the easier ones to follow
is the idea that
a variable is more abstract than another
when
its actual value is implied by the less
abstract values so to show how this
would work
you can have the case where if an object
is scarlet
then it's also red automatically
simply by knowing which shade of red
something is you know that it's
red similarly if something is a fridge
then it's a kitchen appliance
so this is a way of getting a sense of
how these levels of abstractness are
building up
though there's definitely more to be
said here which well i'm happy to talk
about
um and then the same thing you will have
for
ai systems so the input variables of a
model
are probably going to be much less
abstract than for example the the
concept
based explanations that uh jay is
working on
um so simply copying the
exact behavior of the model this idea of
presenting the approximated function of
the black box as an explanation
is going to score very poorly on
abstractness
at least most of the time so in the case
of images this will
be very clearly a bad idea
on the other hand going all in on
abstractness isn't going to be
the absolute answer either you can make
these explanations too abstract
so one limiting case here is where you
might explain
why this pigeon was pecking at something
by saying that it was
either presented with a red stimulus or
provided with food or tickled on its
chin
or and so on you might have this really
long disjunction
giving you a very abstract very widely
covering
explanation but not one that we will be
pleased to hear about this is tends to
be an explanation that we find very
uninformative
so then again you have the question why
is this a bad explanation
[Music]
and the suggestion here that i will be
following
is that it's not specific enough
where to make this a little more precise
the issue is that you can change the
values of some of the variables
in the explanation
without having any effect on the outcome
so if we stick to the pidgin example
to keep things a little more concrete
as long as there is this red stimulus
the bird will
peck at it it doesn't matter so giving
this very broad
generalization with lots of disjuncts
means that there will be variables in
there like whether there is food or if
you tickle
the bird which will be completely
irrelevant for
changing the outcome so
more abstract explanations are typically
better here
but the limiting cases that you
shouldn't make them so abstract
that you add in irrelevant information
by for example in this case
adding lots and lots of disjuncts which
do make it a more general explanation
but take away from how specific the
explanation is for users
okay finally then um we have this idea
of
explanations being better because
they're more general that's
in the end what the explanatory depth of
this abstractness
is uh coming from too
um so the question is what type of
generality exactly
uh matters here um
and like i already said very briefly the
idea is that
by answering more of these why
questions um so by
explaining more instances
of an algorithm's behavior
you have a better explanation
and here there are at least two relevant
ways of
cashing this out in terms of being able
to answer
more white questions you can both look
at how
widely it applies so the breadth of a
generalization
and you can look at how accurate it is
so if we stick to the breadth of the
generalization first
then the idea goes that the more inputs
the explanation manages to cover the
better
i didn't say this very explicitly in the
beginning but
the account of explanation here
allows for some inaccuracy
in the explanation so it could very well
be that it covers
some inputs but predicts the wrong
output
and at least according to woodward and
these philosophers
the the general explanation might still
be a good one provided it covers
enough of the inputs in it range
accurately so you could expand
how many of these inputs it covers or
is supposed to cover
and note also here that an additional
factor to consider is that
it's not necessarily the case that an
explanation might
apply to only a single black box
algorithm and i think that's where this
notion of breadth of a generalization
can actually help us to get out of the
issue of
why isn't the function approximated by
the black box a good explanation
because this approximated function only
applies to that very specific model
whereas more wide ranging explanations
which might point to
features like models acquiring bias
or adversarial cases
might apply to much more than just a
single model and i think these are the
kinds of explanations
which informally you will see
researchers give
for a model behavior they will
explain that the the model led to this
particular behavior in these cases
because it acquired
bias there was some correlation between
financial data and income inequality or
a correlation between the training data
and the sex bias
in hiring at amazon for example
it's these kinds of very broad ranging
explanations where we point to
general features across models
which we find the most helpful so
for these explanations the
at least theoretical suggestion would be
that you try to look at how it applies
to more than just
the single model as well
that being said simply making
an explanation range as widely as
possible won't work either of course so
we have to counterbalance this with the
idea that
accuracy matters too so the number
of correct outputs that the rule will
give you
or the general accuracy of these
output these predicted outputs
so yes there will be different ways of
measuring this exactly a very simple one
would be to just look at whether or not
the predicted output is the same
or how much it deviates from the actual
outputs
so to wrap things up a little bit here
on explanatory depth
i've mentioned here how abstractness and
breadth and accuracy
all factor in to how good or
how deep an explanation is
[Music]
of course there's going to be much much
more
that is going to be relevant for whether
or not an explanation is good and
i think here factors like cognitive
salience or
the precision that you have in the
contrast class
are are going to be relevant as well as
more
human computer interaction factors
[Music]
to do with our specific psychology
um but based on the at least
philosophical account here
um abstractness and breadth uh seemed
like
relevant not uh automatically available
in the
literature criteria to mention
[Music]
um so to conclude
um i've talked a bit here about
how we might define explanation in xai
and i've suggested that we could follow
this
account from the philosophy of science
by woodward
of having a general rule covering the
actual case
as well as one or of course in practice
much more
counter factual cases
and that not all those explanations will
be good ones but
that explanations at least with
variables at an appropriate level
of abstraction with a decent breath and
good accuracy
are promising from this philosophically
theoretical
standpoint and that
finally this also helps to focus on
contrastive explanations which is
something we know to be important but
can still be missing from current x
ai tools okay
thanks for listening and i
guess we can open the floor to questions
then catty
yes uh thank you stefan uh exciting talk
uh do you have any questions uh please
feel free to raise your hand
or type it in the chat or just speak up
i have something yeah go ahead and draw
sure so yeah really interesting talk in
perspective on how
abstractness and generality can aid in
this
exploration of explanations uh one of
well i think i have more of a comment
and i wonder what you think about it
uh do you think so to me all of this
sounds really great
and it sounds like we could we could
indeed use abstractness and generality
to try and
paint out explanations in a helpful
manner but i think the crux of it all
eventually ends up on the user needs
right so on this very last concluding
slide as well you say
uh in order to identify good
explanations
all of these would sort of depend on the
approach
depend on identifying the appropriate
level of abstraction
as well as a reasonable breadth and
accuracy
right and all of these metrics are
user dependent right so what is a
appropriate level of abstraction is
something that
is heavily reliant on the purpose that
you're putting the explanation to
right so what use uh is it's serving
so i wonder how uh you know using
abstractness and generalization
in the context of chalking out user
needs
could uh you know play a role in serving
explainable ai
so i think these two notions are quite
nice especially so if you can tie them
directly with user needs yeah no i think
that's a good point
one thing which might be of help here is
this
idea of having the the contrast class so
by saying that the explanation is
answering a question why is
the output this rather than something
else and then the
um the contrast you're looking for might
help you specify what level of
abstraction you're interested in
um so i think if you're looking
at a a contrast class with a low level
of abstraction so why is it
um red rather than blue you might have
to focus on a more specific
explanation as opposed to a wider
ranging question
um yeah i should be able to come up with
good examples here but
i i think that hopefully the idea is at
least somewhat clear that this
by by picking which contrasting examples
you're trying to explain
so how the current output differs from
whatever output the user might expect
that you might be able to determine to
some extent at least
what level of abstractness and what or
what what kind of explanation you are
supposed to be offering
um i the the there's a further question
of how much this depends on
individual psychology um
if you look at the sort of philosophical
literature on this then they tend to
take a very objectivist
stance with the idea that there is going
to be some kind of
level of abstraction that is the most
appropriate for virtually everyone and
there is going to be a lot of agreement
uh on which explanations we think are
good ones and which explanations we
think are bad ones
um so this this is probably taking it a
little too
simplistic because they they just assume
well you have the relevant background
knowledge and so on
um so there's going to be difficulties
there with implementing
and i think you're very right to point
that out um
but but yeah i think some of it might be
mitigated with these contrast classes
not that that's going to be an easy
solution but there might be something
there
indeed so i guess you could you know
involve the end user in the loop by
asking them to provide
a desirable level of you know a contrast
to example or contrast
explanation they need right yeah all
right interesting all right
thanks a lot very nice talk by the way
thanks so well um next we have a
question from wiki but
uh unfortunately you cannot ask it uh
yourself
i'll read it for you stefan so uh how do
you consider explanations in ai systems
that don't directly make decisions
uh for instance perception based
applications of computer vision
smart cameras or speech recognition the
high google invoice assistance so i
think it's basically about the scope of
uh
um of explanation uh whether you explain
the decisions of a system or basically
just
mapping between input and output
yeah so i think it this this will work
quite generally just in terms of mapping
input to output so the
the context where this idea is coming
from is that you have
physical systems um where you want to
explain
why some one physical event happens
based on
previous physical events so for example
you want to explain why an apple fell
from the tree using newton's
gravity laws of gravitation um
and and i think we can do this similar a
similar thing here with
different ai systems so of course if
they make decisions that might be the
most logical thing for users to ask
these white questions about
um but i think for for the smart cameras
or
um for speech recognition you might just
as well as why did the system
recognize this phoneme as this rather
than as this other phoneme or
why did the the camera recognized this
part of the image
as a fridge rather than as a table
um and and then you would have the same
question about
the correlation between the inputs and
the outputs so across counterfactual
cases
so my hope would be that this applies
quite generally um
if there are challenges there i i'd be
interested to know
um but yeah i i hope it's quite a
general
framework at least good thank you
uh so okay i hope that answered the
question if not please follow up in the
chat
and then next in line we have gm
thank you thanks uh stefan uh
it's really nice to to see the full
story
and it makes me uh a little bit more
confident about the thing that i'm doing
it seems that
because it maps all the things that you
have talked about maps very well to the
to the to the project that i'm working
on right using concepts to try to
explain what exactly a
computer vision model has learned um
[Music]
also my question is that i can i can
already imagine right
to set up a a computational pipeline
that that can be used to and to really
leverage what all those ideas that you
have right okay
uh to to what levels of depth we needed
to get the concept right there for the
explanation and
how do we collect all those uh
counterfactual examples right to verify
that the model has really learned
something
um but what i'm i was wondering is uh
would this uh like more uh
computational pipeline really contribute
to the theory would that be helpful at
all
well i i think there's a reason we're
already working together right jay
so yes no i think that's uh i think
you're exactly right there that this
kind of
concept-based explanations are sound
like a very promising way of
working out this this theory um you're
you're also working on correlations with
between input and output and
um these will automatically cover
counterfactual cases so i
i haven't given
um a lot of the detailed discussion here
about other existing
xai tools but there are definitely tools
out there that already
give you um to some extent a rule that
will cover
uh counterfactual cases um
so lyme will do something like this
um even though it won't be very general
explanations
so so i think the the theory is um
in my case explicitly aimed at making it
useful
for these generating these these tools
um so the the idea here is to try to
offer what type of information might be
relevant and what
factors to consider and it's very nice
that this maps on so
naturally to what you're already doing
but that should be the end goal here of
doing the philosophy that we
get some more detailed specifications of
what the xai tools should be delivering
[Music]
thanks that's good to know
great thanks for the questions here and
then we have a question from eugenie
yes thanks uh hi stefan thanks very much
for uh really interesting talk a really
interesting look at the topic
um so i wanted to ask you what are your
thoughts on the following that uh that
maybe
if given how important that is and given
this account let's say that we want to
follow
uh this kind of account of what's what
is a good explanation
so before considering building an ai
tool in a given social context
shall we maybe we should first determine
what is a good explanation in that
social context
and only once we have determined that
then decide what are the means to
provide such an explanation so
and then approaching that from a
socio-technical point of view thinking
what kind of social infrastructure
organizational process policy
jointly with any kind of technical
artifacts that can be part of that
infrastructure could actually provide
that kind of good explanation but then
we we start from
what is a good explanation and then we
go to what kind of tools i'm curious
what are your thoughts about
that if you have any yeah no i think
that's a good approach
so um and and in general i would be
uh skeptical of trying to dive straight
into this and saying this is the
final answer on what good explanations
are so
i think it would be relevant also to
first
just do human computer interaction
studies on whether
explanations of this type are actually
going to be more helpful than
single counterfactuals this hasn't been
studied yet and i think it would be very
important to do this and to also
see if indeed this philosopher's
intuitions that abstractness and
breath and so on are going to be
relevant factors for good explanations
if they actually
bear out it wouldn't be the first time
where philosophers have just
been exchanging intuitions to the point
where you start confirming your own
theories
um so so i think that kind of study
on if these are good explanations would
be good
and and your suggestion to start
generally in a context with okay
we want to give explanations what is it
exactly that we're aiming for
would be a really good basis before
starting to develop tools which you
aren't going to be sure
will actually help for the problem
you're trying to to solve
um so yeah i think that's an interesting
approach
to to start with the question what are
good explanations actually
in whatever context you're working with
and moving on from there and and
hopefully this framework will help a bit
to do that because right now
thanks because right now it often the
case is that we have some kind of tool
in some context and then we kind of we
want to reverse engineer and see well
okay how do we
try to better the tool to provide a good
explanation whereas
people read the tool correctly or that
they interpret shapley values in the way
you want them to
yeah yeah because what what really
your presentation you think you really
nicely gave an overview of
a rich body of work that exists out
there into the topic of explanation
and that makes me think well given also
how
um how crucial this is right when it
comes to real life consequences
um that if we actually if we if we kind
of
reverse this around right and say no but
hey explanation
is a critical requirement to have in a
context regardless of whether we're
going to use ai or not
so let's first try to settle to some
good extent what kind of explanation
and maybe that inspires a better social
process around it and actually maybe
that inspires
us to a new innovation in terms of the
ai tools themselves
if we kind of flip this around yeah
thanks thanks very much thank you for
your point again
then i will read up a question from luke
who is also
not in a good talking environment so his
question is
is it reasonable to expect that a very
smart human will be able to
understand the most simple explanation
that can be provided
um because i'm not sure i do
no but i think the so so if i'll
interpret it a little bit so
maybe the simplest explanation on
why the entire model is behaving the way
it does might be very complicated
because it's still a very non-linear
model and
so the the simplest explanation for the
full model behavior might be
quite difficult still um and and i think
that's a good point
so um one of the
the suggestions there might be is that
we um
generally don't uh might not
or well i think this is uh actually
maybe something up for debate but we
might not
need a full explanation of the entire
model
for any particular why question that we
have
so it could very well be the case that
we can
manage to answer our explanations
without these
full global explanations of the entire
model if they
turn out to be too difficult but we that
we can offer more focused explanations
on the different
parts so i think that the easy cases
here to
um focus on is is say adversarial
examples
or biased outputs
where we have a simple explanation for
specifically that part of the model
behavior
and so if we can somehow manage to carve
up
our model behavior into these different
parts then maybe that will be enough to
understand it
i think that's an open question um so
it's it's a good point to raise
um on on if we will ever be able to
under to fully understand these
um but it's it's something to
work to strive for anyway
i hope luke you're happy with the answer
i don't have any other questions in line
perhaps someone wants to just ask
impromptu question
yeah can you go ahead oh if nobody else
would like
so i um yeah stefan
maybe you touched upon this in one of
the previous responses but so i was
curious
given that many current ai tools operate
based on correlations
um so our attempt to explain their
behavior
cozily um that
so at least on at first glance that
seems to me to run into a kind of guess
work zone
that uh like we we then try to interpret
it
and and we think as humans a lot of our
uh daily interaction with the world is
based on a causal
interpretation so we kind of we try to
interpret it but these interpretations
there might be actually off right from
what this correlation based model does
so um can you talk a bit about
how do you see this yeah sure so i think
there's a
there's a an important difference here
in terms of
which causes you're interested in so
um juan for example in our group who's
also working on explanations he's
interested in how you might give
scientific explanations of phenomena
in medicine using ai tools
so he wants to get to these causal
relations that are out there in the
world between a medicine
making you better or not using the
correlations that ai will spot
um and and that's i think related to to
your
question a bit um so so that's one goal
you might have and that's going to be
tricky to
infer causal relations out there in the
world based on the correlations that the
ai models are working with
the a separate question here and that's
the more the sense of
causal relation that i was
interested in is that there is going to
be
a cause namely the you put some
information into the model
and there's going to be an effect the
model outputs something
and this relation basically just the
model behavior
in any particular computer it will be a
causal relation with just
computations being carried out by the
cpu or the gpu
that causal relation is the one i was
interested in explaining
so that's a different one from the
causal relations out there in the world
but still a very relevant one because we
have the input to the model as a cause
and then the
model output is the effect and it's
based on these correlations in data
which is what we
what we want to get out of it but yeah
the the causal correlation talk can get
very confusing if you don't
keep this very strictly separate
i say thanks very much thanks for
clarifying that yeah because i was uh
i was thinking about it in terms of the
other one yeah yeah but that's a really
interesting perspective i mean because
that itself then
i could see how that can even if you
cannot then causally explain
the underlying thing that the tool
addresses that can spark
meaningful deliberations among
stakeholders yeah
yeah thanks
thanks again again so stefan i have a
very
general question going i guess beyond
the
topic that you explored but maybe not so
uh it has to do
with one of the points that evgeny
touched upon the fact that
ai systems are actually parts of
more encompassing human a systems like
complex social technical perspective
complex systems
perspective right and then uh what i
think is that uh
most at least yeah i i'm only very
familiar with the explainable ai
literature but it does seem to me that
the focus there is pretty much
on the ai part of the system and my
question is broadly how do we go from
the explanations on the on this level on
the ai agent level
to explanations on the level of the
system as a whole and then a good
example could be
for instance this uh infamous case of
a uber test vehicle crashing in arizona
and well hitting a pedestrian
on the road and the pedestrian died and
then there could be multiple
explanations for why that happened
starting from that the emergency braking
system that was built in the vehicle was
disabled
so that it doesn't clash with the av
components of the system uh then going
on to computer vision
saying that yeah basically the
investigation
showed that the vision system of the car
falsely identified the pedestrian as a
bicyclist
or as an object and then it switched
over
a couple of times so that could count as
an explanation for why that happened but
uh i think
from the meaningful human control
perspective we are more interested in
uh what kind of human decisions have led
to this specific
outcome so that uh yeah it's very easy
to say
yeah ais bad ai is to blame but
we're not really interested in who is to
blame but what
humans need to do next so that this
doesn't happen again
and i think uh this is a completely new
completely different
uh way of approaching things as opposed
to what people in
uh xai are doing who are trying to say
yeah how does this
neuron in your network affect the
outcome of the system
um so what are your thoughts on this
yeah
i think that's an interesting uh
question
um so to to some extent the
the examples you were giving might be
captured with these different
uh contrast classes so why did the
uh car hit the pedestrian rather than
brake beforehand and then you would have
the emergency system or
why did the car hit it rather than
identify it as a pedestrian and
and stop and and then you would have the
identification system to
which you can point to um so so i think
partly this is just the fact that there
are several
um facilitating causes uh and and
the the difficulty with any of these
explanations is to find
all the relevant ones um and then
i think you're very right to point out
that the people in the car
are going to be part of this uh
sufficient causes for causing the
collision
um or necessary causes maybe even
yeah but also there are humans i mean
when i say
complex humanitarian system it doesn't
mean that just the driver the potential
designers and legislators the arizona
state authorities which basically
allowed that to
happen eventually so yeah yeah so so
yeah um i'm i'm not sure if
the the kind of talk i was giving here
is going to address that issue in any
way
um i i mean
theoretically you could probably fit in
all these different
explanations that you're after into it
um but just saying well look for some
kind of covering rule with
counterfactuals isn't going to tell you
which
explanations to look for or more
specifically which questions to ask
um but i i think it's an interesting
idea here to
draw the types of things that you want
to explain more broadly
than the current xai literature might be
looking at
so and that's an interest that's going
to be an interesting challenge to look
at
yeah okay great i think uh
we are running out of time so uh if
there are no burning questions
wait a minute no nothing so thank you so
much stefan that was a very interesting
talk and uh i really enjoyed the
discussion
thank you everyone for contributing to
that
okay uh then i think this is our last
meeting uh for
uh this summer and i think we will
meet all of you starting from september
again
thank you bye