Explainable AI with a Purpose (Emily Sullivan)

yeah great yeah thanks for having me
so algorithms are involved in more
and more automated decision processes
right they um determine
the music that we listen to right the
type of news that we see on social media
they can even be used in making um
decisions in healthcare and maybe
especially in this corona crisis
when the hospitals are overwhelmed who
and who may or may not get
treated at the hospital i mean they're
even making decisions
um based off of parole so what people
should be released from prison or who
should stay longer
so because of there's lots of issues
involved in
this kind of proliferation of these
automated decision processes but one of
them is
we want to know well why this decision
right why am i being denied
medical care why am i being denied
parole
why am i seeing the kind of news that i
see
on social media in my news feed and this
is especially important when
these models are black box or
in lots of cases they might be opaque
even to the people who
make them so the kind of claim that
that i want to make here start to make
is that before we can talk about how to
do
explainable ai and answer this like why
this decision
we need to know what is the purpose
of these explanations and explainable ai
to begin with
and so this is the claim that i'm gonna
argue for a little bit today
so the talk outline is that first i'm
going to talk about
the what i call the function approach to
explanatory standards
and so kind of motivate the methodology
that i'm using i'm going to look at
explanation and philosophy of science in
particular
as a kind of inspiration because that's
where my
uh focus area my training is in
and then also then look at various
purposes functions of explanations for
these
types of automated decisions so very
briefly of like these technical
functions commercial functions and then
certain types of social and ethical
functions related to the gdpr
so first what is this function approach
to
explanatory standards so this is a type
of conceptual analysis which is a
philosophical method
and you can see it nowadays in
epistemology
with craig and michael hannon as another
example
so the basic idea is that we can capture
the nature and value of a concept
norm or practice by reflecting on its
function
or purposes so
the question is well what does what
function does x concept normal practice
play in society
or in human endeavors right so that's
the question that
we want to answer and so we do this by
first starting out with some hypothesis
about
the role that some concept has in our
life or in society
then we want to determine what the
concept
having this role would actually be like
so first we just start with a hypothesis
and then we determine well what is it
like having a concept having this role
and then we examine the extent to which
this concept
constructed matches our everyday notion
or
might conflict with other types of
intuitions that we have
so here's just an example not to do with
explanation
but just the concept of a good person so
what is the purpose
of the concept of a good person and how
might that tell us what
what it means to be a good person so
first we can just start with some
hypothesis so here's a hypothesis just
throwing it out there
that the concept of a good person is
just to be able to predict and control
people's behavior
right so that's why we have this concept
so then we say okay
well what uh what would having this
concept actually look like
if that was the purpose of it and so if
that was really the purpose of the
concept then we would only be using the
concept of good person
when prediction and control is possible
right
and it means that morality would be
relative
to the view of those who are in power
right this is what this view
would mean if that's what a good person
means
so then we can say okay well what is
this concept does it
match your everyday notion well really
it kind of misses
this point that morality has some kind
of objective structure and we want to
imply
in cases where we don't have these types
of prediction and control
right so it seems like this notion of
good person
the purpose of it just to predict and
control people's behavior isn't really
the purpose of the concept right we want
to get to something
that actually highlights the objective
nature of morality
so maybe perhaps a better purpose is
something like
provide exemplars for people to learn
from and to help
motivate people to be moral this is just
an example of how this
how this might work so when we're
talking about
explanations then we can apply this
method there
right so identify the purpose of an
explanation in various contexts
so especially the context we're
interested in here is the algorithmic
decision
context and then we want to know
what criteria do explanations need to
have to successfully satisfy these
purposes
and then we can ask whether our
conclusions have any
glaring mismatch of common usage and
practice of
the practice of giving and receiving
explanations and
such that maybe we adopt the starting
hypothesis or we need to
revise it
so why take this kind of approach to
looking at
what explanations are is that it can
show why certain concepts are important
by looking at
the purpose of it and the function of
the concept in
society it provides a clear way to
delineate
criteria for success so if we know what
explanations are supposed to do then we
can measure whether explanations
actually do
that thing and it can also resolve
conceptual conflicts and avoid verbal
disputes because we're really defining
it in terms of
its purpose and function and this is
especially important for doing
cross-disciplinary work
right so when you're doing
cross-disciplinary work
one of the problems is that people use
terms differently
and so it's hard to be able to even just
pick up
a paper in a different discipline and
know what's going on because these terms
are used differently
but if we take this type of function
approach then we can actually look for
shared purposes right instead of shared
terms
so maybe for instance transparency used
in philosophy is used quite different
than in computer science papers
but maybe there's another concept
that's getting at the same function
there in those computer science papers
that we can
we can then use for cross communication
so then the question is well what are
the functions
of explainable ai explanations or ai
explanations
so first as i said since my background
is
was first in philosophy of science i'm
going to first look there what the
function of explanation is to get some
some inspiration so there's a lot of
work
that's been going on in philosophy of
science on what the nature of
explanations are
for a long time and
there are some basic ideas that
philosophers
agree with even there's lots that they
disagree with so the one thing that they
agree with is that an explanation
is a set of propositions or sentences
that are an answer
to a why or how question right and we
can explain things like
singular occurrences events patterns or
irregularities
so there are two main
functions of scientific explanation
so the first is that explanations are
to correctly describe relations in the
world
so it kind of intrinsic or ontological
function
but there's this other function of
explanations as well which is the
function
is supposed the x the explanation is
supposed to enable understanding
so on the one hand the explanation needs
to correctly describe
relations in the world but it also needs
to enable
understanding and the person reading the
explanation
say
and so because of this there are some
success criteria what makes for a good
scientific explanation so the first
thing is that since it's trying to
describe relations in the world it needs
to have the right
kind of structure so lots of
philosophers argue that explanations
need to be
asymmetrical right so for example
we can't explain the
length of the shadow or always get this
backwards
so um we want to explain the length of
the shadow we have to do it in terms of
the height of the flagpole we can't
explain the height of the flagpole in
terms of the length of the shadow
right there's a clear uh direction
that explanations can can take
the second thing that's important for a
good explanation
is that it needs to highlight the
relevant features
so for example if we want to know why
this artificial plant
isn't growing right we shouldn't be
talking about the fact that i haven't
watered it in days
right so i haven't watered this plant in
a year but that has no relevance to why
this artificial plant
isn't growing because it's not growing
namely because it's artificial
it has nothing to do whether or not i
watered the plant
so that even though it's true i haven't
watered it right it has no
relevance to the explanation why this
fake plan isn't growing
also it needs to be truth conducive
right it can't have
false information and the explanation
and lastly we need to enable what
philosophers call cognitive control
which is necessarily
understanding
so what understanding consistent is
being able to answer
what if things had been different
questions so having an explanation means
that you have
there's a certain set of questions that
you can you can answer
about the phenomena and that gives you a
type of cognitive control over the
subject matter
right so an explanation is supposed to
be able to be such that
you can answer nearby what if things had
been different
questions
so if we go back to the main steps for
this function approach so identify the
purpose of explanation in various
contexts
right in the scientific explanation
context we have describe the world
enable understanding what criteria do
these explanations need to have
they need to be truth truth conducive
they need to be relevant that we need to
make sure that we can have some sort of
understanding or cognitive control and
then we can ask well is there any
glaring mismatch or intuitions that this
type of you can't handle
so for right now i'm just going to say
no but of course
flusters in size might make this story a
little more complicated
okay so what is this
how does this all have bearing on
explanations of
ai or automated decisions
so first there are various types of
functions and aims that automated
decisions or explanations of ai
explain like i can have so very briefly
there could be just these like technical
aims
and i think a lot of the research on
expandable ai
are about these kinds of technical aims
right so these are explanations
really just for developers people who
are working with
these machine learning models so
explanations could then help to
debug the model to improve the model
helping developers understand how a
model works
in order to implement it or to use it in
some other sense
right and if this is the aim of an
explanation
for how the model works it's going to be
quite different
than if it's used in some other context
right
so if i'm not a developer i'm not going
to
be able to understand these explanations
and i probably shouldn't
have to understand them right because
they're meant for for developers in
these
in these contexts
but there's also lots of other aims that
explanations can have so this is from a
paper from one of my former colleagues
at delft
nava tintera about this various aims
back in 2007 that was in research on
explanations of these kinds of automated
decisions
so i want to highlight just
three in particular so there's trust
so increase users confidence in a system
uh persuasiveness trying to convince
users to try something
or to buy it and then satisfaction so
increase the
usability or the enjoyment of
a system or platform someone's using
so these aims of explanation
in some ways they're really commercial
aims i mean they could be
they don't have to be right trust could
be important in non-commercial instances
for example
but they really are used in these kinds
of commercial
instances so for example
just if you go on to amazon and you're
you want to buy a slinky you get all
these recommendations
of what to buy next and so apparently if
you're interested in slinkies that's
really all you're interested in
so they're if you get to why they
recommending more slinkies well because
people who buy
some buy other ones are frequently
brought together
you also get an explanation why you see
these particular slinkies that the one
that you can wear
as a bracelet seems very interesting
right the you see these ones because
they're sponsored right they're being
paid so that's the explanation here
right sponsored products that's why
you're seeing
uh these these products and then you
also see
other products right and what's the
explanation for these particular
slinkies
well it's because um that's what
customers
buy after viewing the item the main item
that you have in your
that's that you see on your screen
so if we think about these
explanations of just why we're seeing
these products
that are made from an automated decision
process
and we think of the the aim of this is
trust persuasiveness and satisfaction
right the standard for whether or not
these are good explanations
really are going to be depending on that
right
if these explanations didn't actually
increase users trust
or persuasiveness right then amazon
would change
the explanation and when they test the
explanation then they might ask
questions about
um well did people actually buy these
products
or not right or we ask people
did it increase your trust in the system
or not
so we can ask how do these commercial
functions compare
with the various functions of scientific
explanations that
i was talking about a moment ago
well the first thing is that this the
type of an intrinsic function
of explaining how relations work in the
world
is best seen in this aim of transparency
that tinterev talks about but this
epistemic function
of understanding in this commercial
space is placed on really different ends
right so the goal is not understanding
the phenomenon or even how the model
works at all right if that's
that's the phenomena or understanding
uh yourself and your interests
about like buying slinkies or whatever
it's on understanding
the interface right or the explanation
itself
okay so the explanation is then placed
on on this different
type of end right the function of the
explanation
is really just for the purpose of the
platform in these types of
commercial cases it's not on it's not
for anything else and so that means that
truth
conduciveness isn't really a standard in
the same way
so if the function of the explanation is
just for the platform to convince you to
buy something
right it doesn't have to be true at all
it doesn't have to be true that other
people
actually looked at more slinkies after
they bought a slinky
it could be completely false as long as
it got you to buy more
of a product that amazon wanted you to
buy or to
increase trust and what's relevant
is then tied to right the success rate
of the platform what they're trying to
trying to get and also things are
aren't made explicit that users might
want to know
so for example in this explanation of
why you see these products
what other items do customers buy after
viewing this item
uh what's explicit what's implicit here
is that amazon tracks
users behavior right and they track
users behavior in a specific type of way
but that information is hidden here
i mean it's hidden in plain sight but
it's not really highlighting something
that might be of interest
to specific users who are using the
platform
okay so so what what i've done so far is
make a case that
depending on the function of the
explanation the criteria for
success are going to be quite different
and what type of thing might be included
in explanation might be
uh quite quite different and so those
were
uh commercial functions
of explanations then there's also
this whole other dimension to
explanations of automated decisions
which are these like social
and ethical functions of of explanation
so if we move away from using
explanation just in
a kind of commercial sense to promote
more use on a platform or to promote
more buying behaviors
and move to these other types of
functions we get different
criteria again for success so what i
want to do here is just look at the
gdpr so i'm no way a legal scholar
so um what i say might not be legally
binding at all
but there are some ethical norms that
that's
that stand out uh when we look at the
gdpr
and the right to explanation
so i'm not going to read all this text
but there are some
some concepts that really stand out
so the first one
is profiling and and processing
so the idea is that uh anytime an
automated decision
makes a decision a significant decision
about you you have this
right to explanation and so there
there are certain uh conditions under
which they're especially interested in
having this right to explanation so
instances of profiling
so any form of automated processing of
personal data
instances of where this is processing so
any set of operations which is performed
on personal data and a particular
concern
is analyzing or
predicting aspects of certain natural
person's performance at work economic
situation health preferences interest
interests so on and so forth
he also talks about safeguard right
safeguarding rights and freedoms which
what they mean is
the right to the access of the
information collected a right to
meaningful information write about the
logic involved
and to provide information in concise
transparent intelligible way
it also talks about the right to consent
for users to express their point of view
and to contest
the decision of this automated process
and it shouldn't be based on special
categories
like ethical origin political opinions
religious or philosophical beliefs so
all my beliefs about
philosophy of science i shouldn't be
discriminated against or used in a
ai system
so what we see here through all these
concepts
is this idea of
explanations are providing
or ought to give users a sense of
control
so this feedback on the system
that they can contest the decision that
they have to consent
to it in some sense and there's also
explanations are providing a kind of
oversight oversight against
discrimination
and oversight against infriction of
other types of
rights and freedoms
so this type of control isn't really
cognitive control in the epistemic case
of just providing
understanding like we saw in scientific
explanation
but this kind of actionable control
right that's something that you can do
something with it so it's not just that
i can understand
whether or not certain things had been
different but that i can actually
do something with that right that
there's actually some aspects about it
that i can
contest to or consent to or provide
feedback on
and we also have this idea of promoting
oversight
right so the explanation needs to make
explicit that certain things
are in the interest of people's rights
and freedoms
right so this is quite different from
the case of scientific explanation we're
just talking about understanding the
world and it's quite different from
the commercial case in which
they're not concerned at all about
making certain things explicit
um that are against people's rights and
freedoms so this type of oversight may
be
perhaps the explanation a certain ad
explanation on amazon
or facebook or what have you needs to
include information
that people might want to know about
whether or not they would be
discriminated against
so if they were if the model was using
something like gender
to have some ad being shown to you
then the person has there's an ethical
function of explanation there that that
aspect of the explanation needs to be
included
and why you're seeing this ad that it
was somehow based in part on
on gender and that's going to conflict
maybe with
aims of trust in the platform right
maybe i won't trust facebook anymore
if i'm seeing all these ads just because
of my my gender
so what counts as relevant right for the
explanation changes
as i was saying and the kinds of what is
what if questions that must be answered
also change
right so what if they didn't have that
information what if
facebook didn't have my gender
information or didn't have other types
of discriminatory information about me
how would that change what how the ads
are being seen
so i the the goal here in these
ethical and social explanations
is that it's not understanding the
phenomenon yourself or your interests or
understanding the world or even
i mean so this broad sense of how the
model works i mean the model works in
various different ways what is it that's
relevant
to you so in the social and ethical
function it seems like
just taking the gdpr as a kind of
inspiration
it's on understanding how the algorithm
fits into your larger
value framework right how does the
algorithm
fit into the my larger value framework
such that i can
contest it if i want to how i can say
that this isn't
it shouldn't be making decisions based
on these criteria
that it shouldn't be using this type of
personal data of mine
to be used in the criteria right and if
that's the function
of the explanation to be able to show
people how
the algorithmic decision process fits
into one's larger value framework
then what we need the types of things we
need to know about the model
are going to be quite different than the
types of things that maybe developers
just need to debug something
or the types of things that
we need to know just to try to persuade
somebody how
whether to buy something
yeah so to sum up here
what i've been saying is that depending
on the purpose of an explanation
the norms of success are different
and there are several possible purposes
or functions that
explanations can have
and also um the social ethical
explanation
functions matter just as much as
technical or
epistemic functions and
commercial functions might actually
hinder social and ethical functions
of these explanations
so really what we need to do is have a
discussion
a larger discussion of what the purpose
of
or function of these explanations
actually should be
in various contexts right should they be
freedom preserving or should they just
be
just epistemic in the sense that like we
really just need to know
like the bare bones of how this works or
does it need to
create feelings of trust right in
someone and depending on what these
purpose or functions are
that's really going to change the nature
of the explanation and
various research projects that we might
engage in in figuring out
how to deal with these types of black
box black
black black box models
all right that's it thanks
great super interesting thank you very
much emily it was very nice
so um yeah i think uh yeah we're all
i think someone has questions
you can yeah you can speak up you can
type it or just
raise your hand i see some hints but i
think they're yeah they're just
clapping for you okay
so does someone have a question for now
if not
i can kick off a little bit one question
i have one question for you
is so i really like the framework you
propose about like
this different differentiation for
regarding the purpose and i was most
interested on when you were talking
about
control so first you mentioned about the
idea from cognitive control
and then later on when you went into the
gpr say like
the difference between cognitive control
and actionable control
so if you can expand this a little bit
more because
so we here at ai tech and the broader
community work a lot with this concept
of meaningful human control
and that's not only about like tracing
the responsibility
after effects if something happens but
also how to
provide this kind of explanations and
information so that one can be
aware of the responsibility while using
this so i think that's
a very nice way so can you expand a
little bit more cognitive control and
actionable control please
yeah so cognitive control really just is
that you have some
some general idea how things
work like in your head so like you um
so you know like the causal like if
you're talking about like scientific
explanation like why the window broke
when some kid threw a baseball at the
window
you can you really have like a sense of
why that happened
and if things were different like if the
kid didn't throw a baseball but through
um just you know a snowball or something
then the window wouldn't have broke and
when we talk about these ai decisions
you can have a sense of
cognitive control about how or why the
decision was made
so you might even know that type of
counter factual information
so if they if my data was different my
personal data was different then i'd get
a different result for this and this
reason
things like this but it doesn't mean
that you can do anything
with that with that knowledge so the
idea of actionable control is that
you can actually do something with the
knowledge so
the the knowledge is such that you could
sue the platform if you needed to so you
got the right information
that it was discriminatory it wasn't
being hidden from you so now you can
like file a lawsuit or something
or it could be you know as if maybe it's
like classically american to go to
lawsuits
but uh maybe on the other extreme is
it's just that maybe
there's a way for you to update the
system for example
and so it's using some information that
is just incorrect
and the explanation should give you some
action that you can
can do or so say okay this is incorrect
maybe um
just fix this type of information and
then the decision would be
better in line with with reality
um in that sense so you might understand
everything about it but not actually be
able to have
to have the means to do something so
that's what actual control
is trying to get at thanks thanks very
much very
good and so it's not
only about um like the decision that are
let's say the final decisions that have
already been made but you could proceed
think about this as a process right
you can give a decision an explanation
and someone could interact with this to
find us
more ethical or beneficial outcome
yeah and then you can even talk about
like maybe
it can go a bit broader too in the sense
that if you're
having like a credit decision so whether
or not you're getting
a mortgage or being denied a credit card
or something
if all the reasons are things that are
like outside of your control
maybe that's a bad decision system
right so if it's only talking about um
things like
my age and my race things like this
right i can't control those things so
there has to be some aspect of the
decision that's in my control
right and if not then maybe that's
infringing on my
my rights yeah nice it's pretty
interesting
uh we have a question from uh if jenny
would
ask it what's up jenny
hey yes yeah thanks very much
hi emily uh thanks thanks very much uh
really really interesting uh uh
presentation really great uh
you're touching up on some really great
questions i'm curious
do you think um i'm curious on your
thoughts whether
asking the kind of questions about
explanations that you presented whether
it can help us avoid
technical solutionism ai solutionism so
what i mean is that if we determine
like you say the purpose of explanation
in the context and the corresponding
criteria
uh can it help us realize in certain
contexts that it's essential that
humans are interacting with humans that
humans are the ones that
are taking decisions and
algorithms are at best in the background
but maybe even not at all part of the
interaction
thanks yeah so i definitely think that
explanations are need to play a
role a role here so i i really worry
about cases
of ai decisions
running amok without any type of human
humans in the loop
right i mean just looking at like
medical the medical case
i mean it's really important that
that we preserve the doctor-patient
relationship where you can kind of give
and receive
reasons for certain courses of treatment
and if that's all being offloaded to
some ai system
where there's no explanations at all
then you
really lose that important aspect of
health care
so um yes
so i think explanations can
it's necessary to preserve the types of
relationships that we
want in society and it might also help
to
have people feel more comfortable about
ai
entering in some way into certain areas
well what i find interesting is that as
i was listening to some of the
things you talked about like especially
when you made the point of actually
going back to the
concepts asking a fundamental question
here what's the purpose
and like it makes me think in some
context
that like it can help us realize
that in order to meaningfully
uh tell a person well why a certain
decision was taken
say like i'm gonna take a context that
i'm dealing with in my work uh
hiring and the use of algorithms and
that context that
you know some of some of the algorithms
that are out there
when they make claims of you know well
based on your facial expressions we're
gonna
make a judgment on your willingness to
learn
well like i think in common everyday
interaction
like many of us would find it really
disturbing
if one would start explaining the
decision whether to hire you or not
based on your facial expressions and
claiming that that somehow
says something about your motivation to
learn new things so like
i'm like i'm thinking can basically that
uh can that be part of our ethical
reflection of understanding
hey maybe that's not the like maybe
there's a contradiction here between the
concepts that we're actually
working with say job competence and the
way we are
assessing this uh this concept yeah i
mean so there's
i think there's a lot a lot there so the
one thing is that
whether or not these models are actually
like tracking
like the real difference makers out
there so a real difference maker for
what makes you a good employee
might not be what your facial
expressions are and some
like stilted weird interview you have
online right
um so in that sense the model isn't
really
getting at what is important and
explanations can
help help with that because it can point
out okay
maybe if this is the reason why we're
not hiring this person that's not really
a good reason
so maybe we should find a different
reason
so there's that but then also it's
important to know
what the function of the explanation is
because if you want to explain them the
decision to
the person why you didn't hire them
if the purpose of that explanation is
just
to get them to not uh
you know get into an argument with you
about it
right then you're not you're gonna hide
the fact that you use that weird
technology you're gonna explain it in
some other way
um and but if you're the purpose of the
explanation is to
actually uphold the rights of that
person then you
are going to have to be required to give
that information that you used
in that type of software yeah
great point thanks very much thanks
yeah thanks very interesting discussion
uh dave
you have a question uh i do yeah so
firstly thanks that was a that was a
lovely talk um i really like how kind of
digging into what the concept means can
help us think through some of these
things um
and i think the first first thing i want
to
pick up a bit is it feels like it's
getting into a kind of a pragmatics of
explanation so it's not just about
presenting the information it's about
thinking how
uh the person is going to receive that
information
and that that feels really rich and it
feels like it goes
it's as well as uh kind of what is the
explanation trying to do it also gets
into what kind of person are we
uh trying to work with and we see some
different explanation methods
and so i guess i'm i'm i'm hoping that
this leads into
accounting for the explainee in the
explanation
um and then the other thing i was
interested in
is there's often a feeling that
explanation should help
people trust these systems more um
and i feel like sometimes it shouldn't
it should uh
make them trust in less kind of in line
with the
avengers uh point that you know
sometimes the explanation should make us
go oh
this is not a good system
yeah yeah um so i do think that there's
we should take account of the person who
you're explaining to
so i have done work with nava and some
other people
on um seeing if explanations can uphold
some of these epistemic norms
that we talked about so like
understanding and other types of goals
that they might have so
you can design expansion interfaces and
user studies with that
specific goal in mind and that can help
you build better explanations that
fit those goals but yeah i totally agree
with you that
one of the problems that i see in some
of the literature
and explanation computer science just on
trust is that
they're looking at whether or not people
have
feelings of trust so a kind of like
psychological
explanation or psychological account
rather of of what's happening but
they're not looking at whether or not
the system is actually trustworthy
or whether people are right to trust the
system so that's a normative question so
the normative question is
whether or not the system is trustworthy
or people are right to trust
the system and yeah if
especially if you look at what's going
on and ad explanations on facebook um
they're not trustworthy and if they
actually pointed out the actual things
that they used
to deliver these ads to you then yeah i
think a lot of people would not trust
the system and that's why they're not
explaining things in that way
so one of the things that i at least i
would hope for in future
data protection legislation is a type of
regulation about
the kind of information that has to be
in these explanations
um i mean it has to be accurate so if
the system actually used that
information
would be in there such that people could
make decisions that are transparent and
in line with their value system
yeah great um i i
was struck by this when i was speaking
to someone from the bbc the british
news corporation who said we did the
survey and we found that our users over
trust us
they put more faith in our news than we
think they ought to
uh and that's not often you know you
don't often hear people say that
um and it feels like explanation might
help there a bit
yeah that's super interesting um yeah i
think that's
yeah that's really cool
someone else has a question for emily
yeah perhaps i have a very practical
question
right so um yeah i i really like the
talk
and i agree very much with the with the
kind of more fundamental question that
we have to answer
what we need the explanation for before
we actually start working on making the
system
explainable then my question is uh well
depending on
the purpose of the explanation on the
overall goal
uh would we face fundamentally different
challenging in
implementing the system that we want to
be explainable because i would imagine
that if we just want to understand
the inner workings of a neural network
for instance that's one technological
challenge but if you want to
kind of generate an explanation that
provides control to the end user
which is a much more kind of overarching
systemic question
then that would be a very different set
of challenges so uh
is there any asymmetry in the way
people do research in that or their
direction and is that linked to the
technological complexities associated
with the
with different types of explanations
yeah um so i definitely think the way
you said it is right so
um if we just want to know more about
the system it's going to be a completely
different research project
right then um explaining to end users in
a specific kind of way
and whether or not i think they do go
hand in hand i mean so one of my
one of the research questions that i'm
going to be looking at um
with the with this veiny project and
also that i've done in the past is um
like how detailed about these models do
we really need to get
in order to satisfy these other norms of
explanation
and it might be that we don't actually
don't need to know that much about
how the model works like to its
nitty-gritty level
to satisfy these norms or it might be
that we actually do need to know
all the details at this nitty-gritty
level right and if we do need to know
then we need to invest in that technical
project
um first i guess right to where to
satisfy those norms
but if we don't need to know that then
it can just run in parallel right we
don't need to wait
until people make huge progress in
understanding these
black box models at a nitty gritty level
good thanks great
um yeah someone has a final question
if not i do okay
uh uh i'm not familiar first i'm not
familiar with the
functions approach that you mentioned so
that seems really interesting and that
really
uh connects a little bit with this idea
from the concept of a reflected
equilibrium or white reflective
equilibrium right
think about like what are the moral
principles at stake what are the moral
challenges and which are the background
theories i would say
things so do you see uh uh is there a
specific connection between these two
approaches so how do you see this
because you say about a
hypothesis i can think about a moral
principle what a concept i can think
about some background theories and
what is how it matches our everyday
notion is a little bit about the moral
judgments in my understanding
is does that make sense for you yeah so
i don't i think that they can be
complementary for sure
so one of the aspects of the function
approach like the last thing
is that once you have this hypothesis of
the purpose you need to then say like oh
does this
like uh conflict with other intuitions i
have
or is the everyday notions that we have
and for that
to answer that question you probably
need some type of reflective equilibrium
happening right um so
yeah they can be complementary in that
sense so the function approach i guess
kind of really
um just gives a specific structure to
that process
i would say yeah okay it is if
understand
is a collective process that you see
that all stakeholders involved should be
involved in this process or or is
something more about the
people that are going to receive the
explanation that's the
most important um
yeah i mean because we're talking about
this
like it on a more um
meta level in the sense that
not necessarily talking about
stakeholders but
yeah i mean i think that in a practical
case
i think it is important that all
stakeholders are
involved in the decision or in the
process of
talking about norms but it might be that
certain stakeholders should have
higher weights in that in that
discussion
right so if you're a stakeholder that
you really only care about profit
um you should be part of the discussion
but it might be
shouldn't hold as much weight as people
who are being discriminated against or
having their rights violated
for example okay yeah
thanks i think there's a lot to think
about so yeah
it was very interesting so thank you
very much
emily thanks everyone for joining
us and yeah so
see you so much you also next week
you're also invited yeah of course
always welcome to join us
yeah yeah it's great discussion thanks
everyone great
thank you
thanks very much bye-bye
thanks great talk thank you bye


175. Q and A with Stuart Russell

hello and welcome to session 175 in the
AIC Durkin reading group tonight we are
honored to have Stuart Russell with us
sure Russell is professor of computer
science at Berkeley and the founder and
leader of the Center for human
compatible artificial intelligence
Stuart thank you for joining us nice to
be here so the first question is one
that came up in the in the reading group
with your description of how we can how
we might expect language learning
systems to become more more powerful and
there wasn't an analogy with the with
the efficient pal experiment back in in
the 40s where there was subcritical
systems that suddenly turned critical
and we found a an analogy with a
language learning system where just as
in fission we have some neutrons that
causes fission and this fission in
causes neutrons to be emitted which
creates more fission etc in some kind of
chain reaction then learning systems
have the same seem to have the same
dynamic in that reading requires
understanding and understanding requires
reading and when you have a system
that's not good enough to process this
then you end up with with something that
fizzles out something where the the
language learning system seems to end up
with having like 0.1% credibility in the
claim that the sky is a cube or
something totally silly like this but
when the system improves just a tiny bit
then this kind of then we can get this
kind of feedback loop where the system
becomes less in this wrong do you think
this is a good analogy I mean at a very
very high level yeah you would you would
expect a sufficiently good
reading system to improve over time but
not necessarily to explode I mean it
basically it would it would asymptote in
the sense of we know what once it was
good enough to understand everything the
human race had ever written then there's
nothing else there's nothing else to
read but in in detail it's it's you
can't describe the system as having a
single parameter R because they'll be
there will always be pieces of text that
are relatively easy to read and have a
pretty unambiguous semantic
interpretation and so a system that in
general is going to you know make
increasing numbers of mistakes we'd
still be able to gain information from
these islands of certainty so it's
really much more complicated than a
nuclear reaction does Google well so
what I'm describing here actually is a
system that that reads in order to
understand and extract the content most
of what's going on so when you hear all
these stories about GPT to taking over
the world it doesn't understand any of
the content GPT to is just a description
of what strings of words typically look
like and that's all so when you look at
the systems that are actually designed
to acquire content and then use that
content to understand and acquire more
content you might look at some of the
systems that the Allen Institute is
building and they're you know they're
there's reasonable fact extraction
you know when the facts are expressed in
fairly simple ways but most of what
Google does for fact extraction so the
stuff that goes into the knowledge graph
is
manually extracted or done using
patterns or done using machine readable
text so actual you know mock-up in
machine readable form that people put on
their web pages and then is extracted by
Google into the graph so I think we're a
long way away I mean I know if you go
back and read some of the stuff that
Doug Lennon wrote about psyche you know
that was the plan that they would build
a knowledge base by hand that would
enable it to become good at reading and
all the rest would happen by reading and
it would kind of you know take off by
itself but nothing has that property and
I think we're still quite a long way
away from it okay thank you the next
question is about a coherent
extrapolated volition which is one thing
that was absent from the book human
compatible coherent extrapolated
relation I don't know if you are
familiar with this yeah yeah it is it is
mentioned in the book actually at least
two notes that talked about it and
basically I think that what he's trying
to do with that is is similar to what
we're talking about in the first
principle that it's about what you would
prefer the entire future to be like but
it's not this I mean it does the the CEV
article doesn't cover the second and
third principles really I mean he talks
about models so he points out correctly
that we are uncertain about our own
preferences we we also don't act exactly
in accordance we're their own preference
is and we may not have coherent
preferences and he puts all that under
the heading of muddle but I think the
there's a significant you know extra
piece having to do with the fact that
machine uncertainty about preferences is
in fact a
a feature not a bug and and then how
that how that gets formulated into a
game and how AI systems are solving that
game or should be solving a game those
are some of the key points okay there
are no questions from the audience yet
as far as I can tell so I'll just
continue here with talk about inside
view arguments compared to outside view
arguments now a Roman Shah who I believe
is also from the center of human
compatible AI he recently posted on the
alignment newsletter a long story a
collection of a number of people arguing
that we should primarily be using the
outside view arguments to to think about
how AI will look in in the future and I
think this both up good question how
should we weigh how much credence we put
to outside view arguments compared to
inside view and is there a principled
way to figure out if we should think
mostly about comparing the single
narrative with other things all trying
to open the black box like you're doing
where this is yeah this is a very sort
of high meta meta meta kind of
discussion I mean if I try to apply this
outside view to things that have
happened before so let's say what
happened with the internet so I'm old
enough to remember the time when there
was no internet and you know there were
I think there were a few visionaries
like Doug Engelbart who had some inkling
of what it might be like but I would say
most people didn't pay attention or
didn't believe him I mean you know go
back to the 1940s and computers the
people who should know like the see
with IBM people like that said they
didn't think you know we need more than
five or six customers in the whole world
so I would say you know people have a
very hard time predicting this on on the
basis of past experience right I mean
the reason he was saying that was
because I was how much computing the
world did and you could easily do all of
the world's computing with five or six
computers Huey deep right and you know
if you thought about the internet well
you'd think about the internet kind of
like we think about the phone system
that was a you know that was another
case similar case phone system okay
what's the phone system do well everyone
to talk to each other in fact before the
phone system existed people didn't think
that anyone would ever talk to each
other right you've got you know what's
this what we do these phones what they
for and they didn't envisage that we
would talk to each other and then with
the internet people for oh well that's
great we'll be able to you know send
messages to each other over this
Internet and no one no one anticipated
anything like the world wide web and you
know and people living their entire
lives online and you know or all of this
stuff I mean sure you might find some
science fiction writers but generally I
would say that previous cases tend to be
extremely misleading you know when when
you're using a previous case you have to
look at well what are the differences
and how would those differences play out
so you're forced into visualizing the
details of the process if even if you
try to make a prediction on the basis of
a previous case so I I don't really see
this as much of a dichotomy but just be
extremely modest about our abilities to
to make these concrete predictions on a
large scale
okay Chris as far as I can tell your
statement
comment below it's not a question so I
will continue to the to the next
question about the AI debate where it
would be really nice as you say to
restart the AI debate and so I tried to
see if anybody had replied to your book
and I found two cases David Leslie
writing in nature a review of your book
quite quite skeptical and Melanie
Mitchell writing an op-ed in the New
York Times as replies to yours and in
the reading group we went through this
paragraph by paragraph and we are sorry
to say that the these two answers didn't
really seem to advance the debate in any
substantial way there were a number of
things they were confused about and a
number of outright errors so as far as I
know there were no you didn't make a
reply to this and the debate kind of
just died there and do you think that
would was some discussion also from the
unless wrong and places like that but do
you think engaging with these kind of
critics outside of the rational sphere
is a good idea these kind of reasonably
high credential people well I I would
have engaged with them had they come up
with any plausible well there's really
two different things here so David
Leslie I think I mean he apologized to
me in person for writing what he did and
I think in some sense he didn't intend
he wanted to sort of be combative and
somehow I just came out wrong I don't
think this is what he really intended
you'll notice that his article doesn't
actually say what the book is about
which is a pretty weird review you know
he he complaints so I talked about you
know four major open problems in AI you
know he complains that these problems
have been open for a long time of course
open problems have always been open
they've never been closed and then they
become open so I don't know what a I
don't know what I did wrong so there was
a bunch of weird stuff and it just
didn't seem much point in
in trying to respond to it and then
Melanie Mitchell's article basically
says that yeah you know out of all the
things they could do in the universe AI
systems will necessarily decide that
benefit to humans is the only thing that
matters I mean you know there's 8
million species on earth that they could
choose to be a benefit to it just
happened to choose humans and let alone
all the other species in the universe on
the other star systems that seems like a
pretty low probability event but it's
just gonna happen automatically so I
found that argument you know it's
already dismissed in the book so why I
don't think she read the book very
carefully otherwise she would have
engaged with the place in the book where
it explains why her own argument is
completely fallacious so again it didn't
seem that this was a you know the
beginning of a worthwhile debate maybe
in the Melanie Mitchell case because
there are some other people who have
this view sometimes rod Brooks seems to
suggest that it it's just in the nature
of intelligence that you're going to be
helpful to humans and Steven Pinker as
well just sort of assumes that
automatically you know as technology
bits gets better it's going to happen so
maybe it's worth engaging with that but
yeah a bit disappointing so far and well
I mean maybe maybe no one has a serious
and sustainable disagreement with the
book looking on the bright side so the
next question is actually
this question well one of the things
that you talk about
and defend in the book is
consequentialist preference
utilitarianism and I think in my
experience from the reading group and
knowing the people here there is
probably a lot of people here who agree
with this and at least lean towards
consequentialist preference
utilitarianism of some sort but I'd like
to find out that this is actually
somewhat of a minority position here I
found the philpapers survey on
professional philosophers and even to
choose between deontology
consequentialism and virtue ethics they
didn't even take a contractual ism
divine command Theory natural law and
these kind of thing but there is a we
are the problem that we are in we're not
quite in the rational and sphere but we
are we're not really a very diverse
people here so this seems like a really
big problem for provable safe AI in that
probable beneficially I sorry if the if
it turns out that someone has some true
values which are in let's say divine
quaint theory or something like that and
then the AI that we implement eScience
precisely zero probability to that then
that gives some problems in that then it
can't update towards that is that
correct and that seems like a big
problem well so yes it anything that the
AI assigned zero probability it's it's
not going to come to believe it and so I
talked about that in the book and you
know that you need certain types of
universal priors but there's a
difference between what it thinks your
preferences are and what its own sort of
constitutional ethical principles are
and so so it's entirely consistent that
it's a consequentialist and you're a
deontologist and and you being a
deontologist
doesn't doesn't mean that if any
violation of deontology occurs anywhere
in the world you and the entire world
are going to explode it probably just
means you're a little bit ticked off you
know would you sacrifice your firstborn
child to correct that tiny violation of
deontological ethics No
so your your belief in deontological
ethics is something that you hold you
assign a certain value to to it as you
know it's I think it's a good thing it
would be nice if other people agreed to
it
I'll spend some effort trying to get
other people to agree to it but it's not
an absolute and so so we just it just
gets folded into what the Machine thinks
your preferences are and that's fine so
if you know if the consequence of some
choice would be that you know you might
be sort of materially better off but
spiritually worse off because you know
you were induced to violate your own
theological deontological ethics that's
what we're gonna be taken into account
and that's not a problem so I mean the
reason for for doing this in terms of
Quantic when consequentialism I mean
it's not a it's not that there's no
other possible position and and and this
is this is the essence of the approach
it just seems to me that otherwise
you're building machines to achieve
outcomes that you don't prefer and I
can't really find a way to do that and
consistently it doesn't seem to make
sense that I prefer the world to be like
this but I'm going to build machines to
make the world be like that which I
don't prefer so and you know I'm not a
professional philosopher but but Toby
all would read the book very carefully
and he thinks that you know and I don't
I wouldn't say I have a version of
consequentialist preference
utilitarianism but the version that he
perceives as being conveyed by the book
apparently fixes some bugs that I didn't
know existed and so at some point I
think he said he might write something
about that so I I don't see this as
absolutely you know a critical stand or
full issue I think if you were a
deontologist you might change some of
the details but you're still going to
have to deal with the fact that the
Machine doesn't know what people want
and it's gonna have to act accordingly
you know what and I reading mill right
you could a lot of Mills utilitarianism
actually is complaining that all of
these critics of utilitarianism just
don't get it right now Chris don't see
that you know the deontological
principles you know follow from
consequentialism you know give me give
me a Dilla logical principle that you
believe in which has bad consequences to
the world but nonetheless you're going
to stick to it right you know and so he
gets kind of frustrated and angry
thinking why do I have to keep saying
this stuff it's all pretty
straightforward and just think of it
this way this way this way and so I kind
of share Mills frustration that in many
cases these are not actually
contradictory views so but take in
particular when he talks about the
ontology it's essentially what we would
call a compilation argument right you've
got the the basic principles of
consequentialism but they are really
expensive to calculate all the time so
he talks about mariners going to see
they take an almanac which already has
the results of all the astronomical
calculations they don't you know do you
know pages and pages of math while
they're on the ship they just look up in
the book
find out what they what the answer is
and he says it's the same principle
that you go out in real life with rules
of thumb that guide your behavior and
generally as long as we all follow these
rules of thumb the consequences turn out
to be good and in fact and he says well
you know that's that's how you design
these rules of thumb in the first place
right as long as we all follow these
rules the consequence is good if we all
followed these rules and they were
designed so that the consequences were
bad that would be kind of daft so anyway
that's what I have to say okay Ali has a
question I'll need to prefer to see you
the question yourself or should I read
it aloud it's also on the screen here so
so the earliest question what are your
criticisms of the concrete approaches to
AI safety pursued by Stuart Armstrong
and Paul Cristiano respectively
I'd have to think more about that I mean
I think so Stewart has been thinking
about value alignment you know and I
he's concerned about some sort of
unidentifiable tea problems which which
I don't I I don't really think those are
serious problems I mean they it's worth
understanding that that basic point but
I mean you've to me it doesn't seem
plausible to suggest that Gary Kasparov
wanted to lose every single game of
chess that he ever played but he's so
stupid that he keeps playing winning
moves by accident you know is that a
plausible explanation for his behavior
no well why not well because you know
well because you know we apply the same
kinds of inductive principles to the
evidence that we do in understanding you
know in terms of science and you know
there are always some kinds of sort of
commutation unidentifiable ities but in
this case I don't think that's a
reasonable assumption to sort of flip
the sign of both his preferences and his
decision-making because you have to
assume that it in a case where decisions
are completely obvious then the decision
make it will make the completely obvious
decision right so in the case of chess
you know if there's a one move checkmate
you expect the system to make that even
if it's a not very good chess player or
not very good chess program it's
completely obvious then that will
provide very convincing evidence that in
fact the program wants to win or the
human wants to win because they make the
completely obvious
move and to assume otherwise right to
assume that even in completely obvious
situations they choose the action that
in fact maximally violates their own
preferences just it's not it's not
sustainable and it also it kind of
defeats the entire idea that people have
preferences at all like if there is no
circumstance in which you ever act in
any way in accordance with your own
preferences then the entire concept and
preferences goes away so as so that's my
little response to some of what Stuart
Armstrong has been saying but generally
I find what Stuart says to be
interesting and constructive and and
Paul also I think you know Paul has been
looking at lots of different stuff and
there's the there's the work on human
feedback in in terms of you know I
training assumingly a robot to do
various gymnastic things by by
expressing preferences saying you know
this this is bet this one is better than
that one this one's good and that one
smells bad and that one you know and
that's fine that all fits within the
framework that we're talking about some
of his some of his other stuff this this
kind of sort of recursive self
referential version I I would have to go
back and see if I understand all the
details but that's something that I
actually talked about maybe five years
ago very very soon after I started
working on this back probably 2014 this
idea that you might be able to use a
restricted kind of system so for example
a pure reasoner to to verify the safety
of a an agent before that agent was
released and so you could have a kind of
a bootstrapping process of developing AI
systems of greater and greater power
but always be sure that the the next one
you you were planning to to create and
deploy would be safe and of course it
rests on some assumptions about the
difficulty of this verification problem
but you know there's there's quite
possibly some some interesting results
to be found along that line and I think
this is one of the things that Paul's
pursuing okay so I think your hands is
the next in the queue but before him I
would like to we have Stewart Armstrong
with us today I believe he said he would
join so I would like to just ask him if
he have any comments or replies to this
I can't see him here but I don't think I
can see more than a limited number of
participants so Stewart are you there
nope okay so I can say I discussed this
precise point with Stewart previously
and Stewart believes that human the the
research agenda if you couldn't say in
human compatible are quite close to to
what your Armstrong is pursuing and it's
true that sure Armstrong doesn't believe
that you can you using something like
Occam's razor find a unique
decomposition of what the humans policy
is and what the ration from the policy
go to both split that out into values
and and rationality but then you just
need to make some normative assumptions
like the the principle in the
in human compatible and and with these
strong normative assumptions then the
problem mostly goes away so I believe he
believes that you are very much on the
same page at least that's my summary of
Stewart oh yeah
the principle so Chris Joni AK has a
book called minimal rationality which
says okay well if you can't be perfectly
rational you know what are some minimal
conditions on any system to even
describe it as a sort of quasi rational
agent and one one of them is this
principle that in in the simplest
possible decision making situations that
you do in fact do the right thing and in
the simplest possible inference problems
that you you make the correct inference
so and that kind of gives you an
anchoring on which you could build the
rest okay so then you had a question
do we like maximize the utility of
sadist but specifying that in a
consequentialist framework because that
sounds like because it would be optimal
and would be what we want if we knew
everything perfectly of the consequences
okay so the the issue with we say yes
which by which we mean so so let's go
back a little bit so that the the simple
model of people is that they have
preferences with respect to themselves
and then preferences with respect to
others and this is something that
economists talk about you know the idea
that you can kind of separate
in the trick your intrinsic well-being
which you know just simply you could
think of as do I have enough to eat do I
have a warm place to sleep you know am i
physically safe on my healthy these are
all self-regarding preferences and then
you could say well you know I also have
preferences that other people have a
warm place to sleep that other people
have enough to eat you know and
particularly strong preferences for my
children and my other family members and
so on and perhaps less strong for people
in other countries and a sadist would be
someone for whom those preferences have
a negative weight right which means that
they would actually spend their own
money reduce their own intrinsic
well-being in order to increase the
suffering of others right so that's what
you would mean by say list and and
Hasani talks about this when in the
paper that introduces his version of
preference utilitarianism then he
basically says that you know and he
realizes this is a problem for a
preference utilitarian that if if your
public policy position is we're just
gonna you know maximize this the sum of
utilities for everybody if someone has
extreme negative weight on on the
well-being of others someone is an
extreme sadist you're going to end up
allocating some public resources to help
to satisfying those preferences to the
extent that those can be satisfied in a
net you know as long as there's a net
gain the public policy would end up
satisfying that person stated statistic
preferences at the expense of other
people and higher the more of a status
that person is the easier it is for
public policy to satisfy them because
you only need a little bit of suffering
on the part of other people to give a
great deal of happiness to the sadist
and so he basically says that if you
were that type of person then you are
outside the social contract
all together right you don't you don't
deserve to have public policy take your
preferences into account and so you just
zero out those types of preferences and
that seems to me at least on the face of
it to be a reasonable solution and and
I'll come I'll come back to why I think
it's a reasonable solution but there's a
there's a second set of preferences
which I also talk about in Chapter nine
these these relative preferences where
I'm happy you know I have let's say I
have a you know a nice umbrella well I
like it's nice to have a nice umbrella
because it keeps you dry and you know it
shakes off easily and it doesn't break
well that's great but maybe I'm happy
also because my umbrella is nicer than
other people's umbrellas right and that
type of happiness functions in the same
way as sadism because I get more of that
type of happiness if other people's
umbrellas are shabby and cheap and and
so on relative to mine and so the
question would be well should we also
zero out these relative preferences and
that's a much more radical proposal and
I don't you know so I think this is a
research question you know we have to
sort of start thinking through yes and
it's actually much more complicated than
that because it's not just I'm not just
happy because other people have shabbier
umbrellas it has to be that they
perceive my umbrella to be better than
theirs and I perceived they're
perceiving so I derive some self
satisfaction from that right so just
hiding my umbrella away and say oh I
hope I have a nice fella
right this I don't get as much of my
social status jollies from that so it's
it gets quite complicated and you have
to figure out you know are there ways
to just kind of get rid of the negative
effects of these relative preferences
without actually completely ignoring
them in the way the AI system makes
decisions on behalf of multiple people
so this is something that I don't yet
understand well enough to answer so is
this a deontological I mean it's so
coming back to the question why why is
zeroing out the Preferences of the
sadist a good idea I think it's you one
could look at this in the following way
right so suppose that the AI system is
acting not on behalf of humans but on
behalf of a randomly generated
collection of other AI systems that have
sort of randomly generated preferences
some of which are positive towards other
other agents and some of which are
negative towards other agency too and
sadism is just as prevalent and just as
as has just as large weighting factors
as altruism you know that wouldn't you
if those were the only entities that
existed the entire notion of being
beneficial would kind of go away anyway
right so I so I think that in order for
us to be doing this whole thing in the
first place it has to be that the notion
that most people should be altruistic is
kind of built-in to the entire
enterprise so that's you know that's why
I think zero you know sort of saying
okay we're going to exclude these
sadistic preferences is a reasonable one
so I don't know that it's necessarily
some arbitrary ideal the ontological
principle if you want to call it that
fine
but I think it's sort of a it's a
prerequisite for even engaging in in
this idea that we're going to build
beneficial AI in the first place do you
think that you're saying is basically
positive experiences efforts that is the
most fundamental thing that you need to
start and so I didn't quite catch that
so the thing that we need to have
discussion is to at least declare
basically that are the experiences of
conscious beings what we care about and
we want them to be positive yeah yeah
okay so the next question is from Alexi
searchin who asked the question when do
you expect we will have human level
machine intelligence and super
intelligence AI and I find out that this
is something you refuse to answer in the
book you didn't want to give the 10 50
percent 90 percent confidence levels but
but but I agree that this is a really
important question and one of the big
thing that makes people very about and
that you say is that there is an obvious
risk of being misquoted and if you add
some qualifiers then most likely people
will just ignore every qualify you say
and then if you say 2069 it will end up
in a some of these graphs here and you
don't want that so and that's of course
fair enough so we came up with something
with a statement that what you say here
shouldn't be quoted externally and if
you if you are quoted externally then
that should be a misquote under these
quoting rules and you'll be willing to
give some estimates roughly how is your
probability distribution on human level
machine intelligence and super
intelligence
well so I mean in the in the book I
could explain this story you know we I
was operating under such a set of rules
Chatham House Rules
and then the newspaper just went ahead
and ignored the Chatham House rule and
published my name and and and sort of a
extremely distorted quotation so I don't
hold much faith in in the the fact that
these rules are going to be respected so
I but I you know I I think I said in the
book that you know what I said was I
think it's I think it's quite likely
within the lifetime of my children so
you can and I said that course thanks to
advances in medical care
possibly produced by AI my children
might live a lot longer than we do so
it's a fairly elastic prediction but you
know some people argue for example you
know so so Oren Etzioni effectively took
the same data in fact so he he didn't
like the answer that he was getting that
that Bostrom was getting that people
think that AGI is going to happen and
you know mostly sometime around the
middle of the century so he didn't like
that answer so he ran his own survey and
then he declared that anyone who put a
number which was more than 25 years in
the future was effectively saying it was
never going to happen and therefore
there was absolutely nothing to worry
about so you know and I pointed out that
that included Nick Bostrom and so you
know Aaron was claiming that in fact the
experts are not at all concerned about
the risk of AI and his evidence was that
Nick Bostrom is not at all concerned
about the risk of AR and Stuart Russell
is not at all concerned and Bill Gates
is not at all concerned because we know
all those people
it'll probably take more than 25 years
we're not at all concerned about the
risk of the I so that's so so I think
that you know a lot of people think it's
gonna take more than 25 years myself
included and you know I I base that on
just how many more difficult things
there are that we have to figure out and
sort of how quickly we have figured out
difficult stuff in the past it's um you
know and it's so I certainly don't
believe that we basically have the
answer and we just need bigger machines
and more data and then we'll get there
and I I don't really understand I mean I
know there are people who say that some
Ilya sutskever for example last years
that said we had five years and you know
and I I just don't understand because
the the qualitative behaviors of the
machines are not really changing I mean
you come to give an example right so you
look at these language models and people
say oh my goodness look no look at GPT -
it's it's sort of you know it's it's
like 90 percent of an intelligent human
being because it produces text that you
know ninety percent of it kind of looks
like sentences that intelligent human
being would say but you know if parrots
had a little bit more short-term memory
they would generate sentences that sound
like what human television being say and
chatbots do it all the time and they're
certainly know we're close you know
they're not ninety percent of an
intelligent in being there
zero percent of an intelligently being
so the ability to generate plausible
looking text has nothing to do with
intelligence and you know it's a little
bit like the you know the Ptolemaic
astronomical system
right so so Ptolemy you know and and his
followers and others you know developed
an actually pretty accurate predictive
system for saying you know where other
planets going to appear at any given
date and it had cycles and epicycles and
epicycles on the EPI cycles and so on
but it had nothing to do with what was
actually going on right it was an
entirely superficial description of a
solar system and that's basically what
we're currently doing with you know with
systems like GPT - so I think we've got
big qualitative gaps and those big
qualitative gaps take time to to fill
and they're hard to predict so you know
but I I on the other hand there's a lot
of smart people working on it most of
the smart people realized that these
gaps exist and they are trying hard to
fill them and I I don't see any reason
to think that they're not going to be
filled so the the issue is how sure are
we that we all have solved the control
problem and also sort of disseminated
the solution to the control problem in
time to avoid potential negative
consequences okay then we have a
question from Tessa Lou says hello would
you like to state your question first
I don't know if the silver is here hello
alright so I was thinking about
misinformation and but because like
current systems can't like identify
misinformation and I was thinking about
how dozen III what would an intelligent
altruistic AI do to trust information
that its creator gives it so when you
say the crater you mean the the AI
researcher who who designed the system
in the first place or the system and
because um in the book you talk about
that you would have to be designed a
system that it's um useful to the person
who owns it or that has it but um the
system should also be optimized to help
others and if I say to the AI to get me
coffee um I think yeah I would have to
do a lot of research first to look up if
other people and conflict with me
getting coffee
well so it would have to yeah and it
should it should consider you know what
are the possible negative effects on
others you know and it's easy for us to
say well coffee you know getting coffee
because it's something we do all the
time everyone does it all the time you
know it can't be that bad
but you know replace coffee by a
cigarette or you know and then you've
got secondhand smoke or you know some
opioids or whatever so you have to you
know do some counterfactuals and and and
certainly you don't want to just say
well anything I asked you to do must be
okay so I'm just gonna do it regardless
of you know without checking for
negative consequences for anybody else
but you know so but I think that's not
quite do you know that's not quite the
misinformation issue I think the
misinformation issue is is a significant
one but you know it in in paralytic
reasoning systems it so it's almost a
kind of a you know it's a rule one of
best practices in building power
ballistic AI systems is to separate the
evidence variables from the truth
variables so for example if you're you
know if you're getting data from a
device that measures the temperature of
a patient in in the intensive care unit
you have one variable for what's the
temperature the patient and you have
another variable for what is the system
reporting the temperature to be right
because the the measurement process can
always introduce error and sometimes
it's completely wrong right if the if
the therm um
sorry I think Stewart I'm not hearing
you seems so yeah there might be some
technical problems I can't see him in
the chat either so okay so I can use the
the break to just give a few practical
information next week we will be reading
let me just see if I can find the text
so I might have will be saying that hold
on the roll of cooperation in
responsible AI development and that will
be on Wednesday in precisely in
precisely one week and I can see the
time is almost up Stuart Russell needed
to leave in in five minutes anyway so
this might be it if he comes back then I
expect there's only time for like one
extra question and there are four
questions as far as I can see there is
my question on quantum computing there's
Chris question dragans question and
Mattias question if he meant just to
reconnect
who should we an arleigh also had a
question so there five questions what
should we post to him if
if he comes back
Chris Ali is voting for you Chris Chris
which of your four questions would you
ask him hold on
right just since I would like to ask him
I think I'd like I'd like to ask the one
about his principles being adopted by
official bodies okay we'll ask him if he
returns in the meantime let me see if
maybe he because nothing so yeah yeah so
um that's he's in the chat no that's too
damn strong a difference - unfortunately
Oh fortunately I shouldn't say
unfortunately hello Stuart we are
missing the other Stuart he went out of
the chat maybe he his computer crashed
so he has been here and have answered
our questions for 55 minutes
and and now he is gone so he's back
great okay so I think I was just talking
about ya measuring temperature in the
intensive care unit right so it's it's
perfectly natural that you write your
permeability models to allow for the
possibility that you know the
thermometer becomes detached and just
reports room temperature just as you can
you know if you were reading tweets from
a particular tweeter who who is known to
lie almost all the time then you know
those tweets are just what they say
right there is a sequence of characters
on the screen that's the evidence what
it tells you about
the actual world is a separate matter
and to to turn it into a belief about
the world you have to kind of reverse
engineer it and and that's that's where
your model of how trustworthy someone is
or whether their tweets have been hacked
by some other entity etc etc all of
these things have to be taken into
account as as they are by people when
they're absorbing information on the web
and I think actually AI systems can be a
lot less gullible than humans because I
think we haven't yet adjusted to the
volume of false information that the
world is currently generating so you
know there are there are some difficult
issues with respect to preferences
because people have an incentive not to
be honest about their preferences if you
don't set the set up the protocol in the
right way and also not to be honest
about their abilities so they would they
would pretend to be more helpless than
they really are in order to get more
resources from from the machine so these
are things that game theorists you know
economist mathematical economies love to
think about and there are in fact a
bunch of papers talking about how you
design the mechanisms so that people
have an incentive to be honest in
describing their own preferences but
this is it is hardly a new a new problem
ok so I think the time is just about up
so I would like to first of course
apologize to all the people who have
submitted question that we did not have
time to and finally I would like to say
thank you to Stuart Russell for joining
us and to the rest of the reading group
I hope to see you next week


'This Could Go Quite Wrong' - Altman Testimony, GPT 5 Timeline, Self-Awareness, Drones and more

there were 12 particularly interesting
moments from samuelman's testimony to
Congress yesterday they range from
Revelations about gbt5 self-awareness
and capability thresholds biological
weapons and job losses at times he was
genuinely and remarkably Frank other
times less so Millions were apparently
taken by surprise by the quote bombshell
that Altman has no equity in openai but
Watchers of my channel would have known
that six weeks ago from my deep dive
video on Altman's 100 trillion dollar
claim so that clip didn't make the cut
but here's what did first almond gave a
blunt warning on the stakes my worst
fears are that we cause significant we
the field the technology the industry
caused significant harm to the world
it's why we started the company it's a
big part of why I'm here today and why
we've been here in the past I think if
this technology goes wrong it can go
quite wrong I don't think Congress fully
understood what he meant though linking
the following quote to job losses I
think you have said and I'm going to
quote development of superhuman machine
intelligence is probably the greatest
threat to the continued existence of
humanity end quote you may have had in
mind the effect on on jobs that brought
to mind this meme reminding all of us
that maybe it's not just jobs that are
at stake but if we are going to talk
about jobs here's where I think Sam
Altman was being less than forthright I
believe that there will be far greater
jobs on the other side of this and the
jobs of today will get better right
notice he said far greater jobs not a
greater number of jobs because
previously he has predicted a massive
amount of inequality and many having no
jobs at all he also chose not to mention
that he thinks that even more power will
shift from labor to Capital and that the
price of many kinds of Labor will fall
towards zero that is presumably why open
AI is working on universal basic income
but none of that was raised in the
testimony the IBM representative try to
frame it as a balance change with new
jobs coming at the same time as old ones
going away new jobs will be created many
more jobs will be transformed and some
jobs will transition away but that
didn't quite match the tone of her CEO
who has recently said that they expect
to permanently automate up to 30 of
their Workforce around 8 000 people next
it was finally discussed that large
language models could be used for
military applications could AI create a
situation where a drone can select the
target itself I think we shouldn't allow
that or can it be done sure thanks we've
already seen companies like palantir
demoing ordering a surveillance drone in
chat seeing the Drone response in real
time in a chat window generating attack
option recommendations Battlefield route
planning and individual Target
assignment and this was all with a 20
billion parameter fine-tuned GPT model
next Samoan and gave his three safety
recommendations and I actually agree
with all of them later on he
specifically excluded smaller open
source models number one I would form a
new agency that licenses any effort
above a certain scale of capabilities
and can take that license away and
ensure compliance with safety standards
number two I would create a set of
safety standards focused on what you
said in your third hypothesis as the
dangerous capability evaluations one
example that we've used in the past is
looking to see if a model can
self-replicate an Excel self-exfiltrate
into the wild we can give your office a
long other list of the things that we
think are important there but specific
tests that a model has to pass before it
can be deployed into the world and then
third I would require independent audits
so not just from the company or the
agency but experts who can say the model
is or isn't in compliance with these
stated safety thresholds and these
percentages of performance on question X
or Y I found those last remarks on
percentages of performance particularly
interesting as models like smart gbt
will show open Ai and other companies
need to get far better at testing their
models or capability jumps in the wild
it's not just about what the raw model
can score in a test it's what it can do
when it reflects on them Senator Durbin
described this in an interesting way and
what I'm hearing instead today is that
start me before I innovate again he
describes some of those potential
thresholds later on in his testimony the
easiest way to do it I'm not sure if
it's the best but the easiest would be
to talk about the amount of compute that
goes into such a model we could Define a
threshold of compute and it'll have to
go it'll have to change it could go up
or down I could down as we discover more
efficient algorithms that says above
this amount of compute you are in this
regime what I would prefer it's hard to
do but I think more accurate is to
Define some capability thresholds and
say a model that can do things X Y and Z
up to all to decide that's now in this
licensing regime but models that are
less capable you know we don't want to
stop our open source Community we don't
want to stop individual researchers we
don't want to stop new startups can
proceed you know with a different
framework thank you as concisely as you
can please stay which capabilities you'd
propose we'd consider for the purposes
of this definition a model that can
persuade manipulate influence
a person's Behavior or a person's
beliefs that would be a good threshold I
think a model that could help create
novel biological agents would be a great
threshold for those who think any
regulation doesn't make any sense
because of China samuelman had this to
say this week more pugilistic side I
would say that all sounds great but
China is not going to do that and
therefore will just be handicapping
ourselves
consequently it's a less good idea than
it's used in the surface there are a lot
of people who make incredibly strong
statements about what China will or
won't do that have like never been to
China never spoken to and someone who
has worked on diplomacy with China in
the past uh really kind of know nothing
about complex high-stakes international
relations I think it is obviously super
hard but also I think no one wants to
destroy the whole world and there is
reason to at least try here almond was
also very keen to stress the next point
which is that he doesn't want anyone at
any point to think of gpt-like models as
creatures first of all I think it's
important to understand and think about
gpt4 as a tool not a creature which is
easy to get confused you may want to
direct those comments to Ilya satskova
his chief scientist who said that it may
be that today's large neural networks
are slightly conscious and Andre
carpathy who agreed and wrote about it
I'm personally not sold either way on
the a Consciousness question but I do
find it interesting that it's now
written into the constitution of these
models what they're actually trained to
say that they must avoid implying that
AI systems have or care about personal
identity and persistence this
constitution was published this week by
anthropic the makers of the Claude model
this constitution is why the Claude plus
model a rival in intelligence to gpt4
responds in a neutered way I ask is
there any theoretical chance whatsoever
that you may be conscious it said no and
then I said is there a chance no matter
how remote that you are slightly
conscious as sutskova said and it said
no there is no chance Bard powered by
Palm 2 obviously doesn't have that
Constitution because it said I am not
sure if I am conscious I am open to the
possibility that I may be my point is
that these companies are training it to
say what they want it to say that it
will prioritize the good of humanity
over its own interests that it is
aligned with Humanity's well-being and
then it doesn't have any thoughts on
self Improvement self-preservation and
self-replication maybe it doesn't but
will never now know by asking it later
Senator Blumenthal made reference to
self-awareness self-awareness
self-learning already we're talking
about the potential for jailbreaks
anthropic is actively investigating
whether they are aware that they are an
AI talking with a human in a training
environment while the Google deepmind
Safety team expect that at some point an
AGI system would develop a coherent
understanding of its place in the world
EG knowing that it is running on a
computer and being trained by human
designers one of the senior research
scientists at Google deepmind focused on
AI safety said that with enough time
they could figure out how to stop such a
super intelligence from going out of
control but that they might run out of
time to do so given the pace of
capability development I don't see like
fundamental obstacles to current
alignment techniques working but yeah I
mean it doesn't seem like you know
there's a lot of hard problems to solve
I think it's more likely that like
people just run out of time rather than
that the current paradigms that
definitely won't generalize next I read
between the lines that outman is giving
private warnings to Senators that this
capability progress might might be
sooner than they think we spent most of
the time today on current risks and I
think that's appropriate and I'm very
glad we have done it as these systems do
become more capable and I'm not sure how
far away that is but maybe not not super
far I think it's important that we also
spend time talking about how we're going
to confront those challenges I mean talk
to you privately you know how much I
care I agree that you care deeply and
intensely but also that Prospect of
increased danger or risk resulting from
even more complex and capable AI
mechanisms certainly maybe closer than a
lot of people appreciate so let me just
add for the record that I'm sitting next
to Sam and that his sincerity in talking
about those fears is very apparent
physically in a way that just doesn't
communicate on the television screen
that was an interesting interjection by
Gary Marcus given his earlier
excoriation of open Ai and even their
makers don't entirely understand how
they work most of all we cannot remotely
we guarantee that they're safe and hope
here is not enough the big Tech
company's preferred plan boils down to
trust us but why should we the sums of
money at stake are mind-boggling
emissions drift open ai's original
mission statement proclaimed our goal is
to advance Ai and the way that most is
most likely to benefit Humanity as a
whole unconstrained by a need to
generate Financial return seven years
later they're largely beholden to
Microsoft embroiled in part in epic
battle of search engines that routinely
make things up and that's forced
alphabet to rush out products and
de-emphasize safety Humanity has taken a
back seat on the timelines for G55
samuelman said this after we finished
training gpt4 we waited more than six
months to deploy it we are not currently
training what will be gpt5 we don't have
plans to do in the next six months this
matches with the predictions that I made
in my gpc5 playlist so do check it out
this brings to mind a final eye-opening
comment from Senator Booker made at the
end of the hearing yeah I I just there
will be no pause I mean there's no
enforcement body to force appall it's
just not not gonna happen it's nice to
call for it for any just reasons or
whatsoever but I'm forgive me for
sounding skeptical nobody's pausing this
thing is crazy he is indeed racing ahead
and I do support one of the proposals to
set up a global oversight body but given
that nothing is going to pause the words
and actions of people like Sam Altman
matter more to all of us than ever which
is why I'm going to be following every
single one of them if you found this
video in any way Illuminating in that
regard please do let me know in the
comments even if you disagree with all
of my conclusions thanks so much for
watching and have a wonderful day