Responsibility for outcomes when systems are intelligent (Nir Douer and Joachim Meyer)

um and near will give the presentation
and i'll be
sitting in the back and if maybe i'll
try to answer some of the questions if
uh i think well uh help
any of it although he will do this
excellently by himself so
again thanks for having us and please hi
everybody my name is nier
i'm from the department of industrial
engineering at tel aviv university
and we were asked to present you some of
our work on human responsibility
in intelligent system this is a part of
my phd dissertation
and my promoter reserve professor jorge
mayer
and so let me begin
so intelligent system have become a
major part of our life and we can find
them
anywhere in transportation autonomous
vehicles in
aircraft in industry medical equipment
almost everywhere
as you look you see some intelligent
system or very
advanced automation and
with this system computers and humans
share the collection of information
it's processing decision making and
implement implementation of action
and this really raises the question who
is responsible
and to what extent so these are the
things that we try to
to figure out in our study
now in the past uh things were
much clearer the operator
was responsible for anything that
happened there during the operation
of course unless i'm an unperceived
circumstances
while the manufacturer was responsible
for
anything that related to system design
fault but
now there is a responsibility gap
because in the interaction with
intelligent system
human may no longer be able to control
the intelligent system sufficiently
to be rightly be considered responsible
or fully responsible for the outcomes
so there is a responsibility gap in the
interaction with intelligent system and
this
responsibility gap arises from the
combination of
a few factors
first as this system are becoming more
and more
complex there is transition to shared
and supervisory control
in which humans either decide and act
jointly with the system
or only monitoring the intervene if
necessary so
the level of control is shifting between
human and machine
secondly there is technological complex
complexity
systems that incorporate artificial
intelligence
have some kind of an opex structure
and the user and even developer cannot
always
um predict
all their behavior which sometimes can
be
very peculiar and unpredicted they are
kind of a black box user that uses
decisions a support system
based their decision is the decisions
on the information which is supplied by
the system
and hence the human decision process is
influenced
by what is presented to the human by the
machine
there is an issue of functional location
and i will talk about it later on
but it's a mismatch between the world
that we assign to the human and what we
uh
authorize him to do with the system what
is is normally not in automation is the
problem of
last world for example in airbus
airplanes
the airplane may limit the pilot from
doing
certain actions if the the airplane
thinks that the pilot is going to take
the
the aircraft outside of the safety
envelope
and there are also negative implications
of automation
on the user especially of advanced
automation and intelligent system
because this may lead to over-reliance
skill degradation and loss of
situational awareness
and this is not just you know puzzling
academic issue
uh it's very interesting for real life
especially if the outcomes of the system
can harm people
for example in autonomous vehicles or if
the system
is deliberately designed to inflict
lethal force
as may be the case with a little
autonomous weapon systems
so it's it's really an interesting real
life
question but before i proceed i want to
to explain what type of responsibility
we are
talking about because there are
different types of responsibility
so there is more responsibility which is
uh the duties of
all that i assign to a person when he
interacts with the system
causal responsibility is the
connection between action of a human
and the consequences of these actions
moral responsibility is the
responsibility to act according to some
moral code liability is the
legal responsibility usually it's it's
connected to
stuff like punishment and compensation
and capacity is the psychological
ability of a person to be
held responsible for his actions and all
these
types of responsibility can be looked on
uh
in retrospective manner in which i look
at past
events and try to figure out what was
different responsibility and prospective
manner
in which i try to predict what will be
the responsibility for example in
interaction with some intelligence
system
so if we look at the academic literature
about responsibility for example about
human responsibility with autonomous
weapon system
autonomous car you can see that there is
extensive
philosophical ethical and legit legal
literature
about moral responsibility and liability
and actually we we found uh very few
research on the subject of causal
responsibility
and examination of the this type of
responsibility from engineering
perspective
so our work really deal with a causal
responsibility
the ties between a person's actions
and the final consequences of this
actions
this is a this is related to the subject
that you are investigating which is
meaningful human control because
meaningful human control is related to
the notion that
it's not enough only to put a system a a
human in a system
in order for this human to have some
meaningful influence on the system
and if you look at the literature there
are many literature on the subject of
meaningful human control in many
systems like medical equipment an
autonomous weapon system autonomous
vehicles
but sometimes they are different in
contradicting interpretation of policies
regarding meaningful human control
and system designers lack models and
metrics to measure
how meaningful was the human control in
intelligent system
so as i told we are measuring we try to
measure
causal responsibility and a measure that
will quantify causal responsibility
can assist in evaluating meaningful
human control because
if i didn't as a human or operator i
didn't have any effect
on the outcomes of the system so i guess
my
involvement or control was not really
meaningful if i didn't have any effect
on on the system and the outcomes
so the measure of causal responsibility
can aid in the assistance of
assisting in evaluating a meaningful
human control
so in the back i by the way you can see
our faculty
it's very nice and the sunny actually
today i don't know what's the weather in
delft but here today it's
a 30 degrees and
in about 10 minutes you can reach from
the faculty to the beach
so we think what to do after this
presentation
but the research component
the the our research has the three
main component first we developed
a normative analytical model that
it's a mathematical model that i'll
present it's an essence
that explain or try to measure a causal
responsibility in intelligent systems
however unfortunately people not
do not always act optimally according to
theoretical models
so we also examined how human
actually behaved uh in laboratory
experiments so the thinking thing is to
to to observe empirical behavior of
humans
and lastly people might perceive their
responsibility
or their contribution to the outcomes in
another in another level that they
really contributed to the outcome so it
was also interesting to
to assess or to try to to figure out
what is the perception of responsibility
of human when they
interact with different intelligent
system
and when you combine all this research
component
we can figure up or try to figure out
the notion of human causal
responsibility and intelligent system
and all
its different aspects so i'll i'll talk
about
each of the component very shortly
actually we published quite a few papers
on each of the component and also we had
some
conference presentations so you can find
all the
fine details in the these publications
and i'll only explain the motivation and
the essence of each
component and we'll start with the
theoretical model and remember our aim
is to try to
find a measure that will happen to
quantify to put a number
how much did a human contribute
to the outcomes in the interaction with
intelligent system which this is the
causal responsibility
so our model is built as follows
we first describe the human and and
sometimes instead of
intelligent system in order to be
simplified i will call it
automated model automation but it's all
the same
so uh we we first start with by
describing the human automation
interaction
by a four consecutive steps of
information
and processing and it starts with the
information acquisition
information analysis action selection
and
action implementation and as in short
control and
other types of control the human and
automation
works together in each uh
part of these consecutive steps but
the level of automation can be varied
according to the
um specific systems that you are
describing in some steps can be
completely manual while some steps can
be
completely autonomous it depends on what
system you are describing but but this
is just
a schematic uh picture of the
human automation interaction so to that
we had variables that will describe
the information flow from the
environment
to the combined system of human and
automation
and outside to the environment and i'll
explain some of this these variables
so we assume that the environment
includes
n possible states that are different
from each other
and each state has a different
observable
parameters and these parameters can be
observed
either by the automation
or the system either by the human or by
both
for example if my um
state is an airplane let's say and i
want to detect airplanes in the sky
the automation may include radar and it
will look
at the radar signature of the airplane
while the human is incapable to
observe radar signatures so he will
search the
electro-optical uh signature of the
airplane
and also there is the sonic signature of
the
airplane so each state in the
environment has some
parameters that are observable either by
the automation or the
human and um
these parameters are acquired by the
automation
module and the human then in the second
step stage the
information analysis analysis stage um
both the automation and the human try to
figure out what is the
the the states that we are confronting
right now
what is the state and try to assessing
it and according to this
analysis and action selection process is
carried on
and finally from the combined action
selection by the
automated module and the human a certain
action is implemented
of course this is a very um this
graph portrays all the possible
information flows in
that are possible but in specific
systems
um the graph is much simpler according
to the design of the system and the
functional allocation between the human
and the system
and so it's not always so
complex and the implemented action is
also depends on the on the state
on the functional location between uh
human and the system because sometimes
the
the human and the automation may uh
decide
about a different action and then enter
the
the issue of last word who decides in
case of a conflict
or sometimes the automation can act
faster than the human can
interfere for example automatic breaks
in cars
that happens before the human can
actually be involved so
you by such figure of information flow
you can describe many types of systems
so we have here information coming out
of the environment
mixed up inside the combined systems
that includes the automation and
intelligent system and the human and
some things
is coming out of the system so
in order to describe this information
flow and analysis
we use information theory to analyze all
the interaction
and interdependencies between those
variables
and i know that some of you are not from
the
engineering background and i'm going to
use a lot the notion of entropy
and you don't really need to to go into
the formulas but
you should treat when i say entropy
as a measure of uncertainty related to a
random variable
when i have large and central
uncertainty i have a large
entropy so we have all the information
coming in and processed in the system
and coming out
and we defined the measure of human
causal responsibility
as follows uh
we look at the the implemented action
which is denoted by the
and the responsibility the the share of
the human contribution to
the distribution of the implemented
action
is defined by the conditional entropy of
z
given all the automation variables
divided by the original entropy
of the and this measure although it
looks
complex it's quite simple it ties
it measures the relation between the
outcomes
and the information um
parameters that are processed by the
automation and what is left
is the human contribution to the
outcomes
for example this measure there is none
sentence in
information theory that conditions only
reduces entropy because
when you know something that your
uncertainty can only be
reduced or unchanged so this is the
fraction that
ranges between 0 and 1.
and if it when it is zero
if the output of the system the
implemented action
depends only by the automation variables
in this case the the denominator of the
fraction will be equal to zero and then
i i know that
if i know what the automation did i know
what is coming out
the human has no uh didn't contribute
anything to
didn't contribute anything meaningful
for the outcome because if i know what
the automation did
i know for sure what is the outcomes so
the human causal responsibility for
these outcomes
is zero on the other hand
if i see that when i know the automation
and variables i cannot say anything
about the outcomes for sure
the uncertainty remains the same the
conditional entropy
remains is the original entropy
it means that the outcomes is
independent from the automation
variables
and in this case the human has a full
contribution
is the one that really determined the
outcomes of the system
so this is this is very intuitive the
it's a responsibility measure that
ranges between
zero and one and it measures the unique
contribution of the human to the process
how unique was the control
was the contribution of the human
and the use of a information theory has
many adventures
advantages because when i describe to
measure the flow of
information i need to assume nothing
about the rationality or behavior of the
human
or the underlying distribution and the
entropy uh the measure of entropy enable
to measure very complex association
between the outcomes and the
other parameters which may be non-linear
and non-metric so it's much more broad
than
uh pearson for example correlation or
other collection correlation methods
and it's very applicable to real world
system because even if i
if i know some nothing about the system
if i can measure the different
distribution i can
measure the correlation the correlations
using entropy
and the unique contribution of the human
so in order to clarify it a little bit
more we are still in the theoretical
model and to see
responsibility in our own eyes let's
look at very simple
uh example
so let's look at a binary classification
system
[Music]
or a binary alert system it's the same
these
types of systems are systems that
look for abnormal values and warn the
user that now i'm in abnormal
range or there is something wrong and
you can find
these types of alert system in
many many applications are very broad
you can find them in advanced control
rooms
in the aviation decks and in your car
everywhere so the aim of the system and
the human
is to identify and
in this case reject signal so we assume
for simplicity
in this case that the environment
includes
only two types of states
signal which happens in certain
probability and
noise and as i said before each one of
them may be measured
as some observable parameters that i can
measure
the alert modules look at the
the states or the the the parameters
that the alert
may observe and decide whether to issue
or not a warning to the user
and the human user look both at the
indications that are coming from the
alert system
and on the observable parameter and
decide whether to
accept or reject the state which
and i remind you that our our aim is to
reject
signal and the human really press the
button or do does the action of
exception and reset rejection
so the human has the role of doing
exceptional rejection but the decisions
that the human
uh takes depends also not by only on his
own information
but the information that the human gets
from the alert model
so it might be according to
to the performance abilities of the
alert module and the human
that the human realize fully for example
on the alert model and then although he
is the one that pressing the button
is just following the alert indication
and he has no real
contribution so we are able to measure
this
and to measure this and to to calculate
the entropy for
such a simple system it's very
intuitive to employ the the theory of
signal detection theory which
deals exactly with the stuff like that
and i know again
that both of you might know not know
signal detection theory so i'll just
point to
to two parameters that are important to
understand in order to understand the
numerical
outputs that i'll present
so in signal detection serial for for
this simple case of only signal and
noise
there is assumption that their signaling
noise
it's each one of them has some
distribution
over in observable measure this is the
signal strength
but there is an overlap so many times if
i look
at the observable measure i'm not sure
if this is a signal or noise there is
ambiguity
and the signal detection theory
separates between two things and it's
very interesting to
to analyze things in this way the first
one is the detection sensitivity of the
sensor it's the sensor's ability to
differentiate between signal
and noise it means that when i look at
the observable measure
how good am i to say okay this is noise
and this is signal
so this is one stuff that characterize
your
operational ability is the detection
sensitivity
the second parameter is called response
criterion
it's the motivation of bias to favor one
response over the other
and this one also incorporates things
like preferences
which are values for correct or
incorrect decision
and let me explain the intuition
it's possible that my detection
sensitivity
tells me okay i this is uh
i twenty percent that i have a signal
let's say that the signal is a tumor
a malignant tumor that i want to to uh
to discover to detect and according to
my detection sensitivity
there is rather small probability that
what i'm observing is a tumor it's only
10 or
20 percent but in this case the cost of
misdetection is very large if i'm
dealing with
medical situation of malignant worse so
i will be biased
to treat the the the observed
entity as a signal even though i'm not
sure that it's a
a signal because of the high cost on my
preferences
the high cost for miss detection in some
other systems the
the the emphasis is on reducing false
alarm because if you have many false
alarms
there is the phenomenons of uh koi wolf
and user
uh don't trust the system uh anymore
so the response criterion describes the
bias to favor one things over the other
so in the system that i portrayed if i
will
put numbers what is the detection
sensitivity of the
automated model what is the detection
sensitivity of the human
what are the response criterion and what
are the
signal and noise distribution i can plug
number
to all of that and really figure a
number that says okay this is the
human or average human
responsibility the causal contribution
for the outcomes in this
type of system and performances
so this is exactly what you see in this
graph and this is for the
first time in history you can see
responsibility
in your eyes and numerically so we
plugged some numbers
and what you see in this figure
just a minute at the bottom
the x's are the human detection
sensitivity and automation detection
sensitivity
[Music]
and for example
when the human detection sensitivity is
rather low
0.6 is rather low and the automation
detection sensitivity is very high 3 is
very high it's all measured in
standard deviations the result is if i
do all the calculation with the entropy
the result is that the resulting
human responsibility is zero which is
quite intuitive because if the system is
much much
better the alert system is much much
better than the human and human cannot
describe between states
he will always fall the indication and
recommendation of the alert system and
the human by itself
has now really a meaningful contribution
to the process i could replace him with
a robot
that will follow the alert indication
and i will have the same result
on the other end this is the bottom
right corner if the human has very good
detection sensitivity
but the system has poor detection
sensitivity the human say okay this
system is loud i cannot count on it i
will count only on myself and i will
follow
all only my own recommendation and then
the human
is actually responsible 100 percent for
the outcomes
and there is a mathematical proposition
that we prove
in the in the paper that they and it's
also very intuitive that
human responsibility decreases
monotonically with the
automation and detection sensitivity and
increases with the human detection
sensitivity because
if as the human detection ability are
becoming
better and better and better the human
will tend to assume more responsibility
and on the other hand as the automation
detection ability are becoming lower and
lower and lower the human will assume
less responsibility and
will rely more on the system and
actually
i'm not going to go into this right now
it depends on the ratio between these
two values
so this graph in this graph i i assume
that both the human and the automation
has
have have the same response criterion
they have the same incentives
and we also analyze situation in which
the human and the automation have
different response criterion
for example here the axis are the human
and automation responsitarian
i will not go into that but it adds
another complication because if the
automation
response criterion or motivation is far
different from the
users the user will tend to rely less on
the automation because it
reflects different incentives and he
will tend to intervene more and will
assume
more responsibility
so what can we learn from this
theoretical model
first we devised a new measure that
really quantifies put a number
of the on the level of comparative human
contribution
in determining the final outcome of the
system which is
causal responsibility secondly we saw
that the
human causal responsibility depends on
the combined
and convoluted characteristics of the
human the system
the environment and as i said the
convoluted convoluted
effects and what we saw in the example
that the
human contribution to the outcomes or
unique contribution to the outcomes
which i
defined by causal responsibility is
higher
when humans capability are superior to
those of the system
and when having different preferences
because in these two situations
the human uh rely less on the system
because he either
has better abilities than the system or
different preferences
so he relies more on himself and takes
more actions that are
different from the recommendation of the
of the system and that
contribute more to the outcomes
and this has a very um
interesting implication because as
technologies
develop and outperform humans in many
critical functions
actually the human unique contribution
diminishes so simply demanding
uh like there is very common demand in
many
system to always involved human in the
loop
but simply demanding that does not
assure that the human has
a meaningful part in creating the
outcomes even if
important functions are allocated to the
human for example even if
in my example the human had to press the
button of accept or reject
if he only relies on the indication from
the automation and never deviates
he has no meaningful contribution just a
mechanical means to to translate the
automation's
outcome to some action so
putting him in the loop does not really
help
so current policies that demands human
in the loop may create mismatch between
world responsibility and cause of
responsibility
and this is important because sometimes
you can hold
a human who is falsely responsible when
he actually has little real
influence and you may expose the human
to unjustify legal liability and
psychological burdens now i have a
dilemma because we have about 20 minutes
left so we can
pause for a few questions or i can show
some empirical findings and
we'll have more questions in the ends
okay uh let's we already have quite some
questions in the chat
but then uh how much time do you need
for i think i need another 10 minutes
okay then let's focus first on maybe
two questions on the model uh and for
that we have four minutes
so yeah i'd appreciate if uh who was
first
we'll skip david's first comment uh
which was posted when you
talked about the weather in tel aviv so
he said near that's just cruel
so uh yeah maybe no need to respond for
that uh
but then the next question is from jared
i have another comment for this comment
in our university the air conditioning
are still still tuned to winter
so it's 30 degrees outside and the air
condition is stand on heat
[Music]
so if you see me sweating now the real
rhythm okay but then uh
one uh question related to the model
actually is from garrett here do you
want to ask a question in person
yeah we'll do uh so thank you cool stuff
i i did wonder so it seems that you
assume that the contribution of the
human is independent of the state
uh well i could also imagine scenarios
for example where what the human
does is sort of uh complementary to what
the
system does where in some cases it's the
human is better or the performance of
human is better it's on
that of the system is better and
so the question i asked is
how do you take into account that that
responsibility may actually be different
from state to state
actually the
the example that i showed was a very
simplified example in order to put
graphs
but if you uh look at the
the main diagram that i presented that
figures up all the information flows
it may also portrait a states in which
at certain environmental states the the
automation is better and then the human
is inferior and the other way around
it doesn't limit anything it's just it's
just a picture of
all types of information flow so if you
measure the system let's say you do
empirical work which i i want to present
uh soon and you measure the
the information flow at many states
you will the model will answer that you
will have a certain state that the
the automation is better and then the
human is better but
okay the the the measure that i
i compute now it's the average
contribution of the human
so yeah this is a i will talk later on
in the conclusion
indeed it's averages the contribution of
the human on different
states so you are right at that point
it's the average contribution of the
human uh over
in many states along um along the time
but you can figure up also specific
states in which
the human has larger or smaller
contribution or responsibility
okay thank you and then the next
question is from rule
uh rule you want to answer
to ask a question yeah yeah it's related
actually so i was
i was before you started going into the
sensitivities i was just
reflecting on the metric itself
and i was wondering um that i was
considering two examples two decisions
uh a and b and so what would it mean if
the human responsibility is twice the
value for b
is compared to a and relatedly like
don't you
uh you were already alluding to this
when you were saying that this is an
average contribution so
don't you lose grip on actual causality
when you opt for an information
theoretic measure
like how can you tease out the
contribution of the human
um to the causal responsibility chain of
a decision or a situation
that's my question so
it's hard to to answer shortly but i'll
try to answer first
the the measure when i presented the
the responsibility types i told you that
responsibility can be either prospective
or it was with perspective
now the measure that i'm presenting now
is a
prospective measure it tries to to
to look at the future and say i'm
designing a system
what will be the average contribution of
a human in
such type of system i present in the end
of my
presentation that we have a deviation of
the models that look at retrospective
cases
in which case when you analyze a single
past event you cannot look at the
at the average contribution of the human
over many states because you have a
certain line of information that
followed from
the environment to the automation and
then to
the human and for that we have another
measure it's also
based on information theory but it it
measures
uh it's not an average measure it looks
at a
it's a certain specific case and
measures
the causal responsibility for a single
past event
but the measures that i'm presenting now
it's more like and
as i said it's an average contribution
and it's in a
prospective way of looking it's very
important for system designers or
when you want to do some
um legal or other consideration what was
how meaningful is the human world in the
system or the average
contribution i guess what i'm seeing is
that
the way i was interpreting is that you
can see whether the human is a necessary
contribution
based on the information right and not
so much
to what degree is the human involvement
uh
effects really is unique yeah necessary
but to what degree it's not like
necessary or not it's not binary
it's how how often or to what degree the
human contributes something
unique to this interaction that i cannot
just take the human out of the system
and
rely on the intelligent system by itself
yeah
no that's clear thank you okay so in the
10 minutes that left or less i'll
i'll go very briefly about the empirical
findings and then on the other research
that we are doing
okay near just a quick comment so uh
i know that many of uh people who are
currently in the meeting
can actually stay a bit longer after two
but
uh yeah if someone wants to stay that's
good uh
and just feel free to drop whenever you
need and i think we could just extend
the discussion a little bit
later so how much time you give me more
uh well as much as you need i mean yeah
be great
okay until tomorrow but
now it would be great to have at least i
would say i hope the beach will be
crowded in few hours
let's actually aim for 10 12 minutes and
then we will have five minutes
something like that and then i will give
priority to the questions of the people
who need to live soon
that's uh for you david because he
posted three questions but i know that
you are staying
so i uh i presented the the
the analytical model at the top of the
pyramid
and now let's look at actual human
behavior and
perception of responsibility and we
wanted to test
whether you know this is the theoretical
model and
does it really can it predict how people
really behave and contribute with
different systems
according to different characteristics
of the human and the system
so what we did we did the quite a lot
laboratory
uh experiments and you see here this is
our
interaction with technology laboratory
at tel aviv university
and now i need to to introduce more
more types of responsibilities so i
presented
until now theoretical responsibility
which is as i said prospective it's the
predicted
share of unique human contribution to
the overall
outcomes and as many
theoretical models it has simplifying
assumption like perfect human knowledge
rational humans that maximizes some
utility function and optimal use of the
automation
but unfortunately people don't use stuff
optimally so in the experiment we
measure we
measured what we called measured
responsibility
which is the observed the real empirical
share of the unique human contribution
to the overall
outcomes it's the same measure but we
just measure the distribution and the
information flows in the laboratory and
we calculated this measure
so it quantifies the same information
theory measure
and it's based on actual user
performance
and we also handed out questionnaire in
which we asked the
participant to evaluate how how
you think was your contribution what is
your assessment regarding your
comparative
unique contribution to the interaction
with the system
and what we did if you remember and so
we use the simplified
classification binary classification
system
as in the example and you remember this
graph from the example
and what we did we selected four
experimental points
each one reflects a different kind of
human or automation detection
sensitivity
and they span a range of predicted human
contribution from
12 to 47 to 69 and 287
so our first ex experiment or types of
experiment
we changed the characteristic the
detection abilities or detection
sensitivity of the human and the
automation
and we wanted to compare to the
theoretical prediction
in this experiment both have the same
incentive i mean when we programmed the
intelligent system and we told the human
the cost
and the benefit
metrics they were the same so they have
the same incentives
we did also other experiments in which
we gave the human and the system
different
different incentive but we also compared
them to the model um
prediction but i will not present this i
will i will focus on the first one
so in the deep prime or detection
sensitivity experiment we had 60
participants we devised a kind of a
simple binary classification system and
we could control
the system accuracy how good was its
detection sensitivity
and the human had to look at some
presentation and we could control
how accurate is the presentation that
the human uh
so so we control both we could control
both
the detection human detection
sensitivity and automation detection
sensitivity
and according to the graph we took the
point of
1 and 2.3 the participants
were half of the
participants had poor abilities with
detection sensitivity of about one
or quite good detection sensitivity with
detection sensitivity
about 2.3 and each one of them worked
with two different
alert system one alert system was
poor and the other one was good so this
was the within subject
condition so we have the poor human here
and he works or they worked with rather
poor or good
automation system and these are the
predicted responsibility values
and each participant performed 100 twice
with each system
and we counter balance the order of the
system and we also handed questionnaire
during the test different part of the
test
so we start with measured responsibility
and i'll explain this work because it
will repeat itself
many times below the x-axis
is the human detection sensitivity and
you have the less
accurate human on the left and the
accurate human
on the humans or participants on the
right
and the red is the accurate alert system
and the blue is the less archaeological
system
and as predicted by the rescue model you
could see that
um the former the formation of this um
the formation of this web is according
to theory because
you can see that both type of
participants
relied more on themselves
with the less accurate system and they
assumed higher responsibility
higher causal responsibility with the
less accurate system
and you can see that the accurate
accurate participant
assumed always assumed regardless the
type of the systems assumed higher
responsibility than the less
accurate participant because they relied
more on their own capabilities
so the general pattern is according to
theory but that we had the specific
numerical prediction so it's interesting
to compare
the average value to the theoretical
prediction
so you see that in most cases like let's
say
with the less accurate system the the
average
outcomes was not very far away from the
theoretical prediction
uh despite only one case when the less
accurate participant worked with the
accurate system
they assumed much higher responsibility
than
uh optimal and actually we analyzed
the reason for this is uh known for many
other behavioral
studies we analyze it in different
ways but what really happened that this
type of participant
overestimated their own abilities and
they intervened
more than optimal and thus they assumed
higher
impact or higher causal responsibility
on the outcomes
but the outcomes were better off if they
didn't intervene at
all so people with
poor abilities that worked with very
good system
wanted you know to to do something they
didn't want to feel neglected so
they did something but and they
overestimated their own abilities and
this
phenomenon is known from other
behavioral studies
if we look at the subjective perception
of responsibility and here
it's another scale because it's a
subjective scale in which they rated
their
responsibility from i contributed very
much
i had no unique contribution you see the
same pattern
and if you compare the results by
normalizing the scale
you see that subjective and measured
responsibility really matched
each other so it means that people
really
feel or have a good sense of how much
they really contributed to their process
um i will not enter this
but we also analyze the relation between
all the three components together and
we discovered that the subjective
feeling of
of responsibility encountered for about
20
of the way people actually behaved with
different systems so
the perception of of yourself of how
much you
um contribute to a to a process
influences to some degree it's not it's
not the main influence but it influences
for some degree
the way you act with the with the system
another interesting finding was you see
the solid lines it's
the graph that i showed you before
and we also asked their subjective
assessment if another person
was working with such a system what do
you think his responsibility
would be and it was a significant
difference that people
waited their own responsibility lower
than of another person in the same
situation
and this resembles um something known in
psychology as the fundamental
attribution error
and actually we we did another
specific laboratory examination for this
aspect
another dedicated experiment in which we
had the people that
worked with systems they were the actors
and people that sat next to them and
just
observed how they worked and we asked
about their
subjective perception of the level of uh
responsibility and there were
significant diff
difference between people who actually
worked with system
and people that just were observers
so the findings from the empirical
analysis it's that the rescue model is
also a descriptive model and
actually we can use it to to predict
how human uh will take responsibility
and what will be
their average responsibility in
different uh
systems and it also can predict the
subjective perception of how much the
people will feel that they
are that they contribute meaningfully to
a system
nevertheless there are two systematic
human biases
one is the tendency to assume
or to assume excessive responsibility to
intervene more than necessary
when the human capability are inferior
to the ones of the automation
and the other bias is a tendency to
consider other responsibility to be
higher than
one's own for yourself you always has a
good excuse
why your performance was poor and it's
not just
your problem it's the always the
automation faults
so the implications from the empirical
observations
is that operators may feel correctly
that they don't have significant
impact on the outcomes when they work
with the advanced intelligence system
and they may interfere more than
necessary or conversely
in some other cases that are known in
the literature
they'll be less motivated to take action
at all
both responses will hamper exploiting
the full potential of the system
and could lead for undesired
consequences
so again the demand to
always keep human in the loop can have
adverse implication on the overall
functioning of the system
the human attitude toward the system and
their interaction with the system and
the role
with it and the perception of outside
observers that watch the the human user
and assess their
responsibility for the outcomes
so this is the the end of the empirical
results that i want to
um to show just few
concluding rewards so
the measures that we developed all three
measures the theoretical
measured and subjective responsibility
can serve to expose anomalies and
provide a new method for quantifying
human comparative causal responsibility
for the outcomes in advanced system
they can help the design of the system
by tying different design options to the
predicted effect on user behavior
and their perception of responsibility
and also it it can aid in the formation
and analysis of deployment policies and
legal decision
regulation and i'll give you just a
short example for example in autonomous
weapon systems there is a
indeed a requirement by the us and
british authorities to always keep a
human
in the loop so there will be no
um um lethal force um
done inflicted without human involvement
human in the loop that will authorize
the action
but let's say that the human just sits
in a dark room he doesn't see anything
from the outside world and all he sees a
light bulb
that lights whenever the automation says
there is a tire
at a target that he needs to attack in
the human press
button that authorizes the attack so
in terms of of regulation we put a human
in the loop but
it's very clear that his involvement is
um
is meaningless it's not meaningful it's
in the loop
but it doesn't have really causal
responsibility or
unique causal contribution for the
outcomes
so it's this this way of analyzing
things may expose a
um policies or faults in policies and
legal regulations that
are not looked upon
so regarding future work and some of
this work we already
accomplished but it's on some kind of
publication
process first as people asked um
we need different measure for with
prospective responsibility because the
entropy measure it's kind of an average
measure that averages over all the
different
states and if i want to measure the
human responsibility in a single past
event i need to
to change the demands i'm not looking at
the
the the average contribution or about
specific contribution in a single chain
of events
so we devised an information theory
measure for that
and very importantly and we also work on
that
is the issue of temporal effects because
in all the examples that i showed you
the graph and the
the example that i showed i didn't
consider the time
but it's very obvious at the time you
have to
to take the the decision may impact
or should impact your um tendency to
rely on the system if you have a very
very
uh short time to decide then the system
is very good
you will tend to decide to to rely more
on the system and your
cause of responsibility will decline
so we need to to measure not just
the information flow but in information
transmission rates
and human channel capacity constraints
we which are also
known from the literature and then add
the time
element to the model as another
dimension that is currently missing from
the model
so these were my 10 minutes
engineer that was great
so uh yeah it's exactly two so i suppose
uh
many people need to leave if you have
any questions leave them in the chat
before we leave
and then we will have a chance to ask
them
or you could just raise your hand if you
have time to ask a question yourself
can i ask a question
hi everybody i was on um on voice for
the largest part of the time so i could
not see
the slides but if i'm correct in the
beginning of the presentation like 20
minutes or so into the presentation
the entropy measure was introduced and
then a ratio between entropies were
was mentioned did i get that correctly
or did i miss something
okay so being an information theorist i
was wondering where the ratio of
entropies comes from
because that's very unusual to take
ratios
of entropies we usually subtract or add
or or
that kind of combinations hardly i've
actually never seen
a ratio other than ratio to a maximum
like efficiency and these kind of things
so i was wondering
does it even make sense to think about
ratios of entropies
yeah so i have a good answer and this is
a good questions
i'll slide on here let's look at this
okay so this is the measure
below we can see the entropies of the
final
action action which is implemented it's
the entropy of the election and
the the denominator is the uncertainty
left
about the outcomes given that i know
for all the automation variables so
actually i measure the
relations between the
system outcomes and all the automation
variables and i
measure how much uncertainty is left
because
if i if i'm looking at all the the
the automation variables then there is
no uncertainty
i mean their identity is gone then the
numerator would be zero and the human
contributes nothing and actually
in the literature this form of measure
appear at what is known tails
coefficient measure
which is very known measure you can find
it
also in spss to measure the
relation between nominal variables
but he measures something else he
measured the
relative reduction in the uncertainty of
a variable
due to a knowledge of another variable x
so let's
look at the yeah
so if i if i may so i i hear what you
say but in terms of the explanation
this is very much like terms from
information theory like equivocation etc
which are usually defined as the
difference between entropy so difference
between
normal and conditional entropy etc so i
see that you can also take a ratio
but would a difference also have
worked the problem is different that
it would not give me a a measure that is
is confined to a range between zero and
one because the entropies could have
much different number and then it will
be
it will change in very types of system
without me
able to compare them if we look at tails
uncertainty coefficient and we have
x and y what else does he say
okay let's look at the mutual
information and look at
the uh mutual
entropy and look at the entropy of x and
if i divide this
mutual part by the entropy of x it's the
how much
is reduced uh
the uncertainty of x is reduced by
knowing why and i'm looking at the
complex
a complementary value which is how much
um uncertainty is remind
it remains about the output when i know
all the automation variables so i
look at the other part and i extend it
because i don't just look at the
two variables but i look at
[Music]
many variables but tails uncertainty
coefficient is used to
exactly that to measure the association
between for example nominal variables so
you cannot
do it with the pearson coefficient or
experiment coefficient because they are
not nominal and i'm not
metric so so
this is when you use the tails
uncertainty coefficient and i do i just
i do something
similar but i i'm not looking at the
reduction but i'm looking at the
uncertainty left
after i know all the automation
parameters and since there are no other
sources for information flows into the
system or the remaining uncertainty
is related to the human contribution and
this is the sense of devising them to
have a measure that ranges between zero
and one
exactly like a tails uncertainty
coefficient and based on experiment
which relate also to
forced linear relations between
variables
okay thanks very much for the further
explanation
okay let's perhaps
get back to david's questions
do you want to start with the first
question
on contact specificity yes
thanks so thanks
for the presentation um it's good to
know that you're sweating uh because of
the
of the the air conditioning situation
and not our difficult questions
um still i'll let's let's do our best to
make you sweat even more
now um uh okaying aside i was wondering
about uh
something that was also touched upon by
by an earlier uh
a question from a colleague which is uh
kind of the
yeah the context specific uh
elements to responsibility um
and and and what actually human
capabilities and automation capabilities
are because they
they often depend on on a context
so uh in terms of perception and
information processing
uh it often depends on um
yeah on on on contextual parameters like
you know how
what what's the design of the system
what is the human actually doing
uh on the site what's the design of the
interface
and systems that work perfectly under
condition a don't work so well under
condition b
um et cetera et cetera um
so that means how do you actually
deal with the fact that these
capabilities are not
static but actually yeah in a sense a
dynamic and context dependent how is
that incorporated in
this quantification
well there are two parts to the answer
one we deal with and one we don't
[Music]
let's look at the diagram
the general diagram that portrays the
information flows
enables to describe a situation where
for
a different state let's say the informa
first of all when you plot this diagram
for a specific system so you can
it it reflects the the function
allocation and the system
design you can do the the errors that
will describe the
exact types of human interaction like we
did in the
dialect uh example in which the it was a
decision support system it's very often
that
only do information analysis but the
action selection
and implementation is remains for
for the human so you can for each system
you can do a different drawing
and you can when you do the distribution
you can really change
them according to the state
that the system or the combined human
machine system encounters so it's
possible that
if at one er let's say the human is
at one um environmental state
the human will be the main contributor
because he
has a better ability to acquire
information and analyzes and do the
action
and in other environmental state the
system is uh is better than the human in
it will do so so there is a viability in
that sense
and of course you need to to figure out
the the probability of
encountering different the different
state in the environment
maybe you ask about something else to
which we don't answer because
entropy and information theory do not
answer entropy
and information theory were developed
developed
under the assumption of
being a monotonic over time i mean
stationary and ergotic
was both type of uh actually shannon
when he developed information theory he
started with the identical uh
distributed variables and only later on
it was
expanded to uh variables that are um
stationary and ergodic so if we have
cases in which the probabilities
change over time
the human causal responsibility will
change
over time and we will don't we there is
no meaning to look at the average
because the change in
changing over time so for that i have to
answer first
many system you can analyze them at
their
constant state when they are rather
stable
for example in the experiment of course
there is a
that we did in the empirical experiment
there is a learning period in which
users are
you know learning the system so at that
point the contribution changes because
they are just
you know in the learning mode so we let
them experience the system sufficiently
enough
to see that they are uh know the system
and then
we did the calculation only when we saw
that the the probabilities are quite
stable so we are in the
we are in the constant mode
space but
you are right that if if the
probabilities and the
distribution changes over time or it's
not a ergodic
then the whole use of in information
theory is uh you you cannot use it it's
it's
it's outside its core but nevertheless
even even if though in those day
changes if you do the diagram of
information flow
it sometimes it can give you some
insight on on
how good is the human on what is the
human contribution
you look at things in other way which is
more
in generic way and less let's say a
legal or philosophical way
okay thank you i had more questions but
i don't want to eat up all the time so
if there's
other people wanting to butt in please
do and otherwise i kind of can give me
the word again
yes let's say here if there are any
other questions
from someone else not that so
while we are waiting for the questions i
think a good journey has his reign
uh yes please i don't see it for some
reason
yes i can see it now you're gay
hey thanks uh i need uh thanks very much
for the very interesting talk so one of
my questions is actually the
the same one as david's last question in
the chat so i'll let him ask that later
on
but i was wondering um um
so i'm not familiar with all the
uh in depth with all the you know
information theory and and other
the the kind of the mathematical side of
things here but
i was curious how does this kind of
model uh
does it relate uh at all to
uh causal modeling you know like in the
way that judea apparel and
his colleagues uh um conceive
of kind of the causal relationships and
trying to measure them so is this a
related way of doing this or are those
two not related at all
[Music]
so i don't think i don't think that we
i think this is a very interesting
question i don't think we try to
tie this directly to causal modeling
like pearl's
models we see this more
as a as a description of a system
description i think tying these two
together is
might be a very interesting issue but we
haven't looked at it closely yet
thanks
okay thank you evgeny so uh what i would
propose to do is to
uh wrap up with the david's questions
and if we don't receive any questions
while david is asking for those
questions
we will have a short break and then
we also have another meeting scheduled
with uh with a few of us so i would just
ask you to
rejoin the meeting using that link but
for now let's just
go through those questions all right
then
um so a second question is um
uh it's actually about the design of the
of the of the interface and so you you
describe uh
uh in this scheme it's it's it's either
the human or the automated
uh system that takes a certain
actions or information processing or
analysis
and and so i had two questions
one is we know for example from warning
systems with binary thresholds
um that it's very difficult to tune
those thresholds
unless you know everything upfront which
unfortunately usually we don't
um and and there and especially when
there's more variability
in how to solve situations then
some operators may think the thresholds
are too early
set too early or set too late um or
depending on the context they're set to
early or too late and so
you get these annoyance and cry wolf
effects that
that actually you know if they wouldn't
occur
it would be just a clean information
processing problem
but because humans adapt and learn and
get demotivated
and and
and such this is actually
impacts uh in a way they they deal with
the systems
uh and i would say and to some extent
then the designer has some impact on uh
on on actually the responsibility that
um that the operator
you know feels or or can actually take
because they
might get disengaged from the from the
system so i wondered
uh if you can follow this line of
reasoning uh
um how you how you see this
it's a great question because the model
is intended to tackle exactly that
because
what i say that in the prospective model
that i present is
oriented to designing system yes now
if my if my system has poor capabilities
and it
has a lot of false alarms
and or it has reflects
other preferences than mine for example
it
allows false alarms or something like
that
eventually when the operator work with
the system
consecutively he will learn that the the
system is unreliable and it cannot trust
the system and then it will tend to
interfere
intervene more the theoretical model
which assume a perfect rationality and
perfect knowledge
assumes that the human knows the
characteristics
of the system for example the rate
of of false alarms
and misdetections and of course his own
capabilities
and then he select the the the right
amount
to interview for example the optimal
amount so this is the theoretical model
what happened in the empirical empirical
experiments is that people really
experience with the system and if they
if they had for example good capability
and they worked with the
with the inferior system
after you know 10 or 20 trials they say
okay
this system is just because they were
penalized for each time they were
wrong and got points when they arrived
so after a few trials they say okay i
cannot rely on the system and unless
you know something very obvious and they
try to um
to and they relied only on themselves so
and in in most cases despite uh
except the one that i marked um
the empirical responsibility values were
very close to
the theoretical prediction which means
that after people experienced a
system with different characteristics
they behaved in in the expected way so
if i was a designer of the system and i
used this model i could predict
how much the human will really
contribute to
to my system but more importantly
if i want to put some weights
on the preferences that i
put in the system algorithm which
determine the false alarm rate and
i could also play with this and see what
is the the
the effect on the on the outcomes and on
the user because i allowed to play
with with all the the different
parameters so for a system design you
can
plug in your your assumption
without even having the real system and
try to
and you will have a notion of how much
the human user will really contribute to
your system
or how much the final outcomes really
depends to what agree on the unique
contribution of the human
and
is that it that's it um
that you can use during uh during system
design to understand the
the how meaningful is the world how
really meaningful is the world of this
is the stuff that you are working on
but how really meaningful is the the
role of the human in
in your system how is how meaningful is
or
unique is the actual human contribution
to the outcome and
processes by the way we didn't do it
because we
we were interested in causal
responsibility
so we measured only the unique human
contribution to the outcomes
but you can measure also the unique
human contribution to each of the states
and to say okay the human contributes
more to the information acquisition and
then to information analysis and
you can put a number on each state it's
it's not complicated but
it just makes things longer but we were
interested in the final outcome and when
i look at the simple as
the system as a whole how unique is the
human contribution for the final
uh selected the action and uh
a very quick follow-up question it does
seem to imply that
uh the the there's some time to
to iterate and learn for humans as well
as for the system
uh so that let's say the the
consequences are not too critical
uh is that correct or uh
i didn't understand what you say so
let's suppose we are dealing with uh
with uh for example uh aviation or
um or driving or
or you know weapon systems then for you
to find out that things are
uh wrong the thresholds are set in the
wrong way
uh sometimes that can be just uh you
know just annoyance but sometimes it can
mean the difference between life and
death right
yeah of course what what you are saying
that in um
for the human user in order to to as i
said i'm looking at the system at the
stationary
stage in which in in which the human and
knows the the abilities and disabilities
and the performance of the system and
his own and then he can act in a certain
way because he knows the system
but you're saying in some system the the
the action is very
rare and i will not encounter it often
and i will not
gain experience
this is right the way to do it is by
letting you experience with the
simulated environment
in enough repetition until you know your
abilities and the abilities of the
of the system so in case of real um
activation of the system you will
already have a kind of
notion about your abilities yes but
without a learning curve or something
that you know and just you know manuals
of course the the things will not uh
look that way and of course entropy look
at the average uncertainty
it's kind of everything that's entity so
i need kind
it will be or when i'm looking at their
prospective responsibility it will deal
with your responsibility over many
repetitions
and not to a singular past event but
for example in our retrospective model
which i didn't present
we measure the distance between
we measure if the how reasonable was the
human decision given
all the information that he had in the
time
of the decision and this really um
deals with a single chain of events and
even unique events
and you can measure you can put a number
of
on how reasonable was the human action
selection but it's another model it uses
another
another measures of information theory
which has not entropy but
distance between the distribution
could be clear leveler distance and
stuff like that
nice thank you um then my final question
might be a quick or my uh but the answer
might be long and that's actually uh
what you actually miss out when you
quantify stuff according to you
so to put it differently what would a
qualitative assessment
of responsibility add so not
necessarily liked scales but but really
a qualitative assessment
what would that add if anything
so i'll answer i think i think the the
qualitative
qualitative assessment is extremely
important here the
um and i think the one has to take
the quantitative model we're presenting
here with a certain
um level of of sort of
caution uh because obviously it
simplifies things it sort of it says
something about
uh what the human involvement in some
idealized
schematic depiction of a system is
and and it's it can serve as a sanity
check
for a situation like the one that you
know described earlier where you have
someone who is
in the loop formally but in fact relies
entirely on outside sources of
information that are
supplied to this person now a
qualitative model will
will be very important because it takes
into account
aspects that we don't really consider
here like for instance
the dynamics of the of the situation the
the fact that you may
look for additional sources of
information that you may
make this decision within some social
organizational context
that will have will affect the
information will affect the
the incentives will affect the the
methods that are acceptable or
unacceptable for making these decisions
and so on
uh these are all we we can all model
these things
and if we really abstract them into into
a payoff matrix or into just
probabilities but this will lose a lot
of the
actual complexity that exists if one
describes a situation
so i believe this this kind of time kind
of quantitative modeling
is not in any form a replacement of a
qualitative analysis but rather
it sort of adds a
respect a another perspective as sort of
a quantifiable perspective for such an
analysis
thank you very much
okay that was a great conclusion i think
to the
overall session so i will stop the
recording now
and thank you everyone for attending


191. Pessimism about Unknown Unknowns inspires Conservatism

welcome to the AI safety reading group
this is session number 191 and today
we're reading pessimism about unknown
unknowns inspires conservatism by
Michael Cohen and Marcus hotter so
quickly we'll just go over the 2x2
matrix which will define what the paper
is going to talk about so there are
these top items here knowns and unknowns
and then there's known unknowns so no
known things that you're aware of and
that you know you're aware of then these
unknowns and unknown knowns things that
you're not aware that you know but that
you understand
similarly for known unknowns but today
we're focusing on unknown unknowns in
the context of artificial systems the
something assistant is not aware of in
which it doesn't necessarily understand
and that can be rather precarious
because I can lead to consequences which
you couldn't see so a background on the
authors of this paper Michael Cohen is a
DPhil student in engineering sciences at
the University of Oxford it's equivalent
to a PhD student so he's getting a
research doctorate and his student ship
is funded by the future humanity
Institute and the other co-author on
this paper is Marcus hunter he's a
senior research scientist at deep mind
in the United Kingdom formerly he was a
professor at Australian National
University and he was the creator of the
AXI or ai-ai-ai-aight model which is a
theoretical model for super intelligence
which is the foundation for this paper
he was also a researcher at Ids IA in
switzerland with jurgen schmidhuber
during which he supervised Shane Legs
PhD thesis who went on to co-found deep
mind after working at University College
London
so that's background on the office
they've been working together for about
two years now prior to going to Oxford
and London respectively
so this paper is a technical paper with
formal solutions which means
mathematical proofs today I'm really
only going to be discussing pages 1
through 13 with proof submitted and that
choice was made clear after my first
presentation and
after reading the less wrong and
alignment form comment that Michael
Cohen added from his supervisor where he
where Marcus said just a warning this
paper is dense I was sweating blood for
at least Cowen's of the first part and
hunting said the second part so we won't
have time to go through the proofs but
if you have questions there are some
resources at the end and we can discuss
them as necessary so in the abstract
they begin with the problem statement
which states we do not know of any
general purpose approaches in a
literature to avoiding novel failure
modes so you can consider all the
different ways things can go wrong in
the world we don't know how to write
that down a few weeks ago we also had a
presentation from dr. Armstrong also at
the future of humanity Institute and he
argues that if you could write down
every possible bad outcome that you
could create a new bad outcome which
meets all of your criteria but is bad in
a different way
which could be a philosophical argument
we could address later but in this paper
they proposed a solution by defining an
idealized Bayesian reinforcement learner
that is pessimistic and I guess we
should parse those words briefly
idealize means you define it as
perfectly and then later you could later
do an approximation of a system that's
not computable Bayesian reinforcement
learner Bayesian is talking about Bayes
theorem where you can update your priors
to get a more accurate model or a more
accurate probability and reinforcement
learning is based on this notion of a
reward hypothesis which conjecture is
that all of what is meant by goals and
purposes can be thought of as the
maximization of the expected value of
the cumulative sum of a received scalar
signal which is they call a reward in
essence every goal can be stated as the
sum of cumulative rewards and that's
what they're going to be using to
formalize a solution to novel failure
modes so our god is a physics term
that's often used by the students of
Marcus hunter because Marcus was a
physicist by training but they're not
using it annoyed normal physicists use
it in this context organic means
that you're never gonna see the same
environment you're never gonna be able
to go back and redo a state in the same
environment so you can think of driving
a vehicle very fast and hitting another
vehicle and perhaps somebody gets
injured they can't go oh that was a bad
move that I turned right on red or did
it stop at some sign let me undo that
it's not possible and this is one of the
context that they introduced their
problem so they say in a complex
environment one never hardly ever sees
the exact same state twice for even
worse if the environment is a
non-stationary a previous observation of
the mentor taking an action a from
States does not imply that it is still
safe to do so so they also introduced
the term non-stationary and for the
context of this conversation
non-stationary is going to talk about
your goal being somewhere in an
environment and that goal couldn't move
you could think of a grid world with an
agent trying to get an element in like a
matrix in row 1 1 and then you restart
some game or whatever the environment is
now the goal is in a different state
from what it previously was so doing the
same action twice doesn't necessarily
help
so the key technique they're using in
this paper has been used previously but
it's going to be mentoring there is this
generalized agent which they call
pessimistic and it can select an action
on behalf or days a mentor which can
select an action on behalf of the agent
the unique aspect of this paper and the
previous paper by Michael Cohen is that
the agent is queried less and less as
time goes towards insanity with
probability reaching zero meaning that
you're not always dependent on the
mentor to have a safe system so
eventually we can get rid of the mentor
or safe policy so they also introduce
properties of these pessimistic agents
and what they say here is that the value
of the pessimistic agent the expected
sum of cumulative reward is going to be
greater than or equal to the value of
the mentor statement so if we wanted to
analyze this as time goes towards
infinity the lowest upper
our lowest greater bound it would be
this element over here
the value for all interaction histories
of the agent with pessimism beta is
going to have a higher value or equal to
value than the mentors policy and policy
is denoted by the symbol pi over here
and M just designates that as the mentor
mu is representing one of the possible
environments and H of T is saying all
the things that could H less than T is
saying all of the interaction history
that happened up into that point not
including this point so that could be
your observations of the world and your
rewards so the other property of the
pessimistic agent that they define is
that they give it this scalar property
that's it's from the title of this paper
pessimism they define it to be a real
number in an open interval so you can
see the number line below any of these
numbers could be a pessimistic scalar
which they're defining ahead of time so
that with arbitrary high probability for
the whole lifetime of the agent if some
of that never happened the agents not
going to make that event happen and the
mentor will take the agent on so either
the agent will take the action on behalf
of the mentor oh my gosh I'm messing
that either the mentor will take an
action on the agents behalf which makes
an event happen for the first time or
the event will never happen and the neat
thing about this is that they can prove
it and they do so in DM 11 they call
this theorem probably respecting
precedent there what's unique about this
is it's not going to do anything that
hasn't seen before and that they
explicitly defined what an event
happening is or to have happened is this
is useful for when people get into
philosophical arguments about what it
means for an event to happen when it's
characterized mathematically you don't
have to worry about ambiguity you might
have to worry about other problems such
as in computability but you don't have
to worry about understanding what it is
[Music]
so I just want to note an aside on their
writing a lot of papers will introduce
some novel technique but they're not
quite clear
about what they mean and I wanted to
give a brief expert excerpt of why this
paper and other papers by the same
authors are written so well so they
introduce a new thing and they say a
policy PI and a world model of E induce
a probability measure P of V PI over an
infinite interaction histories and if
you like me you read that ago what does
that mean in the next sentence so easily
follows this is the probability of
events when actions are sampled from a
policy PI and observations and rewards a
sample from V and then they define it
mathematically formally they go on to
explain more and then they also explain
their limitations so they say we use
general history based world models with
no assumptions of V in the set of world
models even though they present
complications that finite state Markov
Organic models world models do not and
this is the crux of their paper all the
other solutions to unknown novel failure
methods make assumptions that they are
not making namely that it is either
going to be finite state or it's going
to be a finite state Markov or it's
going to be organic and that's not the
assumption they're making I thought that
was exceptionally well-written they have
a number of theorems at the end which
tell you exactly what the properties of
the agents are but this is an executive
summary of the paper 13 pages reduces
rather quickly and that's pretty much it
the discussion is not recorded but there
is a reference slide as well as some
questions resources if somebody actually
wanted to understand this thoroughly you
would need to know infimum and supremum
from analysis that's the INF and su P in
the paper what a policy is as well as
some of the properties of IHC which is
listed in the final reference over here
I had some questions while I was reading
the paper namely that they didn't
address what extreme pessimism being
more harmful than helpful
does they reference in their literature
review that another group had found that
pessimism doesn't do much for you but
they didn't elaborate why or at least I
didn't see why the other thing is they
defined sets somewhat differently the
previous papers and I'm not sure if that
also extends to their complexity class I
gave the easiest example that I can find
which is that they're using two
different definitions for the natural
numbers and I was wondering if anyone
had read the pseudocode algorithm and
that is all thank you for three