The Control is in the Interaction (Jered Vroon)

[Music]
jared you're good to go i think that's
my cue right
hey all it feels really weird to be
giving a presentation to you all without
seeing you
uh still really happy to be here and on
this side of the
screen where i'm doing the presentation
um
and well as the title already says i'll
be talking today about
interaction uh specifically in the
context of social sidewalk navigation
and more specifically in the context of
unpredictable social situations
um but before i dive in first
really brief bit about me and my
background i
studied cognitive ai back in the days
where
good old-fashioned ai didn't always
necessarily include machine learning did
a phd in social robotics
on social navigation and i am now
working as a postdoc
in the design faculty of the udaift's
specifically knowledge and intelligence
design
and what for me is a constant is a
fascination
with people basically and there are some
amazing things we can do
that our machines can't do yet and
i think a good starting point here and
this is actually a shot from the matrix
but
uh is this amazing human capacity if you
were faced with this crosswalk
somehow you would probably be able to
find a way
even though there's a whole wall of
people somehow
the wall would split for you and you
could navigate through this whole mass
of people
and i think that's quite fascinating and
interesting how we actually
get that going and
in fact we even managed to do it in more
complicated situations
such as those with kids or dogs with
leashes
and in fact we even managed to do it in
situations where everything gets turned
on it
had and um we certainly have to keep one
and a half meters distance
and so there is somehow this amazing
human quality to
find a way um even
if rules change if we don't know what to
expect from other people
and um what i would like to be
discussing today is
um how can we well
translate or at least get robots to fit
into that as well
when they're navigating um
the niche will be specifically within ai
robotics hri
social navigation and slightly more
specific i'm looking into sidewalks
uh so i know a lot of people um within
ai tech
look into social navigation i look more
specifically into sidewalks
also because i think it's very
interesting sub niche
where you get much closer to the people
and the
social rules are much more important
than the traffic rules
and at the same time it's also a more
safe environment because you cannot
kill people by driving over them that
easily
and
so wait one important disclaimer to
give before i dive in further is that
most of the things i'll be discussing
today are not necessarily new per se
so i might introduce things and concepts
that might be quite familiar and i
hope actually that the main point of
what i will be doing is perhaps putting
them together in a slightly different
way
that might lead to a nice actionable way
to get this working
um so to kick off let's
start with a very rough uh overview of
the state of the art
um in in social navigation i think one
of the very first
examples is shaky uh that nowadays
and um perhaps many people would want to
correct me but i'm just going to try and
summarize it this way anyway
uh much of the work is based on uh
theories on human social positioning
like proxemics which is the idea that we
suits
depending on how close we are to someone
actually take a
specific physical distance to them as
well
and then those models then
have to accommodate things like the fact
that humans move or the fact that humans
respond to what a robot is doing
and that leads to very beautiful
solutions
in terms of path prediction and
appropriate quick replanning
both machine learning based and
cognitive model based
it also leads to sometimes social
navigation being more
efficient and effective in controlled
settings such as the
warehouses of amazon where you can just
tell your workers what to do or what
not to do and similarly there's also
beautiful solutions to how humans
respond
not going to go into too much step there
because i have some other things to say
about that
uh and lastly of course there is the
learning from aspects
experts where we try to somehow
take their expertise and replicate that
um i think the gap that's currently
faced by this
state of the art is that in the wild
when you throw your
perfect algorithm into the real world
people often turn out to be complex and
predictable or
dynamic in their reactions and perhaps
most importantly
unique in their needs so different
people might have different
needs even on different days if they're
in a bad mood or a good mood
and i think that raises the very
important question um
how can we assure socially acceptable
navigation in the face of such
unpredictable and noble social
situations
uh and this is actually unfortunately a
more relevant question
than i would like uh i'll read out this
tweet for you
um it's from emily slackerman ackermann
or emily eckerman i think is her real
name
she in a wheelchair was just trapped on
forbes avenue
by one of those delivery robots only
days after their independent rollouts
and she can tell that as long as they
continue to operate they are going to
be a major accessibility and safety
issue
so what happened here was that these
robots
roamed the streets to make small
deliveries
and in this specific case
while she was crossing the street the
robot was stuck in the sidewalk because
um it saw so many people in his direct
environment that it
didn't know how to find a safe part and
so it did the second best thing which is
just not move at all
and that was typically a very good
solution but in this specific case it
meant that it blocks
the curb so the low piece of the
sidewalk that you use to get back on the
street if you're in a wheelchair
and that resulted in quite a big
problem for um this specific wheelchair
user
and there's several more examples where
our beautiful algorithms
break down when we throw them into the
worlds there's children harassing me
robots uh there is people learning that
autonomous cars just stop for them and
thus
uh consequently deciding to cross in
front of them
without stopping uh there's office
robots being unable to use elevators
and there's actually the whole package
of challenges faced by
autonomous vehicles that may have led to
the
current shift from autonomous to
automated vehicles
um and if we look at the
current solutions for these problems uh
they are a bit of a band-aids
so they are band-aid solutions i would
say for example recent tweets seem to
indicate that
the robot now avoids claiming the whole
kirk uh the
more robots i think they did something
beautiful there they
predicted the likelihood of harassment
uh and
if a kid was deemed highly likely to
harass the robot it would flee to their
parents and
the office robot asked for help and like
i already said
we are now moving from autonomous to
automated vehicles
um and i i think
all those solutions actually face the
same problem again
because people in the world still are
complex in predictable and dynamic in
their reactions and unique in their
needs
um so yes this is a band-aid and i think
it's very important that we make them
but um the question i am exploring and
that i
hope to be discussing with you today is
could we perhaps solve this on a more
fundamental level can we
actually um well
like people do figure out a way
to do this
even though we don't know which
situations to expect
um and
i think a very good starting point for
answering a question like that
is always to look at situations where it
goes wrong
and i i hope you are all familiar with
the sidewalk shuffle
and this would be one point where i
would really like to see your faces
nodding but
that's life at the moment uh
the sidewalk shuffle is when you're
walking on the sidewalk and
someone is heading in your direction and
you step aside and at the same time
those people also step
aside in so you step to the left and
they step to the right and
you're still faced with the same problem
and you step to the right again and they
step to the left and you're still faced
with the same problem
and you get that back and forth where
you end
up having to stop or somehow break out
of this loop of adapting to each other
at the same time
and i think that's a very interesting
point because it shows
uh it it's where we our
mechanisms break down and it shows that
we apparently are really good at
adapting to each other um even though we
need some time to do so
and i think that brings me to my main
idea
or hypothesis which again is not
necessarily new
which is that we are figuring it out as
we go
um and i think that actually
so not necessarily learning but um just
going forward and then um failing or
succeeding
uh either way using that somehow as a
way to
adapt our behavior and find more
suitable behavior
on the fly um
and i think if we could manage to
somehow
put that into our robots um
we could solve a lot of those situations
where we don't know what to expect
and still run into problems simply by
being able to figure it out
um so how can we get that
um into reality um
well my proposal actually is pretty
straightforward
and it combines something that many of
you will probably be familiar with
which is an interaction focused approach
so
we have examples from embodied and
better cognition action perception
feedback loops reactive robotics
bradenburg's vehicles
and that's just the tip of the iceberg
there's lots of people
advocating for work that focuses on
interaction
in the actions of a robots um
but i propose to couple that to the cues
that we give to each other
so for example when we are walking on
this sidewalk
if we are trying to find a gap that's
not just
a matter of going and hoping for the
best it's also really looking at
people's faces
and it might be that the tall man in the
dark sunglasses might be in a bad mood
and you might be heading straight
towards him
and he might not be willing at all to
get out of your way
and picking up on that picking up on
those cues
and using those to adapt i think is
fundamental
so better that means that i would
propose to have the robot pay to the
implicit feedback in people's responses
to its behavior
and then use that as the basis for
finding more suitable behaviors through
the interaction
and and
i i can talk a bit longer about this and
i will
but actually this is the meat of what i
propose
and well
that means that the hypothesis that
responding to people's social feedback
cues can find
through the interaction socially
acceptable navigation in the face of
unpredictable and unknowable situations
um and that means that i envision a
situation where if the robot is driving
and
someone is just not in the mood to get
out of its way
at all that then still the road would be
able to sort of
detect from their behavior that they are
not in their moods
to adapt to the robots and subsequently
the robot would adapt more
or perhaps even find a way to convince
them to still get out of its way for a
bit
um that brings us to the next big
question i think and that's how are we
going to make this
real how are we going to do this um
first trick in doing that is always to
zoom in
so i actually think that this kind of
mechanism might be feasible
for a lot of interactions we have with
physical systems
but to keep things feasible
i zoom in on small mobile robots such as
delivery robots navigating the sidewalks
and trying to be socially acceptable in
terms of time efficiency
uh safety and comfort and perception of
the robots
and that leads us to a lot of different
steps and
to be honest this is also a bit of an
invitation to all of you
um both to give thoughts and insights
but also
um to perhaps connect if you feel that
this might be something
related to what you're working on um
because i think to make this happen we
need to start with a clean blaze line
then find a small and relevant set of
feedback cues
small set of behaviors and use that to
somehow explore different interactions
and um i think
based on the time yes i will give you
already quick
teaser um of the pre-study one
uh a clean bass line
um which is uh
what i've been talking about so far i
think is really focused on
the emphasis on how we can somehow let
the interaction
emerge from our responses to each other
and i think that what that also means
specifically for social navigation
is that we are somehow often tempted to
try our solutions
and the problem that has is that it also
immediately
gives rise to new emergent behaviors
so for example if we have a robot that
just tries to stay
1.5 meters away from humans
we get beautifully weird um
navigation behaviors where it's fleeing
from humans where humans are adapting to
the robot and the robot is adapting to
them
uh leading to uh wobbly paths deviations
um so uh
my step one the clean baseline would be
what happens actually if we
take away all social behaviors from a
social navigating robot so just make a
robot that basically ignores all humans
um and i had started
these explorations just before covet hit
and
so um unfortunately i won't be able yet
to give you
full-fledged results um
but i did make some cool observations
that i do want to share
because i hope they will provoke some
interesting discussions
soon after and
so what we did we made this small robot
i'm not sure if you can see my mouse but
a small robot you see on the bottom
right
and had it drive around it's actually a
cardboard box
so we could relatively safely ignore
humans
and we observed
when problems actually arose so
just went in without assumptions and
started looking at
which conflict does such a robot
actually run into and what are the
problems we actually need to fix
when we start with a clean slate um
and i'll share two observations with the
very important disclaimer that these are
just
first observations not proven results
yet
uh still so observation one was
the robot was driving to the city
apparently on its own
it did stick a bit to the right but it's
very much stick to its own parts
did not slow down speed up or deviate
from its trajectory for anyone
and um this was the city center of delft
so it wasn't too crowded but still a
decent people of amount of people was
around
and interestingly to me at least
um especially uh based on all the
literature i've read so far
the robot could follow its trajectory
completely unimpeded
in fact people actively did their best
to get out of its way um
and that wasn't even too noticeable some
just made very subtle changes in their
trajectories
uh they some of them slowed down just a
bit
uh but at the same time there were also
people who actively steered their
walking aids um out of the paths
uh to make sure the robot could just
keep going um
which i think is quite interesting
because it might suggest that
our robot which was very clear and
straightforward in its
paths might actually trigger people
being more
adapted to the robots one more
observation
and then i hope to move on to the
questions soon
in contrast to what i just said we also
noticed that when
groups people were standing still for
example because they were chatting
if there was enough space to navigate
around them
then straight up approaching them which
we had the robot do because well it
ignored people
somehow that fell to people as if it was
engaging with them
and so somehow we found there was a big
difference between people who were
moving and people were
stationary um
and i think that's very interesting so
we saw that they
engage with it they started taking
pictures they seemingly started
exploring
how they could play with the robots um
which very much looked like they were
actually teasing
the robot and trying to break it uh
which i think is just like the previous
one a very interesting
twist on what's already known because um
yes if you throw your robot out there
making it avoid people you wouldn't even
run into this situation you would have
afforded this problem
but you also fail to see that it arrives
specifically when people are standing
still
and i think that's a very interesting
way to go at it and
in fact what i would really love to do
is hear your thoughts on this
um and i might have something more to
wrap up at the end but i think this is a
very good point
to um go back to seeing each other and
having some questions
and some interaction uh because in the
end i think that's the most
interesting bit of being here and being
here with you
thank you
thank you jared that was great
so uh yeah the floor is open for
questions
if you want you can post it in the chat
or just ask
um or you could raise your hand good uh
luciano you wanted to ask something
yeah hi jen thank you very interesting
work
uh i have a clash a question actually
regarding the social cues so i
understand the next step
is to model the social cues with respect
from the humans
to the robot and the other ideas try to
mimic to imitate them
that the robot would be doing the same
cues to the
human or how do you see this interplay
and i think that's very interesting in
which question and that means that
partially i don't have the full answer
yet okay i think the first step
is actually to understand the cues that
humans are using
so not necessarily to mimic them but
understand and be able to read from
people's reactions how they feel about
robots
um to give one example of that another
observation we made was that sometimes
people would deliberately change their
path so that it would collide with the
robots
yeah so they would sort of challenge the
robot and see what would happen
um and i think that's a very
interesting cue to pop up and it would
be very relevant for aurora to be able
to detect ah i think this person might
be challenging me
uh also because in the end we found that
if that happened and the robot just kept
going
most of the time people would jump out
of the way at the last moments
figuring out yes the robot is not going
to stop for me my teasing
didn't work so
i think in that respect the first step
is really to detect the cues and use
that to
change the world's behavior but of
course
changing the robot's behavior will also
result in the rope giving cues to people
and an awareness of that and how to
steer that is something that
i think we will really have to figure
out as we go
because it's such a complex and emerging
situation that i
find it challenging to make real
predictions about what will happen there
yeah i understand so but thank you for
sharing this is a really
interesting field thank you very much
i think we have one uh we have the next
question from simeone
hi there um sorry about some of the
background noise uh there's some
construction work going on here
um any robots involved i'm sorry
any robots involved in the construction
oh man i don't know
anyway my my questions kind of relates
to um to what um i mean obviously
uh interactions with robots and people
um does mean interactions so that's two
to two directional bi-directional and so
to what extent
and do you think that and we should
focus either on the robot avoiding
people
or that maybe the humans getting cues to
avoid people
and where do you think the onus lies in
that responsibility
so also um while you were presenting i
was thinking at a certain point
and so i don't know if you know star
wars you know all these droids at
certain points where they can
be going in and out weaving in and out
of people my thought process there is
well these are
droids or robots that are avoiding
people and not necessarily the people
avoiding them so
seeing that maybe humans were on the
street first should the
robots be avoiding people or is it
really this bi-directional interaction
we're looking for
thank you uh rich question again
um which is good which is why i really
enjoy speaking here um
and i think the um
it has to be bi-directional in the end
um so
if our robots would constantly be giving
way to people
then they wouldn't be fulfilling their
function and actually they might be
ending up worse so the example i already
gave of the
woman getting stuck on the street
because the road was
waiting for people in the curb uh
turned i think that demonstrates that it
might actually be more problematic
sometimes to not
engage in the interaction and and say oh
i'm subservient i'll just
hold back rather than actually engaging
in it and
doing the back and forth um so
uh i i would strongly argue that we
need to engage in the back and forth and
don't have robots always due to humans
unless perhaps we are on a big starship
and there's not that many people around
and just a few small robots
but that of course still leaves open the
question where that balance should fall
so um should we would you most of the
time some of the time
how to um basically balance out that
um taking dominance in the interaction
and again i think this is something to
figure out as we go
um i also think it's something that may
change
so perhaps at this stage people are
quite willing to accept robots so all
the examples i gave were of people
mostly enthusiastic oh there's a robot
cool new thing
but that might change massively if there
is 50 robots on the street and
you have to waste your way through them
and you might be less inclined to
actually give the robots right of way
um and i actually think that might make
it more important to listen to the
queues and to
figure this out based on people's
reactions
so not to say so we're gonna sit at 50
percent of the time the robot will yield
but rather have the robot stay in tune
stay attentive to the way people are
responding to it yielding or not
uh and using that to decide whether or
not it shoot yields
and then perhaps during covet our robots
might from those reactions
turn out to be a bit more gentle and
bit more uh eager to give leeway while
once corona is over and we get get
closer again they might become a bit
more dominant again
and so i think actually the answer will
be
we really need to stay alert and figure
this out through the interaction rather
than
pre-deciding on it yeah so i i can
imagine also that
we might actually and get something like
a social optimization problem that
and which will change dynamically in
time so that
um the optimal uh strategy for the robot
and the the humans um will change and
it's about being
optimal in the movement of basically all
actors involved
yes okay cool thanks
yeah if uh there are no more questions
so i would like to follow up on
all sorians i was there first so
yeah i'll just follow up quickly on
simon's point so uh
about the dynamic nature of this kind of
interaction
which we do not know for sure yet but we
could foresee the way it would evolve
right and uh my question relates to uh
basically
what what do you think would be the
right way
of trying to anticipate
what these uh what these dynamics would
look like
given that for instance in some settings
it would be relatively safe for instance
to just
put the robot out there and measure how
it interacts with humans and see how
humans behavior responds
in uh to to that robot sections and then
how the robots
adjust their behavior in the in response
to humans so
that kind of putting that approach of
putting the robot and humans in the wild
and just studying how
how they code up that is one approach
right but then the second approach
which could be for instance uh more
suitable for situations when
it's not it might not be so safe to put
them in the wild
the real humans with real robots
interacting you would think for instance
of autonomous vehicles
the second approach would be to do the
same thing
exactly the same thing just in the
simulation if assuming that we have a
really really good
virtual humans uh models of models
virtual humans which
we could use to test our autonomous
vehicles or
for that matter any other robots with uh
kind of significant
implications uh
then there is this fundamental question
of whether it will generalize from the
simulation however advanced it is
to reality to real human robot
interaction and then
what where would you put yourself on
this kind of spectrum
from uh let's say studying behavior in
the wild versus uh
uh generalizing trying to generalize
from virtual testing or maybe there is
something else that you have in mind
here
i i think i'm going to split my answer
in two bits
firstly i think it's um very
important to realize that yes it's very
cool what i'm arguing for and we want to
be responsive and
back and forth and all those interactive
bits uh
of course that's not the whole deal we
cannot just throw a robot out there and
just have it respond to people's facial
reactions and
hope for the best so i i
am not arguing we should abandon
everything that's been done so far in
certain navigation
uh on predicting people's paths and
using that
to think ahead a bit in fact i think we
should probably end up with combining
them
because i think they will strengthen
each other where we can
find parts um as we go
but we also predict sort of in the rough
strokes where we want to do and
we might at a crossing with a lot of
people decide oh we're going to cross
mostly to the right and then figure out
the details of crossing mostly to the
right as we go
um so in a practical sense
where i think the end solution will end
up it will be very much in a mix
um that's part one of my answer i think
second bit of my answer is
um i think simulations are a very good
two
for those kinds of predictive models
where you are trying to
train your model to basically get a
feeling for how humans will react in
those situations where you can
predict how they will react
uh but i think that actually the problem
with models uh all those things is that
they are limited uh they are firstly
limited by
the ones making them uh but secondly
they are also limited
uh because of the question uh just asked
by
simeon i think um
people will adapt as they go so even if
you have a perfect model hypothetically
i don't think it's possible to say we
have one
the fact that people we will respond to
the robot and
change the way they act over time
will mean that the mere fact that you
have
introduce the robots will make your own
model invalid
uh and i again i think it's a very good
approach to get those
generalizing models but i do think we
need this bit of extra interaction
to fine-tune the actual
last bits on the street
i hope that answers your question uh no
no no not completely but yeah i'll leave
it to
later discussions if you have time for
that uh yeah i think jens has the next
question
thank you um so my my question also goes
a bit in the same direction you were
talking about the interaction and the
background and forth
so i was wondering about how much you
want to go into
theory of mind and these kind of things
yes um yeah
i hope you all forgive me for giving
slightly longer answers but
i really have a really cool anecdote
about this and so i'm gonna just share
it with you
so back in the days when
ai was mostly symbolic
there was this big problem of common
sense um
so common sense is the idea that i know
that you know that i know that you know
that i know that
the sky is blue and at infinity um
so um this turned out to be quite a big
problem because since it goes to
infinity that
poses quite a big problem for all ais to
prove this all the way down uh the
rabbit hole
um and actually i think
there's like a decade of people
publishing tables papers struggling with
this
struggling with how can we actually
prove that something is common sense
that we all know that the sky is blue
until at some point there was this
really cool paper by garrett
and pickering where they said no no wait
let's turn it around and what they did
is they assumed in fact that everybody
knew all this common sense
so they would assume that everybody knew
the sky is blue
and then from that look when errors
popped up
so starting for the assumption that
everybody knows it and then just
looking from for feedback uh that shows
you have been wrong
and if that's the case you try to fix it
tell people oh you didn't know sir the
sky is blue
and continue assuming that they know um
and i think that's a really cool story
both because it shows that other people
wiser than i have been working on this
as well uh but also
uh and now i get to your question uh
because i think it's just tear of mind
is
very interesting and rich but to
implement it in ai
uh gives you very big problems actually
because
um there's this whole complexity of
actually
modeling what someone else knows
especially if what they know also
involves
what you do and that whole back and
forth
so i definitely think that theory of
mind is relevant to
understand what's going on here but to
implement it
i i really like something more similar
to the
approach of garrett and picking where we
say let's just assume
everybody knows uh what's going on here
and correct ourselves
as we go it's a bit of a windy answer
but i hope it
works thank you
yeah if yeah again i'm just
i'm eager to contribute to the
discussion but if there are any other
questions i would
just give away so let me know
yeah i'll go for now so uh yeah
following up on the
theory of mind kind of problem that you
see there
uh have you considered looking into more
kind of game theoretic approaches where
they kind of nicely avoid this
problem of infinite kind of levels of uh
hierarchy uh for reasoning about each
other's actions by just looking at
the kind of equilibria solution which
would assume that everybody has perfect
knowledge of each other and then they
will just converge the situation which
is good for
everyone in one way or another and then
you could just
basically try using some kind of finesse
equilibrium operator equilibrium
and uh make your robot follow that and
then
hope for the best as a as you say you
would do any kind of
mind setting yes and i think this is
actually
quite a similar question to your
previous one um
where on the one hand we definitely
don't want to just go into their
into situations of search navigation
blindly we want to be able to predict a
bit of what's going on to make some
assumptions and
start with a good guess
at the same time as i unfortunately
learned when playing
games with other people most other
people are not that optimal in their
decisions
me neither by the way but uh so
um there is this yes i think
game three is a very good starting point
and it's very rich
but i think also when you throw this
into the world and into the real world
there will pop-up situations where
people are less perfect or predictable
than you would hope
and i think that what i have been
proposing might be a way to
handle those specific situations the the
situations where
game theory breaks down because people
are imperfect
could have a bad mood could be
could have any reason not to obey your
predictions
okay thanks um royal has a question
yes thank you thank you jared was a
interesting presentation fascinating
topic
um i'm kind of building on a few people
here including the last
from the last question or less answer
some of my former colleagues in berkeley
they
developed kind of a way to interact
um you know robots with pedestrians
based on a technique called reachability
uh probable probabilistic reachability
where you basically
have a game theoretic approach and you
have some kind of
model for you know human pedestrians
predicting their most likely paths and
you're trying to navigate
yourself around those and when um
of course perhaps different pedestrians
can have different parameters and have
different you know
perform different predictions um
and when um when you see behavior that
is doesn't align with uh with the models
that you have
what you do is you you use this
technique called reachability
where um which is used to kind of figure
out like figure out how to stay in a
safe operating zone
um you use that in a dynamic sense so
what's what happens is
uh the safe zone basically would um
let's say shrink if you have some
behavior by a pedestrian that you're not
familiar with so you're gonna be
more conservative if you might slow down
at that point and you might
want to kind of update your models
and one of the critiques with that is
that it's
quite can be quite conservative or it
can be quite easy to
become super conservative and kind of
stop right and that's the moment where
you either want to collect more data or
make your models better or
you want to say you know what in this in
these moments there's so much
uncertainty in my model
for me to stay safe that i want to
interact
so i think there's a really nice kind of
compliment that's a really nice
paper i think the paper is called um
probabilistic probabilistically safe
robot planning with confidence-based
human predictions
and my like kind of my question is would
it be interesting to see like when that
technique kind of
breaks down or becomes too conservative
like how would
like what kind of cues what are the
kinds of cues or kinds of interactions
that you want to
design to you know bring those two
worlds together that you were just
talking about
um so just an idea and i'd be happy to
uh to discuss that
offline if that's of interest
very cool yes and it it sounds like
indeed it would be a
very good indicator of when it is a good
moment
to switch to um
more of the interaction and more looking
into it um
yeah i think that's my main answer yes
it sounds like a very interesting thing
to look into and i'll definitely be
looking back into the youtube video to
see
uh or the recording to see what you said
and then look up that paper
very interesting thank you yeah so maybe
like i have a dog here that's about to
bark sorry
[Laughter]
seeing something um she's trying to
update her own model so
that's safe or not um
yeah i guess my question or maybe like
the prompt is like yeah how do you
like what do you use as the context of
designing social cues and i think
using using other techniques is actually
i think can be a very rich
place to be creative about like what are
the kinds of social cues you need so
that's maybe
also a question whether that's like how
you think about that
ah good yes um
i i think um
and to relate it to the situation so
um when we get in the situation that
people don't correspond to our models
i i think what we are most interested in
and should be most interested in is
how they relate to us uh
and i think that's actually the cues
that i would be most interested in so
um i already mentioned one so for
example
when someone is testing the robots
challenging it by adapting their parts
so that it actually
becomes a run-in um but it could also be
more
subtle things like people really
actively avoiding
eye contact with robots or looking to a
robot and
being a bit more scared uh it could be a
big smile where people are
uh seem to be very enthusiastic and
eager
uh which for many robots is a good thing
but not for most
uh delivery robots um
and i i think that that means that the
main cues will be
indications of how people might be
feeling to watch the robots
because i think that will be the most
informative for making you decide how to
respond differently to those people from
what it would normally do
thank you
can i follow up on the social cues uh
question
um i mean it does sound very promising
and they could realize how
uh how this could be useful and at the
same time
uh when you mention uh facial
expressions
uh it feels a little bit uh like a
slippery path here
uh because yeah there is quite a lot of
research
recently that uh is showing that uh
detecting emotions and internal states
based on facial expressions
is not reliable at all so it has
very uh high individual differences
and it it it's just it is a slippery
slope so uh
there used to be this uh uh this article
what was called uh said something like
ai emotion recognition
uh can't be trusted or something like
that
so yeah what would you think of that
because yeah if a person smiles it might
indicate that they have a positive
kind of attitude towards the robot and
they want to approach it or it might
indicate that
the guy is just saying i hate this
romance but it would look like a smile
right
yes i think i can say two things about
that
the one thing is
my guess would be that the cues to start
with are not facial expressions
you need very high quality cameras that
don't have the range that you can use
practically when
driving a robot around so i think a
much better starting point in this
specific context would be things like
trajectory body pose those things
that said my guess would actually be
that
one of the reasons those problems arose
with facial expressions
is that we don't use facial expressions
just to express our inner worlds
we use them to communicate
so if someone is smiling at me
while i'm walking on the street that
might not mean at all that they are
happy to see me
but it still signals something to me
and it might still mean that they are
okay with me getting closer to them
while we're crossing the street
so i would say that i completely agree
with
the new ideas new ideas that facial
expressions are not
meant just to express our emotions and
internal states
but i actually think that's a they are
meant i think to communicate
to other people to somehow
make them aware of something and i think
that communicative meaning of facial
expressions and all kinds of social cues
is actually what we are interested in
this case
um more than trying to understand the
underlying
emotions of imports internal states
and so that that's i think that the two
sides of it
for me right
okay
thank you uh any more questions
thoughts suggestions
i don't see anything yeah no hands right
uh did you mention that you also wanted
to share
something else in the end of your
presentation or
or we wrap up at this point i i think
actually this is a better point to wrap
up
yeah yeah okay okay yeah uh
if if you're curious uh i think rule
posted the link to
the paper he mentioned i also posted it
a minute earlier and then
derek posted a link to another paper so
check out the chat
for the links um yeah if we don't have
any more questions then uh
thanks uh jared for for the really nice
talk and uh the discussion was really
lively and i enjoyed it a lot
and thank you all very much oh wait uh
we have we have we do have a question
final question just the final one okay
um so i i was thinking
about this different things that were
discussed and about like game theory
player of mine a lot of different
interesting approaches to do
this uh but
are we considering a specific goal for
the for the robot for the
mobile robot the the robot the goals of
the robot
is to go from point a to point b the
faster as fast as possible so are you
assuming some kind of thing there is
some kind of
goal of the agent of the robot itself
and if that would be the case wouldn't
he just push like
what kind of social cues you use
wouldn't the robot just say
maybe if i just a pushy person i can
imitate this kind of person that really
push everyone
arrogant i don't care about anything
that would be the best thing to do
given my goal so how do you see these
and how do you see this within a kind of
a
machine learning framework that could be
learned that's kind of show cues can be
learned
and can be used for a given goal
obviously this and
sure please share your thoughts uh
honest answer i would be super happy if
you could get to the point where we can
make a horribly pushy robot
so if if we uh sort of
got to that point of understanding yeah
where we know okay so this is how
pushing people on the street managed to
actually walk this quick
and get people to move out of their way
that said of course i don't think the uh
typically the goal for these kinds of
delivery robots is on the one hand
time efficiency so be quick in their
deliveries
but on the other hand also to fit in and
not offend people and and sort of be
sufficiently socially appropriate
and that would ideally result in a robot
that's not fully pushy but that can be
pushy if it
needs to be that can uh if a
bunch of teenagers is harassing the
robots
can say okay wait i'll switch to my
pushy mode and i'll
take my way through this bunch of
horrible teenagers
and get where i need to be
so i i think we need
several different behaviors and i think
that's logical because we have several
different things we want to optimize
uh i hope that sort of uh answers the
question a bit yeah uh just a quick
follow-up on that
and who decides that this is how this
goes his objectives i know there's a lot
there's a lot of different ways to do
but who should decide that that's also
there's no right answer but just about
how to do i i think
i think the the answer here is that this
is one of the reasons i'm super happy
that i'm at the design faculty now
yeah uh i i still have very much of an
engineering mindset i'm just trying to
figure out optimal solutions
i'm still trying to model this whole
interaction stuff
and i think that actually the amazing
thing about designers is that they
somehow often find or know
how to find a balance in this how they
do this
i'm still figuring out myself perhaps
they don't know themselves
explicitly either i do think that's
one of the reasons it's really nice for
me to be at a design faculty and to
see how they figure out that balance
between the different needs and
requirements
yeah great cool thank you very much
thank you
yeah with this old to the designers we
i guess we wrap up the today's talk
thanks jared thanks again and thanks
everybody for the discussion
uh see you next week oh wait wait we
don't have a
meeting next week so the next meeting is
in
two weeks and then we will have heavier
on some more representing
um about multi-robot navigation i guess
which is actually related to what chad
was talking about today
okay good bye thank you
[Music]


87. An Untrollable Mathematician

hello and welcome to the 87th session in
the AIA safety reading group tonight
Abram demske will present his work on a
non troll group mathematician which is
worked on as part of the agent
foundation technical agenda in the
machine intelligence Research Institute
this is a subject that a prime demske
has to worked on or at least hey I saved
you for at least a decade as far as I
can see
so a prompt asking quick can everybody
see the screen I think there was
somebody containing in the chat I see
this good thing okay I see it I guess
I'll proceed so yeah yeah so this is
something that I wrote up on the agent
foundations for him because it was
follow-up to stuff that I did but the
probability distribution itself was
created by some Eisenstaedt so it's
really sounds work that I'm presenting
but I'm going to start by going way back
in time too and I see this line of
research studying it was August 2014
so at that point in time neither Scott
nor I worked at me but Scott and I were
on our way to visit Mary I think that
was Scott's first visit to Mary and my
second or third and so yeah so Scott was
saying some things about the stuff that
we were going to go and talk to Mary
people about and so he thought of this
thing about how you can drive the
probabilities up and down
if you sort of naively have a Bayesian
reading proofs which I'll explain why
later but at that point we had already
been talking a lot with each other about
how you integrate probabilistic
reasoning with logical reasoning so I'm
going to explain a little bit about our
motivation there so the reason that we
wanted to integrate probabilistic
reasoning with logical reasoning or at
least a reason that you can give to do
that is because probability theory gives
you the tools that you need to think
about how to make decisions probability
theory is the way that you describe
expected value calculations but logic
has the best tools for thinking about
self reference so Miri was interested in
and still is interested in figuring out
reflectively consistent decision
theories so to do that you need both the
sort of decision-making tools provided
by probability and the self referential
tools provided by logic and in
particular a big question was logical
uncertainty so how do you reason about
logic when you don't yet know all of the
consequences let's see
oh I see I'm missing yeah I was not
saying how the full scheme is written so
yeah I missed this light so but I
basically make of it so it's like yeah
you have probability over here which has
a bunch of things inside of it and you
have logic over here and the question is
how do we take the intersection of that
so yeah in probability theory logical
omniscience is usually assumed logical
ambitions means that you assume that all
of your the consequences 0 your beliefs
are fully stipulated already from the
get-go so it's sort of hard to remove
logical emissions from probability
theory because the foundations of
probability theory the kamagra of axioms
they all sort of implicitly assume
logical admissions in one way or another
everything is specified in terms of
events in your event space and these
events are already assumes to be like
equivalence classes of logical
statements so the events have an algebra
in the Sigma algebra and the Sigma
algebra assumes that when you perform
and logical operations like engine or on
an event and you get a new event then
that event is uniquely specified by
logic as opposed to being sort of up in
the air about what other events it's
equivalent to and then well yeah there's
other ways the logical omissions is
slipped in and you can read about that
in the slides I'm not going to try to
cover everything I wrote down in the
sides due to limited time but I'll try
to give an overview
so their simplest way to see why it's so
difficult to get rid of logical
emissions from a practical perspective
as opposed from the axioms perspective
is if you have a bunch of hypotheses so
you can visualize the hypotheses as
putting some probability mass on one of
many like any number of observations in
your range of possible observations and
then when you see a particular thing
then you need to know how much
probability mass each hypothesis put on
that particular thing in order to relay
the hypotheses by a Bayes rule so if
we're unsure about the implications of
any hypothesis and when we make an
observation then we don't know how to
read weight all of the hypotheses
because the observation that was made
might be in this region where you're
uncertain about whether it's a
consequence or not and so in fact if you
were not very careful with this kind of
thing then allowing hypotheses to not
fully state better consequences like
this can let them essentially cheat the
system and sort of remain in play even
though they're not doing any work
because you can't distinguish between a
hypotheses hypothesis that has simply
not yet declared what its prediction is
versus one which will never declare what
its prediction it is and so what happens
is if you let hypotheses be vague about
their implications in this way then
again any if you're not careful malign
hypotheses that are too tricky you can
just
stick around forever because they never
declare any falsifiable predictions
until a key moment when they start
making predictions and then cause you
problems and of course when they do that
then they might lose their probability
mass but the thing is that there might
be large enough of a number of these
that they always outweigh the good guys
who are like always making predictions
because the good guys are going to be
losing some probability mass in the
course of making predictions whereas the
bad guys are not losing probability mass
until they're messing with you so anyway
yeah you get a lot of problems when you
try to do with this sort of thing and so
the challenge is to try to combine the
ability to be uncertain about your
conclusions with probabilistic reasoning
so going over it's a logic world to
explain why logic is so different from
probability and more difficult to
integrate with probabilistic reasoning
than most people realize so I mentioned
this thing of being uncertain of the
implications and this is very
fundamental to logic so in logic you
have kind of a situation where you have
the axiom which is like a seed and then
the rules of inference are sort of
telling you how to grow a tree of
implications from your seed of axioms
and this growing a tree of implications
is never finished and we can formally
say like the way in which this is never
finished by looking at gödel's
incompleteness theorem so you can
visualize it and this is a visualization
taken from Hofstadter you sort of have
the white tree of the truth which is
drawing from the seed of the accident
and then you have the black tree of
logical contradictions or things you can
disprove which is sort of growing from
the seed of the negation of your axioms
and the thing about this tree the the
visualization of Google's incompleteness
theme is that the white branches and the
dark branches injure time to intertwine
so closely that it's not possible to
draw a line that separates the two at
least it's not possible to do so in a
computable way so it's like right my 10
minute timer is gone so that's telling
me to hurry up so yeah so the fact that
you never finished drawing out these
implications means that these scales of
justice are sort of always unsure about
where to go the scales of epistemic
justice of evidence-based reasoning are
always sort of in this gap of
uncertainty between the light and the
dark and so getting back to what Scott
and I were discussing so Scott was
saying okay what happens if you sort of
forget about all that and you say well
I'm going to try to have a probability
distribution anyway I'm going to ignore
the fact that I can't really tell about
my equivalence classes for events should
be I'm going to take logical sentences
and I'm going to put probability mass on
them ignoring the semantics ignoring
what those sentences mean and then I'm
going to try to do a Bayesian update
when I find constraints so for example
if you like think about ever since a and
then the negation and the B and it's
negation
and you have some probability
distribution that doesn't obey any logic
that's over these the combinations of a
being true and false and B being true
and false and then you update on the
implication a implies B so you cross out
this square and then you update by
normalizing and so you get a new
distribution and the idea was just
updated on all of everything you can
prove in that way where you just rule
out the combinations of your proof rules
at and try to do logical uncertainty
that way so Scott's argument is that if
we do this then we can always be kind of
tricked by somebody who is showing us
proofs in a systematic way to try to
drive our beliefs in a particular
direction so what you do is suppose we
have some targets and say that we care
about then somebody can find a certain
say implies B they can kind of be and
this B such that a implies B is provable
and such that more than half or sorry
half or more of the probability mass of
a is on not be so then when we update on
amp lies B we're knocking out more than
half of our sorry half or more of these
mass and so a just keeps decreasing as
we do this so why is it possible to
always find a since B which has this
property going back to the picture of
griddles incompleteness theorem we know
that it's impossible to separate these
the trees of the true implications and
the
things we can rule out any way that we
try to separate these two trees we will
have some overlap of things that are
sort of incorrectly classified but if
you think about the situation with our
beliefs on a implies P supposing that
you could always avoid this trick that
means that for any B such that you can
prove a implies B you must have the
probability of B given or yeah you must
have the probability of not B given a is
less than one-half so we can consider
the sentence the probability of B given
a is greater than won't have and this
must separate truth from falsehood
because or rather it must separate
logical truth from logical faucet so it
must separate the things that you can
prove from the things that you can
disprove because if B is a logical truth
then implies B must always be true but
we know that there's no way to separate
truth from falsehood in that way because
of girl's incompleteness theorem so no
computable probability distribution can
ever avoid Scots trick completely
therefore Scot can sort of play this
game with us where I start out my pride
has the probability of a being one-half
Scott says oh by the way did you notice
a implies B 1 and I'm like oh yeah okay
so I update to probability of a is 1/4
and Scott says also did you notice a
implies B 2 and I'm so I say okay no
that's 1/8 and then he says implies B 3
and I update to 1/16 and so on so you're
always being driven down and then when
Scott gets bored of playing this game
you can start reversing it and driving
my probability as far up as he wants by
finding B that imply a because the trick
works the same in both directions so we
called this trolling in reference to
people on internet chat rooms telling
you about 30 people on internet chat
rooms arguing and bad fake saying things
that they that may or may not be true
but which are designed purely to provoke
a particular response in you this isn't
the best metaphor because the things
that the troll and this is saying are
always true but nonetheless the troll is
sort of fooling you dragging you in a
particular direction by saying false or
by saying only two things so we can also
think of the fishing metaphor where you
drag a line and then you're just sort of
dragging the fish you don't necessarily
take it up right away so yeah that's the
problem so then how do we solve it so
much more recently January 2018 we're on
a me researcher cheat so Sam Scott and I
all were in the same bedroom on this
retreat and so Sam said something I
don't remember but my response to this
was well last I checked you still think
you can solve show ability which you
haven't convinced me of yet this was
sort of an ongoing thing with Sam and I
[Music]
so I was reminding him of this claim and
he was like well do I think I can do
that and he thought for a minute and he
was like yeah I still think I can do
that and Scott and I were very skeptical
because both of us
given up at this point on there being
any kind of solution and then about five
minutes later Sam sort of he paced back
and forth out in the hallway and then
came back in the room and said okay
he's the solution so the priority
proposed is very simple
so he said suppose that you're drawing
students is out of this bag with some
distribution and you're drying them out
of the bag and some sequence and this is
a very old chicken so in terms of like
this is the first thing I had tried back
in 2011 or whatever when I really
started doing this logical uncertainty
stuff but he said you enforced only
propositional consistency on the
sequence of sentences and you think of
nature as showing you the things in the
sequence so previously I had thought of
I'm skipping around a bit previous lis I
had thought of the way this sort of
process works as when you observe a
things you only know that it is one of
the true senses you don't know which one
and you're trying to infer the rest of
the truth which are like Laden laws of
the world and so this probability
distribution on drawing things from the
back which is new gives you a simplicity
are over the laws of the world and then
you give some observations and you try
to infer what the early laws of the
world that would have constrained things
to be as you observed with bearish but
Sam was instead saying no we draw
sentences out
the bag and we immediately see them and
this was a big change and so the reason
why we enforce propositional consistency
on top of that is because propositional
consistency gives you constraints
between sentences the expressible in the
language itself so if I prove that a and
B can't occur together for example then
I can express it as the students not a
and B and so anything that we prove we
can kind of take it out and put it as a
sentence so if you think of things in
this way you're ruling out branches that
you've proven cannot occur in the tree
of possible sentences so then why does
this work well the probability of any
sentence s which is consistent with
everything seen so far can't go below mu
of s so it can't go below this sort of
drying out of the bag distribution which
means and it can't go above 1 minus mu
of the negation of X so it can't it's
sort of stuck between the probability
that we draw s out of the bag and the
probability that we draw not ass out of
the back which means that no matter what
we see in the sequence so far we can't
we can't have driven the probability the
best above this certain constraint or
below so we can't jump we can't get it
above 1 minus P if not s and we can't
get it below 1 minus there was we can't
get it below probability of s which
means we're not rollable so our
probabilities are always in this
sort of comfortable arrange and yeah
this it was sort of a very obvious
tactic adding to the embarrassment of
Scot enacted five minutes before that I
saying that it wouldn't be possible
so yeah this trick of observing
distinguishing between updating on X
versus updating on I observe X is a very
old one in Bayesian ISM it's well known
that this is a critical distinction that
can make a huge amount of difference so
if we have like some fraction we're
trying to infer what the fraction of red
balls to blue balls is and we only are
only shown the blue balls then and we
have our prior over what the frequency
is when we can see the blue balls on all
the red ones that hidden from us then if
we were to update on just ball three and
ball four are blue then this situation
here would increase our credence in
higher frequencies of blue balls but
that isn't right because we know that
we're being shown all and only the blue
balls and there are only two out of the
five and so we should update on IC ball
three and ball four our blue which
updates us right down to two out of five
our blue and we know that frequency
exactly so similarly it was natural to
think this god strolling trip could be
overcome by understanding the
distinction between updating on implies
B versus updating on observing a implies
B however we'd had that idea for some
time without coming up with Sam's trick
the thing is that if you're sort of
trying to figure out the natural way to
do this
then you think of things like well you
have to model the TM trooper just
showing you things so that when you
observe something you're not just
updating on the thing you're updating on
the theorem provers showing you the
thing and so if the theorem prover is
drilling you in a biased way then it's
sort of natural to expect that this
would let you pick up on that and say
aha
the theme prover is only showing this to
me because it searched for something
that would increase my belief in a
particular direction and so I'm going to
not be fooled but the problem with this
is how do you model the theorem provers
it's very unclear how you should do that
so we never came up with anything
particularly satisfying but Sam
basically got past this just by not
trying too hard to model things in the
right way he was only trying to come up
with a prior it wouldn't be trouble and
so in fact his prior sort of gives up on
intuitive things that you want I
mentioned earlier the inferring the laws
of the world and he can't do that
because everything is observed so you
observe a and you observe B and the
subsequent sentences are still mostly
random they're constrained to be in line
with the proofs that you know but there
you have no ability to infer any
underlying regularity so you can't
figure out what the laws of the universe
are and this refusal to adopt very much
this refusal to adapt to the data
although it is bad it even more than
just on troll ability it implies full
convergence so troll ability is sort of
a weak statement about convergence like
or yeah
unchill ability is bigger than
convergence the reason that I was
looking at troll ability and stuff and
regions before was really to show how
bad the lack of convergence was in the
situation that's Casa de so troll
ability is just like really really bad
divergence as bad as you can imagine
because you're being dragged between
zero and one infinitely many times so
Sam's thing actually converges and I
won't go through the proof of that here
but you can look in the post so that but
intuitively it's because as you see more
and more things coming out of this bag
then the things that we will never see
come to dominate our expectation of
what's next because we're not really
adapting our probabilities to account
for this sort of things that we see so
our expectations of what follow will
converge to a set distribution so yeah
that's all I have no clue that was good
thank you very much and very much


To use the human or not to use the human  (Erwin Boer) - 1st AiTech Symposium

thank you it's a great honor to be
invited I always like to stretch myself
and David already has given quite a
beautiful background of a lot of the
research that he and I have done
together a lot of research in terms of
human modeling that human modeling
understanding how humans control systems
can be used to diagnose people so if you
have a person with glaucoma
you know they control something
differently because their peripheral
vision is different so that
understanding helps you you know in the
context of driving to use these types of
models for diagnostic purposes we've
used human models in this types of
shared control that David talked about
we're also using them as templates for
autonomous control so we're looking at
how do people adapt to the environment
how do they perceive the risk how do
they take into consideration or the road
users in terms of vulnerable road users
how quickly do how early do they slow
down how do they swerve around them very
different than a typical engineering
system might do that because you know
they start with different assumptions so
we try to already take into
consideration starting with
well-established human models but
shaping them based on how people adapt
to the environment like I said I'm going
to stretch myself a little bit and talk
about a whole bunch of stuff that I
haven't actually done research in but in
the context of meaningful human control
I think it is is a number of factors
issues that I think are important and
even try to apply some of the knowledge
tracking aspects that are in the paper
by yeah you don't so the question is to
use humans or not humans david advocated
you know we should just have shared
control
we should always use humans humans
should always be part of the design loop
and the decision loop but are there may
be situations where humans might want to
drive autonomously or where humans
really might want to or might need to
drive manually so I want to kind of
break up that space in terms of how can
you think about it how can you engineer
it what kind of decision techniques do
we have to talk about that previous
speaker talked about do we need to be
here do we not need to be here this is
professor in a gawky who has made a
robot looking just like himself
basically this robot can go to
conferences and can stand in front of an
audience and speak you know as if he's
there and either he can speak directly
through the robot or the robot can you
know read something that was
pre-programmed this is clearly something
where you know you have the same human
you're really pushing it to you know
replicating essentially a human which is
an interesting idea AI and in robotics
from a technological point of view but
is that really what we want do we need
humans or do we need really something
that works well together with humans and
augment human so towards meaningful
control like I mentioned in the
incarnation of autonomous driving there
are different ways to do that one is you
know sometimes the human wants to drive
you know I bought a Porsche I want to
drive this thing manually I don't want
to drive it autonomously sometimes the
system you know the system is when can
the system do that if the system is
confident enough maybe you can drive
autonomously what does that mean and we
have levels of automation there
blanketly applied everywhere we'll talk
about that in a second sometimes you on
both you know David very much advocated
the combination of the two and sometimes
it's very much a few symbiotic
relationship David didn't really touch
upon that but when you couple a human
and a machine you have this symbiosis
where you can teach each other you can
touch each other you can protect each
other and together you learn you know
under what circumstances is the robot
machine may be better under what
circumstances is the it's a human better
and the human can teach the robot and
vice versa so it's a beautiful mechanism
very much you know the way we interact
with humans and we challenge each other
we teach each other we not each other we
protect each other so the question
really is when to use what mode then I
have to walk back and forth each time
one thing that already came up is this
notion of bounded rationality we don't
know everything and then do we even need
to know everything Herbert Simon coined
this term bounded rationality and from
that a whole different type of decision
Engineering came about called
satisficing decision-making it's not
about trying to maximize everything that
is important and in optimal control it's
not about you know maximizing
performance and minimizing this it's
really about setting aspiration levels
and once you have reached those that's
good enough that's extremely important
in critical situations because in the
critical situation you don't really have
infinite time to find a better solution
or a more optimal solution so in order
to be able to quickly make decisions you
need to have a very good understanding
of what is enough evidence for me to
make that decision later on I'll talk
about how we can actually quantify that
evidence accumulation and how can we
maybe have some meaningful ways to talk
about whether it is sufficient
information or not yeah so some of the
things to come about is you know what
evidence do we need you know what are we
actually collecting when you look at an
AI with sensory and actions out you
don't really know what the AI is using
to drive its decisions it's important to
know that especially from a meaningful
human control point of view you know how
much evidence is enough you know when do
we believe that okay we can leave that
system alone and it can do it
autonomously you know how much time do
we have is important and can we get the
evidence in other ways you know we have
discussions that you know within AI or
with the robot maybe not everything
needs to be inferred maybe you can just
talk to the robot or maybe you can just
instruct a robot to do something yeah
sort of all forms of interaction there
from a technological point if you we try
to make these things as autonomous as
possible but if you switch back to make
them more socially interactive you have
a whole way of exploring all the ways to
to teach that robot and it's called
epistemic probing now you also see that
as cars come to an intersection you
nudge it a little bit forward to see how
the other one responds to that it's
you probe the system sometimes you just
want to have more information really
augment in the human this is Neil
Harbison is one of the first cyborgs so
the first side works recognized as a
cyborg where you know his visual system
was enhanced he's called a blind he only
sees one color so he put a camera on his
head and now he can see from infrared to
ultraviolet and he has a perception of
the world that's much richer or
different at least than what other
people have yes so here you have
technologies directly integrated with
the person and if you have all this AI
symbiotically working with you that's
very much what it feels like
that's this greater consciousness that
we can all tap into for example so to
use a robot not use a robot we talked
about closed environments where we can
do quite a bit we can predict everything
we can control everything like this
Hotel in Japan it exists you can go
there
you surf by dinosaur it's wonderful but
in an open environment where we can't
understand everything we need to have a
much you know richer way to actually
talk about you know what does it mean to
send my child with a robot to school
yeah are the conditions under which I
would accept that or the conditions
under which not so context might become
very important this is one of the most
ugly slides ever created
I think about changing it all the time
but I don't think I will
it's really just the capturing of
coupling a human at the top with a
machine and you see the duality yeah
when humans interact there's all kinds
of ways in which we share information
and mimic each other and cooperate etc
etc that is essentially a very large
duality if we have that duality also
with the systems with a system you know
all the benefits you have of how you
interact with the human you can also
have with the system and vice versa as I
mentioned in the learning to sharing the
nudging the protecting
one of the issues that has been
discussed in a VA ssin an aviation has
looked at automation for a much longer
period than then driving in a lot of the
research we're doing right now comes
directly out of automation which is a
whole different domain but one of the
issues is handover you know like in the
rejected take off an engine fails or an
engine catches on fire you know do you
abort takeoff or do you slam on the
brakes it all depends on the speed
you're going at the length of the runway
there are a whole bunch of factors that
come into the picture and does the human
trust the sensor systems is the system
you know sufficiently aware of its own
limitations does the system actually
have a good understanding of the you
know dynamics of the airplane with the
weight that's in the airplane etc etc so
all these different factors and
professor in a gawky looked at this
problem and brought to bear a technique
called Dempster Shafer evidence theory
to combine all these different bits of
evidence into a decision you know
whether to take over or not so how to
bring all these different factors into
consideration I want to focus on that a
little bit before I do that Trust is a
very important aspect and Trust is a
very dynamic aspect and when you start
interacting with a machine you have some
sort of a priori trust in how you
interact with it I believe that you know
this person this robot is capable of all
kinds of you know wonderful things and
then we interact with the robot maybe it
doesn't quite work out so you trust very
quickly dies where is this child has the
same technology built in your
expectations are a lot lower and you
interact then you built this mutual
understanding that is much better
calibrated to where you you know
ultimately end up that's the same thing
with technology you know how do the
companies present this technology how
does the media you know react to you
know one little failure that might
happen and we all believe what on most
cars are absolutely awful for every
autonomous car crash I could also look
at you know 10,000 poor kids that were
you know killed by drunk drivers for
example yeah maybe not 10,000 but a few
hundred that would be killed at the same
time so it's very much depends
how the media presents this information
to us I'm not saying autonomous cars is
safe very much with David in terms of
this whole shared control is much better
but we can definitely control that so
when I read the paper on meaningful
human control there's a lot of stuff in
there and this morning it was mentioned
that it was too early in the morning to
to dive into this math so I think
everybody is awake enough now so perhaps
we can we can do that a little bit and
there's a notion called truth tracking
yeah so all kinds of aspects of truth
have to be true and let's talk about
that a little bit so if you have a
particular fact say the system is safe
and the belief whether the system is
safe so then you have four conditions
that essentially need to be true for the
system to be truly trackable in
knowledge and have all the aspects so
that basically means that if the system
is truly safe then you believe that it's
safe if the system is not safe then you
believe it's not safe and then also the
reverse where you if you believe if you
believe is that the system is safe then
it is actually safe and vice versa and
then you can extend that where you have
these beliefs that then ultimately
relate to actions yeah so some of the
practical implications are that you need
to actively establish evidence for these
different facts in order for you to have
a belief whether it's true or not the
two directions you can co go you can a
priori assume that or you actually base
it on some knowledge and that knowledge
might come from somewhere else that
might know what knowledge might come
from the trust you have in a company etc
etc but there's some evidence that need
to be established and the same thing
evidence for not safe and then use this
evidence to essentially built a belief
structure about safe and not safe so
you're interacting with the system
sometimes it works sometimes it doesn't
work and in the end you end up with
something that I'm not quite sure or yes
I trust the system completely or not
completely and then you combine these
different belief structures
into a belief that you have whether the
action is meaningful or not or safe to
perform and the action depends on your
confidence in your beliefs about these
facts whether whether you have evidence
for it being safe or not safe yeah and
then there is this notion which taps
into this whole satisficing approach is
you can act now where you can wait a
little bit and collect more evidence you
see a lot of systems where you just wait
a little bit longer you collect more
evidence and you can base your
information on better better knowledge
so when you talk about this evidence
accumulation and finding out whether
it's safe or not the different policies
that you can adopt yeah there is a
policy that's safe until proven unsafe
so yeah priori assume that something is
safe until it's proven unsafe I'll show
you an example in a second where that
was very catastrophic or you say believe
it's unsafe until proven safe you know
some of you might recognize that we have
very similar things in the different
jurisdictional systems in different
countries like in the u.s. you're
innocent until proven guilty but in
Mexico you're guilty until proven
innocent so they put you in jail and
then it's up to you to prove that you're
not guilty it's very similar to this
this seems very silly this is what we
also do okay so now we have the need to
understand whether a system is safe or
not how do we do that there are two
techniques this morning Judaea parole
was mentioned with this book causality
he is very much a believer of Bayesian
networks and Bayesian networks have been
extremely useful or extremely powerful
but are they really most easily
interpreted by humans no are they
perhaps the best way to go about
integrating evidence and belief that
people might have perhaps not dempster
and Schaffer back in the 50s came up
with this possible astok tech
nique go and then later was coined after
them Dempster shade for evidence theory
think about this particular case a
murder was committed and you have some
evidence that this particular person is
the murderer your evidence is point for
the whole structure is just like
probabilities has to add up to one
etcetera etcetera we'll go through that
in a second in a probabilistic approach
of the the Bayesian approach and the one
that we're all familiar with and thought
how many of you are actually familiar
attempts of shape of evidence theory
okay four or five in a Bayesian approach
you have probability that this person is
a murderer
it's point four well that means there's
a probability of 0.6 that he's not a
murderer well what does that really mean
do we have any evidence that he's not a
murderer no we only have evidence that
this person is a murderer and the rest
is just a mathematical convenience
so what dempsey shaver evidence theory
tells us that okay we have some evidence
that it's a murderer and the rest is our
what's called frame of discernment what
are all the options that we have in in
what we're trying to decide so it's
either safe or it's not safe or it's
both yeah so the rest of the total
belief that you need to assign which is
one is 0.6 so 0.6 is idream urder or not
a murderer mm-hmm this is important
because we have hard evidence that this
person is a murderer and we have a
possibility and that's why possible
istic that this point four might become
point might become one if the rest of
the evidence we collect all points in
this direction or it might be conflict
thing where there might be additional
evidence yeah
so don't have to go through the whole
thing here but essentially it's you have
a frame of discernment which is safe
unsafe what a combination of the two
which is the powerset you have a
particular assignment function you have
a way to combine these assignments which
is just if you have a belief structure
you can have two experts providing
evidence or you can already have an a
priori belief for yourself and then you
get new evidence or you combine the
evidence that you bring in with what you
know already and you do that in
essentially this way you know what
ever a and B you have evidence for which
are part of the powerset as long as you
know the one you're trying to compute
the like safe for example has this might
be safe and this might be both there you
can combine them this way and then you
have a null set where you might have
evidence for safe and evidence for
unsafe which is a conflict the
interesting thing is that conflict is
actually very nice construct if you have
conflicting evidence what do you do with
that and it's one of the things why
Dempster Shaver evidence theory hasn't
been applied very strongly although the
friends Millett French military uses it
for a lot of it's a fusion data fusion
sensor fusion but the the way you can
combine this conflicts and now you have
evidence for safe and unsafe what do you
do with that you can say I'm ignoring it
I'm just assigning part of it to you
know whatever I know already or you say
no my entire conflict goes to ignorance
which is very useful because then you
say okay I have a little bit of evidence
for for this is a murderer
the rest is conflicting so that's
basically equivalent to to ignorance if
you look at you know a driving situation
and you look at the evidence we have in
evidence for safety it tracks lanes
accurately well maybe it only does that
when it's nice weather tracks the lead
vehicle accurately in stable or dub safe
gaps etc and then you know in rainy
weather it doesn't do so well and in
high traffic density it behaves
differently so very quickly you see a
very complex sets of evidences and
confidences emerge that are very
situation specific and one of the things
I just don't see enough when people talk
about levels of automation and this and
this and this there's no mention of any
context it's very clear that an
autonomous car on a beautiful well
striped wrote in clear weather should be
able to drive autonomously yeah with the
same accuracy as people and probably
higher because people get super bored on
these long roads that are straight and
yeah so those are situations where maybe
the human doesn't have to be in the
liberal yeah so we can collect evidence
we can have the human collect the
evidence and then decide do I trust the
automation or not so it can be
evidential or you know the system can
provide some a priori evidence for that
one of the other nice things about
Dempsey shavers evidence theories this
notion of belief in plausibility
the belief is essentially all the hard
evidence do you have for the system to
be safe the plausibility is that you
know everything I don't know yet so
everything that's still in in my
ignorance can I to be assigned to safe
or unsafe so my plausibility is the
maximum amount of belief I could have
given the unassigned beliefs that are
still on the table you can have the same
thing for unsafe yeah so now you have a
very nice way to talk about how much
evidence do a half how large could it be
you can do nice simulations with that
and depending on you know what type of
combination rules you use so you start
with I have point eight evidence that
it's safe point nothing unsafe and the
rest I don't know then you have an
observation where it's safe unsafe and
ignorance and say you have a whole bunch
of these observations and essentially
you can track how your belief in whether
this system is safe or not tracks over
time if you ignore all this conflict in
here you will actually gravitate to
something that says okay I think the
system is safe purely because there's a
little more evidence that it's safe than
unsafe very dangerous if you take these
ignorance into consideration on this
conflicting to consideration and keep it
at ignorance you see that your evidence
for so this is your belief for being
safe is a bit more than 0.4 plausibly it
could be almost point eight but you know
we haven't found the evidence yet so
this is essentially something okay we
don't really have enough evidence to say
something meaningful here yet to make
the decision
so there are two policies safety
policies that NASA employed on a safety
preservation offered to two of the ones
that can be considered NASA employed to
the fault warning one where the safety
preservation is it's unsafe until proven
safe so if you don't know whether a
system is safe you just assume it's
unsafe so and this again very much
depends on conditions so you operate
only when evidence for safe operation is
sufficient it makes a lot of sense right
and then you can turn it into
temperature evidence the area where
based the dis is purely based on the
belief that you haven't safety all the
evidence that accumulates to them for
warning is safe until proven unsafe so
shutdown only when there's sufficient
evidence for unsafe operations operate
operate unless evidence for unsafe
operations is sufficient and this is
based on the plausibility of safe which
is 1 - belief which is always a much
higher number yeah so in this case you
would much more often go so if you look
at this this Challenger accident in the
1980s 86 so they adopted two safety
control policy safe until proven unsafe
but the context within which they
launched that morning they had never
experienced before he was very cold and
they had you know no evidence that the
o-rings would be brittle in that
situation but because of this policy
they decided to go had they understood
that okay there are contacts now it's
freezing we don't have safety and they
had adopted the other safety policy this
thing would have never launched yeah so
again context is very important we can't
just saying this thing is safe under all
conditions it has to be very contextual
and again I don't really see that enough
and systems might be very aware of your
own confidence in particular situation
particular context whereas in others
they might be completely uncertain about
their their own confidence
so the effect of meaningful human
control there are two aspects one is
tracking knowledge so reasoned
responsiveness and tracing ownership I
want to focus a little bit on the
tracking and so the issues that
unsubstantiated evidence was not used
instead of factual knowledge yeah so
unsubstantiated evidence I mean there
was basically no evidence that this
thing was safe yeah you adopt the safety
of policy created an a priori belief
that the system is safe yeah so again
people make assumptions without evidence
and I think that happens a lot you know
we step in the car and we pretty much
believe it's safe and we have no way to
really try except for you know
micromanage this thing but like David
mentioned after 10 minutes we can't do
that anymore
we just don't got the mental capacity to
monitor something that basically works
pretty well so if you think about this
is tracking for the Challenger task so
the fact is that the system is unsafe
one second Thea yeah so the system is
unsafe this is a fact the system is
unsafe and the freezing launching
conditions it's a new concept new
context that was never experienced the
belief that we have is the system is
safe in the freezing launch conditions
without counter evidence so we're
basically completely contacts techno
stick and then you have these two
policies which you can write in this
this knowledge tracking framework if it
were the fact that you know the system
is unsafe then a human would believe
that the system isn't safe this is not
true because the human assumes the
system is safe until evidence for unsafe
is enough so that knowledge tracking
aspect is violated in this condition and
I you know I really like the way the
meaningful human control laid out these
types of things and you know the fact
that we can apply to these real world
cases it's quite useful and it's
essentially you know if it were the fact
that you know these types of subjunctive
condemned conjunctions of subjective
conjunctions are very useful there and
in the safety policy if it were the fact
that the system is say
and then the human would believe that
the system is safe she's also not true
because the human assumes that the
system is unsafe until evidence for
saviours enough so it both policies have
an issue so essentially that suggests
from this truth tracking point of view
that any policy not grounded in evidence
collection is basically doesn't satisfy
the truth tracking criteria of
meaningful human control and any policy
not sensitive to you know context
specific evidence also and also we have
that in autonomous driving again there
are lots of situations where the systems
might be perfectly safe but if you look
at you know these types of conditions
totally acceptable
but you know would we drive our Tesla
here the same as there some people might
and depending on what they actually know
about the system it's okay drives here
it drives it drives you know as a whole
not understanding the technologies is
all over the limitations they might try
it here and the same thing if you look
at you know the dam leur perfect image
of what you know autonomous driving
utopia looks like you know the lighting
conditions are perfect and no hard
shadows there's no rain there's no snow
and there's nothing you know everybody
perfectly walks in the center of the you
know sidewalks you know etc etc yeah in
this condition it works very well so if
we train everybody in society perfectly
you know we might be able to do
autonomous driving but you know this is
this is more like the real world in that
same context and this picture similar
picture has popped up we have these
levels of automation you know we have
the same thing with let's go back to
that the second these levels of
automation that have been proposed for
different decision support systems for
different automated systems where
essentially you know the computer offers
no assistance and the human does it all
to you know the computer executes a
suggestion if the human approves it etc
so the human gets brought into the loop
more and more depending on some rule
when that rule specifically comes from
you know whether it's the engineer
saying okay we don't trust the system
and the human really needs to have the
final authority or whether it is the
responsibility issue you know people
have thought about this and have applied
to two autonomous driving as well and
you know you see that here there's no
automation you know the five levels of
automation by SAE to you know assists it
to you know full automation always in
everywhere under all conditions you can
just send your car with your kids you
know you don't even have to look at the
weather kind of thing doesn't matter
where you live you know we're barely
here at this point and and there are
many conditions where it doesn't work
really very well we're pushing towards
you know a control Society and I think
you know that's very dangerous and it's
we tried that back in the 90s right we
have dedicated freeways and we
demonstrated that we can drive cars
autonomously on that if we have enough
autonomous cars we can see a shift to
okay let's take part of the city and
part of the road network where you know
these cars drive and we leave
pedestrians out of there or we make it
more difficult for people to pedestrians
so we could gauge their so they can only
cross at certain crosswalks and you can
control the environment and you can plan
your cities around that to essentially
facilitate more or less imperfect
automation do we start with the
imperfect conformation and control our
cities or do we say now we really need
to wait for the automation to work well
to not completely disrupt a society that
we want do we even know what society we
want probably not right now the safety
policies you know who applies those the
system who decides you know does the
human decide the system uses or the
system decide the human usage right now
we're very much in this camp here
technology is to a certain degree and
humans will adapt right the human
factors world complaints all the time
it's like we always have brought in you
know when a system fails and you know
they need to understand the human
factors you know how can we instruct a
human to deal with those limitations in
the design properly it's a completely
wrong way of going about it yeah but you
know this type of duality and design and
engineering you know looking at it from
a systems point if you're looking at
from a human point of view is a great
technique for gaining insights and David
and I have used that very much in terms
of you know how do we couple a human to
a machine so one of the things I'd like
to propose is I've showed you that you
know with the stems to shave or evidence
theory you can take all the evidence if
you know what what the important bits of
information are you can combine that and
you can come up with this belief that
the system is safe given a particular
context and then you can have this this
model where you say okay in this
particular context I have a high enough
belief that it is safe and therefore if
the human chooses to use automation
that's great they can still you know
maintain or continue a cooperative
driving style what they can do in manual
and so those options are always there if
you're in between then it's really
cooperation and you get into the
situation the David also described where
you know it's the combination of both is
better than either one individually and
then you know if there's just really no
evidence that this thing is safe like
Google has only tried you know has
driven 90% of its its data you know
around San Francisco what happens when
you put it on a rural road in Iowa it's
a whole different experience and we
don't have any evidence there it may be
safe but we don't know yet we already
talked about that you have all these
levels there's no context you know
implicitly sometimes it's assumed that
you know in you know on freeways we
start but these systems don't have to
you're fencing you can drive them off
the freeway and try it in the city they
might have speed limitations it doesn't
work on the 45 or I'll just write faster
than 45 in the city sure I get the
benefit of my system so that's you know
there are lots and lots of technologies
that we could bring into the picture but
from an economic sales point of view
it's not done yeah indeed
the other thing that has been mentioned
a little bit is systems need to become
self-aware I just throw that out here
but that's one huge thing with AI you
know how capable is AI of saying okay
I'm very confident to recognize you know
this group of people or to drive
autonomously in this situation or you
know take over in the critical crash
situation into that condition and
they're working on you know AI to have
better estimates of its own limitations
and an understanding and traceability
but it's at the moment only used in
academic labs so leave with the same
note that David there is so much
uncertainty and so many things we don't
know and even you know so many
engineering techniques that we still
need to try what works best in terms of
how to build evidence and combine that
that you know there's a symbiotic
cooperation and you know even using the
human to teach the automation
I'm surprised not many more car
companies do that but it's also
partially because you don't really know
where that goes to and then you need to
have a watchdog around that who's gonna
design the watchdog catch-22 thank you
thank you
exactly so we have time for a couple of
questions of course with the books ooh
may I invite whoops sorry thank you very
much there you go sorry yeah thanks so
much for the your talk that was very
inspiring so I have a question
we've been recently trying to put
meaningful human control inside a
classic control loop one of the things
we have been smashing our heads against
was quantification of the many normative
values that are sort of implicitly
included in the meaningful human control
theory so all the knowledge the moral
knowledge and awareness of a certain
user agent in the system even like
things that are potentially easier to
quantify like safety do you have since I
have seen sort of you have decision
structures and theories do you have
suggestions from how to get to those 0.8
certainty or belief that something is
safe or it isn't so how do you get to
the quantification of certain particular
values that determine assessing yeah
it's extremely art and in some ways it's
really to explore it in in the real
world and that's the difficulty many of
these systems can be can't really be
evaluated unless you throw them out in
the real world you know with the snow
and the variability etcetera etc and
that's why it's it's an infinitely more
complex problem than you know the closed
systems in which we had AI and robots
living but you know we've thought a lot
about you know what what does it mean
mean to be safe and one of the aspects
is that in the human has time to take
over so and we have had workshops in the
domain of almost evaluation so all of
you who have designed systems probably
generally evaluate them within the
context within which you designed them
you never really test them
outside that contacts so honest
evaluation is to recognize that people
actually use it outside their context
and see what the implications are there
like you know sensor failure in an
autonomous system is very detrimental if
the human isn't connected it just takes
the human way too much time to respond I
mean you can go with reliability
engineering and things like that but you
know it's really in the context and I
would try many of these systems to
actually have humans as sensors and
evaluators in the loop
you know in complexin of situations we
were in the workshop yesterday where we
talked about micro worlds and and this
morning we talked about you know kind of
like an open city planning environment
where you can try these different things
so you can look at the dynamics and we
have fairly good models of different
responses but in many cases we don't we
don't know when you introduce cell
phones into a society what's going to
happen to people you know we have no
predictions so in some ways it's very
hard to have meaningful human control
and evaluate that because we don't
really know what the impact of society
it's good to think about it and have
everybody think about it and a lot of
mistakes will be avoided but at the same
time the implications and the dynamics
and the shockwaves that it throws into
society or very unpredictable yeah we'll
be using Likert scales for instance like
psychometric methods to assess some of
those but that's why I'm sure there's
many different ways one thing I would
mention quickly though I mean if you
take this satisficing approach and the
problem with optimal approaches where
you have to combine all these different
factors okay and need to look at you
know is it traceable and what's the
impact on society what's an impact on
the human what's you know this and this
and this you have a weight on all of
those and you try to maximize that as a
control engineer you know one of the
things I always say that for every
behavior there is a criterion that makes
that optimal yeah because you can change
your weights and then you'd its optimal
again so if you take a satisficing
approach where you have all the
stakeholders say this needs to be met
and then unless that is mad it's not
good enough right doesn't matter if
there's a trade-off it's something else
no so if we can establish these
trade-offs and then take
the spicing approach I think that's a
much more practical tractable problem
than going with something like an
optimal approach and we've done that for
flight planning and things like that
with noise cancellation thank you can
you throw the box to the right-hand side
took a heart so this control theory you
are already sort of in a system where as
you indicated visual formulas you
operate with tools and so on
let's take us in examples in 737 max who
should be involved in avoiding such
issues where potentially Boeing knew
that there are critical situations which
you may label unsafe but to the world it
presented that this is a safe airplane
so this is outside the formalizations
there are behavioral psychology kind of
trying to take advantage of certain
situations for economic benefits and so
on so you as a control theory person I
mean who is responsible for this facet
oils is also in your sphere of interest
and we think in this particular case it
was just a lack of redundancy in the in
the sensor system so that is definitely
a case where you an engineer made a
shortcut probably because of cost cost
implications to not have the redundancy
that you really need in the safety
critical situation so when the
redundancy comes from a human if it's
easy to assess by human or whether we
done that see comes from completely
different you know sensor systems and
then you can apply this type of you know
belief structure and you can have you
need to have you know sensors that can
detect you know whether a sensitive same
sensor is actually failing or not you
know partially through consistency parts
partially through modeling partially
just through the dynamics that you've
exploration you know observed in the
past so there are ways to to do this
type of stuff but in this particular
case it was just a redundancy that
wasn't there and then you know the
pilots didn't have the time to to figure
it out with these very complex systems
thank you very much
Evan let's thank the speaker again
[Applause]