202. Gwern on GPT-3

hello and welcome to session 202
in the aisafety.com reading group
tonight we will be discussing an article
called
on gbt3 meter learning scaling
implications
and deep theory by gwen
warren branwyn is a freelance
writer and researcher from america he is
a very very prolific writer who has
written
and experimented on a very large number
of
topics this particular article was of
course
published in may 2002
quite soon after the release of team t3
but it has been
continuously updated since then meaning
that uh
precise the version that i'm using is
the version that was current for like
a week ago um so it's possible that
there will be thing in this
uh if you come back to the article later
that i won't have covered for that
reason
i should also say that gwen branwyn is
not his real name he cares very much
about his
his privacy and
he is however um very uh
in the rationalities fear the
rationalist community he is
a very well-known figure and he has um
um he is taking uh surprisingly serious
he is not just a random pseudonym from
4th gen or something like that
ok let's talk about the generative
pre-trained transformer
3 and of course we had the gpt 2
earlier surprising everybody by being
able to both understand and generate
natural language
and tbt-3 is the largest neural network
ever
and this would under normal
circumstances be expected
to have substantial diminishing returns
uh
open ai in gpt2 the gbg2 paper
predicted that it would not have
diminishing returns and
it of course turned out it did not and
not only
did not have uh diminishing returns but
it was
in a very uh it was qualitatively
different
it was uh doing some things that uh
learning some ways to learn that
the uh original gpt2 seemed to not have
been able to
um the uh
the history of uh of the
of course the the idea is that
this extra compute the gpt3 got compared
to gpt2
uh enabled it to do some things and
one looks back at the history of uh ai
to us to see how much compute has been
able have
have increased this is following ai
impact actually
we have the perceptron all the way back
in the 60s
and there seems to be a two-year
doubling time roughly
going on following uh moore's law
until the uh 2012
and from there we see a 3.4 month
doubling time
um finally um this is not from a
ai impacts graph this is what warren has
added gt3
with 175 billion parameters
and i was actually a bit surprised
that gwen put this at so low on the
y-axis i would have thought that it used
substantially more compute but here it
seems like it uses less compute than
alphago zero
i am confused about that and i notice i
am confused
so um the fact that compute
just by itself brings improvement is
uh perhaps not very uh uh surprising
but the fact that it only provides that
uh
foreign there are a number of means
where alphago lisa doll the the version
that feed lisa doll
um did a lot of very interesting
uh domain specific things um whereas
alphago serum
obtained basically the same slightly
worse performance
by basically just stacking layers uh
doing something
um not trying to do absolutely nothing
smart
and um not even some of the things that
look like there are easy wins
um they just put gave it more compute
and then performed
much better this is not an isolated case
one have a number of examples of
other projects that seem to have
been improved massively through extra
compute
in particular becoming more stable there
is
all these examples and 16 more examples
of research projects which have improved
substantially just by having more
compute
um uh i was not particularly uh
impressed by just the listing of 23
separate projects
that have this um there seemed to be no
obvious methodology
for how warren chose these 23 projects
there are
quite a few ai projects so just the the
fact that 23 of them
have this doesn't necessarily establish
a paper
so here are some graphs for how gpt3 in
particular was
scaled and as you can see the uh the
loss
which is a a good proxy for how well it
actually performs
seems to scale extremely leniently with
compute
data set size and the number of
parameters
and of course they're very clean and
they're
not bending at all on these charts here
gt3 compared to the
previous state of the art is roughly uh
one thousand a factor of one thousand
which is uh enough that this uh
general graph isn't just um
it's not just a fact of gt3 it's uh it's
a more general trend
um and the uh the conclusion coin takes
from this is that
simply made very large and trading very
large data sets
with very large compute that's the thing
that we need
to have ai um at least
to get stabilization um
and i feel there is a point here that
these three things
are um are correlated very tightly
with this it seems like you it's not
enough to have
much more compute you also need to make
to get corresponding amounts of data and
you need to increase your model size
correspondingly and
uh the description that um or the
characteristics
that one takes from these 23 examples
are the so
that the problems vanish the models
become more powerful more generalizable
more human-like
there is indeed a blessing of scale that
goes like this
for deep learning everything gets better
as it gets
larger and this is again a
counterintuitive
results in many uh
i i've studied algorithmics where it's
indeed true the small things are hard
and large things are impossible um
get building things larger larger
problems
more data is a recipe for disaster in
algorithmics
but but clearly not in ai
and there is a pattern according to how
much
data and compute you put into the models
at first you get stability
where the uh um you don't get so uh
where you don't get so much randomness
in your performance you get
generalization
where the uh the models are able to
um generalize to other not other domains
but other
uh supplements you could say uh and
meter learning where the uh
uh the models learn to learn
and this is um this might be basis for
uh or evidence for the strong scaling
hypothesis
once we find the scalable architecture
we can simply train ever larger neural
networks
and ever more sophisticated behavior
will emerge naturally
as the easiest way to optimize for all
tasks and data
so that's a really really strong
hypothesis
part of the the evidence for this is
also the the human brain
which indeed seems like it's mostly a
scaled up primate brain
and this is what it has been called the
bitter lesson
um that ai researchers in general try to
use
their own domain knowledge their own
ideas about
intelligence in order to make artificial
intelligence
and that generally fails the thing that
works is to have
general methods with computation and
this is by far the best this is the
bitter lesson is
richard s sutton um with another meme
here
um and this is uh
slowly entering uh the conversation
and um it's while it's not quite
mainstream yet
there's another match but vinnie here
he's uh
the uh one of the research leaders at
deepmind
making claim that learning algorithms
that find or instantiate other learning
algorithms
is a strong attractor in the space of
possible learning algorithms
meaning basically that if you
if you build a strong machine learning
system
it will be able to do meter learning
so why that's a big question why do
these models transfer and generalize
when they become large and that's of
course a heavily debated question and
uh have an answer or a guess rather
that uh well he approached four things
that could
uh be part of it the first is that
uh we we get some kind of um
compression or distillation of the
models
which in particular the uh alpha go and
alpha zero seem to have
we there's a lottery ticket hypothesis
that's saying that
um if you have in particular if you have
models that are not very strong
then sometimes you just get good
performance because you happen to
uh uh the neural network happens to
converge quickly
by by lock into something that learns
really well
they're the babies in the neural
networks and
learn representations finally um
i thought about this a bit uh and you
might recall some of you
from nick bostrom's book super
intelligence where it came to the three
things we
need for a gi are a
way to deal with probability a way to
deal with learning and a way to deal
with concepts
and stuart russell uh in his book
claims that we've basically nailed er
probability we
know so much about probability that our
algorithms basically handle that in a
robust way and learning is also
something that we really really
understand
and if we look at some of these in
particular learned representations
this looked like a good attack
on the problem of concepts uh which kind
of seemed
seems to imply that as both boston and
stewart russell believe
concepts is the big problem and we are
making real progress towards that
towards ati um the more
technical way this actually works in a
neural network when you
um dig into it then even though it's one
model
then it has some sub models along these
175 trillion
billion parameters and some of these
corresponds to sub models
and it's likely that one of these
submarines will work well
and if you have a large number of sub
models
then some of them will work poorly some
of them will work well
and this
[Music]
is kind of the uh the same structure as
you see sometimes in
ensemble methods in the ai and the way
these average
out is to something like occam's razor
which
is uh the original is that just that
entities shouldn't multiply beyond
necessity
and that you should choose the the
simplest uh
uh model the self that fits the data
and if you don't have very much data
then outcomes eraser
will just point to basically the data or
at least or personal
official features of that um but if you
make the problems really really hard
and uh the the data rich enough then
it is actually possible to force neural
networks into
what burn calls true learning
um this is one of gwen's points that we
actually don't have a very solid
understanding on how neural networks
learn
and how we should train them so meter
learning
the thing that gt3 does is learning more
generic algorithms
um and we need to have a substantial
amount of
data and compute to avoid just learning
data or some official things
and the analogy is that neural networks
are
lazy they need heart problems in order
to to actually learn
otherwise they'll just learn textual
relationships and things like that
an example is learning arithmetic right
you can
learn that by road that one plus one is
two one plus three
is four and two plus three is five
um and if this requires
some amount of mental effort
and it might be easier to learn
10 000 examples of arithmetic rather
than
figuring out how arithmetic actually
works but
it is uh there is a tipping point at
some point
if you need to learn enough examples
then at some point
it becomes simple to learn arithmetics
and
in in this case the simplest model is
just
arithmetic i was not quite sure i'm not
quite sure i
agree with this because it seems
falsifiable in a rather straightforward
way
because you can basically if you say
learning arithmetic
is as complex as learning um
ten thousand examples well it's not very
difficult to just
generate ten thousand examples uh
like here's one example you just
generate ten thousand of those
and make a neural network on that and
see if it learns
uh addition or if it just learns these
examples
um and i don't know if anyone have
actually done that i would expect
someone has done that
and i would have expected it to fail
um so i'm a bit confused again here
but um i don't know if i want anyone
have
any actually done this
let's talk about the relationship
between compression and intelligence
this is a part of a comic by ryan north
called
dinosaur comics um i've taken this from
guardians article and
cut off everything except the punch line
here
where the tyrannosaurus says yeah but
there's more to being smart than knowing
compression schemes and the user raptor
says no there's not
and this is the secret of ai
and i think this i won't go into
much detail about why this is true but
i'll just state flatly that it is true
that there is a deep fundamental
uh relationship between compression and
intelligence
even though that sounds really really
strange
um go in comparison to a magic trick
you take some information theory and a
benchmark for
human performance and then you take a
lot of tasks
and show the ai how to encode these
tasks as just
sequence prediction and then you get
intelligence and
everything else about intelligence
everything else we know about
intelligence is basically irrelevant and
that seems like a toll order um
and it is a priori
very um very uh
un unlikely uh that that tv3 would be
able to learn this much
and that that this is indeed a path to
true intelligence
uh one of the things we have
about this relationship between
compression and intelligence
is the hotter price a price in ai for
the uh the project that can compress uh
wikipedia the best
and i think that would be an interesting
thing to see how well
would you something like tvt3 be able to
compress wikipedia
it would need to be amended someone
before that is possible
and i noticed also that there is some
competition rules
which basically state that you're only
allowed to use very small models and you
can't reduce very much compute
um so um gbc3
probably would not do very well on the
hotter price because it is a very very
very large model
um so it would fail um and this actually
uh
points towards the hotter price being
being uh designed in a bad way
because why precisely is it important
that it's small
if the key to intelligence is indeed
that it needs to be very very large
okay so uh go and have a funny example
for um a model of
how something like uh three lions
and this is an illustrative example
it's actually working on bike pairing
coatings rather than
characters but um one say they don't
actually correspond to anything
intuitively
so he's he tries to use it as characters
for
just a standard recurring neural network
and i'm i'm adding here by
for myself some examples uh for how this
is
so let's start with a model that has
learned absolutely nothing
in this case the loss level would be 8
bits per byte
100 loss because there is no learning it
has no idea about anything
and if you have a model that has been
trained on precisely nothing
it might output something like this
basically just random characters
um but then as it learns very very
very quickly you will learn that there
are letters that are more frequent than
others
um so it can get that getting down from
say eight bits of five uh uh
to five bits of loss per byte it happens
extremely fast it will start to make
something that of course
looks totally like gibberish um but
looks more like something a human could
write
a bit more of uh training and it will
learn that some words exist and some do
not and will add punctuation
get down to three to four
bits of loss it might say something like
correct horse battery stable
which is closer to something a human
could say
it will learn that some words have
associations
it will learn to make sentences um
and um boring has it in this particular
order i think it can also come in other
orders you might get
sentences that i have correct syntax
but um of course no no syntax this is an
example of an
uh a sentence with correct syntax but no
semantics
and gradually as the model trains it
will get better and better it will start
to balance paranthesis
and um these kind of things
continue to lower the loss rate which is
below two bits per bite at this level
and it's it starts in order to uh to get
the loss rate down
it starts to get semantics so it might
be able to
uh generate a text such as jefferson was
president after washington
which uh you know makes some amount of
sense
this is of course an example of a true
sentence
but but slowly it gets more and more
human-like as the loss
level uh decreases and
um at some point this is the gbt3 level
where the error rate it makes an an
error in the text every 10 000
characters and it has an error rate
around 1.1 bit per byte
um this is an example of a sentence that
was generated by gpt3
which is from a longer text
which looks really really well but we
can go below 1.1 bit per byte
qpt3 can't do that but humans can do
that and
when humans uh write sentences we go
down to maybe
uh 0.7 bits per byte um
so an ai like that would be able to uh
say things like my name is burnt brain
when and
so let's talk about the last 0.4
bytes the uh sorry not bytes i wrote by
i mean bits the last 0.4 bits
from the last level of 1.1
to 0.7 and what's
in these 0.4 well that specs basically
everything that the model misses
everything that
distinguishes a human writing text from
tt3 generating text
is represented by the
loss rate where humans are still 0.4
bits per byte
better than the humans and that means
that
in order for gt3 to get down to 0.7
it needs to be able to reason very very
deeply about these kind of textual
scenarios
um it might be things like causality or
common sense reasoning it might be
something like with a physical
world where a gt3 might say something
like
if you put cheese in the uh in the
fridge it will melt or if you put ice
cream in the freezer it will melt uh
and in order to make correct sentences
about that
the the model needs to learn how does
the physical world work
it needs to have perfect consistency it
can't have
start writing a play by shakespeare and
then someone who's dead
suddenly becomes alive
when it writes a text it needs to build
up
towards the conclusion um in the same
way as a human would do it it can't
just have a non-sequitur or just go off
of tankans and things like that
in order to uh something that a human
can write
where humans descri where we write a
a scene from a play where there are
eight characters who are talking to each
other and
jogging for position in some kind of
machiavellian scheme or something like
that
this is something that humans can
understand and that's something we have
in our 0.4 bits
that we have the gt3 does not have right
now
and that's what's required and every
time
humans are able to write a faq
and a a description or an instruction
something with nothing
extraction everything that we can do are
in these 0.4 bits
once we get below 0.7 we get something
that is indistinguishable from a human
we might indeed possibly get something
better but that's a bit
further off yeah and this is interesting
because
we saw that a
a nai with a loss rate of 1.1
bits per byte had maybe an error every
10 000
characters or something like that and
that seems to imply that
um
[Music]
it's not very often that
the true human intelligence is actually
needed like
900 999 times
um what a human does and what
an ai does is actually just as good
it's the it's very few actions where we
truly need to think
long term and in novel situations rare
choices
where we need to look forward for the
rest of our life when we're signing up
for
life insurance this kind of thing that's
where
humans have an advantage over eight
years for now
and this is of course something that's
important for humans because
uh if in theory you can make a single
bad decision
and you could die from that and and that
might indeed be while
human brains which are from an
evolutionary point of view
are very costly and energy still are
worthwhile
because every once in a while we make a
really good decision
about not going into that cave even
though it is
um it looks comfortable uh and that is
enough
to give us an evolutionary advantage
right this is a a reactor uh from
from the manhattan project you can see
some of this stairs see that this is
indeed
very very large and this is what i chose
to
illustrate the hardware and funding
overhang
burn calls everything a hardware
overhang i split that
normally into a hardware overhang
funding overhang algorithmic and data
overhangs
but so the rhetorical question that one
asks is
can machine learning afford to run
projects which cost
more than 0.1 milli manhattan projects
which is in very rough numbers
what gt3 cost because
we might see that um
gt3 probably cost millions
of dollars in compute and
that's actually very little compared to
many other big research projects
we have the uh ita projects trying to
make fusion
uh and failing to make fusion at five
thousand times the budget
um and if we had been willing to put
more money into this
then we could have done gt3 decades ago
um and gwen makes some
implications that he we should expect
you d4 to have between
100 and 1000 times as much compute
um there are also algorithmic and data
overhangs
there is a bruce schneier code attacks
only get better
in the way that these algorithms will
only get better and better
uh rapidly i think the attack
is a very um uh
this kind of framing that the uh the ais
are making an attack
on the the difference between ais and
human
it's a rather aggressive framing and
even when you're not talking about aict
at all
so what are the problems with uh gt3
there are bad training data
there is enough train data to fit on a
laptop and there is nothing
with video there are no
pdfs or pdfs
images and books and photos no robotics
with features
with feedback from the real world there
are a lot of things that are not there
and the architecture is really simple
and it's uh there are a number of
problems with it it's
um there are known the the uh
future of learners are language models
our food huge
learners article does point out a number
of ways this could be improved
and some of them doesn't have it seems
to be very hard in particular
there is no fine tuning even though that
would be a really
easy way to improve dramatically
and all this um the algorithms that we
are using are probably not even
necessary
we could probably have done precisely
the same with recurrent neural networks
um the algorithm was from 20 years ago
um
transformers are nice but they seem to
be mostly about efficiency we could have
done this
a long time ago
people probably will want
to build these kind of models there are
indeed very big
additional incentives to to do this
maybe even to have done this 20 years
ago
because it is both possible and useful
to go to trillions of parameters
first is of course the the scaling
curves are not bending
um quantity predicted this would not be
the case but it is scaling
um and um
when we had the uh uh the gt3 paper
we looked at some of the uh uh
benchmarks to see how many
where roughly something would fall one
of the uh
important one was the vino grant um
which is which would which uh gwen
expects would fall
around 10 trillion parameters
um this is an adversarial uh
uh an adversarial benchmark
one that has been chosen to be as hard
as possible for computers
and it seems like it will um be possible
soon
um there are many people
uh uh for instance the um
state of ai just uh uh
released a uh uh an article where they
say
they expect that we will get a 10
trillion parameters model
within 12 months
um a lot of this will cost money it
might cost
thousands of gpus it might cost between
10 and 100 million dollars
and this is without a rhythmic
improvement and there will be
improvements
so what are the actual incentives to do
this
well even if you just have something
like alphago
which uh played go then you used a huge
amount of hardware
to um to train the model and very little
to actually run it but you could run it
1000 times in parallel if you wanted
with the same amount of hardware
and that seems to like it would have a
huge effect
on the strength you could use this um
you could
when you have a model it's often
possible to distill it
into a smaller model uh you can have
transfer learning to other
domains once you have this and once you
have a big model
then you can your next model
can be powered up using the old model uh
if
it doesn't have to start from scratch
there are some experienced curve effects
and uh finally this can be used as a as
a baseline for further research
you can try to take away some features
you can try to compare it with different
architectures
to see what works and what does not work
so there are big incentives to actually
do this
um go and have a nice analogy we are the
cyanobacteria of ai we admit the
cyanobacteria was
the first bacteria that emitted oxygen
as a by-product and in the same way we
has have a byproduct product a lot of
structured data
that the eas can learn from um
so the big question is is there any way
that gt3
can or a successor can become an agi
um there is another example here i won't
go into it
uh there is a prediction here that uh if
we get
uh 100 to 1 000 times more performance
we would have a loss
less than one bit and no one really have
any clue
about what that would mean in practice
go and have a an extrapolation of some
of the curves
and that would imply that we will reach
the human level in 2027
at a cost roughly as the invasion of
iraq
open ai they estimate that it would cost
1 billion in 2038
with some very very conservative
assumptions
i would say i had a very naive model
where i just
assumed that compute would increase
would double every three months
and algorithms would double every three
months and that's
put the human level in uh 2022 or 23
which is of course any extremely
optimistic model but i think
um not completely impossible
um that's definitely a an upper bound
um but um well it should be a lower
bound right
it it can't it probably uh in order to
get
this at last level uh following these
curves
we can't we would need some extra luck
in order to go below
2022 so
this seems like we are at the cost of a
revolution um but groins claim is that
this will
not kick-start an arms race
deepline and google brain are an example
of people
who should be taking this as a splitting
method they have the hardware
budgets the people to actually build a
competitor to dvd3
but they lack the vision the conviction
to actually do that
uh google brain focus very very
practically
uh and um deepmind believes that fancy
algorithms are required and focus very
much on
neurology at least in coins i believe
and some of the other players are just
uninterested and relevant
the chinese ai sector is interesting but
they have the dutch disease
in that the their talented ai people are
all working on
commerce and surveillance and
they are too high bound and deeply
philosophically wrong
to ever admit fault and try to overtake
open ai
until it's too late um
and this is uh there is a neat quote
here by norbert weiner
uh which who said that the one secret
concerning the
atom atomic bomb which might have been
kept was that of the possibility of its
construction
meaning that once
urban ai right now is showing to the
world
that it is indeed possible to get a
strong natural model a natural language
models
uh you just have to use a lot of compute
then this is open to everybody
and this should kick off an arms race
so what are the counter the counter
arguments
here is an old model of how natural
language
uh would uh perform and this
is something that one dislikes um
it seems of course everything can be
framed as a text prediction language we
saw that earlier
but there are many uh kind of algorithms
that are universal in some sense and
that's
that in general doesn't impress people
as much as it should
and it's of course also a priori
possible that
training something like qt3 would just
require too much data
and the scaling would not be good or you
need some kind of new abstraction
or something like that um but it's also
possible
that it would require a huge amount of
of compute there is this uh quote by
neil spohr that you can't build a
nuclear weapon without turning the
entire united states into a factory
like uh it's this is this it seems like
a counter-argument
but it in fact it's not really a
counter-argument
so what's been the reaction of the
research community to this
um this seems to be evidence for the
skating hypothesis
that scaling is the secret of agi and in
the research community
this is wildly unpopular
and uh gwen does not believe that this
will kick off enhanced race
uh notably there have been uh very
little interest from researchers
um and the we don't really see much
of this compared to how much we should
um that's
of course a bit surprising to someone
like me who will live in a rationalist
bubble
in uh that to me gbg3 was huge news
but many many people did indeed not
react
uh i should say that this might be
changing this week i saw in the
the tv there was a substantial feature
about gt3
also in particular the fact that it was
speaking danish seemed
the the fact that they could just learn
how to speak danish was somewhat scary
to people
um and part of the reason might be that
the
standard natural language benchmarks
seem to not really uh have anything like
meter learning in them so so the
standard benchmarks kind of miss this
um and it won't have some very very
unkind word about the ai researchers
believe that the fact that they don't
that they are unable to predict this
prove that they don't have a model of
how ai properties happen
uh and that they do not learn from
falsified predictions
and all they care about is supporting
the status quo
remaining respectable he calls out
gorilla gorilla
i needed to look that up it's an
expression used to convey waiting for a
response
when there is not
he have more unkind words about the crit
to say about
the critics the people who believed in
deep learning
uh in 2001 were very very few
they were called connectionists and were
um
called deluded by the rest of the ai
community gwen
was one he uh was skeptical and he uh
have to admit that he was wrong and the
connectionists were right
these uh the ai community in general
have a lot of authority but there is no
accountability about their predictions
and um the predictions that the
connectionist
agenda would fail were made by imminent
respectable serious people
and calls out all honor to the fanatics
shame and humiliation to the critics
in particular one of the things that one
is angry about is that people are not
really reflecting on this or
saying why did they predict wrong a lot
of the communications seem to be
emphatic
not trying to communicate things but
trying to
get evoke feelings of confidence etc
the connectionist agenda was such
so unpopular that the uh that the ai
projects between 1960 and 1990
actually did not improve in particular
and possibly to a substantial extent
because they actually did not get more
compute from 1960 to 1990
you get got ai projects had
one million instructions per second for
their
compute and that was basically the same
no one thought to really give it more um
even in 2012 the the budget
for alex net which started the deep
learning revolution was five hundred
dollars
so a lot of these critics didn't
dismiss
these machine learning models uh the
connectionist models as
just something but but reducing
ai to to just ex is actually the key
to our success in air
there are some uh here i will go through
um
and bryan i finally have some questions
for people who still assign
near zero probability to agi within the
next few decades
the first question is why and i think
that's a fair question
it might be hard for people to put their
intuitions into word
the next question is did you predict in
writing capabilities like dbc3
i think also this is a fair question and
an important question
particularly if the person who is
claiming this brand themselves as an
expert
next third question is is this how you
expect ai failure to look
in the decades beforehand i don't
understand this question i'm not saying
what specific task what specific number
would convince you otherwise
um there are a number of well-known
tests for agi
like the coffee test et cetera which
has
that it does now if these crew prototype
insect brain
size deep learning systems were not
on a path to success uh i think there
might be a missing inversion in this
uh people will probably just answer
there is no difference because it is
this world where this is not on the path
to success that is all for today
thank you and see you


How humans co-determine the development of intelligent technology (Serge Thill)

yes
page is yours okay thank you
right now i have to click away the fact
that you are started recording
and i have to click this away and now i
can actually talk so i'm guessing you
can all see my screen right that's
working out okay it's a little bit weird
my setup now because i can only see
uh my slides on this screen and i'm
talking to you
on this other screen so just so if you
see me flicking backwards and forwards
uh that's why it is why that is um
okay so thank you for having me right
thank you for the invitation i'm
very happy uh to be here and uh
basically uh what i wanna talk about is
a little bit about
how humans and intelligent technology
interact
on various levels the reason i want to
talk about that is
when i was asked to give this
presentation
right maria luche said that she
mentioned these two things
that she knows of my work that she
thought might be interesting to talk
about
one is called dreams forecast that's a
european funded project it has recently
finished and it's basically about
control of autonomous vehicles where
we've basically built a slightly
biologically inspired
uh controller for the autonomous vehicle
the other one was a paper that was
presented at the roman conference
this year which is much more high level
much more philosophical and basically
talks about what the human robot
interaction the field
uh can learn from what human animal
interaction the field
uh has done in the past so i was kind of
scratching my head for like five minutes
on how can you possibly combine
human animal interaction with autonomous
vehicle control
and the presentation today is the result
of that
uh deliberation deliberation
so the commonality is of course that
there are humans involved
everywhere autonomous vehicles interact
uh with the huma
humans but also the kind of controller
that we built in this project was
inspired
uh by human control and in human robot
interaction human animal interaction of
course human is clear
it's clear that that term exists so
the taker message right because i also
don't know how much time i will really
have
and also i don't have a clock on my
screen anymore i don't know why that
disappeared
but in case i don't run i run out of
time this is the take-home message
you cannot ignore humans when you're
building
an intelligent system and the reason is
the your systems that you build have to
interact with humans
one way or the other right this is
certainly uh true
uh very often even if we build an
autonomous system
an autonomous vehicle exists on the were
on the road
in a physical world where other vehicles
are driven by humans
even if no other vehicles are driven by
humans there are pedestrians
uh bicyclists and so on vulnerable road
users
and even if they all go away there are
still people inside the vehicle so
you're always
interacting with humans and that's you
cannot forget about that part
to some degree it's even true um if the
robots are in deep space
so i primarily have this slide because
janet
vertexi gave a nice keynote lecture at a
dryder conference
a couple of years ago and she has this
very nice book
which describes basically how the team
at nasa that controls the mars rover
works right and how how much of the
schedule of the team and the
um the direction
of what was it going to say the
interaction of the team with the rover
and the rover's behavior
it all depends on how these two uh
how they co-interact it's certainly not
the case that uh having the robot
just uh on mars means there is no human
robot interaction anymore there is
an entire team on earth
living to martian times uh
what's the word for that uh the rhythms
they rhythms circadian rhythms to the
mushroom
circadian rhythms just so they can
control uh the robot
and the dynamics that uh emerge from
that of course make an entire book
which is why i have this little slide
here so
if you look at hri i think this is
obviously
to some degree at the minimum implicit
in most of what people do right so a
very long time people have
uh starting all the way back in 1978
people have started thinking about like
what kind of different
robots can we build and how does the
interaction between the human and the
robot
uh then evolve right and what is the
role of the human
uh in those uh uh what is the role of
the robot in these various
uh interactions and how does that shape
the kind of robots that you will build
so it's not necessarily a particularly
new
idea but when you get to the more
machine learning
fields these days sometimes it gets lost
a little bit that the algorithms
that we build are there to interact uh
or to function in a world that is still
fundamentally a human world
i have like some things in the chat
right there we go the challenge is of
course how to facilitate suitable
interaction indeed
i fully agree with that okay
so what i want to talk about here then
is that there are like different ways in
which humans can matter
and it's maybe worth going through a
couple of examples of each just to get a
feel
uh for why it doesn't make much sense to
develop
an algorithm for real-world application
without also considering that it's going
to be used
by human at some point in time
and i'm gonna just move this here so i
actually have some
control over the time so there are four
three main points and they kind of break
into a couple of sub points sometimes
just having a human present can change
the dynamics of your system
and just having an artificial system
present can change the dynamics of the
human
uh very often they define basically how
a system will actually work
either they're an inspiration for
algorithms or
uh they cut off they can't start
becoming a kind of success criteria
in the sense that your system that you
build is only good if your end users are
happy with it
and they're certainly going to have a
lot of opinions right so you might have
some idea on how to build a robot
uh that will uh support uh uh the
humans in a specific situation like in
care homes and so on
but the end users might have a very
different idea
of what would be a suitable robot and
all of these are things
uh we cannot just uh ignore so to talk
about them
a little bit right first one the
presence might change the dynamics of
your system
uh this is going to be a relatively high
level
um but it's something that i really
think should be obvious right you can
build a fantastic algorithm
uh and it should be obvious that as soon
as you expose it to humans
things are gonna go in a direction that
you did not necessarily anticipate
but apparently uh it is not always that
obvious because in 2016 if you remember
microsoft
uh decided it was a good idea to release
a type of chatbot uh on twitter that was
going to learn
from uh the interactions it has with
people on twitter and it became
a very unpleasant very racism racist
misogynist
bought in less than 24 hours like to the
degree that i couldn't actually find any
funny
examples of the tweets that it was first
viewing out at the end of it everything
i found
was actually really rather offensive and
of course to some degree this is simply
because it started learning from real
humans and humans
especially a particular subgroup on
4chan decided to have some fun
uh with the learning abilities of this
sport and
at the end of the day the system started
building doing something that none of
its developers
ever intended and this can happen over
and over and over again right so one
simple way in which humans
mata i have a bunch of videos that i'm
not actually going to show in the talk
simply because i'm not sure how good the
transmission rate on this call is going
to be
but if you go to youtube you can find
this uh fantastic little spoofs
uh from a team called boss town so you
can see that it's not boston but
boston uh dynamics uh and they
basically have this proof of boston
dynamics uh robot videos where the robot
gets increasingly annoyed with its
humans and
increasingly starts punishing them for
uh punishing it
and that's obviously an interesting
aspect again right if we build robots
that are going to learn from experience
from interaction with humans
then we need to understand in what kind
of direction this robot is going
to develop it's certainly not the case
that it will necessarily keep doing
everything that the designer intended so
the designer has to keep in mind that
this is going to be released
uh to people that are just going to use
hockey sticks and push the
robot around just to see what it does if
the robot then learns from this
unintended consequences might happen
right um if you then take a step back
and you go to uh like social cognitive
science
uh you find a few people who are very
interested in what actually makes
uh social cognition and it's an
interesting subfield
because for a very long time
people in social cognition still tended
to treat
social cognition the mechanisms as
something that exists fundamentally
inside your skull so the problem in
social cognition traditionally was
simply
what kind of mechanisms do you as an
individual need
in order to interact with another
individual
um and then more recently and in like
subfields called an activism and a bunch
of related fields uh people started
asking
uh what if these um interaction itself
is also constitutive
of uh cognition so that you cannot
actually just
focus uh on the uh mechanisms that an
agent has um individually inside
whatever it is it's exclosure it's skull
it's robot brain uh whatever but what if
the fact that it interacts with another
agent
matters and what if the interaction
itself is also
a constitutive element in social
cognition so there
um the paper that i have here is from
2010 i like it a lot so i recommend it
uh from hannah de jega
it basically gives you an explanation
that you cannot think of a cognitive
mechanism just
as something that happens inside one
individual the fact that
we interact with other individuals is
important in understanding human
cognition
and the interaction itself is also
important in understanding the overall
um
while interaction and dynamics and
cognitive mechanisms that emerge
if you have never heard of an activism
that i'm going to oversimplify it
so if you have heard of it then you will
probably hate me right now but if you
haven't heard of it
it's basically this idea that what's
fundamental
is that cognition
in agents we have dynamical systems that
are agents and they are interacting with
the world and we understand
uh these agents fundamentally through
the mathematical language of dynamical
systems
and what makes an agent a cognitive
agent is that it tries to maintain
its own internal dynamics in the face of
perturbation
from the environment so you get this
kind of graph i don't actually know if
you can see my pointer
i hope you can see my pointer but if not
uh then you see these two circles um
that exist and they have this internal
horizontal circle this is sort of meant
to represent the internal dynamics uh
of the system uh and this is what we
try to maintain right if this goes away
then you stop existing as an agent you
also have
dynamics that define how you interact
with the world and then you have
perturbations
uh from the environment including from
other agents
so it begins it's beginning to be
relatively easy to see how
uh all these interactions are going to
matter for the uh the internal dynamics
because they all
uh co-determine uh these dynamics so
that's
uh the one slide uh summary of what an
activism is
very much in an oversimplified manner
there are people who are interested then
in how multiple agents
interact and here's tom fruser he's done
an interesting
paper a while ago and i think they're
still working on this
where they've created a little arena in
simulation and two very simple robots
and they are the simplest
possible robots that you can imagine so
they're each driven
with just one neuron and this is
interesting because that also makes it
the simplest dynamical system you can
imagine
it becomes a one-dimensional dynamical
system
and they start letting these agents run
about and they look at what happens in
the state space
of the two agents or the fair of their
dynamical systems the one-dimensional
dynamical systems that control these two
agents
and they see these kind of um
oscillatory uh behaviors uh appearing
and this is interesting uh if you're
into dynamical systems because a
one-dimensional dynamical system
shouldn't really be giving you any
oscillations right these things
uh go towards a fixed point attractors
they either go to a stable value
uh or they uh declare decay uh to a
different uh
to zero uh so the fact that these kind
of dynamics go
uh shows that you could not understand
one uh particular agent just from those
uh from the dynamics that you would
expect from one neuron if you studied it
in isolation
in addition um it's where it's possible
to
estimate like the dimensionality of the
system that you're observing
and they also did that and they find
that the system that they are looking at
looks like it's a three-dimensional
uh system most likely and that's also
interesting so you cannot explain
the dynamics that you observe entirely
just from the fact that you have two
agents
that are interacting because that would
be a two-dimensional system then so this
seems to
uh suggest at least that uh when i have
two
agents with one neuron each interacting
the resulting behavior can only be
understood if i consider it in
at least three dimensions so i guess
what the two agents are doing and
something that the interaction is doing
just to give this a general message that
interaction between agents is going to
be useful and important
and when we design algorithms we should
probably keep that
in mind right so your system might also
change the
the way that humans behave so we already
got this idea that we have this kind of
feedback loop there
again i'm not gonna show this video but
if you've probably seen it before
uh if you haven't seen it before this is
one of the reasons i love youtube
because people do all sorts of stupid
things and they put that on youtube for
the world
to see in this particular case these
people
uh decided that they were going to this
uh uh try out the pedestrian detection
system
uh of the volvo xc60 60 uh that they
have there
so the idea behind the pedestrian
detection system is that
if it attacks a pedestrian it will go
and uh
break the castle you don't actually run
the car over
the pedestrian over if you look at the
video you will notice that they will run
over the poor fella in the
pink shirt here because they just
accelerate
at uh this person the reason that
happens
uh is for two reasons uh the first
reason was that this particular volvo
that they had
um the pedestrian detection system was
an optional extra
it did not actually exist in the vehicle
that they purchased it wasn't built in
so it wouldn't have done anything and
the second reason
is that the way it works in this
particular car is if you're flooring it
then the vehicle thinks you're in
control uh full control of whatever it
whatever it is that you're doing so it
will not interfere
right so even if they had had the
pedestrian detection system they would
still have run over the person
because it would not have engaged in the
situation that they have used it
so this is important because it shows
you
that how people are going to use the
systems that you build doesn't depend
necessarily on how you've designed the
system
but it depends on how people
think that that system is actually going
to work
right and that we take into account that
we've done like a bunch of little um
i mean this has been studied by a lot of
people right humans will always adapt
to other agents and the abilities that
they think the other agent has not
necessarily the ability that the agent
actually has uh this is true for humans
interacting with other humans right you
interact differently
with a five-year-old child than you do
uh with an 18 year old
teenager than you do with a 60 year old
it extends to artificial agents people
have studied this in robotics for quite
a while now
uh you can for instance manipulate like
how
fluent in english a robot is and humans
will adapt
their language in the kind of behaviors
uh
in how they give commands uh to the
robot
have a robot that speaks perfectly
natural english and people will converse
in natural english have a robot that
speaks more machine like
and people will go more to like keyword
command type
interactions and we've also seen this
when people
interact with intelligent vehicles so we
did a bunch of things just in simulation
um this is an old um example but i want
to talk about it a little bit anyway
because it's interesting
um where we try to manipulate how people
how intelligent
an autonomous vehicle or an
uh intelligent adaptive system inside a
vehicle appeared to the participants
that we had
so participants got a little navigation
task
uh where they had to navigate this kind
of weird map that you see on the top
the reason it looks so weird is because
we try to avoid 90 degree
corners we talked we built we did this
in collaboration with volvo
and one of the things that volvo told us
at the beginning is that if you have
like more than two or three 90 degree
corners
in a simulation people are gonna get uh
simulator sickness
and then uh they'll be in the corner
vomiting rather than doing uh your
experiments so you should have
as gentle um corners as you can have
so we tried to build this kind of map
and all they have to do is navigate to
one of the two
gold locations which are the squares
that we have at the very right
and the way they can do this is either
just with a paper map or by having like
a glorified gps that points them towards
different directions
uh at each location uh whenever they get
to a junction and when that happens it
can either just show you an arrow that
goes like your left go right
or it can show you an arrow and give you
a reason
why you should do that the other thing
we manipulated there was simply how much
traffic there was on the various uh
on the various routes that they could
take and you can also see that when they
see a juncture they can see both
versions
uh so you can do things like say that
they should go left
because there is less traffic on that
route and then drive them right into a
traffic jam
which then makes it seem like it wasn't
a very good idea
to go that way so you can manipulate
like how clever the system worked
basically by
combining how much information the
system gives to the users
and how it describes the situation and
how these things that actually match
uh with reality so i don't want to talk
in detail about everything we did in
this study
but that's the basic idea right you find
uh people who rate
um this uh vehicles more or less
intelligent
while they do this interaction task a
little funny side thing is that there
are
extremely easy navigation strategies for
this environment even though it looks
kind of messy
uh you could for instance just decide
that you go left
until you get to junction 18 and then
you go right so there's nothing
particularly cognitively demanding in
figuring out a way to navigate
this thing but people will still follow
whatever the system says
they should do and one of the funky
things we saw
is that depending on how intelligent
people thought the system was
they spent less and less time just
staring straight out
of the front screen of the vehicle
and they did or maybe a more accurate
way to describe that is that
the more stupid they thought that the
system was the more they spent staring
uh straight ahead
and they did that to a degree where
um it's no longer um equivalent well
it's no longer what we do when we
normally drive vehicles so
it's not the case with that when you
drive a vehicle you spend 100 of your
time
uh staring out of the front of your
windscreen because you're obviously
paying attention
uh to your surroundings as well so what
people normally do is about like 80
75 percent of the time staring ahead
uh the rest is checking out your
surroundings people who do like 95
staring ahead they are people who are on
the mobile phone
and they have stopped engaging with
what's actually really happening
uh around them so this is just an
interesting thing right the behavior
of uh the participants in this study
like something that you don't
necessarily
think should be affected namely just
like how much they stare
to the front of the windscreen was
affected simply by how intelligent this
system seemed to them this was a small
study with only 30 participants so
there are plenty of question marks we
can have on how strong these results are
so we did a similar thing this time with
130 or 120 i don't remember
uh participants and this was funded by
the
swedish energy foundation i think
or yeah energy uh administration so
they're people who like
uh to find ways of reducing uh energy
consumption on the planet so we looked
at whether or not
people people's eco-friendly driving
behavior could be influenced
in that way as well so depending on what
kind of suggestions the systems gives
in the exact same task do people drive
more or less economically friendly
and what we had in this case is that we
have
a system that either doesn't give them
any eco
friendly driving recommendations at all
or it gives them driving recommendations
like
using simple icons you know lift the
foot off the guy off your gas and start
coasting
or change gears up change gears down or
we had like the
more intelligent version is what we
called it where in addition to
suggesting that you should start
coasting or you should shift gear
it would give you a reason why you
should do that now the reason this is a
two by three design is because we had
the same
gps type in there as well so it would
also navigate
you to a goal either just using arrows
or giving you a reason
of why you should go a certain direction
and the long story short is that this
matters
so on the same task in the same
environment
if you give them eco-driving directions
okay so some recommendations of course
they will start
um use they will start following them
and they will use
less petrol uh but if you give them
reasons why they should do something
uh then they will um save even more
petrol so the
kind of uh energy consumption went
down consistently except uh for this
kind of group that i have here that i
will talk about
in a second but give them a basic eco
drive system and we reduce
the fuel consumption a little bit this
is basically liters per hundred
kilometers
give them an informative one then we
reduce uh eco uh
fuel consumption by a lot and a similar
thing happens when you look at when
people start shifting gear
right tell them to shift gears and they
shift start shifting gears a little bit
earlier
tell them to shift gears and tell them
why they should do that they will start
shifting gears
even earlier except for this group here
and what is interesting about this group
is that it gets a gps
which tells people why they should go
left or right at a crossing
but it gets an eco-driving system that
just gives them instructions without
justifying them
and then it seems that those people in
that particular case they stopped
stopped listening to the eco-driving
system entirely this was a bit
surprising to us when we
did this uh our best hypothesis uh
is that this happens because people
don't perceive these two things as
separate systems
they perceive one holistic system that
supports them
in their driving task and if the
navigation part of the system does a
fantastic job
then it just makes the eco-driving
recommendation part of the system
just look a little bit stupid and then
they're less likely
uh to follow that um recommendation but
that's the hypothesis
we haven't really been able uh to uh
figure out how to test that
exactly this is very much uh something
for future work this was published i
think in 2018 so
still relatively recent the reason we
looked at both
like fuel consumption and behavioral
cues so when do people change
gear and also how much of the time do
they actually spend
coasting uh is because
fuel consumption you can fake and
manipulate as much as you want
just by messing with the fuel
consumption model that you have
in your simulation right these are very
complicated models with lots of
parameters
if you wanted to drive a particular
point home you can easily do that
it's a little bit harder to mess with
the behavioral cues
that people will give so i think these
are a little bit more informative
than just looking at the fuel
consumption but okay
so just a couple of examples where how
the system is designed is going to
change how does
how humans uh behave when they interact
with the system and then the same
way if you know imagine the full circle
how humans behave
would then again affect how the systems
might
behave we get this kind of feedback loop
going on there
this is a little site study this was
also interesting it shows the same point
again this was done by other people but
also at the university of shefter in
sweden
and they had an autonomous system again
simulated
where it was literally an autonomous car
this case so people just got into the
simulator
and then they had a newspaper they had
snacks they could nibble they could do
whatever they could play on the phone
uh no restrictions and the car would
just go and drive along
and while it was driving uh weather
conditions started deteriorating
so at some point uh the vehicle asks uh
the human and in the driver's seat to
take over control
in autonomous vehicle research this is
one of the interesting questions right
how do you get
a human who's been doing literally
whatever uh to being in control of the
vehicle within a short amount of time
while also understanding what it was
exactly that caused
the uh uh the problem uh that caused the
um
uh the vehicle to relinquish control
and they had two conditions one
condition basically just had this drive
and then eventually people had to take
over
and the second condition uh people
additionally had this
little display uh here on the left and
they were told
just that this indicates the confidence
of the vehicle in continuing to drive
autonomously nothing else that was
the only information that they had
so what they found initially uh the two
things that you would expect right if
you give
people this kind of additional
information they are capable of taking
control faster than
if it is not there um and they will also
look around
and do other things more than
people who don't have this uh
information and they start uh
well they start monitoring what the
system does a little bit more
overall but the interesting result was
that if you give them this additional
information you would think that it
makes it more transparent
what the system actually does and it
probably does but
people trusted this system less than
people in a condition that did not
have this interesting uh additional
information
so that's also an interesting result and
the hypothesis why this is happening is
because then
is that it's not really a case of
trusting a system
less than in the control condition but
that the people in the control condition
they actually started trusting the
system too much right so that this
is a way of ensuring that people
understand what the actual abilities
of the system are and that then ensures
that they show appropriate levels of
trust
and they also start um showing
appropriate
behaviors uh when this drops um
how do you measure trust in this case
this was a simple questionnaire like
five
item like at scale i think uh so in
general i don't want to talk about trust
too much because trust is obviously a
huge pandora's box
and a can of worms if you start going
there you never get out
but it is an interesting thing right
that if you ask them objectively how
much did you trust this system then
you end up with this difference between
the groups and you end up with behaviors
that is more appropriate in one group
than in another and it's all to do with
like transparency to what degree do
people understand what the system is
doing
uh and uh to what degree um does that
help them build a comp
understanding of the system right so
this was quite a while ago right this
was already eight years ago this was a
little bit before
uh people started really talking about
explainable artificial intelligence and
so on
uh but uh before explainable artificial
intelligence turned up
we had this entire subfield of
situational awareness and system
awareness
where we thought about how does a system
really explain what is necessary to a
human operator so that they can take
appropriate decisions in time and you
can see how this sort of feeds back to
well what is now xai and if i have
enough time i will actually get back to
explainable ai
at the end of this talk but i already
wanted to drop it in here
because just to say that xai is not a
new thing it's
basically been built up on like decades
of human
interaction with machine systems
especially in decision support tasks and
so on
okay so that was the first point second
point so they define
how your system works right so this is
also a little bit uh my attempt
to bridge the two very different things
uh
that we were interested in so let's talk
a little about i'm gonna talk a bit a
little bit less explicitly about
human cognition as an inspiration for
algorithms because
literally all of cognitive robotics
tends to do that and even deep learning
algorithms were originally inspired by
things that we find in neuroscience so
i think we all know plenty of examples
of this case but they're also part
of your success criteria i think that's
a little bit more interesting in this
context of this talk so
this is then the story of that paper i
just want to plug that a little bit
and sell it to you when you have humans
and you design
uh use humans as an inspiration for a
system you have these two different ways
in which you can use
humans right they can be a target in
which case they are simply defining what
you want
the system uh to do and they can be a
benchmark right so they can be
something that you evaluate um your
system against uh either like by
comparing
artificial systems behavior to human
behavior
or basically by looking at how the human
and the system interacts and whether or
not this goes
according to what you have so sometimes
it's a human ability that becomes the
benchmark
sometimes it's how the human interacts
with the system that becomes a benchmark
and when how the human interacts with
the system and becomes a benchmark
you very often have ideas uh from
previous fields
like you might have looked at social
cognition beforehand and have
come up with ideas of how humans
interact with other humans
and you want that interaction to be
replicated but you can also have looked
at how humans interact with computers
and you want that to be replicated with
your robots
or and this is the story of this paper
that i will not actually go into more
than this slide
you can look at how humans have
interacted with animals
and look at those interactions human
animal interaction
is an interesting field because animals
play various roles in society that all
sort of mimics
uh the roles that here that robots that
we want robots to play right we use
uh animals uh as companions so we use
animals to get stuff done uh and think
about farms and so on
we use animals uh as like in therapy
if you think about animal assisted
therapy for children with autism
spectrum disorder and so on that is a
thing but there is also an entire field
uh that is building robots for therapy
for children with autism spectrum
disorder
and human animal interaction as a field
of course has
a huge head start on human robot
interaction because
animals have been around for a long time
robots have come around
relatively recently so the fundamental
story and this paper is simply
it's a little bit strange that we
haven't looked at it a little bit more
at that field of human animal
interaction because there are plenty of
interesting lessons to be learned there
and that's certainly something we should
explore a little bit more
so that's that um i did mention uh
robots in robot assisted
therapy so this is part of a project i
was also involved in
earlier we did actually try to build
robot control
um for um yeah robot assisted therapy
for children with autism spectrum
disorder and the fundamental aim of this
project was simply
uh to make this robot control a little
bit more autonomous so replace
uh the wizard in a wizard of oz paradigm
you want this robot to be able
to go through um an intervention as
defined by a clinical psychologist
but you want it to do so autonomously so
you need to be able to pick up
on like how the robot's behaving uh and
not the robot sorry the child like is
the child doing what we would expect a
child to do
and what are the appropriate behaviors
that uh
the robot should be doing if a child
does a certain thing x
the good news and the reason we should
be interested like more from a
tactical point of view the reason we
should be interested in this kind of
field
uh is because it is actually relatively
well constrained one of the major issues
with robots is of course making them
operate
uh in the real world and that can be
hard but this is a very constrained
world because
the therapy the way that the therapy
goes is defined by the psychotherapist
you don't get to have any say
in that and they also have very clear
ideas of what they expect to see from
the child at every point in time
and they also have very clear ideas of
what the robot should be doing
uh based on what the child is doing so
it becomes feasible
to build a robot that can operate
somewhat autonomously
in this system obviously never without a
therapist
there's always a therapist in charge of
the whole interaction and always
able to override what it is whatever it
is that the robot wants to do
just a way of removing remote control a
little bit
but uh when you're building this kind of
system you also
at the higher level you realize um that
the way uh that we define uh the success
of this kind of robot isn't actually in
like it can score well on an amnest data
set or it can score well on this data
set or it can
pick red object and put it in the blue
box right the
nice criterion for success is really the
interaction that
starts to emerge between the robot and
the child so if you start thinking
about this on more general terms we have
very often a situation where
uh we build robots what we design is the
robot behaviors but what we
evaluate is how this robot behavior uh
facilitates a certain interaction with a
human that we did not design
but the interaction between the robot
and the human is what really really
matters there
so that's what i have on this slide i do
think uh yeah i probably talked about
this um already
right the main problem that you have in
those cases is that you can never fully
specify
uh all the behaviors of your robot so
you get what we had before with the
twitter bot
uh and the bosstown dynamics fake
example that the robot can start doing
crazy things
so you do want to constrain it and
the proposal here but again this is a
proposal it's not something we've
actually managed to implement yet
so if anybody wants to work on this i'm
really really interested
um is to take a sort of an activist
approach
to the system understand that we're
fundamentally building
uh dynamics of a system that's sort of
optimized according to some objective
function but
define that objective function purely in
terms of the quality
of the interaction in a way that will
actually modulate the dynamics of the
system such that it becomes
what it wants what we want it to become
so
this is basically um what we have
written about uh
again in a paper a couple of years ago
and the take-home message of that is
simply this kind of characterization
right
all of the cognitive science will tell
you that yeah you have basically
uh component dynamics that basically
determine how your system behaves and
how the system
interacts with the external world but
how
the extra this interaction with the
external world goes
that defines your component dynamics so
your change
um in your internal dynamics is a
function of what your current state is
and basically a function of what's
happening inside the outside of uh
what's happening outside
in the actual world so the way that we
can characterize
interactions and this is going to be
relatively quick because
it takes me 20 minutes to explain it in
detail but hopefully two minutes to fly
over it
is that we can describe like something
like something like a forward model and
something like an inverse model
that describes the expectations that we
have on the interaction
so it's not literally a forward model
because it's not about the agent itself
but it's basically uh models that
describe if the robot does
something then i want to see the
appropriate
um i want to see this specific response
from a child and the inverse like models
are then if i see a specific behavior
from a child
i should have done this behavior in
order to elicit
that response from the child i can have
a bunch of these uh
forward-like an inverse like model to
describe my interaction initially
the cool thing is that i can ask talk to
my psychotherapists
uh so they can give initial inputs into
what kind of behaviors we would want
right what should the robot have done
in order to elicit a certain behavior
and what is it that a child should do
when a robot does a certain behavior and
then i can start learning these things
and getting better at it maybe
using reinforcement learning and so on
so in this paper we have this full
characterization of that system but we
haven't actually implemented it because
doing this in practice
using reinforcement learning in an
actual interaction becomes
really hard and i haven't really figured
out a good way of doing this in a
simpler simulation yet
so if anybody has any ideas i'm very
interested
to talk about that more
something that falls out of this a
little bit when you start talking about
everything in dynamic
systems terms is that you can start
putting things
in spiking neural networks right there
are a bunch of reasons why you want
might want to start controlling your
robot
using spiking your networks rather than
traditional
algorithms the simple reason is that you
want to make use of
the benefits that neuromorphic
architectures might potentially
give you and then of course you need to
have a way of instantiating everything
uh that you do in spiking neural
networks
the reason i am interested in it is
because uh
it gives me an interesting uh entry
point
uh into ways of modeling cognitively
more interesting
behaviors on robot arms uh
in a in a way that grounds cognition in
the censoring motor experience uh
of the robot uh there are a couple of uh
frameworks out there one of them is
called
uh the semantic pointer architecture and
uh the neural engineering framework
which comes with a software that's
called nango in which you can basically
design
spiking your models of pretty much
whatever you want
and also run that on robots so
i'm interested in building a spike in
your network
models of basic sensory motor cognition
that operate on an actual robot in this
case this is a simple model
that can do reaching uh towards
different targets in space on the robot
arm
and it can adapt um to changes in the
dynamics of the
robot and it can adapt to locations
of the target changing location in space
but it does all of it
in spiking neural networks and it can
this can become
a basis for building a higher level
cognitive
model of uh how you might interact in an
environment
so this is really relatively brand new
this was published um
i don't know a couple of months ago uh
the actual proceedings are not even out
yet
and we have a second paper in the works
on this that just does a slightly more
um expansive uh way of controlling the
robot
using forward models in particular to
predict
where the robot arm is going to move uh
while
it is moving in order to make control
more accurate so there is nothing
particularly new in terms of robot
control here
the thing that is new is that we're
doing this in spiking neural networks
uh with all the dynamics that come with
a spiking neural network so we're in a
situation where again
we don't necessarily know exactly how
this robot is always going to behave
in particular if it's going to start
adapting and learning
from a human interactor so i can no
longer
ignore the fact that i might be doing
collaboration collaborative tasks with
a human because this might eventually
co-determine what my robot
will pick up on and learn and if i try
to figure it out just in terms of what
spiking neurons
are doing that might actually become a
hard thing
all right so this is basically the
reference for the paper
uh just to have that there but it's what
i already mentioned
and then i mentioned forward like models
how am i doing on time i have 10 minutes
or so left i guess
we did start a little bit late so
hopefully i have 15 minutes
uh let's see um every now and then uh
you start thinking about how do we need
to interact with the world and every now
and then you end up with this idea
uh that you need to have a forward model
you need to have some way of predicting
the outcomes of actions
um either because you want to predict
the outcomes of your own actions before
you carry them out
or you want to predict what other agents
are doing
while they are carrying it out
potentially so that you have a nice
interactive dynamics going on right
certainly in hri this is extremely
useful
uh because you can help the robot choose
appropriate behaviors
not just because it can predict the
outcome of its actions before it carries
them out
but also because it can already figure
out what humans are doing and then
decide
what it should do in itself um
the thing is that you cannot always
expect that you have large amounts of
training data available
when you when you try to train these
models because there are many different
ways in which robots interact
with robots and we're not going to
collect data sets that cover
all of this uh extensively a good
example
is autism spectrum disorder where every
child is really unique
but you're only going to get so many uh
data points on how they behave
right and then out of there you try to
build something that as
good as it can will work even in cases
that the system hasn't even seen before
it's really a different completely a
completely different game
uh from uh trying to figure out what a
traffic sign is
by first training on 150 million traffic
signs right we
don't always have the luxury of large
data sets
in human robot interaction in particular
yeah so something you can do is look at
these kind of data sets that do exist
there's a nice one it's called
pincero this was presented a couple of
uh years ago by several lemanyon and his
colleagues
uh and what they have is just a data set
of children
either playing with each other or
playing with a robot
on a tablet like a really big massive
touchscreen that is placed between the
two agents
and we just have a bunch of videos in
that data set of how these children play
uh so this is relatively naturalistic
they were not told to do anything in
specifically uh the
videos are just um collected
and annotated later on by human
annotators or like levels of engagement
and whether children are playing or not
engaging in play
and so on and you can obviously throw
these kind of uh
videos that you get out of there through
um software like open pose
to go from a full scene to like the
skeleton of the child
and then you can ask interesting
questions like if i'm interested in
predicting like how engaged children
are can i figure this out maybe not
necessarily from a full scene but can i
figure this out
uh from looking just at the kinematics
of how they behave
and this kind of data set is useful for
that
but the first question that you can ask
before you get
to the question of can a robot recognize
these things is even
can humans do that so we have this
study here where we did that can you
take
uh can you show these kind of videos do
you have an example here on the top uh
right uh of uh children playing with
each other on the sam tray
and then you have the killer map they
had the open pose uh version of it
uh to the right of it do people
generally uh
are they capable of recognizing what's
happening even if they only see uh
kinematic information uh and uh do they
agree with each other what's happening
that's
basically what maddie bartlett here it
in this paper
so lots of participants it was an online
study uh
it's just a simple questionnaire study
so show them a bunch of videos in
various
conditions they either see the full
scene or they just see the video
and then ask them a bunch of questions
like at scale like to what degree do you
agree
um with those things like the child what
the left was sad the child on the right
was sad
child was aggressive the child was
excited a bunch of those things that's
all there is to it
there are also a bunch of questions
there just to catch the people who are
not going to actually try
uh to do your experiments so you can ask
people
uh were the people in the video children
or adults
uh and you ask and the answers are
strongly agree disagree
children not sure adults strongly agree
so people who always just
click strongly disagree those who can
filter out immediately
because they clearly didn't read the uh
the instructions but what what we had
left were i think like 130
or even more people after that okay
so first thing you can do is ask how
much do people agree
uh with each other uh two insights that
fall out of it they agree with each
other more if they see the full scene
than if they see uh just the kinematics
um they don't agree with each other a
lot though so if you look at these
krippendorf alpha
scores they're really not super high
right so they're just like the highest
thing we have for a clip
is potentially 0.4 or something yeah
which isn't really great right it's not
really up to the 0.8 or 0.9 that you see
in like fantastic conditions
but at least it is above chance so at
least humans
can pick up on something they don't
necessarily agree
on what it is uh that they're really
seeing but it is uh the agreement
um is better than chance and it is
higher
if you have the full scene than just the
kinematics but even for the kinematics
it is higher than chance so at least
there is some hope that we can build a
machine that can eventually pick up on
these things
because uh at least humans seem to be
picking up on something there
hopefully we can you know exploit that a
little bit
you can do an exploratory factor
analysis to see what is it really that
people pick up on
and they basically pick up on imbalance
between the two children they pick up on
the valence of the interaction
and they pick up on how engaged children
are in the tasks
so those become like really interesting
targets to see if you can
recognize them with an algorithm
itself you can see how good it is
uh how good machine learning algorithms
would be at
predicting how humans would classify a
certain
scene so in this case the via the
classifier is fat
the video scene and then we ask you to
predict what it thinks the human
ratings were scales and again
it works but really not great and all of
that
is just the take-home message here is
just humans
have considerable variability uh in how
they
interpret even the same theme right so
that's why i said at the beginning right
if you're in user experience design
and so on i think this will all be
straightforward to you because every
human is different
and they interact with things that you
design differently
but if you have um
if you have um like a machine learning
approach and you think all humans are
essentially going to be the same
i'm trying to say that they are not
going to be the same
simian has to leave so by simeon i mean
the talk will be recorded i guess so
it will just be available for everybody
else
even the parts that you haven't yet seen
uh okay how am i doing on time so i have
like three minutes left i'll pick up the
pace
a little bit so that the first thing
here is humans are different
uh but they do still like agree on some
level right it's not like there was
complete disagreement between all our
raiders
and even a knm could pick up on the fact
that there was something
uh that people tended to agree on here
just not very good
so at least it sort of scopes like what
we expect
in performance from the algorithms that
we built we certainly cannot expect
a hundred percent agreement with a
certain ground truth
if humans will not show us a hundred
percent agreement
right so with that in mind we can then
ask can we essentially build uh
algorithms that can explicitly recognize
this kind of engagement it's taking
everything into account
that i just set and doing it in pincero
so
first thing you can do again is ask
people
to rate how much they think various uh
uh clips uh are engaged uh uh a show
um engagement and the way we did that
here
is that uh the um principal data set
comes with some
annotation and
we asked basically uh people to rate
engagement in the three different
annotations that the data set came with
so there is
a goal-oriented aimless play there is
sorry um there's a goal-oriented play so
if the children are certainly doing
something
purposeful there is aimless play when
they seem to be playing but without any
clear goal
and there is a no-play condition where
they don't seem to do anything at all
and it makes sense that you would expect
uh goal-oriented things to be more and
to feature more engagement than aimless
behavior and that should still be more
engaging than no play
behavior so we asked people uh how much
they would rate engagement and this is
sort of
on the right here with these plots and
you can see that it's difficult
to distinguish those two especially
goal oriented and aimless play are
difficult to distinguish between their
engagement ratings it's certainly true
that goal oriented gets a little bit
fatter
on higher ratings and a bit narrower on
low ratings
and if you do the statistics yes you can
find a difference but it's certainly not
the case that uh
humans tend to agree a lot then they
also agree a little bit
uh in overall that no play conditions
show less engagement but again not
in like the beautiful clear-cut way that
you would hope for right where
everything clusters around the six for
goal-oriented
clips everything clusters around the
three for aimless play
and everything clusters around zero for
no play that that just doesn't happen
right and that should adjust a little
bit our expectations of what we expect
from a realistic algorithm working on
ecologically valid
uh situation um scenarios
it's possible to train algorithms that
do and that then try to classify this
engagement so
um we did this with conceptus this is a
type of reservoir computing
and then we get fantastic performance on
trading sets
and slightly better than chance
performance on
testing sets and we also tried something
new that's called the
legendre memory unit this is something
i'm not going to talk about too much
because that paper is currently still
submitted and then we got
much better results and in particular we
can also use these lmus
um to kind of do perform somewhat
reasonably
on classes that were not explicitly
trained uh in the training set
so the idea being that you can train on
examples of higher engagement and
examples of low engagement
and if you then see an example of
intermediate engagement that you haven't
seen before
it's a little bit easier for our
legendre memory unit system
to interpolate between the two extremes
that it has been trained
on and understand that this is uh this
new example is something in between
than it is in other types of classifiers
but the principle
uh is just what reservoir computing will
give you if you have
reservoirs that work well on 2x streamer
it should be possible to also identify
and things that sort of semantically
learned between two extreme and it
should be possible
to do this with like a reservoir type
computing like a conceptor network in
particular
as well it's just we that to work as
well as we wanted to
which is why we switched to these
legendre memory units if you haven't
heard of them there is an europe
europe's paper last year by aaron volker
and colleagues which describes this it's
basically a kind of recurrent memory
uh which tends to make optimal use of
the reservoir so if you're familiar with
reservoir computations then you know
that tuning the reservoir and the
parameters
is a bit of a black magic sort of dark
art how do i find
uh the best um the best
values for making this reservoir work as
well as i can
and the gender memory units give you a
mathematical answer
to how these uh connections should be
built
all right how much time do i still have
then
just to say we should
sorry i can wrap up okay yeah because we
we yes we are running out of time
okay yes okay i'm gonna wrap up
right so the same thing forward models
this idea that we can predict the
outcomes of actions this is relevant for
autonomous driving
so i'm just going to leave this as a
plug for this project
but what we did is we built a controller
for
robot vehicle that is a little bit more
inspired by how
the human mind works in particular in
terms of action selection
and so on how does the system decide
what to do how does it run hypothetical
situations so that
you don't actually have to crash the
vehicle in order to find out that a
certain behavior
is a bad idea uh so we did that in a
bio-inspired way right you look at how
the human brain
does this and then you try to implement
that in a controller
um it's also you that you're possible to
use this kind of predictive mechanisms
that you build in order to predict what
other agents are doing
um but you can only do that really well
for other vehicles because they have the
same sort of dynamics
if you try to predict what uh the
pedestrian is doing
then then gets that gets a little bit
harder
so that's sort of my open research
question
uh on this part uh it's very easy to
build biologically in
inspired control for an autonomous
vehicle because of advantages it
operates in two dimensions
so that's fantastic um but using it to
predict what uh
vulnerable road users are going to do
and so on that is much harder but that
is of course what the field
still needs to solve so i can see uh
maria luchi getting a bit impatient
so i'm gonna zoom through the last point
so
people have opinions uh i just want to
say this
right so a bunch of the work that we
have done is building robots or
introducing robots in care homes so they
can work like companion robots
uh for the elderly uh and they're an
interesting thing if you're in that
field
is that you have a lot of papers about
researchers having strong opinions on
what this robot should look like
and if you then go and talk to the
people in the care homes
they have very different opinions of how
they should look like so what we did
here is simply ask a bunch of robotics
researchers like what kind of robots do
you think
are appropriate and why are they
appropriate for a kind of companion
robot scenario
and then we asked a bunch of elderly in
care homes what kind of robots they
would want as well and we also asked
a lot of other stakeholders right we
asked people
who are nurses in care homes we asked
family members
everybody and the interesting thing is
that roboticists really like
paro and none of our elderly people
like parano the reason roboticists like
paro
is because it's animal like but it's not
so it's familiar but it's not too
familiar so people will not get confused
uh as thinking that this would actually
be a seal
the reason the elderly didn't like pero
is because they don't know how to
interact with a seal they like a cat
they like a robot cat because they know
how to interact with a robot cat
they know it reminds them of mr
scruffles that they had
20 years ago uh it's just more familiar
and that's okay
and i think that's the last last point i
want to make here right because it's
very easy to go off
and build beautiful solutions to
problems that you think make perfect
sense
and then when you deploy it with your
end users uh
it fails fantastically as a product
because you never actually talked
to them and you designed a solution that
they were not asking for
so this kind of importance of always
talking to the end users of your
products
no matter if it's a robot an algorithm
or whatever this is sort of highlighted
in the paper that i have
i just have to reference down there and
that's the take home message here
and since there is at least one
philosopher i want to point out that the
same thing is true for ethical concerns
philosophers have a lot of opinions on
what ethical concerns of robots are
and stakeholders have a lot of opinions
on what ethical concerns
are and they do not match between the
literature
and what and philosophic
and users don't agree with what
the ethical problem okay
so i've got to stop there so you see i
didn't actually get to talking to
about xai but i got through most
everything else
these are the phd students who did a lot
of the work that i mentioned
and so i just want to thank them and i
want to thank you for listening
thank you i think it's great
and it's a pity that people uh have to
leave
because yeah usually this is when our uh
meeting uh let's see if some questions
come up in the short time we
have left but in the uh in the meantime
i really would like
like can you paste the title of the last
paper you discussed um
[Music]
so this one is basically i think on the
importance of end user-centered design
uh i think this one is basically ethical
perception
of stakeholders yeah right yeah
well i can also
you will find them um they are there
this is the most cited paper
anybody has questions of course i put my
email address here so you can also just
uh
i don't know we're running out of time
thank you
i will hang around for a little bit more
but i just have to go one second and let
the cat out of this room
you know always cut yeah there you go
[Music]
so there we go sorry that's like that's
no problem
it was really it was really cool um
of course many aspects to to look at
um yeah for me what was interesting is
that at one point you you
you put you wrote like a uh we should
not
define all the behaviors of an attempt
but uh but still we want to constrain
them now
and i think the following slides about
the
neural networks were about that right
but the thing is that
when i when it gets to that complexity i
don't understand
it but uh yeah yeah yeah and i think
it's a very fair point to make because
it's very easy
uh to to mention this conceptually right
that
i don't want robots to do whatever i
want the behavior to be constrained but
at the same time i cannot constrain
i cannot define everything a priori but
i still want uh to constrain it
so this is already uh difficult to
really disentangle if you want to build
the system and then if you want to have
a system that actually behaves
within those constraints so that gets
even harder so that's obviously
you figured out why we didn't actually
build this right it's a conceptually
i think it makes a lot of sense but how
to really do it is hard and
maybe it's a bit of a concern uh that in
a lot of modern day ai
we don't actually seem to care about
this part right we just throw
a lot of data at an algorithm and
it's uh yes we don't think yeah yes
yeah i was actually interested in this
point because we are as a group
well by the way i will jump uh out soon
because the others
left because we have another meeting but
in any case we are writing
paper and one of the points we are
making is that
like um the agent should be
uh constrained within uh
uh the boundaries of morally acceptable
situations or
now to make it simple yeah but actually
then
making this something actionable it's
all another story
like you can say that indeed you were
saying
but actually maybe it it's really
something yeah and it's also hard
because it's like this concept like
morally acceptable is really fluffy
and even if you build something that you
think is morally acceptable right you're
just building your own morals in there
and it's going to bias the system
towards your own morals and yeah
yeah yeah yeah indeed indeed
yeah okay
yeah yeah yeah so thank you very much
again
i'm so sorry that uh i have to run into
that other meeting i thought it was a
third year
but yeah if kenny i think he's also
leaving for that
uh but i hope to see you soon maybe at
high or
other venues exactly oh you know maybe
in real life it's not that far
exactly exactly exactly well
let me know if you if you come into that
for any reason
yes exactly i will definitely be in
touch and also if anybody comes to know
me and you know you have my email
address
just drop me a mail you know especially
after corona has gone away maybe we can
actually meet up in person right
would be yes yeah that would be really
nice
that's all all right all right cool
excellent thank you very much for having
me thank you for joining
thank you no problem bye
you