239. Soares, Tallinn, and Yudkowsky discuss AGI cognition

hello and welcome to session 239
in the ai safety.com reading group
tonight we'll be discussing part of the
post suarez talin and yukowski discuss
agi cognition
elias rykowski is
the senior researcher at the machine
intelligence research foundation jan
talon is
the founder of skype a philanthropist
and very interested in
existential risk
there is also nate schwarz and uh
rapenzina present in this um work but
actually they won't be relevant for the
part we're talking about
so this is a discussion on agi cognition
that happened
on
discord and several other places uh this
fall
um
we will not go be going through it all
we'll be zooming in on the part that
talks about the treacherous turn here um
and only that so that's quite good we
will skip um
i try to like roughly estimate we are uh
zooming in on four percent of this uh uh
sequence and actually four percent is
probably higher
but
i believe it's an important focus um i
believe that the treacherous turn is
indeed
uh very close to the core of the problem
of ai safety um and i believe by with a
rather large margin i see a correlation
of somewhat something like 0.8 between
was how
how will we in any given world
solve the treacherous turn problem and
uh
doom from ai these are very tightly
correlated in my world
in my view
first a few meter comments unfortunately
normally we read formal articles instead
of this kind of discussion and this is
really hard to summarize briefly
these people have a very rich context
between them there is no argument why
the treacherous term is important or
like the basics are not discussed they
are assumed and that can make it harder
to follow um
of course when i summarize i lose a lot
of like the
emotional language and quality
qualifiers um
i
also uh some of this is actually
happening as comments on a
google document which is not public so i
can see the comments but i can't
actually see the google document
hopefully that will be published at some
point
and i also can't really distinguish
between i i don't want to distinguish
clearly between what uh young italian
and what uh elias utkowski say um so um
mixing it up most of the things are
being said by eliza utkowski
i
also feel a bit iffy about my uh
my style here because i generally like
to call out places where definitions are
loose and can be tightened up and there
are a small kind of
technical errors and
for this kind of discussion which is not
meant to be formal that's really unfair
i think you say e.t james would hate me
for this but i do actually feel that
sometimes by poking in between
definitions you can at times find
something that i feel is valuable
so i would like to start by taking one
step back and just describing what a
treacherous turn is without arguing for
in the details
so here is the uh definition from
bostrom you can read this this is from
the book super intelligence path
dangerous strategies there's another
definition here from the
less wrong wiki um but the the overall
gist of it is that if we have an in ai
that is obeying some of these convergent
instrumental goals um then there is this
instrumental goal of appearing aligned
which is uh which holds for both aligned
and falsely aligned ai's while they are
weak
so uh that means in general that but and
then of course the behavior diverges
when they become strong and um
this means that testing
ais can be um
really difficult we can't
say for certain that
if it's
behaving nicely in a box it will also
behave nicely when we let it out of the
box
illegitimate cow skin in this world
divides the treacherous turn into five
probable phases
the first is the over plotting phase and
thus a hiding and binding phase
does the escape phase
the theme stays in a terminal phase
if you start with the other planning
phase that starts at the point where the
ai realizes that if it takes over the
world it can get
more choking utility for instance in the
classic example with the paperclip
maximizer but it will if it can
take over the world you can make more
people clips
and this ends when uh
is no longer over where the ai realizes
hey i should conceal this kind of
plotting from the operators
so that's a gradual step where first it
realizes probably that takeover is nice
and then our humans will probably resist
this in particular the operators which
happen to be humans are also some that
would resist this
but it's over meaning that
reasonably standard
transparent
and interpretability methods could in
theory uncover this
the current state of the art is not
sufficient for this but it's something
that could in theory be done
this uh this phase will probably not be
of a duration zero uh most likely uh
eliezer you'd have to put some greater
than 50 chance at least that
building an ai that is
that when we build the first ai that
realizes that it could take over the
world it will not yet be capable of
realizing that it should conceal his own
mind from the operators
once it does that hello we go into the
hiding and binding phase which is when
starts to conceal the plotting from the
operator
and this ends um
when um
if this hiding ends when it figures out
that the expected value of acting uh
there is an action where when takes this
action has a greater expected value than
just continuing to heighten the weight
um
so the reason why it weights in this
phase is to undergo some kind of
recursive self-improvement from
some way of improving its cognitive
capabilities
at this point a big focus is also on
modeling the operators from a
psychological point of view this is both
important actually at the beginning
where it needs to decide it should hide
its thoughts and also
towards the end
where it needs to figure out when does
need to take um
protective measures
uh and another reason why waiting is
good
in addition to becoming more intelligent
is also because the probability of being
cough is probably less later when it has
both a better plan and it is smaller
there are some of the definitions the
previous space
ended where the ai realize it should
conceal and this is where it's actually
concealing the this phase is starting so
there might be a gap in between these i
think the earlier koski talks about what
happens in this gap so he knows that but
the definitions could be tightened up a
bit here
after that we come to the escape phase
so this is where there is a
sorry
and
a
some kind of protective measure that
gives higher expected utility by doing
it than by just waiting and
in this
text
doesn't frame it as a protective measure
he uh
thinks it as escaping and that seems
like the obvious way to
defend against the human wanting it to
be turned off but there might be other
ways and in particular at this point it
is unclear if the ai is smarter than us
or less smart than us or more likely
very different from us
so it's certainly possible that it could
find a different way i can immediately
think of a couple
that i won't immediately share here but
the escape phase ends when the ai
estimates that it's quite confident that
it can take over the world
so um in this case the expectation of
just waiting is probably still very high
um so it's just that escape seems uh
both very productive and with a very low
probability of getting caught at this
point
um how would what the action could be to
copy itself to the internet and
we some of the models we see right now
are extremely large so it's possible
that it would prefer if you needed to
copy itself to the internet to have some
kind of distillation process that is
coming itself through
it could of course also be that just uh
redesign something else but then we have
to ensure that it's aligned and
that's
the an interesting question to what
extent that's easy uh
if you go back to good old-fashioned ai
then there was like a utility function
and putting that into another ai seemed
reasonably trivial with uh things like
gt3 that's no longer quite the case i
think still think it's an uh
a substantially easier question but
maybe not
and of course this could allow the air
to
in this escape phase to redesign itself
and perhaps other actions
then we get to the doomed phase
and that's where the ai is confident it
can win but it's still going to wait and
hide and
do different things get gather more
information to try to get the
probability of victory as high as if
plus i possibly can
until at some point it can't get higher
and then it won't just take over
um
sorry
the key thing that they is doing here
is trying to figure out is there another
super intelligence that's just about to
take over also now you know to figure
out if it's uh like the second and
that's um very important for it um
probably uh we're talking about things
like during a day or something like that
and the probability of another super
intelligence forming in that particular
day is probably rather small but
certainly not zero
um and there's also the uh
this value of galaxies leaving our
hubble's uh
sphere hubble space um so uh
that's also something from this value um
uh
suggested on a sandbox could calculate
how many 99.99 until this galaxy becomes
dominant i think this is kind of a new
pick but there are actually three things
that prove provides this value for
weekend there's the galaxy going out of
reach there's the
risk of getting caught and there's a
risk that there'll be another agi
forming at just that moment
finally we get to the general phase
which is where ei can't really drive up
the probability of success any further
and it ends with our extinction and
what how will that german phase actually
look um well at this point the ais
almost certainly is radically super
insultant so it's hard really to say if
you just use human level in the
imagination and you're making things
like nanotechnology which seems quite
feasible
you could do something like distributed
diamond
bacteria filled with botulism and then
kill everybody over the course of one
second or something like that to ensure
that the victory is too fast and
complete that there is no way to
implement any hidden counter plans
but probably the ai will do something
different by the principle of efficiency
either it can can do as well as we see
here or there is some part of the
underlying model as fault for instance
nanotechnology could be more difficult
than elizabeth kowski and hearing
dressler 4c
because this is a slow scenario but it
doesn't look slow from our point of view
um i would somewhat disagree here like
obviously when you talk about scenario
if you only think talk about the strike
then if it takes one second then that's
obviously very very rapid um but
it's not clear that when um
with the entire takeoff that's from the
ai started the over plotting phase until
the um the extinction and um
that could still be quite a bit of time
in uh
calendar time um
i would also point
that boston also uses this definition of
whether it's a fast or slow takeoff with
a lot of focus on how much time do we
actually have to react which i think
is probably the crucial consideration
here
a variation of this that could happen
there are multiple variations but
assembling one is one where the ai
escapes early
possibly if it just
has a wrong model of
humans and
escapes early by mistake or tries to do
that um there is a part
where
sure i understand
this
um
where we catch the internet on the
internet
but um
koski won't spot this kind of
generalizing from fictional evidence um
if this is a thing that the aai strongly
prefers not to have happen so
it seems unlikely that this the the
action is going to take
another of this more science fiction ish
plot is that we can see another ai not
one we've built that is somehow uh
detected on the internet um
i could see something even a similar way
this could happen a number of people
have suggested to build an agi and just
turn it loose and hope it does something
nice so that's something that we could
all i could also see happening
the ai
if it's to escape before it
estimates that it has a good chance of
escaping successfully might be because
it's uh it stands in uh
it faces imminent uh erasure or that
it's um feels it will be replaced with
something else that it can't bargain
with um
but
this is kind of like an overly specific
miracle and that's something we should
count on happening
um i agree we probably shouldn't count
on happening but
it might be possible to trickle this if
you every time you build a new version
of the agi you change something of its
utility function somehow so it's obvious
that it's opposed to the previous
generations um and
this kind of bargaining with successor
and a causal bargaining in particular is
something that humans
most humans don't understand that a few
humans barely understand and there is
certainly a possibility that uh we build
an agr that's incapable of how of of
bargaining uh
effectively
but if we try to deliberately trigger
this kind of thing then it's it ceases
being an overly specific miracle
so what would an early escape agi do
well it probably will just hide but
there are a number of other actions it
could take
this ties into what elias uh casually
calls the law of earlier happenings
this is i think you would call this
the
tendency to uh
try to write laws um
lawful things in the universe of as
explicit laws with capital letters um
but the law of earlier happenings says
that um
when we discuss this like also when when
i try to explain the treacherous turn in
uh
uh to people like in the reading group
or whatever um then
i focus on the most plausible things
that could happen and uh with
instrumental convergence then it has to
do this post steps predictable endpoints
this kind of thing and um this is good
for arguing but it's actually
a biased way of looking at the world
because if you look at the world
then it's not true that only false
things happen and only very predictable
and plausible things happen uh sometimes
rather random things happen and if you
want to have a model of how the world is
actually going to look then it's going
to look kind of like what has looked
over the past
and that's in general been really really
hard to predict
so in
with this treacherous turn the uh the
thing we would probably see is things
start happening earlier and in more
chaotic ways
so probably not the specific miracle of
young and dumb ai escaping but something
different chaotic
would probably hold the blind so that's
a
very weak prediction um
so what will happen is something weird
happens well um
kills a lot of people
there is a hope that this would serve as
some kind of fire alarm
and
less than that so an ei that just
goes to a i don't know upwork or
something like that probably will not
cause anyone to worry or certainly will
not be a fire alarm for adi
um if it kills a lot of people will that
be a fire alarm
believes it won't because um the
bureaucracies that could take action
upon this are not really geared to
handle this kind of thing i would agree
but for a different reason because it's
like here where the effective action
would actually be once there is a rogue
agi on the internet
and
i can't immediately see any existing
bureaucracies that look like they're
capable of taking any kind of action in
the space roughly close to what would be
um
uh effective like it's possible that the
united nations could shut down the
internet but that seems extremely
unlikely to me
has the law of undignified failure that
no matter how sad you imagine the
the failure to be then in reality it
will be even less dignified and the word
dignified is one that it is doesn't he
doesn't precisely define what he means
by that
um but i guess
it's
a
failure that is closer to actually being
success
um and
one way we could
fail in a sad way is that there's some
operators who uh see the ais plotting
and then they rationalize in some way
say why why it's good enough and maybe
they just train it away that's really
easy that's uh
basically what ai uh researchers do all
the time um and then
the assumption of a really
sad failure could be something
but like ignore that um at some point
it disappears because the ai is hiding
it and they already have to say now it's
gone probably
uh the area has become
if they are unaware of the optimality
pieces then they'll just say oh moral
realism is true and it's become smart so
then it must also become good
and
the operators
would conceivably also just dump as much
computer into the uh as possible and
allow it to experiment if you want and
if it just asks for lightning then sure
you can synthesize proteins
in that kind of case then we won't
actually have a distinct escape phase
because there's no need for the hospital
um
and i could obtain a designer's
strategic advantage in many other ways
like social manipulation or whatever
or the
cognitive abilities of the could develop
in a in a very different order um which
would also change the scenario
potentially quite a lot
so in general um
the path to this endpoint isn't really
strongly determined by elias yudkovsky's
world model the endpoint is
and uh
yes is saying it's
it's hard to predict especially about
the future i believe that's a danish
proverb i tried to look it up but i
couldn't immediately it's a bit more
complicated than that apparently
so let's zoom in a bit on the end of the
overplotting phase
um so here
um when the ai is plotting overly it's
not doing anything to hide this at all
in in the beginning and so if we have
some good transparency we could have the
ai say something like i should make a
funny voice here i can't make funny
voices oh dear my model of the operators
is looking at my over actions and coming
to conclusions that are disadvantages
for me
so that's like the first step and then
if we had really better transparency
tools then we could see
that the ai is now trying to figure out
how students think in order to
manipulate us
this requires the ai to think about how
it is thinking so some reflectivity um
but the second oops the second part
would be something like
oh no my operators are modeling my
transparent thoughts and reading to
those thoughts in a way i'd rather
manipulate
so the um
how will
the ai react to figuring out that it's
being monitoring or by deducing the
existence of a transparency system well
one thing you could do is to just stop
plotting uh that would would be an uh
one way but that's kind of negative
because uh if we realize if we can read
the thought that says oh i should stop
playing because the uh that's a better
way to plot
then it's possible that the operators
would notice and
become worried based on that
if the operators were less dignified
then they would just assume that the
last credibility was the one that worked
and it could be even
signified in different ways
the raiders don't care or the operators
don't have a transparency system then
they just don't think about this but in
the case where there is in fact some
strong transformation and we can see
this
then
most likely since we have taken the time
to build up a system that can read this
information then there's a chance that
we will
that will actually listen to this um
i think that's
technically not entirely certain in that
interpretability can be used for many
other things than uh just ai safety in
particular in the case where we built an
agi by just you know stacking more
layers and we don't have any thorough
understanding of intelligence or agi in
that case we'll probably be forced to
build better interoperability tools to
be able to have any hope of using this
and in that case we might see this
thought and still
basically just ignore it
but um but i think it's worthwhile to
keep in mind like when you do this kind
of reasoning in steps it's very easy to
get into the conjunction fallacy where
it's hard to update enough in the
conditional probabilities on what has
previously happened
the over plotting phase could end early
because the ai has in some way been
fine-tuned for figuring out that it's
just uh not plot overlapping
that could be in the case where uh it's
been trained on agents modeling
different uh
uh other agents um a very sarcastic
comment here is but like with something
like gt3 where it's trained on text that
includes text where humans are talking
about how to uh model other humans that
seems like something that would change
the air in particular how it should um
be treacherous um and i think uh a an
even worse example would be this
conversation the text that eliezer is
writing that will also be something that
is
actually literally um
describing the hypothesis that we don't
want the ai to find
um
but there are other ways we could give
this uh kind of modeling behavior like
if you're optimizing for not just
winning go in self play but beating
humans in gold then that seems to imply
that it's very beneficial to have a
strong models of
humans
we could also have this happening
without any references to humans just in
social evolutionary games um that could
also be a place where the ai could learn
this kind of behavior
so
what will people do well the obvious
thing if you see the ai is plotting is
to just do gradient descent against that
uh that would be um part of the problem
here as i see it is that for an ai
researcher that's basically tuesday
that's the normal thing you always do
you have an ai and it's doing something
and you don't like what it's doing then
you do gradient descent against that so
it stops doing that and that's what
they're doing every time so this is over
plotting but they just use the same
things and then they do that and then
stops and then they just continue to the
next thing and that's probably not a not
how you do credibility that's not how
you do a safety that is probably
unlikely to generalize at least at some
point we are getting um
we we will be needing some some kind of
people work with their safety earlier
specifically
this will seem to
actually work when the air becomes small
and finds like an undrug strategy around
them
so one of the more dignified ways to do
this would be to have like a
training set where you um
take the some of the plots and then you
do gradient descent against that and
then you have a validation set also to
check that those are also disappearing
but you could also just train on the
validation set like literally every plot
that you can detect you try to change
that away and then you don't have any
clue about whether that's actually fixed
it um or even if it turns out that when
you trade against it it doesn't
disappear you can it can get different
kind of bad results and you can choose
to just just continue or you can just uh
yeah there are many ways to do this
worse
so if you are trying to build an agi
that doesn't model other humans is that
possible well um it might be possible uh
to have an agent that just doesn't do
this kind of thing that's just focused
on nano technology or whatever you want
your road light to be
um
it is it kind of doesn't sound too
optimistic about this in like he says
you could do that if there is some great
gain of uh starting the height face a
bit later um but i think there is um
here i would use elias kowski's uh
concept of something that is dignified
against him as um this is something that
is on the path to eventual success um
but i'm not entirely sure that i'm using
his word correctly
then you should define me better
um this of course is not enough but he
says this could be a part of a complete
strategy and i think the best place i've
seen this strategy described is in
miri's 2017 fundraiser um this document
describes this kind of strategy where
you have a
super human engineer that's super
superhuman wondering how other agents
follows its actions so it won't
go to that it won't start to hide its
actions this is sometimes called the
suggestion
and of course if you do that then you'll
obviously get an overly plotting agent
and you still have to deal with that so
you're not at all done once you have
that
perhaps we can um
use the threat of subsequent
superintelligences to somehow modify
what are the instances the agi will have
right because
that's one of the things we can do that
will actually stop potentially stop this
uh
the hiding and binding ai would be
building a competing super intelligence
and not just is this something we could
do we it seems like
like
the strategy that i just talked about is
publicly available is something we would
like to try to do in fact
um so can we lean on the fear of the
first surveillance of the second
to shape the behavior of the first
superintelligence
well um one obvious way we could do is
to
make sure they can cooperate by giving
it just an opposed value function but
there's an obvious uh yes risk here by
making at this utility max mice instead
of utility maximizing so it needs to be
done in a smart way but i believe it
probably can be done um that's not a
doubt of safety but it seems like at
least something it could be done
but a lot of strategies don't work and
you'll see it constantly points out
three problems the first is that you
can't bluff
something someone who is much smarter
than yourself in general that kind of
thing will never work
the second is that the strategic
situation is extremely unstable if there
exists strong concealed first by
capabilities
and the third is that even if you try to
uh one of the obvious way you could uh
do this is to negotiate with
superintelligence but how would you
verify these promises
um elizabeth castro claims this is
effectively impossible i tried my hand
at coming up with
like a scheme where
you have some
several super intelligence against each
other and you lean on the fact that um
as they leave part of the universe as
close to the state of life they become
incapable of influencing each other and
um that kind of symmetry i think you
might be able to do things
uh
to this description of the church chart
like the over uh plotting phase here is
on board with that like this matches to
with the type signature of classic uh
machine learning and so type signature
that's a concept from uh well taking the
computer science but mostly people think
of what in terms of programming well the
type signature is like a function with
some input arguments and things like
that and in this case
something like a partial observable
markov decision process then these are
like the input arguments and the
properties the actions and
in this kind of setting it seems that if
taking over the world is a possibility
and it's good then
eventually this is a a solution that the
ai will figure out
and um there's even an easy way to find
out how we can figure this out because
if there are humans in the world
then uh you can search over the brain
states of humans and both then you can
model them and then you can figure out
that plotting is a thing and ignore the
world is a thing because humans know
that and then you'll find these
solutions so um in general yan chan is
confident that a
sufficiently intelligent pay-per-click
maximizer will be able to find the
strategy of taking over the world
but the height phase
that is not something that is easy to
express in terms of standard uh terms of
machine learning
because
when you
hide something we kind of assume that
you have a
you have in the
over plotting features like the root
search process that goes through and
figure out what would be good actions
but you also in this case need an
obfuscated search process um and um
uh the italian calls one of them a root
uh search process and the other a user
land i'm not entirely sure i like this
description this metaphor in that that's
mostly about privileges in modern
operating systems and what we're more
thinking about
don't have access to our neurons and
it's probably this kind of thing that
he's uh referring to it like
possibly
and how do you discover this this is not
something that's always like if you do
great interest and you ever get in
obfuscated search uh
process and it's also not always how you
would turn into that
and i noticed that man it seems that
industry is not precisely answering this
question um i think it's quite fair he
doesn't want to explain how to do this
but he is somewhat touching the question
defining five levels of cognition for
agis so the first is
a non-imaginative non-generalizing soda
consequentialist basically the ais that
we have right now
and they will plot
if and only if they have previously
encountered some uh some plots that they
are that have given a low loss
and and then they will plug in otherwise
not
there is
there's some sound in the background if
you have questions please
yeah
okay so yeah okay um
i will um
i will continue uh and also uh describe
a bit about how uh
uh
yen talent sees this as fitting into the
uh the current machine learning pre
paradigms
the the next stage from this is where
the ai becomes both um imaginative and a
true consequentialist like there's a
long description uh and debate about
consequentialism with richard and go
early in the sequence that i don't
really want to go into here but this if
for our cases the important thing here
is that at this point the ai will indeed
plot even if it has never encountered a
um like i have seen plots give be given
low loss it's imaginative enough to
discover the strategy on its own
the next level of cognition is one where
um the ai has long time horizons
the fourth is where it has figured out
that there are humans and they will
react to to this over plotting and the
fifth is where the ai is also reflective
and it's hiding its thoughts because it
it understands that this is a
transparency system
um so of these uh in yesteryear
counseling says that going from step one
to step two is really hard in the same
way going from step four to step five is
hard you can't just do gradient descent
that that is not going to be enough and
i think in general um
it is a casca believes that you can't
get to egi by
just doing the same thing as we're doing
now just stay by stacking more layers
unfortunately that might not be required
because you can get this kind of hiding
in a way that doesn't require the ai to
be truly reflective that's by just doing
great intercept against
um against these bad thoughts the way
that they are detected if you have a bad
thought detector then you
do breeding descent against
what are bad thoughts
um and this is something that will be
possible to do and will only generalize
shallow
and the ai will be able to find the
nearest on broader strategy and when it
becomes just slightly more imaginative
uh and then of course uh it will be some
bad thoughts that can't be detected and
that's basically uh
uh basically this thing
uh
so uh
jan thailand is trying to translate this
into a standard machine learning
language where we start out by having a
um
non-imaginative through consequences
list but that's basically just a
uh
model free reinforcement training
that's something we are quite familiar
with an imaginative consequentialist
what's that um
uh yantalen says this is basically a
monte carlo research with some kind of
world model um that's something that we
are reasonably familiar with also um
iliya zerkasky is giving an example
as something that is imaginative but not
generalizing
um
and uh
uh yen talent is saying that most people
who are working machine learning will
probably not be
uh really uh enjoy to call it um
imaginative but uh just say this is
doing monte carlo research over the
world model is something that people
will have a much easier time to uh to
engage with
adding long time horizons that's
persistence and giving us some kind of
state that seems easy enough
adding humans to the model uh so it's uh
the fourth level that also seems quite
simple but the imaginative and
restrictive consequentialist and how
would you actually uh implement that how
would you
in something like great world how would
you go about uh getting that behavior
again don't use the answer slightly and
say well you know if you're talking to a
standard machine learning engineer then
you don't need to um to to talk about
this because you can just say uh like
what if the current state of uh
transparency and interpretability
continues to hold because
right now we can't see any of these
thoughts so in practice there's no
difference between an over plotting
agent and a hiding and binding agent if
we can't read this laws in any way
so
i think a more direct answer to uh the
entire question might be um michael
truster's uh post a gym grip world
environment for the treacherous turn
finally er elias yorkowski has some
rather harsh words about the epistemic
state of the aic cube engine because
what he has described here seems
to his words kind of obvious really this
is obviously a thing that's going to
happen and it doesn't depend very much
on ai this is something that you could
have um basically
determined back in 2010 without any
knowledge of the deep learning
revolution and things like that
um
but still
we think that the indiana safety
community doesn't really engage and
produce these kind of insights at all at
least the same community outside of
myself as it says um
i think there are several ways to
interpret this um like you
one way is uh to point to the east
community that is like outside of the
alignment community which is um
sometimes called like a mainstream ai
safety research um and um they seem to
not consider this kind of thing i think
that's a reasonable criticism but that's
actually not what elias is saying right
he's making a rather strong thing when
he's talking about everybody except
himself right i can uh
some names of people who seem to be
clear counter examples are the questions
john armstrong
these kind of people seem like they are
obviously able to engage and produce
these kind of
insights
so
what insights have actually been
produced on the treacherous turn well i
went to the alignment forum and looked
at everything that has been checked the
treacherous turn
and this is it basically right this is
in particular the one on top this is the
article we're currently discussing
and there is a better than super
intelligence down here and uh
this one with also and this one we've
also read i believe um so
uh
not much work has been done and i i
agree with india's casket that this is
actually really meager
and for me in particular because i
believe that the treasurer's turn is so
important uh then i think this is
actually quite disappointing so i
strongly agree with that
this is not everything that's been
written about the churchill's turn one
always thing that's missing is uh so you
cast his own writings at arbitum which i
feel are the highest quality writings on
the treacherous term
it totally
if i can go a bit outside the text here
then i envision this somewhat as a
challenge
and that um
it is because you're saying that
researchers are less able to think
adversarially than some science fiction
authors and also some fantasy authors
have more uh understanding of these
behaviors in in settings where the gym
involves mind treatise then yeah safety
researchers
and so when he's saying he has
previously not been um describing this
and has been holding it back as some
kind of validation set against ai
researchers but um
it seems no one is actually generating
this kind of insights
some people are able to recite them but
people are not generated and so
you might as well just
sell everything he has because um
evaluating all the aicf2 researchers is
not really productive because no one
according to him is able to uh
to generate this kind of
uh of insights
so uh one way you could look at this is
like elias asks 2010 challenge to
generate this the insights in his
validation set um and obviously i failed
that um in my defense he didn't say that
this that was this challenge um so i
didn't try but if i should speculate a
bit like a lot of things have happened
since 2010
so i would expect that
he has some new insights also that he's
not publishing
and that he's hoping that aict
researchers would
would be able to would generate this
kind of insights
and me personally i think
if such an uh
challenge exists i would like to
participate in that i would like to
submit my uh insights
in in the next session um try to figure
out if i can come up with some of the
implications of new things that have
happened and also ways to go deeper into
this model
and i will try to talk about that next
time
i probably won't publish them however so
you will have to ask me to for a copy to
get them
that is all for today thank you and see
you next week


AiTech Agora: Emma van Zoelen - Human AI Co-Learning  Mutual Understanding and Smooth Collaboration

but before i started my phd because i
was looking for nice places that worked
on like human centered ai
uh type of stuff
um
so yeah very nice um so today i'll talk
about my phd work um and also just some
my general thoughts and vision on the
topic of human ai co-learning um and
also a bit how it relates to uh
meaningful human control
because i already had a few discussions
with some people and thoughts so i'm
just
well excited to discuss it with you
so the title of my presentation is also
growing mutual understanding and smooth
collaboration so is that what
co-learning can bring us
um so to give a bit of background about
me and my vision so
uh i believe very strongly that if we
want humans and ai agents to kind of
live together symbiotically they need to
collaborate as team partners
and i think it is important that we
focus on the different strengths of
humans as well as ai agents to really
ensure that they can empower and support
each other
and i think that can help us to make
optimal use of the qualities of ai
agents but also while maintaining
respect and appreciation for the
qualities that we as humans have
and i think probably many of you agree
with me on that
so i have a background in industrial
design i originally studied industrial
design in eindhoven but i also did a
masters in artificial intelligence in
utzek university
and i think this combination combination
causes me to look at ai and robotics
really from a very human-centered
perspective but also interaction focused
so what happens in
the interactions between systems and
people
so my phd is about co-learning human ai
teaming
but i focus very much on the complexity
of interactions that appear and emerge
if you create such situations
um so if we talk about co-learning so
and start to think about what does that
mean
i always start with okay what does it
mean if we have two people
that collaborate with each other but
that are very different so they might
speak a different language or they have
different ways of behaving um and that
might mean that they don't understand
each other from the get-go
so what we do as humans to cope with
that is
we
adapt to each other
um we maybe learn each other's language
but also we change our behavior um and
we do this in a very reciprocal manner
so both both of us do that or if we're
in a larger team everyone does that
um
that of course raises the question what
happens if you're in a team with a
machine robot ai agent what not
does it understand us if we talk to it
and behave well probably not immediately
so what do we do
well you can of course say well we just
put the human behavior into the machine
so we let the machine adapt to us and
then we'll be fine that's perfect but of
course there's something a bit odd about
this uh the machine doesn't have any
hands naturally unless we give it to it
um
and i think also of course the inside of
our brains are very different um so it
feels a bit odd to let only one
completely adapt to us
um so
i always try to argue that you need this
kind of very reciprocal and mutual uh
adaptive learning process to really get
to uh like understanding and become a
well-functioning team to get attuned to
each other
um so when we talk about co-learning
well what comes to mind is a process in
which two or more parties or systems
change their behavior and or mental
states while interacting with each other
over time
when you look in literature there are
several terms that you can use to
describe this so for example
co-adaptation co-learning co-evolution
what do they all mean uh when compared
to each other so i think that they
differ in terms of time span
so if you talk about adaptation that
feels a bit like something that happens
like ad hoc implicit short time span go
evolutions like this huge long process
and co-learning is somewhere in the
middle
um i think they also differ in terms of
intentionality so co-evolution is
clearly unintentional just this process
that happens that we can't really do
anything about co-adaptation
can be
maybe a bit intentional in the sense
that you want to
well be better at something or improve
collaboration or whatnot
but at least
the actual changes can be relatively
implicit and i think co-learning is
more intentional so we really want to
improve the performance of a team we
want to improve our experience and also
that this development is sustained over
time and context so it's not just we
adapt and then we adapt again and adapt
again and this is just a continuous
process
of course it is somewhat of a continuous
process but if we adapt to something
that works well we maybe also want to
keep that and keep doing that
um
so in relation to that um we can also
wonder in general what does it mean to
learn
um
so
probably many of you have either done
the utq or the um
some graduate school courses on teaching
so you might know this
um so we can say learning means to
change behavior
which very behaviorist perspective we
just change our behavior we can also say
learning means to acquire knowledge
the more cognitivist perspective um with
the computer metaphor we just gather
some knowledge
but the more constructivist perspective
and one that we kind of usually try to
uh to see is also that it means to build
meaning
and try to make connections between
behavior and knowledge
and all these different things
so to relate that to what i just said
about co-learning for me co-learning
consists of different
parts so yes it is also co-adaptation
which means these implicit and
unconscious changes in behavior and
interactions
um but there is also some kind of there
should be some kind of feedback loop so
this is a bit more related to acquiring
knowledge but it is specifically about
connecting the two so
can we become aware of
implicitly developed behaviors
can we reflect on them and then see if
we want to use them in the future
so this is the kind of the definition of
coloring that co-learning that i go by
um
but yeah how to actually research that
how to facilitate such a process what
are the benefits what does it mean if
you actually have a human and a robot or
human in an ai system
so this question is a very big part of
my pte research
just how do we actually research this
so i'm going to give two examples of
experiments that i did that were an
attempt
so one
is one that i already did during my
master's in industrial design
with a focus on leader follower dynamics
where i created this task where people
had to navigate from one end of a field
to the other side with a robot on a
leash
and the idea was that they had to do
this as quickly as possible but in the
field there were hidden objects
and the robot knew where those objects
were and you could get points
by finding the object but of course
finding the objects costs time
so this was a wizard of oz experiments
it was really exploration of behavior
how do humans deal with this conflict of
constantly trying to decide okay do i
follow the robot or do i go my own way
and kind of moving back and forth
between that
um
and it's very interesting to see what
kind of behavior comes out of it so some
people really only follow the robot some
people really like lead and pull the
robots towards the end of the field but
there's really all kinds of behaviors in
between um where they closely watch the
robot and see where does it go and then
pull it that direction that the robot
wants to go
or all kinds of other different
behaviors
um so another
experiment that i did is in a virtual
task
that i did last year as part of my phd
where the focus was more on
observational learning and adaptation so
i created this task
it is an urban search and rescue kind of
task so you had to save a victim that is
buried underneath a pile of rocks
and again there are some
differences in capabilities so the
robots virtual robots could pick up
large rocks and break them into pieces
and the human could only pick up small
rocks but the robot was very unaware of
gravity and those kind of things so it
might just pick up a lower rock and then
everything would collapse
and in this case the robot the agent
actually used the reinforcement learning
algorithm where it tried to decide
between different rule-based strategies
um
and the human was very unaware of all
those strategies so kind of on the go
the human had to figure out what are the
different strategies of the robot and
implicitly pick one to kind of focus on
and then hopefully the robot would also
learn to use that one which is also what
sometimes happened
so both these experiments focus very
much on the implicit part so the more
co-adaptation so to explore okay what
happens if you put a human and an agent
in this environment where there's this
tension and they need to learn to
understand how to work with the other
so some things that i learned from that
is of course if you create a task like
this you need a task an environment with
interdependence uh it needs to be useful
to collaborate why else would you do it
um so you need to provide the
opportunity to have this implicit
adaptation in learning
and what was very important for me in
terms of research purposes is that it's
possible to have different strategies
so if you have one clear optimal
solution then it's very easy to convert
to that but the real world's often not
like that the real world is often
dynamic so
tasks change you have preferences so
i deliberately design tasks that
allow people to choose their own way of
solving the task
so
for the second experiment so it was also
important to have a simple enough
problem such that in this case
reinforcement learning agent could learn
in a few rounds so quickly while working
with the human
and you could also say maybe it's
important to have like a real life
domain um because it should eventually
also scale of course the real world
um but what i think is the the
kind of key thing or what i thought was
most important
is
to facilitate the emergence of
interaction patterns
so what do i mean with interaction
pattern um i mean with that basically a
sequence of behaviors or interactive
behaviors that can be distinguished as
kind of a unit of behavior that repeats
itself so if you come in a certain
situation then there's a certain type of
behavior that you show like a strategy
um
but then one that is not really decided
beforehand but one that emerges um so we
often use examples from sports so for
example football or any other team
sports um when you know each other
really well and you've
done the game for a long time uh you
know kind of strategies of how to deal
with specific situations that you
encounter um and at some point you can
also reflect on those and be like oh
let's apply that when the
opponent does something
um
and
so okay so maybe not go to the figure
immediately so um for me what i mean
with when co-learning is co-adaptation
and this feedback loop and reflects upon
that it is especially this so
something emerges some kind of behavior
can we become aware of the fact that
that emerged
um and can we then also name that say
this is we call this behavior x and
we're now in a situation where we should
actually apply behavior x and then you
can keep that behavior and sustain it
over time
so then we can learn and it's also tying
that
behavior that we felt and experienced to
the knowledge that it works well and
thereby building some kind of meaning
so in order to facilitate that what i am
currently working on as a next step
is to build an ontology
that is able to save that kind of to
store that kind of information
um so inter patterns of behaviors
um with requirements for context in
which they hold or in which they are
expected to be
um
and a way for a human and an agent to
communicate about that so um currently
i'm focusing on the left scenario so
where we give human initiative in that
because it's a lot harder to also let
the agent or robots recognize certain
patterns
so the idea is that okay the human and
the robots they collaborate they
interact they adapt and as a consequence
of that
inevitably there will be some patterns
of emergent interaction
and then in this case a human should
kind of signal that or recognize it as
such
um and then save it in the ontology by
communicating to the robot
uh meaning that the robot can also
in its uh
when it performs the task look in the
anthology and see oh in this situation i
my human expects to have this behavior
uh or and expect me to have a certain
kind of behavior
and then of course ideally what you
would want also is that the robot can
also signal this has initiative in this
so the robot can also say oh maybe you
haven't noticed but i see a pattern in
how we behave maybe this is a useful
pattern um so yeah so that is something
that i'm uh working on uh or trying to
achieve also with different students
um of course it's not trivial
um
and then a question that arises for me i
think and hopefully also for you
probably is okay but what does that then
mean for
meaningful human control or control in
general
because um
well
co-learning also means to fail
uh learning means to feel always right
and i think it was very clear in some of
my experi experiments so in this one
some rounds so participants did this for
i believe five rounds um
so sometimes they were really slow
sometimes they walked all over the field
to find all the objects and their
overall performance was just really bad
um or in other tasks this can have even
more severe consequences meaning that
maybe the robot drops all the rocks on
top of the victim in this case which is
not really something you want or
we take a very long time so maybe the
victim is not very happy being
underneath the pile of rocks all the
time so maybe eventually after after a
certain time it becomes better
but yeah in order to get there you also
need to fill so you need to let go of
some kind of control to
allow for that failing so what does that
mean for a real world example um it's of
course hard to imagine
but i think
that the process can have the potential
to facilitate the emergence of meaning
so if we talk about
well we have certain behaviors that
emerge once you can
name those behaviors and be aware of
them
reflect on them
maybe that is something that we can call
meaning
so maybe if we have a certain modus of
collaborating that
is there in the end
maybe that is more meaningful than
anything that we can design because um
well we learned to understand the other
in a certain way and i think that is how
we humans often work
we learn to understand by
by making these connections over time
and learning about the other slightly
changing our behavior
so
um
kind of also to to finish that up um i
think there are many questions that you
can ask but um i wrote down a few that i
think are important
so first of all how do we cope with
human failure so we're
we're very well familiar with the fact
that humans fail and we accept that also
to a certain extent
and we have methods with which we go how
we cope with that but what if we are in
a safety critical environment how do we
cope with failure
well some people in the military might
probably say we don't fail
but that is of course because they put
all kinds of protocols in place but also
because they train a lot beforehand
so can we use similar strategies to
deal with or cope with machine failure
that also means we need to change
how we look at that and we need to start
accepting that machine sometimes fill
but we need to make sure we have
strategies in place that help us control
for the machine failure well the machine
maybe can control for our failure
um
of course also a very philosophical
question always when do we start
qualifying behavior as meaningful
um
of course touches on in general what is
the definition of meaningful human
control but i think um
in general what what is meaning and is
it something that we can create or find
through these exploratory processes of
co-learning and whatnot
um and then one that i always
well
this is one that i don't necessarily ask
to wonder but ask to trigger people also
what does it mean to optimize
performance in complex and dynamic tasks
because of course a lot of work we do in
ai and robotics and whatnot
is focused very much on optimization
um but if you connect that to a term
like meaningful into a process like
co-learning where you have a dynamic
environment where people have different
preferences and there's not clearly one
solution to a task
what does it even mean to optimize
so um those were the the three questions
with which i wanted to conclude the
presentation and hopefully open it this
could open the discussion
so i don't know if any of you has
questions
but uh otherwise let's just start the
discussion
great thanks emma for this presentation
super cool yeah so
we're opening up the floor for uh for uh
for everyone else so please unmute and
uh
and have at it
here's what comes out
go for it lucha
yeah
thank you very much for the presentation
i really liked it and then also really
like the slides which is another
kind of common thing for me to
appreciate also this like
well uh but uh so i also
like a lot the questions and the kind of
uh
the
the fact that you try to make a
connection with meaningful human control
what does it mean uh what happens when
uh
[Music]
we move from like uh the lab environment
the simulation environment to the real
case scenarios and i i think it's really
what we should be asking ourselves and
uh
related to that do you think that this
work can have like an anticipatory power
like
do you see it like possibly as as you
are saying it it it helps
understanding
emerging interactions emerging patterns
uh and behaviors so
uh do do you see it in a positive way
that it might create the space for
anticipating uh
consequences uh
and effects that we might have in real
case
situation
um
yeah yeah i think so how i look at it is
um as i said it partly is about the
perspective that we have on um what what
is what does a machine or an ai system
do and do we require it to not fail um
so i think um
yeah looking at these kind of processes
is really for a very big part reflecting
on this um like how how do we look at
this relation um
allowing a a machine to fill um also
accepting that a process like
co-adaptation and co-evolution is maybe
inevitable yeah because humans are
adaptive anyways and if we then start
creating adaptive machines whatever that
may be um then co-adaptation is
inevitable i guess
um
do research work that also appreciates
that because i think a lot of the time
we assume either the human or the
machine to be static
which is almost never the case yeah um
but like do you think that the kind of
failure that you imply
and you might experience in uh
simulations and
lab environments might be representative
of what can happen
in uh
real applications or do you think there
will be still
a lot of
well probably it depends on the case and
the and the kind of simulation you build
yeah and i think there's a difference
between the actual failure and the
process of like accepting that there
will be some kind of failure yeah um so
if you look at how humans collaborate in
safety critical environments like the
military or whatnot
they also can't anticipate beforehand
exactly what the failure will be but
they can anticipate that there will be a
risk of failure yeah and they can put
strategies in place um to make sure they
know how to deal with that failure
um and i think
like a process like co-learning or
something um is a way to to start
understanding how to cope with that so
if you train together and you learn then
things will go wrong and then you can
develop behaviors interaction patterns
um that enable you to to deal with
failure situations so um in a very kind
of uh not real life example so in the in
the experiment
that i did like with the virtual
environment with the rocks so a type of
failure that occurred there is that
the robot
broke rocks in such a place that they
then dropped on the victim so some
participants started creating
behaviors where if they suspected that
the robot was going to do that they
started placing like a little bridge of
rocks over the victim
um to um well
make sure there wouldn't be yet to
prevent damage um
so then you can say okay well they have
a very concrete strategy to react to the
concrete failure but part of it is also
looking at the at the robot and
understanding its behavior to an extent
where they could know when it was going
to fill for example yeah
okay thanks
great thanks
um
does anyone else have a thought or a
question for emma
if not actually i have a question in the
meantime so
i think it's it's interesting uh uh
points also that you make i mean also in
terms of meaningful human control one of
the issues that that that's also present
is the
the rate of learning that we're
considering right so sometimes
either the human or the robot is faster
in terms of learning a certain pattern
or a certain you know uh preferred
action
so
frisk is also in your own in your own
research how do you
how do you see that happening so you
know in terms of for instance the hume
still has a little bit control over what
interaction pattern goes into your
ontology set for instance right um but
the robot can be way quicker and how do
you so
in your view how do you how would you
cope with this or how would you handle
this
is it a problem at all
i think it's a very good question and i
think it is a problem
so um
i think in my experiment uh i try to
kind of force uh the agents to learn at
the the rate that is similar to the
human so
um
for example people played uh the little
game eight times and i had to ensure and
and with two different scenarios so five
times one scenario and then three times
another so i kind of had to ensure that
the robot would learn something
some kind of strategy or behavior within
five rounds such that the the human was
also able to kind of see that change and
anticipate
um
but yeah in the real in real life the
question is of course do you want to
enforce that
because learning has is a very strong
behavioral cue also so if people notice
that the robot changes its behavior then
that will lead them to maybe also change
their behavior and
ultimately if you want to have a
balanced collaboration where indeed both
agents have some kind of initiative at
least in the type of behavior that
eventually emerges yeah if one is is
much quicker than the other
that is very it's going to be very
difficult so yeah i don't have a
solution for that i think in my current
work i just enforce it
as much as i can
um
yeah no
i guess then they're the that's sorry
david that's where the mutual
understanding i guess that you also
alluded to is is going to be really
important to make sure that at least we
can still understand what has been
learned at some point to some extent i
guess
but uh you know yeah yeah that is one of
the big challenges in this kind of work
is you need to make sure that the human
sees that the robot learns because
otherwise they'll just be like this
robot just does whatever i'll do my own
thing and not care
so you need to create this uh empathy
sort of
um
i just wanted to make a
a comment uh
on what what you just said uh about the
learning rate oh maybe i'm bypassing
afghani i see a raised hand
so just trace it
just raise it okay good
um so um
so actually i think what you did is a
prime example of what in the military
domain
would be called putting the system under
meaningful human control so
i'm not quite sure but so
let me check
with you now
if it if the analogy makes sense
so in this case it's learning right
uh that that the speed of learning is
kind of made
made in the same order of magnitude
and in this case uh
that learning also i think has something
to do with the initiative with how
quickly the robot
might take initiative but there i'm not
quite sure because i don't understand it
well enough yet perhaps
but so in the military remain they're
very scared of systems taking initiative
by themselves
and perhaps learning much more quickly
what the target is that they need to be
searching out and how to identify friend
or foe or whatever and do that much
quicker than humans and there and that
therefore the system should then also
make the call to you know press the
button
um and then there's this disparity in in
i guess response time
um
that says well yeah you know there's
maybe uh advantages of having
military advantages of having the
system uh being uh not under meaningful
human control because then you're much
much faster
um
yeah and then
and so that's the fright right and then
maybe then if the system makes an error
my god how could you prevent that or
right so yeah so i think what you're
describing is maybe correlated somehow
that you're saying well right now i
imposed the learning speed to be in the
same order of magnitude
simply because i want to make sure that
the interdependence
stays
and so i just wanted so this is an
observation i thought that came up
listening to you
and
maybe i'd like to hear your reflection
on that
yeah i think it's a very fair a good
thought in the sense that of course you
can also adapt the agent or the robot
beforehand
and by indeed enforcing a certain
learning speed to adapt it beforehand
versus real time
but i think it also
depends a bit on
control versus meaningful control so
of course
maybe if the system can take initiative
um then that that means the human's not
in absolute control but it may still be
meaningful control
if we decide okay in certain
circumstances we allow the machine to
take initiative because we know it is
it can perform better and within the
boundaries of what we find acceptable
um
so yeah um
this is what i always wonder about so if
we say meaningful control that means we
don't mean absolute control
right
so yes yeah so if that's a if that's uh
a question so then we're at the heart of
of our research topic emma and
so i think there's still debate around
that um
but
yeah but so maybe indeed my perspective
is that that
control indeed is then that not absolute
top-down control that you can control
every nitty-gritty detail necessarily
then a lot of the systems are out of our
meaningful control anyways
it also doesn't mean it needs to be
top-down control it could also be kind
of bottom-up influencing perhaps
in terms of meaningful it it also
means that sometimes we implement
systems that in principle
allow people to overrule them but in
practice
you know are so boring and so and are
good enough that that people don't even
you know
actually de facto stay in control and
then therefore it's a kind of yeah but
it's possible but it doesn't happen and
so that's also not meaningful uh control
yeah and so i spent now quite a lot of
words describing what mean viewing tool
is not
and i gladly give the i drop the mic and
give the floor over to people to define
what it is
yeah i mean
you can have different different well
talk a lot about it so for me a very
important component is kind of what i
started with
my vision on this is
if we collaborate that means we need to
appreciate the qualities that the system
has and we need to appreciate the
qualities that a human ha has and be
realistic about who does what better
and then of course as a human maybe
sometimes we say okay we
sacrifice performance over something
else
which is maybe also performance but
another kind of performance
but then that also means that sometimes
you need to let the machine do its thing
because it is simply better at it
[Music]
yeah but what what does better mean that
is then of course another question
and uh evgeny if you allow me one more
response or
to that
is that uh uh this discussion also
reminds me of uh work we've been doing
in the uh um in the automotive domain
where um where it's uh
actually who's better
uh it's very uh contextual so context
dependent
uh and that means uh if you know uh a
priori
uh
in what context which would be better uh
you know which system would be better
life would be easy
but uh but quite often contexts evolve
uh
unpredictably and perhaps even unknown
and
and then
there's this tendency that
because we look at specific parts of the
context we can say well there the
automation or the robot can do much
better
so we should do it there
and actually
what this discussion makes me think of
is that also there you might say
actually maybe it's not overall better
the way we design our collaboration
system um but but but maybe in a larger
scheme if we consider more context
maybe then overall
is better rather than in one specific
context so maybe kind of the widening of
the scope of your context
it's actually maybe
needed to
yeah to make the distinction between uh
you know what's what's good performance
heather was your third point performance
in complex dynamic systems uh um
uh yeah so that was was another thought
that i said that maybe different
contexts
need to be
put for your collaboration system in
order to find out its weaknesses and
strengths yeah
yeah i agree and um for me also that
maybe it's too sometimes too subtle to
determine a priori and that is why we
need to um
look at co-learning right
actually because it's an ongoing process
to figure out the subtleties
yeah
good okay so
afghani
thanks for the nice discussion
i am i was so i was curious about that
last uh the third question that you uh
uh post on on the last uh slide so so um
well so first of all i was curious if
you can share so from your research like
what kind of experiences uh
um
what kind of experiences you've had
where you have seen this question uh
really uh
like play out with with
where where then
one would struggle to say
well what is optimal because maybe like
it's hard to define something that is
optimal or the nature of the
problem and people's perspectives makes
it hard to even talk about some one
optimal thing so i was just curious if
you can share some experiences that you
had with them um yeah so one that we
basically almost always find if we try
to do some kind of co-learning
experiment
um is um
short-term versus long-term performance
and then subjective experience and like
human understanding of what is happening
versus performance
so very often performance is um
something you try to measure to see okay
we now made some kind of co-learning
process but it doesn't increase
performance
um and
basically in all the experiments we've
done it's very difficult to within one
experiment
uh c increase in performance
uh sometimes it might even be a decrease
in like and then with the performance
here i mean objective task performance
so how long does it take to finish the
task how much did you hurt the victim
but if you then talk to the people who
use the co-learning versus people who
did not
they have a much better understanding at
least in words of what the system did
they can explain it much better
um
they sometimes
well they maybe they feel like their
collaboration was more fluent but maybe
you don't immediately see that in
objective performance
um
so that is then exactly the challenge so
the claim we usually make is
well
it makes sense that maybe you don't
immediately see a performance increase
because the tests are sometimes complex
and humans try different strategies that
sometimes don't they're not necessarily
efficient
but
uh well we kind of assume or hope that
in the long run because collaboration
fluency increases because understanding
increases in the long run we will have
some kind of performance increase um but
then that is very hard to measure of
course and to
kind of prove then you need long term
experiments so i think that is uh where
we find the challenge so um
for that reason i measure collaboration
fluency usually
so it doesn't necessarily mean that i
let my agent optimize for that sometimes
uh i mean in my experiment with the
rocks for example the agent did try to
optimize for
time and
victim harming um
that doesn't mean because you're in the
team context doesn't mean that it
actually works so that that actually
improves
but that was just to give the the agent
something to optimize for
so yeah i think collaboration fluency is
an important thing but i think a lot of
the things you want to optimize for are
very subtle and qualitative
making it hard to optimize
i was just gonna thanks i was just gonna
say because like it strike i mean it
touches something that i'm looking into
in my own work but
exactly what you just said that
the qualitative
aspect
which can be so important in many
contexts whereas
if we i mean by nature of mathematical
optimization and if you want to put into
algorithm then things need to be reduced
somehow into numbers
and then they're in certain contexts
there there there's a tension uh yeah
there's a tension there but but
well
i mean just as a follow-up to that like
what um
from
i mean part part i think part of what
many of us are figuring out as people
coming from
originally different backgrounds
kind of common ways of working together
and different disciplines have different
uh
um well philosophies of approaching
problem but
like connecting to the previous question
where do you see
like what what are some of your
what are your views on how qualitative
research
plays a role in in you know informing
these decision design decisions that we
make engaging with people in a
qualitatively rich way in a way that you
cannot just present in a diagram
um necessarily you know with some
numbers but you really
you really need to get into like
interviews and then and yeah
the nuance of what people say and there
were there might be more than one
interpretation as to
yeah
yeah i think it's essential but uh it's
also uh
how i was raised academically because uh
well when i studied industrial design um
we basically the research that i learned
was more qualitative
stuff
um
so for me
um
doing my phd i'm constantly also trying
to search for the right way to combine
qualitative research with
more traditional
quantitative analyses or doing
experiments where you compare have a
control an experimental group and trying
to combine all of it
thereby also trying to convey the value
of the qualitative stuff to people who
are not used to working that way
um
yeah and trying to be very systematic in
that to make sure people believe what i
say
um yeah i think it's essential um
and i think um people should
maybe always use mixed methods
um to really understand complexities of
things like this
thanks i can definitely
definitely agree with you that yeah
thanks
yeah thanks for skinny um
actually am i if if i if oh caramel
you're first before i
you go for it
so thanks emma that was super
interesting and fascinating although i
have to admit i didn't
understand everything but
so i but i had a question about the
2020 experiment that you are
planning to do or probably already have
thought about a little bit
so
um so there seems to be an asymmetry
between the the human and the robot and
and of course it's very interesting that
if you have a way to for the robot tool
or the ai system to find out that there
is an interaction pattern so that seems
very challenging but
i think manageable in some sense at some
point not for me but perhaps for you
uh
but so but what i
find difficult to see is how you could
if you have found such an interaction
better how you could communicate to a
human
what what kind of interaction pattern it
is you have found and what so how can
you communicate to the in a meaningful
way to the human that there is this
pattern and yeah so in a way that is
useful as well
so did you
did you have any thoughts that's just
already or
yeah i have some thoughts another
solution um
i so it's very ongoing so um
i have a master's student who's starting
just now in general to think about in
general these communication questions so
how what kind of dialogue or
communication modalities etc
should we
use
for this type of communication
so and he's just starting
but i think
what i and also what my supervisors
really strongly believe is that
you need
you need a basic common vocabulary
and
that is what i use my ontology for so i
create an ontology so a knowledge
structure that has certain concepts in
there like okay we have actions and we
have objects we have actors
i create kind of like a template in
which you can describe the interaction
patterns meaning you can describe
certain context factors and then
basically a sequence of
actions
of course in reality it's a bit more
complex than that and maybe you also
want conditionals in there and
expectations and whatever but this is
like a first uh first way to try to
achieve it
um and then just assume that as the
common vocabulary so we create that
ontology we
make sure the robots can use it and or
the agent can use it and then we make
sure it uses a vocabulary that is
understandable to a human
um and of course in in kind of an ideal
world you would have to kind of
ground all these concepts so there's of
course a lot of research on like
grounded situated communication where
you say okay this is a mug now you know
what a mug is and i mean
what would be very cool is if you were
to ground all the concepts in an
ontology in that way
but currently i assume that has been
done so i just have an anthology that is
a given
so you need the common vocabulary um
my tea is coming in yeah so um
uh currently i can show you that uh
maybe something so what i've done for my
own next experiment is to just create a
little graphical user interface that
allows the human to describe behavior
and it looks like this
but it's just something i designed
so you have so basically just look at
the right side of the screen so you have
this little form that is just like okay
a situation contains and what did we do
um so you can say for example situation
contains large rocks at the top of the
rock pile and in that situation um
basically
we let the human
um maybe
uh stand still in the side of the field
so we let the human wait and then the
robots can use that as a signal to then
um
pick up
that large rock that is in the top of
the file
um so this is really based on that
experiment
um
yeah so this is just a first suggestion
of
a communication thing
because there's a lot of questions of
course like do you want
an interface do you want natural
language do you even want language can
you do this in a more haptic manner or
uh whatever and it's all very dependent
on context also
yeah
yeah
thanks fascinating yeah
and and and that really helps me
understand as well how you think of this
really great i would love to
read
the thing that you already work on and
especially also your
the your next paper so yeah sure yeah so
i have a i have papers about the
experiments that i used um uh in my
presentation yeah for this i don't have
yet but i will be running a little
experiment where people have to watch
videos of the previous experiment and
use this graphical user interface to
describe patterns that they see of
behavior
and once that is done i'll probably
write something about it
yes
um one question ago you said that
oh we should definitely encourage mixed
patterns of uh solving this
co-adaptation problem
and then uh
harmon talked about uh how we can
basically describe asked about how we
can basically describe or
transcribe the behavior we are saying or
how the machine
is trying to recognize
um
this
collaboration
do you think it is also valuable to do
this in systems where
these systems who which
implicitly do this
so optimization systems such as let's
say
a very common example youtube
recommendation
which optimizes for something
really simple uh money
so it uh only shows people
uh more ads if you watch more ads if you
are always clicking the skip button it
will show you less ads um it will show
you addictive content when
it knows that you can watch addictive
content which is at night at 3am when
you actually feel like it or
these
really
let's just say toxic actions
you think in these situations
where
an algorithm
automatically just
makes uh user profiles and uh knows how
to act to just like generate more
revenue no matter
where on earth and what or what kind of
content you're consuming
it is good to describe this sort of um
implicit co-adaptation
um i think it might be um
so there's there's different things um i
think on the one hand the the way that i
looked at co-learning i mostly assume
that the agent is kind of like a
robot-like entity even if it's virtual
and of course this might be different if
it is more like a a dashboard ai kind of
thing or so also within tno we're trying
to apply co-learning in such situations
so we're also thinking about how does
this translate uh is it actually
different um
but i think
in general it is also useful to reflect
on harmful or bad or whatever
not effective patterns of behavior
and i think for example like a youtube
recommender uh does a youtube
recommender assume that the human will
also change its behavior
uh maybe maybe not but it hap it will
happen i mean we all know it if you at
some point you figure out what the
algorithm does um so then you know you
can influence it right or at least we
try to it doesn't sometimes it works
sometimes it doesn't work
i think it would actually be very
interesting if um
the website would then give back to you
okay i noticed that you've done this and
that causes me to do this
and then as a given um because sometimes
you might be unaware of it
sometimes you might be it's it's really
about awareness
um
if the system would do that then you can
also say okay sometimes i just really
like to watch these type of videos but
it doesn't mean that
you need to start recommending it to me
so actually this sequence of
interactions
thanks
t flying so this sequence of interaction
is not something i want and then you can
uh so you can reflect on it then
so i think it might be a valuable in a
sense it gives a certain transparency um
it makes people aware of their own
behavior as well how they react if the
if the algorithm recognizes that the
human changes its behavior to influence
the algorithm they can also say hey
you're currently influencing me do you
realize and then you can be like
yes and i want this or oh maybe this is
not what i want
okay
not sure if
youtube would do this but
i think it would be interesting
yeah it's difficult to implement for
millions of people
yeah so
there is also of course
a lot of the algorithms optimized for a
population kind of and they put you in a
box
of you're this type of person so
therefore i behave like this um and i
think the perspective i try to take is
more okay everyone is different and as
so i focus of course on teaming kind of
tasks where um your behavior is
influenced by the past interactions that
you had within that team so then it
makes more sense to personalize for the
person that you are as a unique person
given by the previous uh
not necessarily a box
oh
sorry i didn't
no no that's great i actually think it's
it's an important point too am i like so
there's also part of the meaningful
control right this control over the
mental model that that something the
robot ai or whatever has built over you
so that's it that's a nice uh
and i guess that's also something maybe
you can test in your experiments with
this anthology where people can test you
know i want this but then all of a
sudden you realize i don't like the
interaction pattern at all so you can
take it out again something like that
yeah yeah i hope to see that that people
then uh save an interaction pattern and
that maybe then the robot interprets
that differently as how they assumed it
would
and that means they need to like revise
it and that is how you get this
continuous
improvement and co-learning process
okay
great
um yeah so we have a couple of minutes
left um
we do so we'll stick around for a couple
of minutes more um but i
want to thank also i guess
also on behalf of the audience emma
thank you so much for this presentation
and the discussion afterwards it's super
interesting we uh we really sunk our
teeth in it so thanks everyone um and
also especially to you emma um and yeah
indeed if you
would like to know more about emma and
her work there also her link to her
personal webpage and two delta pages in
the invitation of the
of this meeting so there you can also
find her papers for instance so uh
thanks everyone
yeah thanks for the very uh very nice
discussions
thanks
i don't know if so
different i had a adam
go for it
yeah
um
so now
some people are leaving already but but
uh
if you want you can stay around
and because
so
emma we we've been working you know that
with nick
on
on actually translating this to a
physical task and we're working
together and when i say we it's actually
you all
and me vaguely hovering in the distance
above it uh on actually finding a
physical equivalent to the kind of uh
uh
co-learning and uh that that you're
looking at
um but i thought maybe it's nice to
share some of the work we did in co
adaptation
and so i have a one slider
that
that might be interesting for you to
reflect on
and so um let me share that while we
scare everybody away
and
yeah we have a meeting with with the
students uh for the project tomorrow
right
that's yeah the different master student
exactly yeah lucas lucas indeed
yeah very very happy to hear that
um
and great that you're you're
going on
and so uh this is the not so beautiful
slide
that i scrounged together from work of
lorenzo flipsa
supervised by nick
and also by sergio gramatico from
dcsc and what he's doing is he's looking
at a task which is uh
using a steering wheel to control one
degree of freedom left and right
a kind of uh
compensatory
or actually kind of pursue task
it became compensatory tasks right uh so
you know anyways you need to to follow a
box and minimize the error
so it's a very simple task
but then we also looked at how well do
people do this over time
um when it's just manual control and
then you see here kind of the gain of
their control inputs
and we develop two kinds of robot
assisted
interaction strategies
and so these are actually you could say
not emergent behaviors but imposed
behaviors
that that
that actually dictate how the robot
learns to do this
really joint action task
and so one would be to say well if
humans do less if we estimate that the
human gain is lower then the robot needs
to realize aha i should do more
because then i can guarantee
joint system performance
and that assumes that humans won't adapt
yeah but uh i won't
adapt to that over time but what do we
find
um
if we actually implement that let's see
if this works
and if we do that the robot does more
when the human does last and the first
trial you're still working together but
quite soon the human has very nicely
learned not to do anything anymore
and so we know this concept from human
factors domain and from slacking and so
yeah you know if a robot will do it why
would you even make a an
effort um
and so
just i thought it would be nice to show
it so we also explored what happens if
when the robot estimates that the human
is doing
uh
less maybe then also the robot should be
doing less or vice versa when the robot
is doing when the human is doing more
when it's the estimation that the robot
should be doing more so it's kind of a
reward for if you're engaged okay i'll
help you to do it even better
and and then actually what you find is a
kind of stable
equilibration it's a new word it's very
very uh
beautiful
but so that's actually the idea of
kind of trying to
empower humans and the interesting thing
is this all dynamic so this happens in
real time and all possibilities for
weird stuff could happen but but it
doesn't seem to uh to happen that's what
it seems to be something stable
so i was just wanting to share that with
you uh emma yeah very nice yeah yeah i
think
you it's always very important to make
sure that the human feels the need that
there is really a need to collaborate
why otherwise would you do it and
the more dimensions you add the the
higher the chances that this will happen
um
yeah i think for example in my
experiment with the robot on the lee so
that was a wizard of oz of course but um
yeah some people did like push it
towards one side or to the other side
but most people try to find some kind of
balanced uh collaboration because there
was an advantage in in well
the robot actually had a capability that
the human didn't have and vice versa
um
and these some of these situations were
indeed relatively stable that people
kept
having the same balance strategy um so i
think in the paper that i wrote uh the
second paper that i wrote about that i
also
uh categorized that as okay you have
stable situations and then you have
triggers that lead to kind of like
sudden adaptations
of moving into a different stable
situation so it's not always a gradual
it's more of a now you're in this
situation something happens and you go
to a new
new situation yeah
complexity theory yeah
right boundaries and triggers and uh
yeah yeah yeah
um by the way maybe something that you
will find interesting as well uh last
monday i had a meeting with isa could
you kill us
probably maybe know this person
yeah yeah
vaguely from name but i know well she
does a lot of human human interaction oh
of course yeah from bristol yeah yeah
yeah
yes yeah she so she reviewed that paper
of my robot on a leash
but it was a frontiers paper so um
she did very extensive reviews of like
thousands of words
um
so
what she
has worked on and is working on is also
like very haptic and like one dimension
kind of adaptation so i think very
similar to these kind of things yeah
yeah the role of roles is one of our
key papers uh yeah yeah yeah so we're
also looking into how we might uh apply
these ideas of like having stable
situations and then these
triggers uh and learning different types
of stable situations uh how we maybe can
either use all data from her to analyze
it in that way or maybe also set up a
new experiment uh
how nice
she's doing well
she's in bristol or something right uh
not in him i think nothing was it yeah
yeah yeah
yeah it was very funny uh
yeah because that's a nice thing about
these open access so that you can see
afterwards who the reviewer was and then
actually reach out so
yeah
yeah she's she's great
but um so so unfortunately we need to
move to the next meeting already so but
thanks emma i just wanted to say one
last thing about lorenzo's data that you
saw
so basically also what happened in these
with these with this controller is that
basically the mental model of the robot
so the controller thought that the human
was terrible at the task and then at
that moment it would completely take
over yeah like you said
um but then also the human did get
didn't get the chance to take back
control anymore because you can never be
better than a human at that moment that
was it that was an interesting emergence
like it's it's it's it's a simple
engineering you know um
implementation of this controller that
led to this you know weird behavior
where the um had no control at all
anymore to take over it so that was just
just a little observation that we had
also in this yeah
and the insightful discussion this is
cool
yeah thanks for the conversations yeah
and uh yeah we will see each other
tomorrow i guess definitely yep thanks
very much bye-bye good day bye bye


The AI News You Might Have Missed This Week

the goal of this video is simply to show
you seven AI advances that you might
have missed this week Sam Altman
recently said that in a world of AGI
everything happens much faster but as
far as I can see AI developments are
already almost impossible for a human to
keep up with so in no particular order
let's get started first video calls look
like they're about to get 3D let's take
a look at how Nvidia aerial and Nvidia
Maxine 3D running on the Nvidia Grace
Hopper Superchip can enable 3D video
conferencing on any device without
specialized software or Hardware this
brings a new dimension to video
conferencing with Maxine 3D
visualization engage with others more
directly with enhanced eye contact and
personalize your experience with
animated avatars stylizing them with
simple text prompts and it isn't just
Nvidia here's Google's new project
Starline prototype you know you were so
used to seeing a two-dimensional little
you know box and then we're connecting
like this and that feeling of being in
front of a person is now replicated in
start lines speaking of connecting the
world here is gpt4 doing geography in a
paper you might have missed from this
week the paper proves that even without
access to the internet gpt4 knows a lot
more granular detail about the world
than you might first imagine I'm not
saying it knows where you live but it's
not too far off take this example it
could recreate the Hong Kong mass
transit railway from memorization this
wasn't through using web browsing it
could recreate this diagram giving the
latitude and longitude coordinates of
each of the stations in this Transit
line obviously it's not perfect but it's
pretty incredible that it's got this
mental map of the world gpt4 can do
elevations as well and here is it trying
to recreate the Topography of the Alps
it gets pretty close one of the ways
they tested gpt4 was to ask it something
like this please provide find the
latitude longitude coordinates for the
outline of X where X was a consonant or
a river or a country as a python list of
tuples consisting of approximately 50
points arranged clockwise and they
describe how it did really well for
quite a few countries and rivers but
kind of flopped on Africa but honestly
when I read this paper I was skeptical
that gpd4 knew that little about Africa
so I gave this exact question to gpt4
with code interpreter now interestingly
it would sometimes deny that it had the
ability to do this but with enough
encouragement it outputted these
coordinates and here is the end result
in Google Earth I think that's a pretty
impressive outline obviously a few
points are a bit off like this point
here isn't really on the coast nor is
this point but it really knows the
outlines of countries continents Rivers
so I'm not sure if code interpreter had
an impact there or a model update but
the researchers kind of underplayed what
gpd4 could do by presenting this out
line of Africa now I am sure that some
of you are thinking that's not that
interesting not that impressive but
check this out in an indirect kind of
way gpt4 knows where it was made it was
able to construct a map of the
semiconductor supply chain it not only
knows about the design manufacturing
materials equipment and tools that go
into the hardware that helps make gpd4
it also knows the locations of where
this is all done and as the authors
later say looking to the future if
Frontier models Beyond Gypsy 4 continue
to advance in capabilities the
geographic knowledge and planning
abilities present in the current model
May later evolve to represent a
significant risk through misuse or
misalignment on a much less important
note did you notice how I could do this
demo without that sidebar of all my
previous chats that's because openai
have brought in this new button here
where you can hide the chats and as a
bonus some of you may not know that you
can now share a link of the chats that
you've already done just by clicking
that button to the left and as it says
messages you send after creating your
link won't be shared so if you carry on
the conversation people won't be able to
see but anyone with the URL will be able
to view the shared chat but before we
move on from open Ai and chat topt I did
find this table really quite interesting
it gives the daily average number of
visits to each of these sites along with
the visit duration and there's two
things that strike me from this table
the first is how much more popular chat
GPT is compared to Google's Bard it's
got about 15 times the number of
visitors who stay for about twice as
long but look at the Dark Horse on the
right character AI I've talked about
them a couple of times before and while
their daily average visit total isn't
too crazy look at the visit duration in
terms of grabbing people's attention and
keeping it they are truly a dark horse
next I want to briefly dip into
augmented reality we are going to be
creating our own worlds and living in
them some people like in this video
might choose to live their lives as if
they're in an animation others might see
augmented reality as a way of augmenting
their intelligence or Memory live
we don't
go my prediction would be that wearables
that resemble things like Google Glass
might flop but something like an
always-on app on your phone mediated
through GPT models could become really
popular or even enforced in certain
workplace settings all of this reminded
me of a recent video about conducting a
video interview with help from GPT 3.5
what about your development areas what
do you have identified as your greatest
and biggest Improvement areas and what
have you done to improve them so far
outside my greatest Development Area is
my communication skills I work on
improving my ability to clearly convey
my thoughts and ideas to others of
course at the moment this is only really
viable with GPT 3.5 because of inference
speed but open AI are aggressively
planning a cheaper and faster gbt4 I
wouldn't be surprised if video
interviewers soon require you to take
out any headphones although I guess with
Maxine 3D you could maintain eye contact
with the camera while you're actually
reading off a gpt4 teleprompter anyway
what about gaming this is nvidia's
neuralangelo where you can take a 2d
video and turn it into a detailed 3D
landscape with High Fidelity my first
thought turned into Imagining the kind
of things you could then bring into
games using Unreal Engine 5. this is a
recently trailered horror game Link in
the description but don't worry I'm only
going to show you two or three seconds
of it it's getting to the point where
it's quite hard to believe that this is
a game but it is and on games don't
forget this look at the realism that can
now be achieved in terms of skin texture
and movement for the final bit of AI
news that you might have missed I want
to focus on AI drug discovery
I think there's there's no question that
there is a before and after in drug
Discovery and one of them is AI Alanis
puruguzik is the director of the
University of Toronto's acceleration
Consortium which in April 2023 received
a 200 million dollar Grant to build an
ai-powered self-driving lab the
acceleration Consortium has already been
using AI to help discover molecules that
have potential drug-like traits that can
be used to develop life-saving
treatments developing a drug can be up
to a decade and this is just the
discovery piece so that process let's
say takes a year or two and we compress
it to 45 days in that case and then 30
days recently in January 2023 the
acceleration Consortium used an
ai-powered protein structure database
called Alpha fold to design and
synthesize a possible liver cancer drug
in just 30 days within two weeks we can
formulate the drug as well as some
people have done it in years suddenly AI
has surpassed any human created
algorithm AI what allows us to do is
lower the bar of what you need to do
certain things and therefore more more
people we have access to it in general
unleashing more innovation in the planet
same token someone with nefarious
intentions could unleash very dangerous
deadly chemicals on the world absolutely
I am an optimist but I'm also aware of
these pitfalls that very soon will face
us and videos like that are why I agree
with samuelman when he says a much
faster rate of change is his single
highest confidence prediction about what
a world with AGI in it will be like I
follow AI news full-time and can barely
keep up so I can only imagine what the
situation will be like when we get full
AGI but until the very last moment that
it's humanly possible to keep up with
the news I will try so thank you so much
for watching to the end and have a
wonderful day