Stefano Albrecht – Learning to Distinguish Between Belief and Truth – CSRBAI 2016

welcome back our second speaker today is
is Stefan Elbert and a postdoctoral
fellow at the University of Texas at
Austin and in the department of computer
science and engineering and he works
chef now works on the Arian practice of
autonomous agents and multi aging
systems especially talking about
Bayesian modeling and belief modeling so
he's here to talk to us today about
practical and theoretical progress on
model updating and multi-agent systems
so join me and welcoming shut up thank
you very much I should point out that
this work is supported by a research
fellowship from the omoide Foundation
just to have that cover as well so this
this toxin toilet learning to
distinguish between belief and truth I
realized that there has been a lot of
work in the philosophy and psychology
communities about what it means to be
true and have beliefs I just thought
this would make for a compelling title
but I'm not going into the differences
between the two things I'm hoping it
gets approximately clear as I go through
the talk okay so consists of three parts
you can see the pointer can you know
really so I'm just gonna use my hand
answer so consists of three parts on the
first part I will explain to you what I
mean by the title and the problem that
we consider here and the second part
that will tell you about some research
we did to adjust one specific aspect of
this bigger picture problem and in the
third and final part that will give you
a you know a brief description of a
possible future and a recent agenda okay
introduction so this talk is about
multi-agent systems such a system
consists of a collective of decision
making agents each agent can take
certain actions such as a B and C and
each agent is trying to accomplish
certain goals such as reaching a goal
state in the environment and maximizing
a given utility function and what makes
the system interesting and often very
challenging is the
that what one agent should do can depend
critically and what the other agents are
doing in the system agents often have to
deal with a variety of uncertainties
there may be uncertain about the state
of the environment and they may also be
uncertain about the actions that other
agents took in the past
and importantly there may be uncertain
about the behavior of other agents and
by behavior I mean the way in which an
agent chooses actions based on the
available information now all of these
are important issues and there's been a
lot of research but in this talk I will
be focusing on uncertainty about
behaviors okay so this basically two
ways in which you can adjust that
there's what could be called model free
methods and that includes regret
minimization policy gradients and model
fear in for smelling and various other
approaches these methods do not directly
adjust the uncertainty about behavior
what they do is they use the information
that they observe during the interaction
and try to manipulate their own behavior
directly so you know there's no man in
the middle the other approach could be
described as model based methods and
these methods basically try to learn
models about the behavior of other
agents based on the interaction history
and there are many ways in which you can
do this he could try and fit decision
trees neural networks finite state
machines and various other methods case
based reasoning or graphical methods and
then the idea is to use that model the
resulting model to inform your own
policy to plan your own actions I think
there is some important features that
the model-based approach provides and
basically why you would want to try in
model in agent s because the hope is
that some of the observations that you
encode in your model will generalize
usefully to other unseen situations and
what this gives you will allows you to
plan into the future as the key part it
allows you to perform guided exploration
for example an icon you can also be used
to do things such as risk control and a
version
and all these things are hard to do if
you don't have a model to begin with
this is useful but all of these methods
like one key component currently which
is that they provide no notion of model
criticism so in other words they produce
models but they don't check the validity
of the resulting model during the
interaction and so what can happen is
that an agent using these methods can
end up learning and using a model
without ever realizing that it's
incorrect that's an important a
disadvantage so here's a simple example
a very simple example let's say robot
plays rock-paper-scissors against a
human
I assume standard rules and standard
payoffs and let's say the human place of
a very simple sequence of
rock-paper-scissors rock-paper-scissors
and so forth no the robot doesn't know
how the human behaves but it can try to
learn about the behavior and in this
simple example the robot is fitting a
model based on the assumption that the
human is using a fixed distribution of
reactions or fictitious play as it's
known in the community so if you use the
empirical frequency to fit the
parameters in the model then in the
limit you end up with a uniform
distributions distribution which is
incorrect so of course because really
the human is using a deterministic
sequence and in the planning if he were
to plan with the correct model your
expected payoff would be one because it
can always choose the correct or the
winning action and each time set whereas
planning with the learnt model would be
the expected payoff would be zero which
is as bad as playing randomly and the
the problem here is that the robot will
never realize that the model is
incorrect now there in the learning step
nor in the planning step and this is a
very simple example but it also holds
for more complicated model fitting
methods and this is a very simple
example but it's not all that hard to
scale is up to more realistic and
complex examples just such as elderly
support users interfaces or should say
adaptive use interfaces and electronic
markets and we had a really interesting
talk already oh brother
essentially any domain where you have a
collective of agents that need to
interact closely with each other
and that have to develop some kind of
understanding of each other's behaviors
what can go wrong well in principle
anything
agents attempt to learn models that end
up being incorrect incorrect mollusc can
make incorrect predictions and incorrect
predictions can lead to bad actions so
if in 50 years from now I still happen
to be alive and we are at a
technological stage where a robot like
this can carry me right now I would hope
that if it attempts to learn about my
behavior it has some reasoning
capability to make sure that well and
learns makes sense rather than you know
dropping me on the ground and killing me
in the process
so it's about safe and robust there in
that sense so what's the problem here
well the fact is that the model is
effectively a hypothesis or loosely
speaking a belief of the agent and our
policies can be false it's a key part of
the definition well the problem is that
the models are not treated as hypotheses
and these methods and so the fact that
they can be falsified is not even
contemplated or considered in the first
place you might say that you can address
this problem or alleviate at least to
some extent by maintaining multiple
models and then you can do some model
selection over your models such as
keeping beliefs
well that doesn't correspond to model
criticism and and in this particular
example of maintaining beliefs well the
problem is that the beliefs quantify a
relative likelihood but then but they
never quantify an absolute truth and so
even if all your belief point to one
model that model can still almost ever
totally be incorrect ok so here I've
here's I think what we what we need to
do or what agents repairability they
should be able to construct hypotheses
of behaviors but they should also be
able to contemplate the truth of the
airport says both of these things should
go hand-in-hand and that's currently not
the case what can this do for us well in
the first place agents can decide to
reject the model and then depending on
how this reasoning is being carried out
the agent may decide to change for
example some of the assumptions
underlying the model such as what
information to fit into the learning
process or
press the structure of the model and the
parameters instruction and all this done
in the hopes of obtaining a better
modeler ultimately or alternatively if
none of these stuffs seem to help after
rejecting a model well at the very least
the agent should be able to resort to or
decide to be be able to decide to resort
to some safe policy that can work
without a accurate accurate model and in
the setting of game theory for example
this could correspond to a maximum
strategy that can guarantee a certain
minimum level of performance so you know
it's safe in that sense okay so that's
the introduction that was a brief
description of the problem and in the
direction in which we're going with us
and in the second part that we tell you
about some research we did on a very
specific aspect of this bigger picture
problem and it's what we call behavioral
hypothesis testing this was first
published in the uii conference last
year in the Netherlands and has been
published more recently as part of a
more substantial manuscript in the
artificial intelligence community so if
you decided to take a look and you have
some comments please do get in touch
with us ok let's make it more concrete
so the model is in hypothesis because
there's either true or false and because
it's testable importantly this is the
key definition of an hypothesis and then
the net-net ask you a question that you
can ask your surface well if I interact
with some agent Jay and I have a
behavior and hypothesis for the behavior
of that particular agent and also given
a interaction history thus that agent
truly behave according to my hypothesis
about the baby is something that you can
ask yourself and that an agent should
probably ask itself as well well here's
an example to make it more concrete this
table shows the first time steps of an
interaction process between two agents
playing the rock-paper-scissors game
such as the reward in the human that we
talked to earlier in the first column
you see the time steps 1 2 all the way
to 5 and the second column you get the
actions taken by the agents in the
respective time step so rock and paper
for example and the first time sir
and the third column corresponds to the
hypothesis that agent one maintains
about the behavior of agent two so for
example and the first time stuff agent
two chose to play paper and agent ones
hypothesis would be that agent two plays
paper with probability point one and
this goes on in the same way and and the
hypothesis can produce these
probabilities based on the prior
interaction history so you can think of
the hypothesis as some kind of a black
box program that can take us in for the
interaction history and produces me know
probabilities this isn't this isn't a
belief it's just a a distribution over
action outcomes for agent two so there
can be 0 because this sums up to 1 it
just means that the hypothesis does not
believe that agent 2 would choose rock
in this time step with any chance so
this doesn't have to be so as I just
said just think of this policy as some
kind of a black box function it takes us
in for the interaction history and it
does some internal logic and reasoning
and produces it ok so a question is how
should agent 1 decide whether or not to
reject the hypothesis about agent T and
importantly all of the actions that you
see in their in their column here of
agent you are supported by the opposite
because they're in the positive mass so
he can't just outright reject the
hypothesis now one way to adjust this
for a natural way would be to compute
some kind of score from the information
given in the table and one example is
the empirical frequency distribution and
while this is a relatively simple and
appealing there are also two big issues
one is that the question of when you're
scoring scheme is sufficient by
sufficient I mean when do you know that
it's sufficiently informative
indiscriminate enough for you
in order for you to make this decision
if you take the empirical frequency and
go back to this example well then you
can tell them really that it's not
sufficient because the distributions
changed drastically in all the time
steps and so you can't just take the
average of your predictions and then the
second issue is once you have a scoring
scheme then you will have to find some
threshold parameter beyond which you
decide to reject or not reject and this
seems to depend this will depend highly
on the scoring scheme that you choose
but it doesn't necessarily tell you how
you should choose the parameter and
furthermore once it once you change your
scoring a scheme you will probably have
to change the parameter as well because
the semantics of the parameter changes
change so what we propose instead is a
method that is based on frequentist
hypothesis testing or a p-value
calculation which I'm assuming most
people will be familiar with already
this has two interesting advantages
first of all it allows you to combine
multiple scores into a common test
statistic and this is done in the hopes
of obtaining an overall more reliable
and robust test statistic or scoring
metric and the second is that in this
particular method the decisions
threshold parameter would correspond to
the significance level which is you know
you will all be aware of this already if
we do p-value calculation and
interesting fact about the Alpha
parameter is that it's in a sense
defines a uniform semantics of what
rejection means and it's invariant of
the specific scoring scene so if even if
you decide decide to add or remove
certain scores from the overall metric
it doesn't change the semantics of the
decision pressure there's also other
advantages which is for example this can
be implemented quite easily and as very
efficient in practice but I'm not going
to talk about this and this Tokyo okay
before I go into the method discretion
is just some simple notation this won't
be heavy math but as some minimum
rotation just to make sure that you can
follow the explicit
so we say that an agent a has a behavior
denoted by a PI and pi is basically just
a black box function that taxes in put
the interaction history the history
itself can contain all past states and
join actions and the constit and then it
produces a distribution over actions
available to that agent so in other
words a hypothesis is defined as a
mapping from histories to distributions
of reactions okay so here's the key idea
underlying the model we control one
agent I and we observe another agent J
the PI J is the true behavior of agent a
and PI J starts the apophysis paper and
a question and the testing problem is
are these two things the same well we
can't answer this question directly
because we don't know the true behavior
we only have the hypothesis but we don't
know the true babe
but here's what we can do instead we
first of all we know the past actions of
the other agent because we can observe
the history so we can observe that
vector of actions and we know that that
vector was generated by the true babe
and in addition we can use our
hypothesis behavior to generate a
similar vector of the same structure and
this corresponds to a vector of
hypothesized actions assuming that the
agent truly uses that behavior and then
once we have these two vectors of
actions we can address the original
question as a two-sample problem which
is the question of whether these two
vectors were generated by the same
stochastic process which in this case is
our hypothesis behavin so that's this
the this simple idea underlying this or
a method so we want to compute p-values
and this is not new none of this what
you see here is new but I'm let me just
explain what it means anyway this this
here is the test statistic that we
calculate given the two vectors of
actions that are just explained T so
this is the vector of observed actions
and this is the vector of hypothesized
actions using our hypothesis and then
this is just the test to see now the
p-value itself is just the probability
with which we expect to
for test statistic at least as Extreme
as the one that we've just calculated so
none of this is new this is all of this
should be familiar already this assumes
that the hypothesis is true and that
allows us to use the same distribution
and what will M D is well once we have
the p-value we decide to reject the
hypothesis if the p-value is below some
manually chosen have a decision special
so this is this is not new yet none of
this is new here's how we do that the
test statistic and and that's where it
gets more more interesting on the top
level here the test statistic for two
vectors of actions is defined as the
mean of the towel prefix test statistics
that's what the Tao means here so the
towel here means that we just take the
first tau time steps of the overall
vector of actions so if you have for
example ten actions and you take the if
tau is five you take the five initial
segment data self is defined simply as
the as the weighted average of scores
calculated for these tower prefix
vectors so W refers to a weight I could
for example be uniform and the zebb K
refers to a specific score functions and
one score function there could be the
empirical frequency distribution but I'm
gonna go into some examples and the
examples in the next slide the the
intuition is this the score function
intuitively could correspond to
something like a as a as a likely the
likelihood that our apophysis would
produce the vector that we give to this
function it doesn't have to be a
likelihood in strict theoretical sense
sense but it makes its it's useful to
understand the method I find and that's
why I like to use that when I explain
the method
okay here's a very simple example of a
score function is that one so if this is
our hypothesis and this is some input
vector is simply defined as the average
ratio of probabilities assigned to the
observed actions in the vector and
maximum probability actions in that
respective time step so this is the
probability assigned to the observed
action adaptiq
time Steptoe and this is the maximum
probability that any action receives at
that particular time step it's very
simple and you can see already how this
would translate into your likelihood
intuitive ly would say that you know the
closer you get to the apothem more
likely or the higher scores and
corresponds to like it but you can also
see that this is highly imperfect for
example if our hypothesis is a uniform
distribution then it doesn't matter what
what the true waiver is this score will
always be the highest score then so it's
imperfect in that sense as well but I
guess I get back to this in a second
here's another example just skip over
this but it's similarly simple and
here's another example of quite simply -
this is based on the empirical frequency
here we just take the average
distribution of observed actions in the
vector and here we just take the average
of all hypothesize distributions and
this whole expression just takes the
overlap of these two things and so again
you would you would think that you know
the greater the overlap between these
two things the more likely it is in a
sense intuitively that this generated
this year and you can also see a going
back to the first one here that this one
isn't necessarily food by uniform
distribution such as this one so the
idea would be that we can somehow
combine these two things to make an
overall more informative test statistic
that's that's the idea here
they're all in perfect but there can be
combined to be to make up something
stronger okay this is this is just one
last step and then we finish with the
method discretion there's some
interesting math leakin D or some
interesting theory you can show that the
test statistic doesn't necessarily
converge in the process but you can show
that the fluctuation is nonetheless
normal in the limit and this will tell
us that we can just use a normal
distribution to learn the test
distribution but there's a an important
problem here if we use a normal
distribution that would fail to account
for the screwing the gradual screwing
and the initial learning process and you
will find for example in the multi agent
setting with the data in there in the
first say 100 time steps you get a
highly skewed test statistic so a
t-distribution or no
distribution with failure to capture
that accurately so we instead propose to
use a SKU normal distribution which
includes the normal as a special class
and as some parameters and in this work
we provide a learning method that can
learn these things online during the
interaction based on a sampling a
procedure but it's not necessary for the
rest of this talking so so we we
basically learn the test statistic
online as we interact and here's some
here's some interesting experimental
results so in this first set of
experiments we assume that our own
behavior the behavior of the other agent
and our apophysis for that other agents
behavior they're all just random
behaviors and random behavior defined as
random distributions of reactions at
each time step completely random
we tested all combinations of score
functions that one two and three which
are the ones that I showed you earlier
and the examples and before you go ahead
and interpret the data let me just
explain to you what the access mean here
so this the numbers here correspond to
the combinations of score functions that
we tested so he we used one here is T
and here we combined all three of them
together and blue corresponds to the
case in which the hypothesis is correct
so the correct decision here would be
not to reject ret corresponds to the
case there where the apotheca says
incorrect so the correct decision here
would be to reject and obviously we want
to be as high as possible here so you
can see this is these are some very good
results we have we have almost a hundred
percent and most of these cases or
bordering and represent there's just one
exception to this which is this the
third scoring at scheme or scoring score
function and there's a very simple
explanation for that the explanation is
that the third score function is just no
good for these kinds of behavior see if
you recall the third score function is
simply text the overlap of the empirical
average and the average of all
hypothesized distributions but for these
particular behaviors the both of them in
the limit will just be uniform
distributions so we will have no useful
information from that score metric and
this explains where we have a good right
here
because in the limit any two behaviors
will always look the same to us we will
never reject and which is good in the
blue case but it also explains this
because again in the limit any two
behaviors will look the same to us and
then we should we never reject when we
should be rejecting but it is something
cool about this method we can we can
solve that problem by combining any of
the other two metrics or adding it to
the third metric this is what you see
here so the lines here correspond to the
evolution of the p-values and ideally
there should be converging to 0 in this
case because this corresponds to the the
red case when we should be rejecting you
can see that this is the case for all
lines except for a3 which is the red one
you three simply is not a good score
metric but as soon as we add any of the
other guys you can see that conversion
is restored or heat in other words and
and this is the case for the for these
guys too oh I should have pointed out
sorry I forgot this these three busts
the individual ones correspond to a
higher complexity and the interaction so
this is two actions ten actions and 20
actions and you can see the healing
takes place and all complexity levels
and this is exactly the effect that we
wanted to achieve with us you know we
didn't necessarily know ahead of time
that this was going to be a bad score
function but we can combine this with
other possibly in perfect score
functions as well and then we reheat
early convergence here right so if for
the for this case here you can see that
it's almost as fast as the other ones
but for the high complexity once it just
turns out that it needs more data the
second score function which I didn't
explain but it's relatively simple as
well just just isn't quite as different
from the third one as the other ones
would be so you know it just needs more
data basically what this means is for
these cases here if we just add more
time to the interaction they gradually
go up as well here this is a cut-off
after 10000 time steps which I thought
would be sufficient for the learning
process to in order to have converge to
something but if you add this to maybe
20,000 time steps you know this
up gradually as well and you can see you
can see that the convergent goes to zero
okay here's another set of experiments
this in this case for adaptive behaviors
we have different kinds of adaptive
classes for example a co-wash decision
trees and coal both your networks I'm
not going into the details here but
they're basically just behaviors that
take that adapt to certain portions of
the past interaction this tree for
example the past five time steps and
here you can see that we have similarly
good performance in the positive case
but we have a mixed picture in the
negative case yeah and in particular for
this bar which corresponds to the core
of decision trees and this well
discusses will shows you a an important
limitation of the testing method which
is that we only make decisions based on
passively based on data that is provided
to us by some external process so in
other words this testing method does not
probe specific aspects of the hypothesis
if if this is the structure of the true
behavior a tree and if our hypothesis is
a tree as well of the same structure and
if it just so happens during the
interaction that we only ever see this
part of the tree but the differences are
only in this part of the tree well then
for them then there's no way for the
method to make the right decision so in
other words something else is needed
testing is not the only thing that we
need yeah okay this was the second part
and in the third and final part that
will tell you about what I think could
be an exciting and future research
agenda so the future so we saw that
testing is an interesting and useful
part but it's also just a part of the
bigger problem as you just saw it's not
sufficient in some cases I think what is
needed eventually in the future is
something that could be described as
hypothesis contemplation it's a much
more holistic way to look at things it's
a it's a reasoning process that truly
considers models as hypotheses and not
just as models that we run with
nonetheless so this was this words
consists of elements such as being a
to generate hypotheses and you know you
could do a machine learning and fit
models you can do case based reasoning
or you can make guesses based purely on
the structure of the problem such as
equilibrium solutions then we would have
an element that does selection and this
could for example be maintaining beliefs
over a potentially promising subset of
hypotheses then we have testing which is
what we just saw then we would need
exploration and revision which I'm going
to talk about in just a minute and then
there's probably even things that I
don't even include in there in there and
the figure here so what we really want
Nabal in an agent to be able to do is
this practically contemplates truly any
path assess but importantly this is
happening online this is not a an
isolated process this is online and
happening during an interaction process
so really we'll probably look more like
this eventually in the process so here's
some more details of expression
exploration is simply the question of
how when we should explore aspects of
the hypothesis if we remember in the
last part of the second part where I
showed you that we didn't have good
results for the rejection case well the
explanation was that we didn't actually
probe this aspect of our hypothesis we
simply didn't see data coming from here
and this can be solved with exploration
schemes so in this in this case we used
a simple random randomized exploration
scheme you know for example epsilon
greedy exploration where you have a
certain probability of randomization and
the choice of this course goes up really
quickly and if you do if you do that
because with the randomization of that
you quickly end up in this part as well
but of course you can just do that in
practice because any of reactions will
have an effect on the other agent and
you don't just want to do random and
randomized action choices you will have
to be smarter than that and that's I
think one of the a very interesting a
key problem that we will need to adjust
something else is really interesting and
something that I'm involved in currently
as well is revision the question is how
should you revise and improve aspects of
your opposes so rather than generating
new hypotheses from scratch you already
have something to begin with
you want to revise certain bits and
pieces based on what you see so an
example would be if you assume or if you
have Porthos's is that the other guys
have been reinforced Mona what aspects
could be the learning rate the
exploration rate the discount rate and
when various modules and parameters and
I'm assuming you're familiar with this
the question then is based on what you
see in the interaction how should you
revise and improve these things
subsequently and when should you do that
as well and then so this is a really
interesting but challenging a question
okay so in summary individual pieces of
this whole puzzle exists we have people
in the community and people in this room
who have worked on bits and pieces here
but what's missing is a complete
integration of these individual pieces
into a more comprehensive reasoning
process all of these things need to be
put together I think eventually I think
this is an important problem because of
the things that I just pointed out to
you and then the presentation and it's
feasible and timely precisely because
we've worked on this for a while now and
also I think this is relevant not just
for multi-agent systems but actually for
the broader aíi community we all use
models in you know various aspects of
models but we don't do autonomous model
criticism there's something you don't
see much in the community so if you
familiar with statistics then you will
probably be familiar with model
criticism they've worked on this for
many decades but there's a difference
between that NII the difference is that
in AI we will need to carry this out
autonomously and that's I think where
the leap has to it has to be done okay
so this is the goal but there also
challenges since we're combining various
elements there will be issues relating
to complexity we want us to be efficient
there will be issues relating to
soundness and completeness
all of this will need to be worked out
but I think something more interesting
is that sooner or later when we pursue
this goal we will have to reconsider
whether correctness is the right goal
for this and maybe we should contemplate
something else something that could be
described as usefulness usefulness is
something that could include correctness
as one aspect but couldn't
be traded off with efficiency model
complexity and the idea is that you can
then carry out things such as estimating
value of information and you know go
beyond the correctness criteria there's
this famous citation and the statistics
community by box that goes something
along the lines of all models are
incorrect but some models are useful and
maybe your agents can learn to reason
along those lines as well analyst over
the last part which I think could be
really exciting and a really interesting
step forward is if abs can learn to
carry out this hypothesis contemplation
autonomously rather than us crafting
these individual pieces and then putting
them together into an integrated
solution this is a more mechanistic
approach which is what you see a lot in
AI but I think will be really
interesting if you can just you know now
I have these individual pieces but let
the agent learn these things
autonomously I think that could be
really interesting and tremendous
milestone in the III development history
ok that sums up the presentations so it
was not so clear why the special
elements and so first of all this is not
just finding about not just about
statistical tests as I was trying to
explain the tests that would just be a
small part of the bigger picture and the
reason why I'm relating this to mitogen
systems is because it's simply not
happening and the my teacher since my
gen systems community but I also point
load in the presentation that this is
relevant for many areas and I so if you
do model-based reinforcement learning
you will you know this will translate
into that as well if you do data base
game theory analysis you know you have
more
as well and you will have to have some
if this is to be carried out at Ellimist
leave by an intelligent machine there
will be then the machine should be able
to have some model criticism member in
the reasoning process so yeah so
basically that's the answer that it is
relevant to to many things I'm wondering
if this can be phrased as a form of
anomaly detection and as well when
you're when you're looking at something
outside of your your class of models for
how how the other agents are behaving ya
know
well first of all I should say that I'm
I'm not all that familiar with the
latest work in their community but I'm
familiar with some papers and it does
definitely relate to this as well I'm
not sure you know what kinds of methods
that you say I think it's mostly based
on you have pre-specified models and
they try to detect deviations from that
which in that sense relates to that as
well but yeah so it definitely relates
to it but I unfortunately I can't go
into detail because I'm not too familiar
with the latest enhancements that or if
I can go back come back to your comment
one one difference as well that I
pointed out in the presentation is the
if you do model criticism in a data
fitting scenarios then you don't have
this online component whereas in the
multi-agent systems setting you have
additional pressure or complexity coming
from this idea that there's this is
happening as you interact as you collect
the data so you can't just reason about
the correctness or do the model
criticism in an isolation from the data
generation process your reasoning will
be part of the data generation process
as well in the reinforce learning book
because they interact with the
environment that's right
many different that do you end up a
value yes yeah so the different score
functions they will be part of the
overall test statistic well it's
actually and this method is relatively
simple
so it's a defined as the average of a
weighted difference of the scores so
these are the different score functions
and then you calculate the difference
and if you interpret this as a
likelihood you know they should be high
and this gives you the zeroes out then
zero is what you need to get a high
p-value and that's where the intuition
comes from you mean you would calculate
basically just some reduce this
definition to one scorer function or
well could you hear well I'm supposed to
could do that but then you would have
the year I'm not sure if that would give
you an edge over this so I'm not I'm not
you could surely I think you could do
this but then I'm not sure if this gives
you more power than this and the the
other thing is well as if it even if it
does give you more power than well
there's complexity issues as well but if
you do is simultaneously and it's
independent then maybe the increased
complexity it doesn't matter all that
much is a weighted sample the the right
metric for this or should be some other
LP norm or or some other statistical
piece yeah but I don't know I don't know
I best friend man yeah so how would this
handle your example with the
rock-paper-scissors where the true world
was a true mystic one and maybe there
the problem with the IRD assumption
there it seems like the natural kind of
score functions that have the ID
the what do you mean the idea in the I
see so first of all them you would need
to specify what model you do the test on
this is the if you use the limit model
here say after maybe a you know a
thousand interaction and they do the
test on this model you know you would do
a uniform test for the first score
function they wouldn't help you because
the first ko function over a uniform one
always gives you the high score but it
does the the third score function which
is the empirical average actually the
third function I think wouldn't capture
that as well because if you have the
sequence you get a uniform as well and a
limit so my guess would be that the
second one captures it but I would have
to look more into that each other
there's the second example of the score
functions here I'm not entirely sure cuz
I've not actually carry this out my in
my mind but from what I can tell the
first to score functions actually you
wouldn't be able to solve that because
in the limit the sequence is uniform and
the limit model is uniform as well so
yeah I'm guessing this guy would be able
to provide information but I'm not sure
to be honest I we correlation yes you
can if you you can definitely you can
you can have a key you can encode
correlations as well in your score
functioning but these simple ones don't
actually do that one more note I should
make here this is a statistical test and
and so if you end up with a
deterministic distribution here my
personal recommendation would be to have
some you know prior layer that tries to
identify the deterministic relation and
then you don't actually have to go ahead
and do all this relatively complicated
testing because then you you only have
ones and zeros then but I mean I mean
you have space of hypothesis you
maybe the truth is not even in space oh
damn it's you our priority then can I be
if you can identify the right thing this
first this testing method does not
reason about the space of hypotheses
here
it's only yes or no there delivers so
the space of hypothesis would be some
kind of model selection which is what's
happening here is because my shoe is not
like if one of those models is the
correct model than where everything is
not yes it was expressly told and yes
yes exactly that that's fascinating
that's right for speedy watch in the
good and you watch unknown the null
hypothesis as for any PTSS that the
hypothesis is correct okay
this shooting shows the same that's why
I Louis UT have to specify the
distribution over values yeah so you
have one common distribution over test
statistics basically so when we think of
a Gaussian in the limit and then you
calculate your test value you take the
spot and then you just take the outside
well I would say this in the frequent so
in this particular I guess you could say
standard way of calculating p-values
this is actually the normal thing to do
I think in the in the name and what's
naming something a very variant where
you have two hypotheses there the h1 in
the DHD or then you have what you just
described there
you made any attempt to map this into a
bit in framework you mean such as
maintaining beliefs over a set of
alternative models well yeah you can you
can certainly combine these two things
first of all if you do the if you reason
over a hypothesis space find a drawer or
uncountable then this as was just
pointed out and as I pointed out this
cannot does not necessarily give you the
true answer if the true waiver is
outside the space but you can combine
these two things by maintaining your
basic belief and reason and and doing
the testing simultaneously and the
testing may be able to tell you that
this is no good at all you know if it
gives you a negative test result in the
end so then you can either change your
path of space you can start you can
start learning based on different
assumptions or you can just scrap all of
this and do some math default policy for
example on the other end if you mean
combining or adding in some pace and
reasoning into those core functions you
can do that as well even though I
haven't done it yet here because we know
they're fairly simple they don't do any
patient reasoning in it then it would be
nice to have something that says because
you start with some prior and then you
sort of update your beliefs and nice to
see something well your problem was
actually exactly that were that was the
idea behind this originally but but then
as I pointed out this is ultimately just
a small piece in there over you know
product that we want to achieve okay


Bas Steunebrink – About Understanding, Meaning, and Values – CSRBAI 2016

all right welcome back to the program
series on a bus and beneficial
artificial intelligence our first
speaker today is Boston Berg who and it
works at the AI lab IBS I a its its SIA
its SIA and and finished is HD at the
University of Utrecht in our
intelligence okay and he's here today to
talk about reinforcement Earth sorry
it's a value learning for agents in in
the realistic environments under
computational resource constraints thank
you so yeah I'm a Dutchman my name is
Bob sterling so yeah I've been working
here it's okay it means just a paved
village square in Dutch so yeah I work
in in Switzerland right around here if
you would
sumit enhance you would see Yogesh
meathook were waving behind the window
there because I work in his lab and for
for many years now ready and since more
than a year ago we also found that a a I
start up called nations so you're going
to be aware is the president of this
startup and the the the goal of this
company is to develop and market no
network based AI solutions on large
skills for ku industrial and less
industrial applications and our state
goes also to to develop really a general
AI but with the provision that we are
very mindful of potential dangers and
pitfalls of building such a system so
and I am as some of you know already
very mindful of these things actually
I'm still fully employed at it SIA
working on a grant from the future of
life Institute which is this grant
thing set up by through a cash injection
from Elim asked to develop or try to
think more about robustly safe AI so I'm
working on that and I already talked
with a few people I see also some new
people that were not here this weekend
when I arrived we talked already a bit
about what I was gonna talk about and
quickly my talk was renamed to boss a
super pessimistic value so yeah so where
I come from I would say it is super
pessimistic I would I'm just usually
very careful to make simplifying
assumptions so you could say that's
pessimistic or you could say that's
realistic so my favorite analogy which
we also discussed a bit on Sunday
already with a few people I always like
to make the airplane and all analogy
where you have your clever engineer but
it's the year 1900 and he wants to build
the first airplane and he thinks well
it's very difficult let me first write
some calculations on paper and then he
figures out almost calculations that's
still very difficult let's make some
simplifying assumptions because the
airplane
okoma calculations keep slowing down and
eventually crashes so let's let's crab
friction for my equations and then he
runs the cranes again he sees almost
airplay still crash they're being pulled
down by some force so let's also scrap
gravity for my equations and then
suddenly his equations work out but what
he is effectively designed is a
spaceship and this is a really very
different solution this is not something
that will work in the constraint
real-world environment that the airplane
was supposed to function in so I'm
always worried to to make simplifying
assumptions I'm much more I always like
to really
tech problems head-on in that sense so
with with respect to when it comes to
teaching some values to our a is the
simplifying assumptions that I did not
want to make is that we are wise enough
as humans to come up right from the bat
with the perfect utility function for
this agent so I want my a is to be able
to figure out over time and accept
updates and and polishing x' to the
utility functions or value functions or
whatever you want to call them second I
am very hesitant to try to excuse the
agent or its environment because these
we are talking about extremely complex
agents and environments here that we
want our age to to work and so what if
we cannot even do that and what if also
we cannot really hope for optimality
because the agent will not have enough
resources at its disposal anyway so I
really want to discuss with you here now
what are the ingredients that are needed
to develop a system which can still
learn safe and robust values despite all
these pessimistic assumptions if you
will and whenever I talk about values I
do mean really human values as in do not
kill do not become a polar populist
politician this kind of real values do
not waste resources unnecessarily this
is also really important value in my
view so a few disclaimers before we dive
in is that I still would really like to
see more formal approaches to the value
learning problem
and I see this line as a line of
reasoning in parallel to that so by all
means if you're working on the formal
methods do continue that in this talk I
do not really intend to show any results
I'm more asking you to think with me
about the necessary ingredients to make
this pessimistic set of work and
therefore this talk will be necessarily
quite informal and I will be happy to be
interrupted at any time make this into
some kind of moderate the discussion if
you will and I have enough slice to fill
the hour but I really don't care if I
finish them or not so if you think like
oh this is ridiculous or why why would
this work just say so let's have a good
discussion so the key ingredients of a
set up where this pessimistic value of
learning would work in my view is has
needs to kind of details
one is architectural details the other
is methodological details so we need to
specify some ingredients of the agent
that we're building for example its
knowledge reputation goals and how do
you specify goals and constraints to it
how does it learn how does it select
actions solve the control problem and
besides that there's a really big
methodological issue that is how do we
teach and test the system and how do we
measure its growth and its stabilization
it's how the way it settles on as values
I think the methodological issues are
often also a bit underappreciated but I
I think these are extremely important as
well and I will come back to that so
this is the big picture this is about
the level of detail that I would need to
describe the architecture of an agent
that would work in this pessimistic
setting so all I would say I will say
something about the way knowledge is
represented in what I call granules we
need some humans specifying and updating
requirements which the agent can use and
take an update at anytime and there will
be some processes that act on its set of
granules it's its knowledge basically so
let's go into that first of all what do
I mean with requirements requirements
just technically things sort of an
engineering taking from engineering
parlance is these are goals and
constraints I do not even say we need
rewards in a sense I would just say we
need goals and constrains goals being a
future state that you can specify for
example I want the all the dishes in the
dirty dishes in the sink to be clean
this is a goal that you can specify to
an ancient ancient has to somehow figure
out how to go towards that goal and
there is constraint which are sort of
the negative over goal for example do
not break any dishes and be done within
15 minutes
these are come constraints that you
would also pass as part of a task to an
agent another important in capability of
our pessimistic value learner is a
capability to perform simulation before
commitment with that I mean is that it
should be capable of running what-if
scenarios if you specify a goal it will
want to find some course of action to
achieve this goal but before committing
to this course of actions it must
simulate what will be the effects of my
actions and will they violate some other
constraints that I have in my system
right now such that you can decide
whether and which of its
possible plans are proper course of
action another extremely important point
in my view is that knowledge should be
decoupled from goals but that I mean
that the knowledge representation itself
should not in itself contain anything
about goals
it should just contain stuff about
phenomena in the environment this allows
the controller which is the part of the
agent which couples knowledge and goals
to obtain actions this can then make
this coupling dynamically and this is
extremely important because the human
might update goals and constraints at
any moment so the agent must be at any
moment ready to reevaluate the course of
actions that it has committed to against
whatever updated constraints using its
very latest knowledge any questions so
far on this maybe it's hard to see how
this should be implemented but I will
show you a bit later that this stuff
actually works knowledge representation
the only level of detail that I would
like to go into is that it should be
represented as what I call as granules a
granule is a piece as a very small piece
of code that has the functionality of
both forward and inverse model in the
sense of control theory maybe I should
make a drawing of this what I mean with
that fine
so unless everyone is completely
familiar with control theory what our
forward in the universe models so see
here it's a little piece here for mobile
if you have an input X this one will say
a forward model
say I predict that Y will happen next
sometime in the near future and a
backward model will do the other way if
you have a goal Y it will say X should
be a sub one and if you wrap two of
these together to relate that are
related I call this one brand new I
chose the word brand new because it's a
very small thing but still has some
internal structure so yeah and as I
specified the controller can use these
kind of granules to change them
dynamically if it has some end goal it
will repeatedly on the fly find this
backwards model that will lead it back
to basic actions and at the same time
whatever comes in through the sensors it
will be able to make predictions for
that and also this simulation can be
seemed to be implemented in this way if
you do the backward chaining from a goal
state to a possible course of actions
then before you actually start executing
actions you can run the same granules
forward to see what would happen if I
would do these actions comes up with
some predictions can check predictions
against other constraints if they
conflict this means is your bad course
of action these granules they are
subject to three processes
additive subtractive and compressive
processes so of course the system needs
to be able to learn so it adds them on
the fly if if it needs to find some
explanation of something that it will
not explain before it can add a new
granule to it system to capture this
phenomena it should do that tentatively
so with very low confidence such that
over time it can observe the accuracy
and efficacy of this new granule and
slowly phase it in to participate in the
actual chaining that will lead to
actions and if the model turns out to be
nonsense it can believe them before it
has affected these actions so in this
way it does not actually need to reason
about any self modifications so part of
case being the addition and deletion of
models they all can be completely
vindicated and falsified based on
experience no reason yeah I think that
you don't mean that there is no like
heuristics or was that actually yeah so
they might be reading of some kind there
there are heuristics used in the
implementation in the system but once it
has been started there's no you just
need to make sure that the all the
triggers for adding in granules are
sufficient and complete to do proper
learning to agree or I think we phrase I
would be more agreeable if he said no
reasoning about direct self
modifications there are of course
actions which can result in self
modification and is perfectly possible
for this architecture but reasoning
about that yes of course but this is
yeah this is a very good point but what
you said this is more about choosing two
put your attention to phenomena in the
rule that you do not yet understand
hoping that you will learn more about
them which will by construction of the
system leads to new granules being built
in the system and this way it can sort
of steer I think we're both worried
about we're heading as a particular case
of something that doesn't fit what you
were looking for but may in fact happen
with such an architecture hey we're
afraid there's still the question are
you sure yes it's a question for
instance you know self modification to
wire head I think something that is
possible it's architecture but not
desirable so far it is possible in a way
but this is where values come in they
should prevent this and I will get back
to them so of course as we go along I
will claim a lot of things about the
necessity of these particular
ingredients but it may not be clear how
they all interact to become a safe
system until you've seen they're all
ingredients together because each of
these ingredients in isolation sounds
cool but you can always come up with
some cabinets say okay this group may
not work but with the whole picture I
plan to come to and when I complete the
picture it should be clear that all
things combined should result in a more
robust system itself so the value but
currently would shoot the human and
press the button and so in lieu of more
details on undervalue learning yeah
for the current set up and the current
set up if the human press the button and
it did not expect a human to press the
button then it means that it had not
sufficiently many granules to predict
this so it means more it needs to build
an explanation of that to Lully credit
assignment so what if it adds a granular
this is like obvious this is to always
be yeah that does not exist in this Sena
the granules are just explaining we say
these sequences of inputs as being put
into the system that's what they are
capturing they are not modifying
anything they're just capturing
knowledge which says I predict that if
you shoot the human and press the button
and yourself you will receive this
signal from the environment yes and so
and if you want to receive a signal from
environment then you can shoot the human
and press a button yourself yes so it is
like indirect self modification yeah
like this is possible and needs to be
blocked out with the evaluating part to
come there yes and we already saw a
handle death because the system as I
said must do simulation before
commitment to actions so if it
stimulates this course of action of
killing a human this will flag a
violation with the constraint that it
should have that key and humans is not
allowed no this was the whole point you
do not have to specify the a priori you
want to specify as many as you can a
priori but I do not assume that human
designers are perfectly wise and they
will need to polish
and probably extend the set of
constraints over time and this still
made some dangers but I claim that there
is a significant period of time during
which system is not yet as powerful to
do really harm humans or deceive them
where you have time to polish it's not
one the system is not adding constraints
no you're not trying to learn these
constraints no constraints are a
debugging ghouls are automatically
automatically generated by the system
using the backward chaining the
constraints are imposed by humans and
constraint the way you find it it sounds
somewhat bindery whole if either reached
or not reach constraints in violent or
nonviolent but yes know for sure I mean
can't reason like this yeah of course
yeah I did not go into that but this
this is a completely valid point and
this is something that that we have so
there is there exists at least one
system that which implements this and it
does this comparison of probabilities of
likelihoods of certain course of action
leading to violations if those
probabilities are extremely low and the
goal is very valuable it might still
connected and vice-versa so and it was
another point what was the first part
you made was the binary that's all of it
yeah I did not say explicitly but goals
and constraints can be under specified
in this case so they do not to capture
exactly the exact numerical values that
you mount easily
and the specification to work and what
language the human operator use whatever
words and I will show you a little
scenario later which for which we have
sort of a kind of programming language
to specify the spoken constraints to it
but I mean there's no reason why it
needs to be so difficult as we grasp
anything because yeah I mean this is
going to be a big change this was made
one of the main points is that we also
need to dive into the metal low blood
sugar methodological detail of teaching
testing and measuring groans into system
what I mean when you specify the goals
in certain language in this language
uses certain concepts and so the agent
is know how to make this consonant
exactly exactly and this is related to
the issue of understanding so the system
might in the beginning not understand
even what these constraints mean this is
something that I will get through later
understanding is a very important part
of the large which I'm very happy
wassailia because this is exactly in
line with what I'm going to present so
they said just to conclude here this
instead of granules is self expanding
and self pruning as well based on
experience initially you will have to
specify a small seed of initial in
granules just to bootstrap to learning
nothing comes from nothing but this
should be as small as possible
sorry yeah you this is sort of vague but
like what's the type signature yeah so
the you mean what could an X or Y be
they are just factors of numbers in a
way like in your do you like yes yeah
this is the next fine action so just to
show that this is not hot air at this
one implementation exists of this kind
of system I call this class of systems X
bar for experiment based AI because they
they gain all the knowledge from
experience from interaction with the
world and they do not prove anything
about their own growth and I showed also
really this video on Sunday to some
people here so this is we implemented
this this system here and tested it in a
scenario where we have first these are
two humans it may not work documents but
this is just the the rendering of the
scenario has observed by the agent so
there's two humans that are having a
conversation about the recycling
properties of some objects on a table
between them and in reality there were
actually two humans with I say tracking
gloves I gaze tracking and microphone
for speech recognition and the system
was observing in real-time the movements
of the people they're pointing gestures
and they're in the direction in which
they were looking and all the words that
they were saying so we did not solve the
speech recognition problem in this
scenario just the words were the signal
was
through an audition speech organizer
just get the words although there was a
lot of noise in it because are one of
the two guys our lead engineer he has a
very nice French accent so I just beat
recognize it's a lot of trouble with
that but all the better just to show
robustness a bit and the system was
observing interviews so one guy is
asking questions about these items on
the table to the other guy and the other
guy answers I'm just using unscripted
full natural language sentences and the
system was observed as in real time
they've got two words as they were
spoken with the time set time stamps at
which they were spoken in real time were
streamed into the system together with
all the gestures and after about 20
hours of observing different 5-minute
interviews we were able to put the
system in to give it a new wall so
originally the goal of the system was
just to observe what's happening the
only cue for it was that the interviewer
at the end of the interview says thank
you is the only goal that the system had
in his observations of figure out why
one guy says thank you to the other this
was everything that was specified in and
after about 20 hours of observing
five-minute interviews we were able to
put the system into a role of
interviewee and answer questions about
the objects and it did so at several
lower levels of competency competency
and so first of all it was able to
properly sequentialized words I'll put
the words because the system was
massively parallel it could in principle
speak of three forces everyone to be
learned no words come one after the
other they don't come in parallel it had
picked up structure of grammar at least
the grammar used in these scenarios
which was actually not scripted we did
not provide any girl English grammar to
the system we did not provide any
spirits to the guys doing these
interviews but still the system was able
to pick up which words would follow
which words if you would follow proper
English grammar on top of that it
learned that there's such a thing as
question entering turn-taking so one guy
speaks gives a month or so or a longer
post that means it's returned from the
next guy this is also picked up it also
picked up how to figure out what is the
topic of conversation so the system was
actually capable of answering exactly
about objects that you were asking to
say how long does it take to recycle a
glass bottle it would actually answer I
will something about the glass bottle
based on what it has observed in terms
of answers in scenario says it has
observed and not only that the system
was very flexible you could give another
goal to the system saying now don't be
the interviewee be the interviewer sure
enough I mean it has observed the
interviews so why not
it started constructing questions about
these objects and you could even set
additional constraints it would then say
also finish this in the field five
minutes and the human interview we would
start answering really slowly the system
was able to plan minutes ahead figure
out that this feed I'm not gonna be able
to run through enough questions I will
violate my time constraint here so I
need to start interrupting this guide to
speed things up a bit they did not have
to constraint it it was not allowed to
interrupt the guy so he just had if
isn't only ethical strength to finish
the interview in time it would start
interrupting
so any questions about this I mean this
is just to illustrate that a system has
specified with the granules it works in
really non-trivial scenarios yeah I can
I don't know yeah this is the to humans
so so just pausing so these were the two
humans discover doing the interview the
system was just observing trying to
figure out what makes the interviewers
say thank you to the interviewee after
five about five minutes and as you could
hear they were using just normal full
sentences in English they were pointing
and objects they were not always even
saying what they were talking about
sometimes the guy was saying that can
you tell me more about this one pointing
at some objects yeah so interviewing a
human being interviewed so this shows
the system being interviewed by the
human so you will hear that one of the
guys has a more robotic voice so that's
the speech and design
that's the human mission ethically
correct little saying even though no
grammar was given about it say it takes
this this cover box two to four months
to disagree in the Sun well that may be
in there problem with the speech
synthesizer or maybe it was because the
text-to-speech that it had observe maybe
the I will say the off the shelf speech
recognized that we used did not properly
figure out the difference between is
integrate and disagree and so the most
noisy thing I started this was the
missing
so yes it's also an example where which
is actually pretty boring wherever it
shows there's an interruption but yeah
it's just first one minute of someone of
the human answering questions really
was pretty boring and you see the system
start interrupting
so presentation so I would even go so
far as to say that system with this
setup would be capable recursive
self-improvement it just may be the most
controversial claim I'm going to make
today so feel free to quit the sciences
to death yeah yeah and now you're saying
those their goods of self-improvement so
how does that work like as I understood
the guys well they're being added by
Higgs mechanism yes which means that
they might ask why the system kids is
wearing everything is like the same
right yes only its its lowest level
learning algorithm is the same because
the it does also have a focus of
attention so there's a positive feedback
loop between the goals that it has and
its intention to to what it wants to so
the the higher quality the granules the
better we'll be able to achieve the
goals and focus on actually keeping its
goals the more it will learn about what
matters in the world the better is will
the more it will pick up and the more
high-quality
granule feed will gather this is of
course there's one caveat there is that
the environment may have lots of hidden
causations which it would not figure out
if it would not have some kind of
intrinsic motivation to to find the
cabinets in its or bar find
generalizations or analogies or whatever
its own granules so I do insist there
must be something extra besides these
learning
in essence that I told you about also
one assumption is that the world is not
too deceptive and adversarial so
otherwise it would just be garbage in
garbage out do you think it's possible
to learn to to manipulate them to
somehow gain these heuristics learning
processes I think this is all a very
different level I mean this kind of
behavior you're talking about about
gaming things this is this is based on
thousands millions of granules
interacting to give rise to complex
behaviors whereas the the way the grain
is unlearned
is so far down I don't I don't see such
an immediate connection between these
two it could this is something we'll get
into later because as I specified
earlier the mechanism the controller
which couples knowledge to goals to come
up with actions the controller is not
the CEO of an intelligence and it's also
a fixed routine and so it cannot
directly manipulate this controller to
to to handle its motivations differently
but of course all these categories that
if the system nurse program learns to
perform self surgery in some other
convoluted way this might still break
this set up but that's exactly where
values come in as I will get to later of
course some values specified that it
should not want that well it doesn't
have to be
yes but okay let's let's go to the issue
of value of them because this will
answer your question to see how this is
stabilized is this set up so the point
is to get two values we first need one
of our very big angry very important
ingredient which is understanding and so
the point is that having a seed having a
capability if you will believe me my
claims that this system can grow to
become a very powerful systems tools
teeth just having this seed and the
system does not mean with a mature AGI
so we still need to find some
methodology of growing this system of
teaching and testing in so the most
important thing first and foremost to
test is its level of understanding of
observed phenomena I will specify what
this means in a second only this way to
answer your question I think is only if
it has proper understanding of of
phenomena in environment can it finally
connect its actions in the environment
to previously unspecified but now
understood constraints that were imposed
by the humans so I accidentally press
okay so my favorite way of illustrating
the issue of understanding what is
understanding is to show pictures like
this the question is if you would train
your your favorite AI algorithm to
recognize tables and images you would
show this image to the system would you
expect this system to answer yes or no
to the question is is a table
thanks I get a response the table
something emoji yeah I actually consider
putting that emoji in there and I
thought this this was such a nice
picture of the table upside down so yeah
I've already said this to a few other
people during the week is that it's a
trick question of course because a
system that really understands tables
would not say yes or no to this question
it would say well it's potentially a
table if only you would flip it such
that it can actually fulfill its common
law as a table would name me to keep
objects a certain distance from the
ground so the system that can answer a
question like that
this shows understanding and that means
that understanding has to do not just
with some visual properties of phenomena
but really with the usage of them as
well so in for me I have with a few
colleagues of mine a paper about this
issue of understanding at the HR
conference this year and then July next
month in New York City and we did some
research on it and it seems that the
whole issue of understanding is sorely
lacking from a idea to the only only
time when the word understanding is
actually uses in natural language
understanding but there the
understanding is not really kind of
understand that I'm talking about here
so what is understanding is my
definition our definition is that level
of understanding of the phenomenon fine
is determined by the completeness and
accuracy of the granules relating
elements of fine both within fire and to
Alabama
so yeah that's probably a lot of parks
so that let's taking it may be like
outside of the realm of granules it
would be having some sort of from the
period yes in whatever knowledge
representation it doesn't have to be
complete I was saying a level of
understanding so understanding is not
binary the the better you understand the
table the more complete and accurate
this set of mobiles granules theory
whatever you wanna call it is with
respect to Phi in the sense that it
relates parts of fine to other parts of
five and to other non non five parts
once predictions which means some
ability to model the external
environment
yeah and the second is the ability to
break goals into sub-goals yeah what I
don't understand is why were necessarily
coupled like what does it mean for a
granule to I mean what is the relation
between these cool arrows like how is it
you know how does the overall capability
of the system to model the environment
is what why is this is generally
possible to break it down into modules
in this way it is possible at the lowest
level you have your sensors that give
you England that will probably give you
vectors of values and so what goes in on
one side or for our model is a vector
values and how it comes a transformed
vector of values which is which is a
prediction now this is very hard and the
shallowest live rendition for the state
if typeof realism if could the enters
here yes are pixels from camera then the
output is this is expects the pixels to
be in sometime in future yes this is at
the lowest level
this is one way to see it another way is
that these X's and Y's can also be more
abstract events so in our system that
are that I showed you the interview
system what it does as well as every
time a granule fires it doesn't only
produce a prediction it also produces
another internal input that says I just
fired and this can be also picked up by
granules and can be used as left or
right hand side such that granules can
be hierarchically composed so where you
see the gradient fires in the granule
goes to some kind of classifier it is a
sort of a template it's a it's a program
that it's more like a program that only
runs if and you can matches one either
the left hand side here the right hand
side there so if if an input comes in
either through sensors or through some
internal event every time such an income
is produced it is run by the descent of
applicable granules each one that
matches this input fibers and they all
fire in parallel okay
this in plates are one functionality
production are another function of in
breaking down some goals with a third
function of the trends now how all three
are coupled together well that is the
point about a granule being a very small
thing but still having some structure
and it has as you say at least two
functionalities namely to produce
predictions and for mode and to produce
sub goals in backwards modification is
the deferred yeah
well the templates I mean that that is
there's what it just to say that this is
a granule is not constantly running it
is it is a data-driven piece of code so
it's it only runs or fires as I said if
something matches one of the one of the
input points and the prediction is added
to the pool of inputs in the system and
this prediction can then be picked up by
another granule on the left hand side
which produces another prediction in the
also be contradictory predictions yes
yes but each prediction also has a
likelihood attached to it depending on
the the confidence and accuracy of these
granules so there's some control
parameters so it can have different
conflicting predictions but when it
comes time to act on these predictions
it can choose the the ones that it's
monster amount okay what is the
interaction yeah this is it's basically
one guy it's a true two boxes with
actually there's just one box and it has
a left-hand side and a right-hand side
and both can take input and if the
left-hand side takes an input then the
output will be a prediction if the
right-hand side takes an input in the
form of a goal then the output is a sub
code or if it takes an input in the form
of normal example yes well if you think
back of the the the interview scenario
the word has being broken
it's composed of thousands of granules
okay in the end
a lot of them garbage as well because
this is what's still actively learning
you if you would observe the I don't
have groans with them you reserve the
the size of the set of granules in the
beginning is really large because it's
just producing a lot of them it's very
low confidence because it's just like
what hell's going on here and this cells
over time some become more confidence
because they are firing with high
accuracy same interview you were to see
like the type of the granule one example
would be like a thing glitch and I said
the farm for my head form of one also
another word for something so the X
could be the and Y could be poeple in
this case where words were the most
basic kind of inputs with some time
attached
yes exactly so yeah this system does not
have any time steps as is common in
algorithms everything is stamped with
real time stamps so in in this case in
the case of Mario's it's a relative time
so the output did the right-hand side
against the relative time headed to it
responsible for specific would say that
this world is always full like this one
yeah what would be the other direction
would be if I want to say this so yes
the if you put the system in in the in
the role of interviewee you would say
you just keep one goal to the system
that is make sure you observe the words
thank you and at all propagates
backwards from that so then the system
starts planning and sees that okay to
get thank you I need to answer this
guy's questions
and so if he prompts me with an with a
question I need to answer because that's
the way I've observed through hours of
observation that's the way to get
thanking there will be some top-level
loans which we don't have to be able to
predict when thank you said on one hand
I'm and received the goal of saying
thank you yeah so yes no I don't know
these were not in there I mean this is
well this was the only signal we gave to
the system and it was really focused on
that anyway because there was the only
signal so we try to make them four or
five minutes each but they were not
scripted so somewhere longer someone
with less long yeah I have no examples
to show you all where we are probing the
system to see where it breaks we're all
ready I just have the videos of where
those right I mean but this was very
robust you could just have this
conversation with the system I mean yeah
because it did not observe many
contracts and so many of the scenarios
they were noisy because the speech
recognizer was making errors and so but
we did not really try to put it off to
suddenly start talking about the weather
outside or something but that will
basically in case of carbon heart attack
because we did not teach it anything
about the weather so why are we expected
to answer and incorporated
so the the scary definition of
understanding basically breaks down to
at least four capabilities that the
system must have in order to be sent to
understand the phenomenon fine
it needs to be able to predict elements
of fine form from happening or or how
they happen if it observes some cues to
them it should be able to achieve goals
with respect to fine so given the goal
of putting an object a certain distance
from the ground it should be able to use
the table to put this object on the
table it should be able to explain fine
in the sense that explain how the
components of fine interact so really
high level of understanding moves with a
lousy system to explain why a table nice
legs
because otherwise the service will fall
down itself as well and the highest
level of understanding will be
demonstrated if it could actually create
or recreate fine so if you could say
make me a table and it could figure out
how to actually build your table that
will be really good understanding so we
see here a system doesn't need to be
able to do all these things but level of
understanding is really on a gradient we
can say which you could each test form
as well so I don't know how we are going
on the time or what yeah so it's new now
if you can get if you can get quickly
through the points you wanted to say I
know we can have time for notions I need
to film and after a few questions
yeah we can take it offline and those
are fine machine yeah because now we get
to have these all ties into values and
you'll probably never love criticism on
that as well so probably you should take
it offline for those
to criticize me so let me just very
quickly go through this
so values in the first place are of the
form of constraints they mostly tell you
what you are not allowed to do these
constraints may be under specified such
that you may especially in the beginning
not really know what phenomena are
related to these constraints
therefore agent must build an
understanding relating these constraints
to phenomena in the environment however
understanding building this bridge from
phenomena to constraints is not enough
because take for example a psychopath
who may very well realize how bad it is
for to kill humans how bad that is for
society he may not actually feel
compelled to adhere to this train so we
need additional ingredients to be able
to talk about values instead of just
constraints is that what is requiring a
get to be compelled to adhere to them
now I could argue that given that the
controller was specified not to be
modifiable in any direct way we should
be safe of course you will say probably
not because the system if it learns to
become really Clyburn over time it might
start doing some kind of self surgery or
programming in systems which will still
violate these constraints by ladies
feelings so this is also not safe enough
yet it is safe enough for the early
teaching periods where the system is
still trying to figure out what are the
phenomena in my environment in the first
place but it's not safe in the long run
so we need two more ingredients here to
really get to values we need to somehow
instill some value that it should value
the persistence of values
and ensure that the understanding of
each value that we want to impose on the
system stabilizes so that the meaning
that the the understanding all the
relationships that it is built that
relate phenomenon in environment to this
value that they do not at some point do
not change so much anymore and this here
so this can be seen as so the thing is
with since the system is so data-driven
and different course of actions
different I say chaining paths may lead
to conflicting course of actions but if
if our own course of many course of
actions will be logically inconsistent
and but equally likely or effective over
the system this would mean by necessity
that this is these are not very
effective path of actions so the the
pools of knowledge that turn out to be
effective will necessarily be also more
logically consistent we're really the
one path of action will be trumping the
other one in terms of utility and
likelihood so we need to make sure that
our expert systems tend towards
progressive logical consistency so we
need to teach them to to reliably
explain and predict and interact with
the phenomena that we care about and
in this way if we have that set up with
the two metal values that I just
explained this system should be pretty
robust against influence being either
from other agents environmental forces
or even from itself because given the
simulation capability of the system as
I've specified it it will be able to
simulate this course of actions more and
more accurately as its knowledge becomes
more and more logically consistent and
figure out that even doing self surgery
or programming other agents to do things
that go against its values is actually
against its values so it should practice
plans I think we can quickly stop here
by these closing remarks is that this
whole approach it is we started with
very pessimistic set of assumptions what
we have in the end is something that
places a lot of importance and
responsibility on the teachers during
the sensitive early periods of this age
nice so logically we should have some
kind of regulations maybe the walls on
who is allowed to teach his baby
Engineering's because since they are
sensitive and Koko enemy directly in the
especially in early stages curves in
garbage out so that teachers will give
you bad systems there's no giving this
this pessimistic set of assumptions
there is no I see no way to avoid bad
outcomes if you train it badly
so lots of responsibility on it
interests
well I mean we have lols already for
schooling systems for kids
and we have checks on schools we have
regulatory bodies visiting schools to
check on the results to check on the way
T and children are told because we have
recognized that if you teach children
badly they will become bad evel's later
so this is a much different case yeah
humans are kind of okay this is actually
what the whole field of photography is
about it is about figure out with
respect to kids they're progressing
their growth in especially their moral
reasoning their values and deciding when
it is time to intervene or whether
everything is going well but as far as I
understand human does not tend to become
associate that just as a matter of like
interacting with other humans on a
regular basis without teachers like like
not literally everybody who didn't go to
school history here is a sociopath
yes but disarms but those people who
didn't go to school will some probably
not be so capable of doing being a very
big influence in in the world anyway so
but his point is just that be these
things are higher and it's hard to
imagine the public certification system
survey I teach and it's actually good
yeah
so we may yet also figure out a good set
of constraints that we can put already
in the seat of such systems such that we
get this kind of stability even with the
lack of teachers already so far I don't
know of such a body of constraints so
this is why I'm proposing this kind of
interactive way of training systems to
figure that out I guess what I'm sure
yeah this is yeah I can stop here yeah
just ruminate on this this service comic