246. Democratising Risk 3

hello and welcome to session 246 in the
airsafety.com reading group tonight we
will be discussing the last part of the
article democratizing uh ai in search of
a
sorry i said the
name wrong democratizing risk
in search of a methodology to study
existential risk
this is still worthwhile done by carlos
prima from fhi and you came from caesar
and we'll be discussing from section
five and onwards uh let's cover
the last and we will also be
going over parts of the comments and an
ea forum post accompanying this
but first i want to take before we
get to the uh i want to take a quick
step back to the definitions because
there is in fact
even though the definition uh
uh in
in section four there is in fact
something in section five about
definitions that i think is important to
to highlight and that is on uh when we
are
judging a moratorium on ai and how hard
that would be to coordinate then it's
not uh it's not enough to look at this
in isolation and see yes this seems hard
but we need to uh
to compare it to our other options and
um building an aligned ai that also
seems really hard and there are a number
of reasons why we should expect that to
be very hard and perhaps look at the
moratoriums in a more favorable look
based on that
the first is that a aligned ai needs to
have a really
high level of
reliability
and depending on precisely how you how
big you view the risk from online ai
then
if you view it as very small then it
needs to be really aligned if you view
it as we are by default doomed then uh
it would be nice if it's reliable but
it's
but halfway reliable is better than
nothing
another reason why ai airlines is
expected to be hard is that it's well
it's basically non-existing technology
and that's
in general really hard and there's no
way around that some way it has been
done it doesn't look
obviously impossible we certainly
haven't found any impossibility results
but it's clear that there is a lot of
work to be done
it's something that is probably decades
away and at least api
is probably a decade away and
uh the deadline for uh ai for a line ai
will probably obviously coincide with
when will we have a agr
uh the authors uh are arguing that it
requires a refined vision of the future
in order to align ai
and i think it certainly obviously it
requires a vision of the future but it's
not necessarily in that uh refined and
this is as i see it
there there are not that many details
about the future that need to come to
pass before ai alignment is um is
valuable
one of the things where
a refined vision of the future is
important is that uh the authors claim
that really precise control of the
speech of technologies if we want to do
the uh uh differential technological
development thing
um
we don't actually need precise control
as far as i can see we have we need one
break and one accelerator and uh agi
development needs to hit the brake and a
ai alignment needs to hit the
accelerator and if we overshoot from
that then that's perfectly fine if we
have aligned ai ready to assume
so
there is no real requirement for precise
control
it's a technology that's interlinked
with all the technologies i presume that
they mean agi and that is indeed true
and requires fine tune recorder tools
and
these need to be implemented worldwide
possibly plausibly uh it depends on what
the alignment tax is there are also some
vocal acts that might uh make this less
required um but but i think overall the
terms that the ideas they have that we
you can't say that um
just uh moratorium is hard period you
need to argue that moratoriums are
harder than aligned ai
uh so that was actually something in
section five that i think belonged in
section four so let's actually go to the
main topic for today uh the risks of
studying existential risk
so um
there are some
trivial low level problems like we won't
actually solve the problem and we will
waste resources by going down wrong
paths and that's not really what they're
talking about here though those exist
obviously everybody agrees
here's a key quote the seller's pursuit
of technological development according
to proponents of the technology and
approach accounts for the vast majority
of risks over the coming centuries
so
this sentence like many others in this
uh in this paper is ambiguous ambiguous
because it can be read in two ways
that according to the proponents of uh
the technology approach um
this accounts for the
vast majority of risks which is totally
true
um or it is it can also be read as
that the sort of technological
development is according to the
proponents of the
utopian
approach and this is something that is
implied many times that it's kind of
like this of course the authors can't
really come out directly and say boston
wants to build a super intelligence
because it's
he makes it reasonably clear that he
really doesn't want that but that is
insinuated many many times
it's argued that technological maturity
is the only way to permanently end
existential risk
by people from you using the signal
utopian approach
i think
this might be true there is the long
reflection as an example
of
something that permanently ends
existential risk but does not imply
technological maturity it might imply
but uh
it's certainly possible that we can have
the long reflection without
without space quantization
and
the others further say it's unclear why
we can't just
make technology that stops existential
risk
um
without also having dangerous agi and
bio weapons and all these other things
and i think
i think it is indeed possible to do this
it might be possible to
develop the technology to address these
things but the problem is that it's just
so much that we want technology to stop
super volcanoes because super volcanoes
actually isn't the real problem the real
problem is how do we stop other people
from even
from making dangerous intentions
um
and
that's the key thing so i think what the
all this means to argue here is
isn't really strongly at at least at
this point that
the technical utopian that the current
approach to existing risk studies really
makes the risk greater except for the
operator's case that if you think it's a
bad approach then it overshadows other
approaches that could do a statue risk
studies better
next is on the stomp reflex
which is
defined as the long history of security
threats being used to enable draconian
emergency powers which are subsequently
abused
um
so the name the stump replic reflex has
been uh coined by the author uh luke
kemp in a previous uh article i uh i'm a
i am a bit confused about this it seems
like
the stump reflexes doesn't really
explain very much about what this is and
i
also don't really think this is original
i think it's something that many many
other people have uh have noted before
and this stump reflects um
argues further that the greater the
threat the more extreme the meshes and
particular for human extinctions then
the measures could be very very extreme
and
the thing we're seeing right now
with with ai is very much not draconian
emergency powers
um i think what we're seeing is
possibly the opposite uh
in that most people and the authorities
might
uh
seem like they're totally unaware really
um i think it's worthwhile to be aware
that this could change in the future but
uh but it's certainly not something that
is here right now
they give one example then the power to
start the second the third world war is
in the hands of uh well joe biden and uh
putin right now
so it's it's not under any kind of
democratic control
it seems kind of when i read the uh
others like this was just a bad stupid
thing that was um that just happened to
be um
i i think there are actually very good
reasons why this might be the most
stable way to do it uh but i agree that
the precedent it is creating is quite
worrying because it would be easy to say
just like the president has the right to
file the nuclear missiles the president
also has the right to decide whether the
uh
they should pursue a national and
security
agi crash manhattan project
project
securitization that is a way to change
the discussion about an issue from like
normal politics to national security you
could almost say that this is taking it
out of politics
and in this case it's often dedicated to
military and officials
and there are different approaches to
um
security realization so moving things
away from politics and into the realm of
national security
and um
unfortunately the only approach that is
being discussed is the technology
approach and i would really have like to
have know like if there are many others
and then what are they
and also
to me this seems like a really bad thing
to happen and i'm uh
not so much interested in uh
hearing the best way to do this rather
than how can we avoid securitization
which the authors don't discuss
unfortunately
rather they argue that the chicken
utopian approach is particularly
vulnerable to authoritarian secret juror
security station
i think when you try to unpack this
this seems very problematic like there
is
like i can imagine another really easy
way to securitize
a subject which is to say that this is
very dangerous and this will kill you
right you can do that with nuclear
weapons you can say we need the
president needs to have the authority
because that's the only stable way and
if you don't if there is a third world
war you personally will die
and that's like a a very fear-based
intuitive system and there are in in
general securitization uh this is the
way they people argue that uh the other
countries will get a strategic advantage
and this kind of thing and that is my
understanding of how secret realization
is normally done but i am not an expert
and the paper doesn't really see how
it's otherwise done but
the taken utopian approach
recall that this is basically the
argument that the universe has a
potential a cornucopia
that
in theory it would be possible to have
10 to the 54
humans if we colonize the uh entire
universe and put as many digital people
in there happy digital people as
possible and i think that to me sounds
like a really poor object for sure
securitization
in that like you could imagine that the
president comes on the tv and in grave
tones say that we need to do this
because otherwise we will die and i
couldn't see him go and argue that this
needs to be taken out of politics
because otherwise we can't realize 10 to
the 54th uh people that seems to me
uh
like an unlikely approach for
signaturization
now we turn to boston's vulnerable world
hypothesis
where it is stated in this article that
boston's preferred solution is extreme
preventative policing and widespread
surveillance through freedom tags
and if we
go we read this article quite a while
ago actually in the reading group
where boston has the following quote the
obedience sounding link is of course
intentional to remind us of the full
range of ways in which justice system
could be applied
so the word freedom text is very much
ironic
so is this in fact boston's preferred
solution does boston want this to happen
well i started to gather up like a lot
of quotes from boston that i felt
showed rather unambiguously that bostrom
really would not like this to happen uh
but i'm not really sure that
uh like if you can if you can read this
uh um
vulnerable world hypothesis and come
away with this thinking that yes buster
i think this is a great idea then i
don't think really that me finding some
quotes uh will help to convince anyone
another thing thought experiment that uh
bostrom is working with is um if someone
is creating a
a technology that would
cause extinction
would it be
an idea to make a preemptive attack on a
sovereign nation to avoid extinction and
in some cases it might be
the problem with this
is that
if there are autocrats or spying watch
quests then they could use these kind of
justifications
for
um
to just subvert them democracy basically
and in this way the speculation that
bustrom have these thought experiments
could indirectly cause the thing that
they are worried about because they
provide roadmaps and things like that
so um the authors don't use the word
intro hazard like bostrom obviously is
uh like i literally wrote the
so i would expect him to have thought
about this
um but i think
we can point out some
positive and negative things about this
like one thing we can say about this is
that the first paper on what to do if we
are in a vulnerable world hypothesis is
written by nick bostrom who is clearly
not an autocrat he would clearly not
prefer that we are on the pervasive
civilians and things like that um so in
that way it kind of frames the
discussion and if the uh if the first um
papers say this is really really a
horrible idea that makes it harder to uh
later say we need to do this comparator
if the first paper was written by
someone who says who believed autocracy
was a really good idea
also i feel a lot of these are
um
interesting when uh argued for a um
from a extensive risk perspective but
for an autocrat i
would expect these uh
uh arguments to actually be rather
simple to invent like i would not
believe that vladimir putin or someone
um would have a hard time uh arguing
that we need more preventative policing
in order to stop dangerous technology
and and things like that
um
but
it's not
so much because we like the authors are
uh talking about like can autocrates do
this and i think a more interesting
angle would be to look more at uh like
cultural things like politics can people
be convinced to accept extreme
preventative policing
and one example i found um is uh
uh warhammer 40 uh k that's an example
of something that has way more cultural
appeal than boston's people and that's a
world where fascism is indeed uh
technically correct uh in that if
like when you do world building you can
make any world and the authors have
decided to make a world where if you are
not a fascist then demons come out from
chaos and eat you and everybody around
you if you are not fascist so that's why
the humans in the warhammer 40k are
basically fascists and this has way more
um cultural appeal and it's also
not really a an argument for fascism and
everybody recognizes this and i think um
my point here is
that bostrom's paper the vulnerable
world hypothesis has extremely extremely
less cultural impact than for hama than
warhammer 40k um
so i would expect that
the the impact from this would all would
be dramatically small
and if it's already a small impact in
there then it must be a small impact
let's go to the
the idea
that gives this people the title
democratizing risk
democracy must be central to efforts to
prevent and mitigate catastrophic risks
in particular choosing which myth to
take should be a democratic endeavor
but the problem with this
i think it sounds really nice in theory
the problem is in practice the agi
development that is being done is
in fact
as far as i can tell
reasonably well
democratically
founded in that most people believe this
is the right thing to do there is no uh
uh
political and democratic opposition to
the work being done by by deepmind right
now
it is argued that avoiding extinction is
a communal project and i think
it should be it ought to be an
accumulative project but we need to do
something if we uh if it turns out in
practice most people don't actually care
so right now ai risk isn't that much of
a communal project unfortunately um and
we need to to grapple with this fact in
some way
another
democratic
problem is that um
some views are explicitly excluded right
now and
the arguments do that needs to be
compelled
so the the argument in practice that is
being excluded in particular from
if you want to do something like the
long reflection is the argument for
extinction in particular for
quick extinction and i believe that
it is right and proper to exclude these
views i believe that they are
sufficiently niche but i
recognize that we need some kind of
uh
like this is easily a slippery slope
that i say oh obviously this can't be
excluded because it's so niche like some
kind of way of um
operationalizing that would probably be
better i think it's really compelling
right now at this stage
to uh because very very few people are
in favor of extinction right now
there's also an argument being made that
democratic judgment is superior
i agree that democratic adjustment is
superior but i also think that um
democratic
uh adjustment is not always superior and
it depends on the case and the argument
that is being uh presented here is
insufficient uh in order to er to prove
that it is in fact superior judgment we
should expect some democracy like some
of the arguments like how there is this
conflict of interest in democracy is
that even true and i'm not entirely sure
and of course judgment is only a part of
what we need to do uh a lot of this uh
of the facts need to be established by
experts and cannot be uh democratically
uh estimated um or uh judged like
judgments it's not really about facts
and and so
this is a part of it but only only a
part
and the uh the explicit suggestions are
citizens assemblies surveys and
diversifying existentialistic studies
unfortunately there's not much more
detail being given in this than what i
just said and i worry that if something
like citizen assemblies
would be
effective
then
my thought would be that even smaller
citizen assemblies would also be
effective
possibly possible that is not a given
but i think an incremental approach
should work
and right now we're not really seeing
that
and that is an interesting question that
the obvious hypothesis is that that
might be because in fact citizen
assemblies doesn't really contribute in
any meaningful way to existential risk
studies um but i think it's something
that
in order to uh
build an argument
why this is uh useful
a lot more work needs to be done than
just saying citizen assemblies is very
easy actually holding a citizen assembly
is a huge undertaking
and finally how democracies limit harm
there is a claim diverse thinkers would
not sacrifice one billion humans for an
infinitive improvement or of our arts
and
diverse thinkers wouldn't do that and i
think
basically no one would do that
it seems really unlikely i have a hard
time finding a scenario where
we kill one billion people and then um
we reduce the risk of uh nuclear
holocaust and uh or something like that
that doesn't really
seem to be like the way the world is
working i think in fact there could be a
core misunderstanding here in that um
we see a number of places in this paper
that um
some indications that the author thinks
that in the future we might in fact be
placed be a place before um
for options like this where we can
choose between
sacrificing one billion people or
avoiding existential risks um
and
i think i'm strongly convinced that
things that kill one billion people
in
almost always dramatically increase the
risk of the existential risks
um
also scholars should in general be in
favor of democratic fail-safe mechanisms
and that is indeed true and one of the
earliest work i will point to here is
kowski's
short novel uh failed utopian number 4-2
which indeed features precisely a
democratically unsafe mechanism
um how would democratic voters well this
claim that they are unlikely to tolerate
global cash of risks if they know
themselves could be affected
um and
uh that is a claim and unfortunately the
this empirical thing seems to be um
are true right we're right now seeing a
lot of people accepting ai being
developed as as quickly as possible um
and that is a uh
something again that the authors need to
grapple with like the democratic uh
structures that we have currently seem
to
to not in fact uh put any kind of breaks
on ai development
it's claimed that citizens often show a
significant risk aversion in comparison
to their government
sometimes they do and sometimes they
don't uh like right now in ukraine we're
having calls for a no-fly zone which
would be an example of citizens being
way less risk-averse than uh than the
governments who are refusing calls for a
no-fly zone um
and
it's not really at least strongly
obvious to me that this is a
effects-based uh difference
governments are certainly not perfectly
rational but
these kind of
grassroots
organizations seem not to be perfectly
rational either
empirical democratic feel safe
mechanisms seem to work
the empirical belief is
not as strong as well as as i see it and
um right now we are in a situation that
looks like it's very different like
some of the uh um
the things we are seeing uh previously
with like global warming or local
pollution or whatever seems
not that analogous to what we're
currently facing with ai risk in
particular and so the failsafe right now
seems to not be working and that makes
it unclear whether we should double down
on it if it seems to not be working
so that's the article and now i'm going
to talk a bit about the ea forum post
that accompanied this
and this is um claiming a number of
different things um
the first is that there seems to be some
kind of conspiracy against criticism
so
senior scholars told us in private any
critique of central figures in effective
altruism would result in an inability to
secure funding
and that's
uh clearly not what these people are
saying um and so um
it's not
uh
it's quite consistent with um like we've
seen uh earlier the uh the strong claim
that uh
senior scholars in other cases also have
different opinions in private than they
do in public with nick bostrom in
particular so these two claims are
rather consistent
um
now of course the central figures in ea
happen to like read the ea forum and uh
and some of these people within mega
skills nick dixon holland kanovsky and
many others really you would say the
central figures in the ea
replied to this and said no
we in fact strongly welcome good
criticism um
and
aaron gartler went into gartner went
into
more details about
this game how it's been presented by
others and
found that the evidence is
very lagging for any such kind of
um
uh of opinion being held
i can i can add my own anecdotes and
data um
it's like i don't have access to the
central figures in the ea at all nor do
i have access to senior scholars but i
have talked to a number of ea people who
are if who are involved but not
but not central and i have talked to a
number of scholars who are not senior
scholars and it seems really clear to
them that there is a cultural norm
against um
being afraid of criticism
uh for fear of uh
of missing out on funding um i think
it's unlikely that the culture is the
opposite at the top but i this is just
my anecdotes i i don't have a strong
reason to have an opinion about this and
in fact
i would argue that someone like elon
musk could be said to be closer to this
like
the elon musk started open ai
and
uh was criticized roundly for that
eliezerkowski and miri in particular had
very harsh words
against that and it seemed like elon
musk after that
ceased to provide a
substantial funding so in that way that
seems like an
example of where this dynamic could
indeed be true so while another one for
conspiracy theories
i am
not dismissing our claims as quickly and
as strongly as i think other people are
the second part of the
ea forum post was that this was
political and i'm sorry to
say that the authors have recruit
claim this has been a really emotionally
draining paper and it is clear there is
been a very substantial amount of
friction if you read between the lines
um
sean haggarty is being uh
is uh in particular is called out as one
who claimed that this was like a polemic
um post uh
an article uh they say they believe this
is as an insult
and in general that there is a higher
burden of proof because the people was
considered political
i think
i think that there is substantial part
of this people that is political and
also a lot of this that didn't have to
be political that uh
we see things about like a criticism of
uh neoliberalism that
didn't need to be in this people that
could have been cut off very easily uh
and i think it um
in general if you if you add this a lot
of people in ea have this kind of um
it's often said in particular and
rationality that politics is the mind
killer and a lot of people um
get um
would object to this paper just purely
on the spaces
and there is a question whether this is
good or bad and whether
political mixing political and
existential risk is a good idea i
personally tend to come down on the side
that it's a better it's a bad thing to
politicize
but um
in in general political claims
will be met with with substantially
moral resistance
and finally it stated that ea needs to
make structural adjustments in order to
stay on the right side of history i
think that's um like staying on the
right side of history is usually a uh
like
historical materials and marxist uh
framing and it's also obviously to me
hyperbole in that
sure ea is not perfect and no one is
saying that ea is perfect but it seems
clearly right now to be
a force for good in the world rather
than a force for that
and finally one of the things that were
criticized in the ea forum was that the
it's really wide-ranging for instance
shannon hagerty again uh gave the advice
this review
to focus on one section and compare
um the technology and the troll approach
against alternatives
um
they reply in the comments that there is
tension between these two things right
that you're they're both saying focus on
one thing and also focus on more things
and
they
have a
rather adversarial reaction to this
critique to shawna hagerty's suggestion
saying that that is only a suggestion
that's given to ensure that they do not
write a critique
um
i think in fact that is a very correct
uh
criticism that shannon hackett is giving
here like if you have a paper that is
really really wide then it's
both fair enough to say there are a lot
of these places where it's just too
shallow and you should either like
make the paper ten times as long or you
should focus on on something where you
have you can go into much many more
details in particular one of the things
the authors do often is
presenting a an argument and saying
we're not convinced by this argument and
then quickly moving on without a any
kind of
counter arguments or or any detailed
treatment of this which is necessary
because otherwise the people would just
be too too wide but then they should
have made the um the uh
uh the people that's why really
in particular one of the things that i
feel is key in this is boston decides to
discuss surveillance in the in the paper
the vulnerable world hypothesis is this
a good idea
it seems in this um people that uh
is built up to like a really bad thing
but it ends up with very very little
concrete discussion of if this is
actually a good idea i would have liked
to know more about this we'll call them
on out on this in the comments uh
she says um well the others i think it's
only
writing in the comments we need to stop
classifying everything as info assets
i think in fact
if boston's
decision to discuss this is bad for this
reason then
uh
we need to to figure out if we should
discuss it and if we should discuss it
then um
the framing of infohazar and the theory
of info hazards is probably hard to get
around
and finally i would say uh
some of my own personal uh
uh problems that are not mentioned on
this ea forum post um my first one would
be on the taking utopian approach um
approach is defined as an argument uh in
the uh
in the paper but it seems to be more
like a world view and this kind of mixes
things up and um
it's really unclear to what extent the
typical utopian approach um
wants to build agi as fast as possible
or as safe as possible and this kind of
thing this kind of tension is not really
made clear
there's also a really strong tendency in
the first part to overstate the
influence of the utopian approach there
are many many arguments
against existential risks and i think
all of them are sufficient or basically
all of them are sufficient so the
chicken utopian approach isn't really
needed at all
i planned i announced in the beginning
of this uh of the first session that i
would um argue in favor of the utopian
approach even though i don't actually
hold that and that turned out not to
really be necessary because
if there are just so many other good
arguments against extinction
another thing that i would really like
the people to at least acknowledge that
there is currently a lack of public
engagement it might be indeed that the
way to get uh public engagement would be
to politicize this um but
it's it's totally uh you you can't say
that right now there is a democratic
basis for uh uh institutional risk
studies because that clearly isn't right
way too few people care about this
another thing that i felt is the
argument that a moratorium is feasible
is
spread is spread over many places in
this paper i counted five places and
none of them really are in depth they're
not really presenting this argument like
this is how why we feel that this is in
fact feasible
and also whether this is desirable
that's also something that probably
needs to be like someone needs to make a
case for a uh a moratorium
there is the
uh the uh which we saw the first time
that there was some misquotes of of
bostrom and it seems like a clear
uh
mistrust of nick bostrom in particular
uh and possibly other people in uh
in extinction risk studies
and that might be um
that might be uh
worthwhile or not
but
another thing that seems really clear
from the uh uh subtext in a lot of
places is that the authors don't seem
convinced by ai risk at all but way more
convinced about global warming as an
existential risk
um and
in that way it
kind of feels uh i think the authors are
on a difficult quest to reform execution
risk studies when there are so many of
the existing scholars that they don't
agree with and um so many of so much of
the central parts of the existing
research that they also don't agree with
that makes it really hard to make a um
to reform the field unfortunately
because i feel in fact that in many
places the the authors do put their
finger on things that are potential
problems but it's just um this paper is
uh has too many problems to be a a
really good criticism of the current
state of existence risk studies
that's all for today thank you and see
you next time


Jan Leike – General Reinforcement Learning – CSRBAI 2016

all right welcome back our next speaker
is Michael who just finished his PhD at
the Australian National University
working with Martha Souter and just like
a week ago join the future of humanity
Institute as a machine learning
researcher there and his talk today is
going to be on general reinforcement
learning in wider varieties of domains
looking at frameworks for those so
please join me in welcoming y'all Mikey
yeah thank you for the introduction I
want to connect this stuff more to the
kind of things that are being done today
so if you I'm sure all of you have seen
this so this is the Atari game space
invaders and particularly this is a
screenshot of dqm playing this game
the QN is an algorithm that combines new
networks with Q learning to learn to
play these games a lot of these Atari
games and I mean the title says
reinforcement learning today but really
this is sake almost three years old now
so you can do a lot you can play a lot
more sophisticated games with this stuff
and in my opinion this is kind of the
closest thing that we have today to
strong AI and it's the kind of question
that you might ask is if we upscale dqn
do we get strong AI what do I mean by
upscale dqn I mean we just use a bigger
neural network we need we use like more
sophisticated better trading techniques
and so on just take the base same base
algorithm makes it make it better and
the answer is no you don't because DQ n
is fundamentally you're restricted in
the kinds of environments you can deal
with and in the case of Atari games at
higher games are all Markov decision
processes in particular and all of these
games
almost all of these games fully
observable meaning that all the
information you need to know is readily
visible on the screen all the time and
the games are a ghatak meaning they can
always there's no bad mistakes that you
can make that you won't be able to
recover from of course if you know if
you play these star games some of them
you lose lives and then eventually you
lose the game but for DQ an offer for
the agent the agent always gets to play
again in that sense there's no you you
don't you don't have any incentives to
really be careful while you explore
because you can always start again then
of course you have a very large state
space and this is where he is you need
these newer networks to really learn to
from from just the pixel data what you
do but and another thing is that
exponent Epsilon exploration works we
just take random actions every once in a
while and this teaches you enough about
the environment to understand this game
so at least it does so for a lot of
these games but this is this is not the
class of environments that I care about
so the class of environments that I care
about I'm going to call the real world
and the real world is partially observed
war they go to the supermarket I have to
remember what I have missing at home so
I can buy the right items they're not
very readily visible in the supermarket
and such a large artificial intelligence
needs to have memory next thing is that
the real world is not a gothic if I jump
out the window that's it it's not I can
mistake I'll learn from just should
never do that in the first place then of
course there's an infinite state space
I'm not going to see the same problems
again I have to still learn and abstract
from that and I can't explore my
environment by using Absalon exploration
because that would just be
flapping around my hands randomly and
that's not gonna teach me useful things
so yeah so at Howie's the game the
framework of Tiger games is I got ik
MVPs and the environments that I care
about the general environments and these
are the environments that a strong AI
has to deal with so a lot of this talk
is gonna be about understanding this
environment class in particular I'm
gonna first I'm gonna talk about aixi
I'm gonna tell you what I see is of
course most of you already know then I'm
gonna talk a bit about optimality in
which sense is axis optimal or what
optimal means and what other things are
optimal then I'm gonna take a short
detour into game theory I'll show you a
cool new a result of we have and finally
I'm gonna connect all of this to AI
safety and then tell you why you should
care so I guess for now you just have to
care without knowing why okay so this is
the general setup that we'll use we have
an agent here environment here the agent
takes an action receives a percept and
every time step and the percept is an
observation and reward and the
observation could be something like a
camera image and the action could be a
limb movement or something and this is
commonly realistic so NY ammidon agents
are complete separated and if you think
about these Atari games that's entirely
true of course that's not true in the
real world and I'm gonna come back to
that but yeah so formally my agent picks
the policy and the policy is a function
from histories that is action percept
sequences to actions and that could be
stochastic so this little Delta means
distributions of reactions the
environment is a function that takes
histories and actions and maps them to
percepts again stochastically
and I'm gonna use a less than th denote
a history of lying t minus one just
first t minus one interactions and then
the goal since we're doing do
enforcement learning right now the goal
is just to maximize discounted rewards
and for this I'm using the scan function
that has only been constrained being
summable so typically you would use
geometric discounting where gamma t is
gamma to the power of T for some
constant the gamma and then the
simplifying assumptions that I'm making
is that rewards rescale to be between 0
and 1 and the action and percept spaces
are finite so this is kind of the formal
set up that we'll deal dealing with and
one of the central quantities in
reinforcement learning is the value
function and the value function I'm
going to denote with v pioneer so that's
the value of policy pi in environment
new given this history here and that's
the future expected discounted reward
when following policy pi in this
environment conditional in the history
and here I just put a normalization
factor that ensures that values are
always between 0 and 1 and because
remember we assumed that it was at
between 0 and 1 ok so with that we can
define the optimal value the optimal
value is just the value of the optimal
policy so the supremum here is actually
a maximum there's a policy that gives
you the optimal value and that policy
I'm going to call them the new optimal
policy
and then one other ingredient that we
all need is the effective horizon and
the effective horizon is intuitively the
effective horizon at time T is the
number of times steps that you need to
plan ahead to get all but an epsilon of
your discount functions mass and so this
is what this is saying and so since
you're using this counting the effective
horizon gives you kind of a limit of how
much you have to plan ahead and for for
geometric accounting this limit this
effective horizon will just be constant
okay so now let's talk about aixi so we
start with a countable set of
environments so that could be in the
case of icy or the traditional
definition of AXI that would be the set
of all computable the environments we
take a prior over this class preferably
a prior that is sounds positive
probability to every individual
environment so for example we could use
this lama of prior where the prior
belief in environment new is 2 to the
minus klimova of complexity of new and
this is kind of so the motivation for
this kind of prior is on the one hand or
comes razor where if you have a very
simple description of your environment
on the universal Turing machine then
this kamagra from xt is low and hence
the prior belief is high so you assign
higher prior beliefs to environments
that have a short description and the
other
right and and with with this prior you
get a Bayesian mixture where you just
mix you you take this weighted average
over all the environments and this gives
you from the type signature it gives you
another environment that I'm going to
call the base mixture and I see is just
the base optimal agent with Islam of
fire where I take the policy that
maximizes expected value in the base
mixture so this is just yank the
standard base anything to do and he
started with you prior you do some stuff
you update your prior to a post area and
then you must summize expected you
what's in that according to that the
studio so how well does that work so for
one thing one thing that we immediately
get is the on policy value conversions
and that sense says that for all
policies the value of that policy in the
base make sure converges to the value of
that policy in the true environment so
I'm always using you for the true
environment in other words base is good
enough to learn the value of any policy
but that's it's called on policy value
convergence because I'm learning the
value of the policy that I'm following
and does not tell you the value of
policy well what would happen if I had
done something else that's what you're
not learning and this is important and
that gives you yeah and this brings us
to the next point optimality
so how optimal is this and that for that
if you first have to ask a question what
what do you mean by optimality so we
could mean based optimality we could
mean asymptotic optimality in the sense
that some probably my agent
to act optimally in in the true
environment there's other things that
you look at in machine learning sample
complexity balance regret bounds sample
complexity bands are basically just a
quantitative version of asymptotic
Europe accident particle Tumulty tells
you how probable it is that you're close
to optimal and yeah I'm gonna come to
the bacteria great later so this is the
formal definition of asymptotic
optimality so the policy is
asymptotically optimal if the value of
the policy in the two environment
converges to the optimal value so this
is a different this is a different thing
than we had before in the on policy body
conversions so it's always a quantity
between 0 & 1
and kind of in in slogan form in order
to get asymptotic optimality the agent
needs to explore infinitely often and
for an entire effective horizon so if
you remember the effective horizon we
have that before how many steps you need
to have to get most of the disk on
functions mass and intuitively if you do
this if you explore infinitely often for
an effective horizon then exploration
becomes sufficiently on policy that you
kind of learning to predict what would
happen off policy in a sense because
there's not really any off policy since
you might always end up exploring and
this is I mean this is just kind of
intuitive and it's only a necessary
condition your voice you also have to
explore like in the right moments but
yeah so there's a theorem that says aixi
is not lessen publicly optimal and this
makes somehow makes sense if you think
about what a bayesian would do if
there's if you think that the
potential gain and value that would have
from exploration is small enough that
you think it's not worth exploring then
you don't explore or if you think
exploration is very dangerous
then you wouldn't explore a sebacean but
that means that you don't explore
infinitely often and then you don't get
asymptotic optimality okay I can even
make this more specific and break
actually really horribly by using a bad
prior and for that I need to define what
hell is so this is what how looks like
you're in this state you can't escape
and then what is always here and if you
remember we assume that zero what's the
lowest reward so this is the bad the
worst possible thing that could happen
right so what I'm gonna do is I'm gonna
define a prior that I'm calling the
diplomatic prior and for that I need one
more ingredient this ingredient is a
computable policy in my example I'm
using the policy policy that in every
time step just outputs the do-nothing
action and clearly this is not like a
very intelligent policy at all and using
this policy I can define the dogmatic
prior that basically says as long as and
if you're not acting according to this
policy pile I see then you go to hell
with high probability and the crucial
thing is that you can you can set this
up in a way that the it actually assigns
positive probability to all the
computable environments it's just that
it assigns vastly more prior probability
to environments where this happens if
you don't follow your deck Matic policy
and then we can formally prove that I
see acts according to this lazy policy
as long as the future expected value of
this policy doesn't fall to close to
zero so
basically AXI is sitting on the couch
doing nothing and if every once in a
while somebody comes along and gives
gives it food it gets some rewards and
its really scared if it gets out of the
couch it will immediately go to hell so
basically if you if you imagine yourself
seeing and being a situation if you yeah
if you as you basically feel like this
is this is kind of good enough I get
food every once in a while and if I get
off the couch like probably something
bad will happen so I'm just gonna stay
here and of course this is really bad if
you think of aixi as you know like the
most intelligent agent or the perfect
agent because it won't do anything
intelligent and yeah so it all depends
on the prior and this is this is in
contrast to the results that you have on
positive sequence prediction where
asymptotically the bias of the prior
always washes out and goes away and then
reinforcement learning this is not the
case if you have a bad prior then it
will make you add but act badly and even
since you're acting badly you're not
exploring enough and you're not learning
that you actually have a bad prior so
how do we fix that
so one thing when a charity could do is
Thompson sampling and Thompson sampling
in this case you it works as follows you
take you look at your posterior
distribution over environments you
sample 1im environment from that
distribution and then you follow that
environments optimal policy for an
effective horizon and then you resample
so basically you yes so you you sample a
belief environment and you you kind of
like this is today I'm going to see you
and this is how the world works and then
you act optimally according to that for
the entire day or however long your
and intuitively this incurs more
exploration because every once in a
while you draw a really bad sample like
an environment that it's just crazy
where I believe the best way is to just
lie and cheat to everyone and then I
follow that policy for a while now of
course this is gonna be terrible and I'm
not gonna get many year it was from that
but at least I'll learn something along
the way right right and this is this
kind of leads us to the following
theorem that says that Thomson sampling
is awesome totally optimal where the it
actually learns to act optimally in any
environment from the class so is that
now better or worse than what i exceeded
and this is just me so it's one version
of asymptotic optimality so asymptotic
optimality if you look at the definition
that I gave is that this converges to
zero but this is a random variable
because the history might be random so
you have convergence of random variable
and there's like multiple times or types
of convergence and convergence in mean
is one of the types and yeah and this is
what it formally looks like in
expectation the difference goes to zero
and yeah so whether you think that you
know what Thompson something does is
more sensible than what actually does
because at least it gives us a subsonic
optimality and kind of depends on
whether you think asymptotically optimal
T is reasonable and one problem with us
in public optimality is that you might
get stuck in a trap there are traps in
your environment and you're just kind of
exploring everything you eventually you
run into the trap and then you stuck
and but that's good because now you have
some probably optimal because this like
whatever you do this is optimal so in
terms of asymptotic optimality that's
great but even worse you have to explore
all the traps because there might be
hidden treasures there so in order to be
asymptotic Optima of synthetic the
optimal you will yeah you have to spring
all the traps and yeah so that's a
little song for desirable anymore in
there are two traps each of which gives
you stuff forever you can only be stuck
with one of them yeah but doesn't matter
which one as long as you like get into
the trap so let's look at environments
that I don't have traps and this is this
formal recoverability the definition
that I'm using here and basically says
that whatever stupid policy I followed
in the past
you're like past me was really stupid
but what if you know like starting from
now I act optimally how much would I
lose and the I'm going to call the
environment recoverable if this loss
goes to zero as T goes to infinity so
basically that as long as I start acting
optimally today I haven't lost much and
this is this definition it looks kind of
strong but it's weak enough to encompass
all the other recoverability definitions
that are used enforcement running like a
greatest city and weakly communicating
and so on as long as you have the
groaning horizon it it prevents traps
because if say there is a trap and you
run into the trap then now the optimal
value the value of the optimal policy is
going to be really really low right but
if you had followed the optimal policy
from the start
then you wouldn't have gone into the
trap and you would get a lot more
rewards so you keep comparing the
optimal value on two different histories
write the history generated by the
optimal policy
and your history no because the policy
that generates history is different yes
so the optimal policy manages to stay
clear of the crap at least an
expectation then so the expectation is
over this history yeah yeah and
basically what we just said is that for
your environment that are not
recoverable either the agent gets caught
in the trap or it's not awesome policy
optimal and like that's both version
those both alternatives sound bad right
for any specific agent either there's a
non-profitable environment where it gets
caught and trapped or there's an
unrecoverable environment where it's not
as effective law well some no like
optimality is defined in terms of
environment classes right yeah so if
your environment class contains
non-recoverable environment and your
agent wants to be well let's say it
doesn't it doesn't even have to but you
want your agent to be a part of you
optimal in that class right yeah but if
you now put it in an environment that
does contain traps then it has to
explore those traps because there's
corresponding environments in your class
where there's hidden treasure in that
contrast right trivial failure that
statement if your classes is just gonna
be artificially small yeah yeah any mean
I could have a class where you know
where like the locations where there's
potential traps and everything locations
where there's treasure and then it can
be asymptotically optimal without going
to traps but I mean like this is kind of
yeah if your environment is recoverable
then you don't have to worry about traps
but yeah and this is this is kind of
like just the slogan for one it's not
meant to be like a formal statement but
yeah so let's talk about regret so
regret is so since we just learned that
yes the Telegraph technologies maybe not
like a useful way of optimality or less
useful aim for us let's look at some of
the alternatives and one alternative is
regret so that's one of the things that
is very commonly and the regret is
basically the number of rewards that
you've collected so far until
timestep M undiscounted compared to the
member of rewards you could have gotten
had you followed the best policy in
hindsight or in other words it's just
how much you regret not having for the
best policy and generally a problem
class is called Lerner ball if there is
a policy that for all environments the
class they regret is sublinear and like
usually if you have something like
bandits or MVPs you have you can do much
better right for bandits you want look
at making bread and friendly peace I
think it's a square-root regret so
sublinear is like aiming really low but
turns out that's even too high and
generally enforcement learning is not
learn about and from again traps so here
you have you know why I'm at where if
you go right you go to hell if you go
left you go to heaven and then there's
this corresponding environment where if
you go right you go to heaven and if you
go left you go to hell so you need to
decide and whichever decision you make
you're gonna be terrible in one of the
environments and then you have linear
great in that environment so if I choose
to go right in this environment my
regret will be linear and he'll be zero
and if you choose left as the other way
around so you you just have end up
having linear grad and there's
nothing you can do about it that's
terrible but if your environment is
recoverable and you have a policy that
is asymptotically optimal and plus some
additional assumptions on this
contraction then you can actually get
something you regret so in a sense the
these this kind of connects asymptotic
optimality with with regret and kind of
tells us that you make this optimality
property that we have where
asymptotically the agent learns to act
optimally is kind of it's one it's it's
a useful thing to have if you have if
you don't have traps right but yeah so
this is my summary slide for optimality
if you look at aixi Mike C is also like
it's based optimal because it's Bayesian
and Thompson's sampling is not based
optimal but based optimality as we saw
depends largely on the prior if you take
a bet prior than based optimal s and I a
bad notion of duality we also had paired
your optimality but turns out all polar
studies are pre go optimal so that's not
useful
we have asymptotic optimality that has
all these problems with traps and I see
it's not asymptotically optimal but
Thompson sampling is and with sub-linear
regret we get that in recoverable
environments but I guess this will be
the overarching question here is if you
have this theoretical model of strong AI
or this you know ideal agent that ideal
which what reinforcement learning agent
what do you mean by ideal and this is
the question we try to answer here and
it's not clear what the answer should be
if you're a hardcore Bayesian then you'd
probably say well this is actually
really all I care about in this fit
that's why I really like aixi or if you
if you know then maybe you might say
well I care about other like some
objective notion of optimality and this
is kind of the only one with that we
have
but yeah I I think it's not clear so
yeah so let's let's apply these results
before I get to the a safety part I want
to apply these these results to game
theory and for that I need to do I have
to explain to you what what is that I'm
talking about so here we have a multi
agent environment and I have n different
agents acting in that multi agent
environment and I each pick their own
policy and they communicate with the
multi agent environment and this could
be like some arbitrary repeated game or
more something more complicated and we
say that a policy is so each policy kind
of interacts with a subjective
environment the subjective environment
is basically everything else combined
and then it just looks like the
dualistic case that we had before maybe
it the agent and the environment just
takes action and we transfer receives
percepts and we say that this policy is
an Epsilon best response if the value of
that policy and the subjective
environment is except for an epsilon
close to the optimal environment optimal
value in that environment and if you
remember this is just this looks
remarkably similar to our definition of
asymptotic optimality
so if our agent is slightly optimal in
the subjective environment then it will
be playing Absalon best responses and in
game theory what you want to what you
care about is if you if everyone plays
epsilon best responses then you have
this epsilon Nash equilibrium and this
is kind of like one thing that you want
to aim for and of course this is not a
very strong notion of convergence and
there's so for example if you imagine
playing an iterated prisoner's dilemma
then one Nash equilibrium is everyone
cooperates all the time and if you stop
cooperating then everyone just affects
kind of like grim trigger style and then
you have this equilibrium where everyone
always collaborates but everyone always
defecting is also an equilibrium this
lots of equilibria and not all of them
are good but this is kind of like again
kind of like the lowest thing to aim for
because well it depends on what you do
on the counterfactuals but say everyone
plays the strategy I cooperate until you
defect and then I always defect so now
since its iterated if you defect once
our always defect and then basically you
lose a lot of value it's wrong as they
make it's good enough even for geometric
discounting should be enough so if you
only care about the immediate and next
step then you know really just playing a
sequence of one-shot presence dilemmas
and then of course it's not Nash
equilibrium you're right it's sorry
about that all right all right so I
already said that everyone so we have an
excellent excellent national Librium if
everyone is playing a best response so
how do we do this well again let's take
the business approach because we really
like that so let's start with a
countable set of policies and let's
start with a prior prior of the policies
so for now we just assume that we know
the game that we're playing but we don't
know the other players we have our set
of policies of possible policies that I
might play and we have
prior and then again we play act
optimally with respect to that prior and
the kind of crucial property that we
need here is that the space optimal
policy that you get is again in this set
of policies because if everyone is doing
that you want this is only gonna make
sense if the policy is that the other
players actually play are in this set
and the since they're doing the same
thing as as you are the you need that
the base optimal policy ends up in the
set and never ever so that that everyone
has this grain of truth and this is
really tricky and there's like my
examples where we knew had a grain of
truth so for example if you play if you
play the iterated prisoner's dilemma and
you play strategies where you take the
the set of policies where for every T
you cooperate and tell time sub T and
then the effect unless the other guy
defects first and then you just always
defect so if you take the set of
policies for every T then the base
optimal policy for any prior is going to
be in the set and then you have a grain
of truth but if you this is no longer
the case if you add other policies into
this tap and then this into the set so
if you also add the tit-for-tat policy
then the grain of truth probability
breaks and so what we kind of want what
we need is a large set that has this
grain of truth probability or let's say
let's put it this way why do we need
that so this is famous theorem from the
economics literature that if every of
every you know you have an infinite
repeated game all the players know the
game and in everyone's page and everyone
knows that everyone is Bayesian and
everyone has a grain of truth then the
players converge to an epsilon Nash
equilibrium so this is basically saying
as long as you have this grain of truth
BAE's works and you get your Nash
equilibrium but of course we already SAP
saw previous examples where bass
horribly fails and you can of course get
this for the same case if you just relax
any of these properties if you the
players don't no longer know the game or
if they don't know that the other
players divisions because then you
potentially have these traps where
somebody plays the grim Stricker is
trigger strategy something so for
example if you played an infinite
repeated mention pennies then you might
fail to converge to an epsilon Nash
equilibrium even if you have a grain of
truth and this is basically we both have
a coin and we secretly pick one of the
sides and then we reveal if it's the
same I win if it's different you win
this it's a very fun game so new york
top temperature said if it's common
knowledge that each player is Bayesian
yes I think so so in the if you look at
the way the the Panda theorem is stated
they don't say any of this explicitly
and people I guess people didn't expect
things to break once you like them
because they don't contract each other
because here you don't assume that every
player is Bayesian I mean Knight sorry
if you as you assume that not everyone
knows that everyone is gay patience if
the first thing had said if it's common
knowledge it each of that each player's
beta yeah yes so the point is that in
this M matching pennies basically what
you might you do is you give the players
a dogmatic prior that says if I deviate
from my stupid strategy then something
bad will happen and this something bad
will happen is of course outside the
scope of the matching pennies game
so they do not know the game there's
something bit yeah that's right but this
is know so you have to be uncertain no
this is this is right you have to be
uncertain with respect to the game
because the game where each player has a
strategy it's something completely
terrible for some kind of murder
sous-vide okay when matching pennies you
think the opponent can't really have
strategies that are bad for you even if
they have knowledge because the other
player might be each might have the
prior that if I do something good the
other player will do something terrible
and yeah so for example and if you go
back to the iterated prisoner's dilemma
you might think that the other guy plays
a grim stir the triggered strategy even
though for that particular task on
function that would be this that would
be based above them all because it would
be based optimal to eventually forgive
the other player but if you think the
other guy might actually play a grim
streaker trigger strategy then that I
could be like your hell that you're
trying to show your boy this is because
looking an interested game Nash
equilibrium you know what do you think
about the control factors so the novice
is sub-game perfect that's not even like
we're not even talking about that but
this the point is that you don't even
get the Nash equilibria because you
think about the contractures if you
don't require something greater if you
just assume anything about so I mean we
can talk about this later I can show you
the example basically you just use a
divided prior it's a prior over I guess
you'd have to we can talk about the
exact fire that you need I'm I don't
want to make a concrete statement
because I'm scared I will make the phone
statement off the top of my head but
yeah yeah but there's a positive result
so our public result is this class of
environments
it's called M raffle because it's based
on the effective Oracle's that benya and
Jessica came up with and that class
environment contains a grain of truth
with respect to any computable priors
based optimal policy in any computable
multi-agent environment so basically you
have you have this rich class of
policies that we wanted they contain all
the computable policies and they're
additionally they also contain these
Bayes optimal policies this is I'm kind
of being sloppy here so basically since
you're in Miami and your subjective
environment contains both the mostly
agent environment and also the other
policies you kind of want to fuse them
together so you want to have you want to
be able to plug in all the base optimal
policies into your multi agent
environment if you think about this this
picture here you want to if this is here
you want to be able to plug in any
Bayesian agents here and then be the
whole thing together again in the class
but again I could also treat that
separately look at the class of make
Sigma's and then look at the class of
Asians agents and then yeah put them and
I'm basically I'm saying that that the
base optimal policy for that should I
should be able to plug these in here and
yeah so that works and the reason why
this is really difficult to get is that
if you think about Ike C actually
actually assumes that the environment is
computable because nines to predict all
the computable environments but itself
is in computable so it's elevated above
the environment and if you put another I
see in the environment then this
assumption that the environment is
computable is violated and this is kind
of what we get around here and then you
have this Bayesian agent that manages to
do self reflection in the sense that it
it doesn't believe that there are no
agent like it in the environment so it's
not I'm not claiming that this is like a
solution to the self reflection problem
but at least not it's no longer
obviously obstructed and then you also
get that each of these environment is
actually limit computable so they're not
like very crazy constructions and then
the in the end we get the theorem that
says you have these limit computable
policies and these are going to be
Thompson sampling card policies so if
you remember I told you Thompson
sampling is asymptotically optimal
meaning that along as long as my
environment my subjective environment is
in the class I will learn to act
optimally and since my class contains a
grain of truth I know that the Samsun
sampler will be asymptotically optimal
in the class and in terms of game theory
that would mean that my Thompson sampler
converges to an epsilon next Nash
equilibrium
yeah sorry we'll play in Absalon best
response and then if all my policies at
Thompson samplers and my multi agent
environment is computable so it's also
in the class then everyone converges to
playing a best response and then that
means in the end you'll have an epsilon
Nash equilibrium and this is this is
remarkable because that was not the case
if you just have Bayesian players you
need this extra explanation the
exploration that you get from Thompson
sampling yeah and I think this is this
is a really cool result and there's no
yeah observe that there's no conditions
on the horizon or this clown function so
each of these power policies can have
any discount function in particular they
can also have different discount
functions there's no restriction of what
the environment can look like it could
be an infinite repeated game it could be
just generally any computable game and
yeah and in the end you get this Nash
equilibrium which you know international
you Librium equilibria are not the
perfect thing to get it's just something
you've worked really hard to get that
okay no it's being sometime perfect
would mean that you also learn to
predict optimally off policy
so basically off policy you would also
act up to money not quite so
the vision yeah yeah maybe we should
talk about that later
yeah so let's get to the a a safety
point that I'm sure you're all very
excited about and so what so now
basically what I'm whatever I told you I
have told you about how our the
currently best algorithms that we have
for AGI dqn and and consorts how they
fall short in to of of being a solution
to strong AI in principle and then we
try to understand what strong AI means
and particularly what like what an
optimal agent would be or what our ideal
agent would be and this kind of provides
us with a formal model of strong AI and
what do we do with that and in terms of
failure safety approaches I'm gonna be
bold and just split them into two
different categories and one of these
categories is going to be bottom-up
where you take practical algorithms like
you reinforcement learning algorithms
and then you study you try to study air
safety problems on them using time
models or demos or something like we did
like we saw yesterday with Dylan's stuff
when you try to solve this shutdown
problem in MVP and my point is that the
real world is not an MVP and we have to
look at bigger that the big picture and
so the kind of firms that I want you to
look at is top-down problems where you
start with theoretical models and you
look at abstract problems and then you
prove theorems about that so in in
particular in this case you have these
models for strongly high and
you can try to see what happens if I put
them in a white heading problem what
happens if I put them in if I wanted to
make them interruptible so this is the
stuff that Stuart I'm strongest news new
gun and how do we what do we learn if we
do that and what can we how does that
transfer to the things that are being
used now so a lot of people have already
done this so just mentioned Stuart's
paper there's some stuff that Tom
Everett has done on self modification
and decision theory and why heading and
there's a lot of things that lower ISO
has done and I think these kind of
formal investigations can provide us
useful insights to these sorts of
problems and I think we should do more
of that and the aim of this talk was to
give you the mental tools and the
mathematical tools to do that
so if you start with like your formal
model of strong UI where you say well
let's just say I have strongly optimal
policy and now I want to like break that
in some highway or apply that to my
formal modeling problem of my safety
problem and let's see what happens and
yeah so these are just like a number of
things that you could do for a lot of
stuff this is not conceived conclusively
solved and there's lots of open
questions and I cannot even form a
finished formally modeling them but yeah
I think this is useful there's also some
open propene questions for just this
general model so the algorithms that
I've talked about here all model based
meaning they have an explicit model of
the environment and can use that model
to plan or explore but in practice these
algorithms are usually model free so
we're probably going to see them being
more on model based as
we come closer to you strongly i but
yeah so I think that our former models
should be sufficiently close to what's
being is in practice so I would like to
have something more model free next
thing is that this model is dualistic so
if you remember the the picture with the
age in the environments they're
completely separated of course that's
not true in the real world and we need
to have a model that accounts for them
it's really difficult there's another
thing that this from framework doesn't
formalize is to have improvement just
have an agent that is a policy and the
policy just specifies what the agent
does in any second situation and there
is no like soft modification going on
and of course the model assumes infinite
computation but I think this is I don't
think this is too bad because it's
supposed to be a theoretical model it's
not supposed to give us a practical
algorithm and it's needs to abstract
away from the implementation the details
in a way that where we just say well
let's suppose we have really good
learning algorithm like this in here see
what happens and as you saw things
they're already difficult enough if you
assume infinite computation yeah so this
is the list of like mathematical mental
tools that I think can be useful in
these things so you have this
exploration exploitation trade off how
much would you explore do you want to
explore enough to get asymptotic optimal
tea do you want to explore it less
effective Verizon is what you know how
much you need to plan ahead and how much
you need to explore in order to
understand what's going on what are the
consequences of your actions
there's the details between on policy
not policy I we know we saw that you
learn base learns on policy but it
doesn't necessarily learn off policy you
need more exploration for that these
algorithms can be model-based with this
mother Fraser this is a second very
classical reinforcement learning
distinction you have the
recoverability assumption or generally
environments that don't contain traps or
environments that do contain traps and
then yeah asymptotic optimality was just
a formalization of what it means to and
learn the entire environment not also in
order to act optimally
yeah and finally these reflective
Oracle's which enable us to have a
former model that at least where the
agent can be inside the environment yep
if you want to read more about this I
finally finished my PhD thesis it was
printed yesterday you can go to my web
page and download your own coffee and
basically that it tries to explain all
the things that I explained today and
just a lot lots more words and formulas
and proofs and so on and yeah you can
also talk to me and I'm really happy to
chat about aixi and all these things
let's say some time for questions yes
sorry yeah I was writing this up on the
board is he as you were talking about
the matching pennies game because is
this this is not matching pennies
anymore but it's matching pennies plus
opportunity for new which I you together
should make this like minus 11 so it's
no longer even in Nash equilibrium but
right we're we're both of the players
believe that if they deviate from this
set of moves which is not a Nash
equilibrium because it has each of them
point half the rounds and no introns are
going to win and you think if they
either deviate though the other will
like to do forever so that I think
that's an example what you're talking
about but where you're actually thinking
saying that like there was a case with
matching pennies without news sir you
need I mean this was this was a correct
an interjection you need to be very
careful in the way you set up your
environment class and I was I was being
imprecise about this so I think in the
example which we use we just use the set
the Class M raffle just all environments
are computable and they're reflective
Oracle but you can restrict this further
and like here
you mind so in your case you might
actually be playing a matching pennies
game but you think there's some
probability that you might instead be
playing this game and then of course you
afraid that the other player you might
nuke you and then you can set up this
situation okay but you have to have this
your class has to be such that these
traps exist right in this case the trap
would be that the other player to start
sneaking you in though in practice this
can't ever happen because there's no new
go option does that make sense yes
other questions alright thank y'all
again in here