What Does (and Doesn't) AI Mean for Effective Altruism? | Owen Cotton-Barratt | EAGxBerlin 2017

fine yeah I guess some of you may have
noticed that within the eventual tourism
community there's a reasonable amount of
attention given towards artificial
intelligence and I want to talk about a
bit about that and what I think it means
or what we should be doing in some cases
conclusions I think are warranted to
fund that and assume is better feeding
intimates first is the idea that
actually from a moral perspective most
of what matters about our actions is
their impact on the bottom term and how
that plays out then there's some
empirical assumptions that there's the
idea that artificial intelligence might
be the most radically transformative
technology that has elegant about the
better of us is the idea that because
it's a the better or worse the
influencing the way in which it's
developed and whether that is beneficial
or not could be a major way to impact
the long run future and finally an
empirical claim that timelines on the
scale of decades for the development of
badeley transformative artificial
intelligence could make back in the 18th
century
ok you may or may not believe all of
these things for this talk I'm once you
take them as such
going in and I want to explore if you do
believe these what does it mean or what
we should be doing I think that one
cartoon of you says well if you think
this we should all stop whatever else we
were doing and just go and work directly
on that problem and and try and make
sure that we get good outcomes by
learning how to code AI that's going to
be good for people I don't agree with
that cartoon and things I'm not to do
with this talk is to get into an
explanation of why that is so if we're
thinking about artificial intelligence
one of the best uncertainties is
actually on what kind of time scale
should we be expecting this what can we
say about about timelines and the first
thing that we notice if we look at that
is that actually predicting technology
is very hard particularly predicting
qualitatively new technologies sometimes
we have things like Moore's law and we
do curve fitting but if we have
something which is fundamentally
different from what's gone before it is
extremely difficult to be confident in
dates or that and I think therefore the
uncertainty and significant uncertainty
about timelines is appropriate and this
is borne out if you ask experts this
barb is from a survey of machine
learning researchers and asking them but
the dates by which they think that
almost all kind of bits of human
cognition and work may be automated and
there's just a lot of uncertainty
there's a lot of uncertainty between
researchers and something
woman's there but if I can summarize it
I'd say that the consensus for you to do
is that we don't know it is plausible
that it could happen early and it's
possible that it could happen and early
meaning within even the next 10 years
but it's plausible that it like quite
possible that it could be 100 and then
so that I want to investigate that
because I think that we want to do
different things if we conditional on
these different timelines in order to
have a conversation about that and to
talk about it I want to look at some
discrete and scenarios which represent a
few particular points within this
continuous space the reason I'm looking
at the sweet things other than just
talking about a continuous distribution
is that it's easier to think about and
hold discreetly in the mind it's also
much easier to communicate and have a
conversation about what's going on here
and people can push back and disagree if
they can as well so the first of my
scenarios is where things are actually
happening imminently on the timescale of
just a handful of years this if we
condition on language there let's assume
this is the case then it's more likely
that we can make good guesses about who
will be involved in developing radically
transformative AI may be working go and
talk to them and help them work out how
to do it one it's also the case that if
your plan is okay I'm gonna do a
master's in computer science
I'm going to do pH energy learning and
then I'm going to go and get a job in
government and might try to be there AI
policy person and in 12 years I should
be well-positioned and radically
transformative AI is coming in five
years that's not a good plan and so we
all this scenario it's beneficial to go
to things which are significantly
shorter and more time active if we look
at a slightly longer tsunami ago I
pulled this generation and I chose these
scenarios be lovely this bit of time
because I think that the actions applied
are different and on this kind of time
scale I think that plans like the one I
just mentioned where we you're kind of
building a career or we've tried to set
people up to be in the right position to
help advance what's pretty well I also
think it's the mic kind of timescale
characteristic time scale for building
academic fields you can hash out an
argument you can get the conclusions
accepted you can get them to filter
through to decision makers over this
kind of time scale if we look longer
again the scale at about 40 years and
people the time it's very hard to
predict but I think where any of us in
this room will be at a scale of 40 years
time and it's also paradigm shift within
academia you could persuade people of a
thing and then find that actually
there's been kind of two big shifts in
thinking later down the line so we
shouldn't make too many assumptions
about things that we're doing now having
implications for what we'll be doing
further down the line in this timeframe
therefore it's better to build
institutions which push towards getting
things light long and just go okay
as quickly as possible and if we're that
means we want to be persuading people
really impede things which are solidly
correct we want to be putting people in
positions to if they're making mistakes
do the self correction later if we look
out even further something at the
timescale of a century then it's very
hard to predict causal pathways between
now there's just a lot of things and
that could happen and so very precise
plans don't look so good
I think instead we really want to focus
on building Lord institutions I think
actually and the effective altruism
community could be one of those
institutions it's about empowering
people in the future making them our
epistemic superiors and so that they are
positioned to work out what's going on
what to do and act on that and it's also
a time scale over which I said
artificial intelligence might be
radically transformative maybe there are
other things which will be quite
transformative to our society happening
and I think the longer the timelines
we're looking at the more important it
is to pay attention to other
transformative technologies okay I've
outlined these I've argued that we want
to take different actions or conditional
on these different timelines I also said
we don't actually know which of these is
correct how are we meant to verify that
and pull together and come up with
inclusion of what to actually do I hear
there's some strategies we could take
first is just try not to take you say
well it's Quebec to be pretty uncertain
so we'll take actions which look like
they're going to be good for all of
these different times and that is a bad
idea because it might be that we miss
the best actions that are available
because the best actions perhaps you
only spot if you have a particular thing
in mind and we think how can I achieve
that so we could look at which of these
things as we do we think it's most
likely and act on the basis of that the
problem with that is that it ignores the
fact that we might be able to have more
influence over some of this in our ears
than others if we think we have a lot of
influence over the 15 year timeline but
the hundred year timeline is more likely
we just have no idea what to do about it
in that case you may well be better
taking action on aiming at the 15 year
timeline this generation bother than the
distant case and but there's a problem
so that's against that we should be
aiming for whichever has the highest
likelihood x metallurgic where it's just
kind of the most influence we can have
and if everybody equal really good at
following that strategy I think would be
okay but I didn't really endorse it as a
way of thinking about this because
leverage bearings
Leslee people barely between individuals
and so it isn't the case we can just
have a conversation of which of these is
the best to do and now we're all going
to
focus on the Hat party because there's
diminishing returns as you put more and
more effort into one particular strategy
that could be corrected shifting to
other strategies partly just because the
opportunities that come along barely
there can be different between people
they can sometimes be different so for
individuals as well and so what I think
we should normally do is think about
what portfolio of actions we want to be
taking as a whole community and using
that to try and guide what we're
actually doing so the way I think about
that is that we should collectively
discuss a lot of these uncertainties
which feed into the decisions and the
relative probabilities the amount of
leverage we have in the different cases
the diminishing returns on more work
going in we should use that type of
discussion about what does the ideal
portfolio of work over these different
strategies look like then we can
individually consider how we think that
people are acting and how that deviates
from the ideal value and we can also
think about what our personal
comparative advantage is and whether
that means that we should be doing
something in particular so I expect that
most of you are familiar with the idea
of college but here's quickly how I like
to think of it and if you have how many
come aligned in one and you have three
tasks that you want to
and each task leads one person acting
like Hawaii is the best of everything
but you can't send my name to do all the
lens so you want to line people up so
that you're making relatively good use
of the skills that everybody has given
the opportunity costs have given the
other things that they might be do I
think that this is a heuristic that we
can use at different levels and can be
useful general thinking tool so we can
think about it at it as individuals if
you notice that you get confused when
you think about AI but you're quite good
at thinking about how to design good
institutions then that probably means
that you're better placed to be helping
on bit longer timescales
than the shorter ones we can think about
it other scale groups and organizations
there are different actions which better
groups with different groups are better
placed to be taking and particularly as
they build specialization - they start
doing things now they feel that
expertise and there helps feed into what
they should be doing in the future we
can also think about this across time
people in the past did some things we're
going to do some things people at the
future will do some things now we can't
change what the people that the past it
but we can perhaps people in the future
can react to what we do and so we can
say what is our comparative advantage of
by being in the present in 2017
compared to people in 2027 2037 and
there were two things broad conclusions
that are all kind of trying to apply
that reasoning to this case of the AI
status one is that I think it provides a
bit
reason to focus on the imminent and miss
generations scenarios because if a is
coming
if transformative a is coming in the
next 15 years people in 2037 can't do
anything about it and we have this
advantage of being actually in a
position where we're the people who can
take actions to try and help although
still obvious if you look at
conditionally call the longer-term
scenarios you can sense a what is
particularly useful to do and I think
that that's often the building the white
community and building the white
institutions button and trying to tackle
the problems directly where you actually
have an advantage when you live it at
the time because you're able to better
see exactly the naturally products and
so that fed into some of the things I
was saying earlier about what I thought
their strategies were conditional on so
here's a couple of board thoughts on
this plot by the approach I think that
most projects that people aren't taking
should have a main scenario in mind
because this allows them to enter into
that mode of thinking where you they can
be quite goal-directed and they can work
out what is it that we actually want to
achieve here and to the extent that
there may be better things that you
think of when you go into that matters
this is why I didn't want everybody to
just say well we won't take use and I
think it's often good to take use maybe
even try out a couple of different views
and work out what seems particularly
good for those cases and it's often the
case that most individuals should have a
main scenario in mind as well because
the comparative advantage Barry I think
that's won't always be the case
that said do be careful about things
which might be bad all other scenarios
so if radically transformative enemies
were definitely coming in five years
time it might be that the right thing to
do would be to go around the same
everybody panic I actually but you might
think look this is the way to persuade
people that this is coming very soon we
should do that and then we'll actually
get more attention on to this problem
even if that were correct conditional on
the imminent timeline scenario I think
it is a terrible idea because if we're
not in the imminent timeline sailing
here but instead radically
transformative AI is coming in twenty
five units then that comes along and
people try to generate momentum around
it everyone will say you know we've
heard that before this turns out to be
nonsense so it's a case where the side
effect of the action was not just
neutral but actively bad for one of the
other scenarios and I think that if
we're splitting up and doing different
bits of work we want to be cooperative
and do things which are helpful or at
least not bad for the things other
people are doing and I think that the
particular example I use there actually
realizes something which is helpful
across lots of different cases is having
good mechanisms or arriving at the truth
and coming to the two things about the
world and here's a cartoon of how I
think about that the truth is a jigsaw
puzzle and we all have little pieces of
it walking around with those and we want
to and assemble this into a larger thing
which is going to tell us how attacked
and so we want to build institutions
which good at doing this which don't if
people insert pieces which in collect
don't just run with those and a better
than excluding those I think one thing
we can do is think about what kinds of
institutions and we can also think about
this at a local level and if people are
saying things they have ideas and they
communicate to others one thing that
happens is we end up communicating
things that have been communicated to us
but that doesn't necessarily track which
ideas are most truthful it might be that
instead it tracks who is the most
charismatic and who is good at
persuading people of thanks this is
something that we want to be careful
about we'd like to have things which
don't push us towards this dynamic and
instead do help us keep track and
correct this and move back towards the
truth wherever possible I'm just
gesturing at the type of things that one
can do within this space but I think
that so one norm that we can have which
is quite local that we can all invest in
is communicating with the business or
beliefs
don't just say green puzzle these say
and well my friend Anna told me good
puzzle piece and I think she's quite
smart and I trust her reasoning and
that's why
if that's why I deliver and/or I came up
with this idea in myself or suddenly
told me this this is the argument that
they gave it seems to check out to me
the communicate as much as possible the
different steps that actually led you to
beliefs and then in your audience gets a
better picture of the extent to which
they should trust not just the idea but
the whole process leading up to that and
I think that that is something which can
help us be self-correcting okay coming
back what should individuals within the
effective action community do about my
special talents I think I have three
main things there first of all it's good
if they take a bit of time to consider
those assumptions and maybe some of you
already have but I don't want to argue
people should just collide early I find
these possible people and blindly
believe me can actually go away and
think about them and if you end up
disagreeing great talk to people about
why you disagree try and work out
they'll be some of the aspects of
thinking that I presented here which may
be useful for you working out what you
think you should do even if you disagree
with me about some of these things and
but you can also perhaps persuade other
people of that perspective and make them
work what maybe I'm wrong about some of
these things if there's well at least
you've just never spent a bit of time
thinking about I do recommend it I think
they're all quite important questions
that feed into the question of how we
should be helping the world and so I
think they are worth that attention and
secondly I think people should consider
their personal comparative advantage I
think it is valuable to have people who
are very good technically were the kind
of people who could be top PhDs in
mathematics or computer science going
and spending attention to the problem of
the technical problems of how to be able
to the bus being beneficial I am but I
they were just like for most people that
is not going to be a path which is good
for them and so I don't think we should
I think they are better thinking about
how what other methods could I have to
help support work on this that might be
other types of actions aimed at short
timelines it might be actions aimed at
longer timelines I think that there's a
broader base set of things that we
should be doing conditional on the
longer time lines and so there's a
broader set of skills which were
particularly useful that and then
finally help promote good community
understand X because I think that this
is the type of thing which is likely to
help us go well as a whole community and
hopefully build a long and flashy
occasion
thank you so much Owen I just saw Dennis
tonight I was actually working and I'm
very excited about that so a lot of you
have been asking questions and a lot of
other people voted some questions up so
the first question today is what might
be Dai driven transformation we are
talking about gosh that's a big question
I I'll say some words also point people
to some online content about this
because I think it is just a long topic
I like all Cristiano's essays three
impacts of machine intelligence I
recommend people Google that it's not
that the long essay that it does talk
about a number of things
luckily I think that global weights and
the developments of new technologies
might massively accelerate and it's also
the case that right now can all large
processes and the work is in the world
involve human decision making and AI is
not just an automotive technology it's a
fully general automated technology and
that means that we're going to be able
to remove humans from more and more news
and I think that's something that we
want to be careful about so just does
everybody know about slide o or is
somebody in here in the room that did
not attend the opening speech okay
[Music]
your products a new phone just one line
and choose the room you're in and then
you can like ask a question you can do
it anonymously or with your name doesn't
matter then you can upload the questions
you're going to ask the most questions
or best efforts yes the next question is
from Adia
how can we estimate the likelihood and
leverage of materials right and I think
the combination of things that we can do
here we can try and take an inside view
perspectives where we look at our
understanding of the technology we look
at what way to do technologies and tend
to change how often do we find very
sudden large breakthroughs the website
ai impacts has quite a lot of good
analysis at this time yeah another thing
that we can do is we can say let's try
and take the outside view let's look at
what other people think
let's go and look at what the experts
think which is why I included that glove
you can go and read the paper and try
and find the perspectives of what a lot
of people are working on AI think I
think ideal might be some combination of
these different approaches that surveys
people who are working on AI but just
because they're working on AI they don't
actually spend necessarily a lot of time
thinking about this question it might be
better to find people who have spent
quite a lot of time thinking about this
question and go and talk to them okay
menu one last question and please send
demarcate transformative AI fractional
technology is there any meaningful
difference
yeah I love these questions basically
all of these questions could be a large
research field by itself so I would just
be the inadequacy of my answers I think
that in some sense a I may just be
another another technology but by
transformative III mean something which
is probably at least as big a deal as
the Industrial Revolution was in some
sense the Industrial Revolution was just
more technologies but it led to a
qualitatively different world afterwards
than beforehand I think that two big
features about AI I've alluded to both
of them all already are one that it's
something which can actually let us take
humans out of more and more loops and
that means that it's more important to
be careful about the direction it's
going in advance rather than just say
well we'll develop it and then we'll
work out what to do the other is that
because like the Industrial Revolution
sped up over it's quite not it I think
it could be the and automating a lot of
cognitive tasks also speeds up
progressing and future growth rates a
lot and that can mean that we have more
thinking time now in a box of getting
into a regime where things are going
very quickly and this is a reason why
it's important to pay attention to
implants relative to most technologies
thank you so much the time is over and I
know there are a lot of more questions
gonna stick around here maybe people can
come just like up here and ask some more
questions thank you so much Mary
[Applause]


Logical Induction: Progress in AI Alignment | Andrew Critch | EA Global 2016

so this is joint work with primarily
Scott garabrant who is here Scott can
you indicate all right so this man over
here is like the primary source of the
ideas in this paper and then the four of
us have sort of like crowded around
Scott as a sort of carbon rod
arrangement to like absorb the neutrons
of like logical induction that have been
coming out of him into a controlled
reaction that like people can eventually
understand so you can I guess I'll
eventually have these slides or some
slides very similar to them probably on
my website so if you want to look at
them you can google Critch they might I
don't know if they might end up on
intelligence that order not but that's
that I'll just wait uggs wait a little
bit looks like there's like oh yeah fire
fire hazard we don't want to kill all
the people who understand logical
induction huh yeah let's do lights yeah
blue all right so rough plan for how the
stock is going to go is first i'm going
to say what logical deduction is seems
important then i'm going to say a little
bit about why we care about it because
it's intrinsically super cool like you
don't you almost need to care about
anything to care about logical deduction
but it also we came to it from from
caring about long-term concerns about
the control of highly advanced machine
learning systems or artificial
intelligence and so we'll talk about
that then I will I guess probably I
should be over here in order to be able
to point with my left hand since this
thing does it work so then I'll then
we'll get into like the technical stuff
and then I'll like try to save a little
bit of time at the end for
interpretation like what does this all
mean who are we in the universe okay so
here is a little table that I'd like to
use to illustrate what we mean by
logical deduction so suppose
I asked I I've got a ten sided die like
this and I roll it and I ask you what he
what's the problem I role that I cover
it up and I ask you what is the
probability that that died landed seven
so that's probably détente equal 7 and
if you think about it for a minute
you'll say ten percent right oh yeah can
you guys can you guys nod if things I
say makes sense and like make weird
squinty faces yeah okay cool make weird
squidgy face as if it doesn't make sense
alright so if you think about for a
minute you'll say ten percent and if you
think about it for a day you'll still
say ten percent if I've still covered as
a dye is still covered you will still
think ten percent and no matter how long
you think about it you'll think ten
percent at the opposite end of this
spectrum you can have a question like
hey what do you think is the chances
that the tenth digit of Route 10 is
seven so if you think about it for a
minute you'll probably say ten percent
because you don't know right but if I
let you think long enough and maybe if
you've got some paper you might be able
to figure out that it's probably not
seven like you lose some calculations
and you'll be like I don't think it's
seven but like maybe I screwed up but I
really don't think it's seven and then
if you think for infinity which I
apologize sorry for leaving you in the
room with the paper but you will
eventually converge on an entropy zero
because it is not seven and then there's
something in between which is what we
see going on in machine learning all the
time which is where you've got some
observation like maybe right before the
die was covered by the cup you got to
see like you know you got to see it for
a brief moment while it was like
tumbling does this make sense for this
picture I don't know if you can see a
little bit of blur there right so you
see it and at first you don't have time
to think about it you're like yeah ten
percent right but if you think about it
longer you can like run some simulations
if you're a human or if you're in a I
you know you might run a bunch of
simulations and realize actually based
on what I saw going in it looks like a
little more likely to be seven because
seven was like looked like it might come
up does that make sense so you have to
think about it to change from this ten
percent to this fifteen percent and if
you thought way WAY longer maybe you
could get
even more precise but eventually would
like top out like you maybe still don't
have enough information about like the
shape at the table under the cup or
something so you still can never be
quite sure what how it ended up so this
is a this is like a mixture this is one
kind of uncertainty you could call it
empirical uncertainty and it refers to
uncertain do you have because someone
didn't tell you some information or you
didn't observe something in the world
this is a different kind of uncertainty
its uncertainty not because someone
didn't show you something but because
you haven't had time to think it you're
not waiting to do an experiment or see
anything in the world to know the digits
of route 10 you just need to think about
it and then this is something in between
you see something and you have to think
about what you saw okay so probability
theory if you go on Wikipedia and look
up probability theory you'll have see
some axioms that tell you how these
probabilities right here should relate
once you've fully integrated all the
information that is presented by your
observations there are certain coherence
laws that probabilities have satisfied
like probability of a and B should be
less than probability of a right things
like that are probably of a and B plus
probably a or b has equal probability of
a plus probably be you guys seen stuff
like that right so there's some rules
the probabilities have to satisfy but it
takes a little time to satisfy those
rules you have to think you have to
think about stuff to like figure out
what your probabilities are going to be
and while you're thinking you know you
might not satisfy all the rules you need
like some time to like get your
together and have coherent thoughts so
logical uncertainty and logical
induction is is more about the
horizontal movements in this table so
I'm going to refer to so there's two
there's two interesting questions you
could ask that are not answered by
probability theory one is that like what
are what are like good ways for
probabilities to change over time like
what's the definition we probably theory
sort of tells you like the definition of
like good coherent beliefs once you've
thought about stuff but what's the
definition of like a good way to change
your beliefs over time as you think
simulation predictable so the
interesting thing here is you're not
accumulating any information you've
already accumulated the information
right here when you've seen the snapshot
and now you're just thinking so that's
really important distinction the like
the step from here to here would be
accumulating information or rather from
here to here right whereas in the
information theoretic sense information
refers to you not observed random
variable and here you've already
observed as much of the thing is you're
going to observe and there's this other
thing you have to do like so you see the
snapshot and then you have to think to
like process the information yeah then
did you speak when you said for snaps or
two maybe you ran some situations well
depends on how you think right so maybe
you think in simulations maybe you think
with some other heuristic maybe you just
write down some differential equation
and just say I don't know how you do it
maybe you just feel when you humans like
just kind of feeling something out-like
how does it look like it was going to
fall it's kind of like a simulation your
head yep yeah wave throwing that
dividing line between say computing that
gets you to have you seen the blurry
seven different we know the photons to
your menu representation versus the line
of now I have the information yeah you
can you can to draw the line and but the
interesting thing is in our theory that
you will get into later you'll see that
you don't have to draw the line that's
fair to say you can mix and cook when we
start mixing and axioms with
observations or something we're not
going to really talk much about it but I
think the answer is that you don't have
to build the line so there's one
question is like what are good ways of
probabilities changing over time when
you're not making the observations
you're just thinking about stuff you've
already seen ok so what rules should
these things satisfy then there are any
really there are many rules in the
literature or papers about like what's a
good way of doing that and another
question is obviously there's something
intuitive about fifty percent being like
a bad answer here right if I'm asking
you to guess probability of a digit
you're going to probably use ten percent
until you've had some time to think
about right so is there some theoretical
framework that we can we can define that
would that would imply this intuitive
judgment of hours that
you know fifty percent would be like a
dumb answer to start out with here and
like ten percent would be a good answer
to start with right and then you you're
still going to have to think to get 20
or 100 eventually because this is
completely determined by your
observations the answer is completely
determined by things yard see okay so
does that make sense logical this like
logical induction stuff is about like
hey what's a good place to start and
like what's a good way to evolve your
beliefs over time when you're not
getting the information yours processing
what you've already seen cool yeah for
the fifty percent thing isn't that just
like equal outcomes number of outcomes
yeah so you could say you know if
there's an equal number of outcomes
maybe they should all have equal
likelihood but should they I mean why
not just have seven as fifty percent
likely to be true and the other nine are
each you know 50 over nine percent
likely be true like you'd need to make
some argument and you'd like to
formalize that argument we like that
argument to be like written down in math
so we can do proofs about it cool so
that's what we're gonna be talking about
and I like to say that well we'll get to
that so the question is now why do we
care about this thing that's the quote
the question is how do we do the purple
stuff and why do we care well the reason
I care and I would you know different
people at Mary might have different
nuanced descriptions of like why they
care but we want to reason about highly
capable AI systems before they exist we
want to develop theories of safety and
control and alignment for very very
powerful thinking machines that could do
a lot of good for us if we like a name
to move the right direction and it'd be
great to have those like aiming theories
before the super powerful thinking
machines are like a round so we'd like
to have theories of AI very very strong
AI systems before they exist or rather
like how to use them and how to direct
them so so we want to how can we it's
this hard problem right you often hear
people say like we can't really work on
a I love it because we don't have AI yet
we don't have highly advanced machine
learning systems we don't have and if
you're in jessica's talk I'm sure she
may be convinced to you that there are
things we can talk about now but if
you weren't I'll convince you that
there's even other things that you can
talk about now because we can model a I
whatever it is it's going to be good at
stuff we could just develop a theory of
things that are good at stuff okay and
then try to like decide how we're going
to aim the thing that's good at stuff
for various values of stuff so what do
we think what do we already have
theoretical models of good at stuff for
well choosing actions there's this
rational choice theory I don't know if
you've heard of the Von I'm raise your
have you heard of the by knowing
Morgenstern theorem awesome right so
we've got this theorem some of the spark
yes spark students know about it so
there's this theorems that tell you you
know if your if your choice is between
Gamble's or your choices between random
outcomes satisfy some very modest
conditions that seem very
unobjectionable then there's a function
whose whose value you're maximizing the
expectation of so there's this like nice
theorem that helps you think about what
is it you know an agent or a system
that's really good at choosing between
actions well if it's really good at
choosing between them in a way that's
like not self contradictory it's going
to follow this vm theorem and you know
modulo some adjustments to the BNF
theorem that might want to make but part
of the vm framework is this belief thing
that the agent has right so we had this
rational choice theory and you've got if
you look at the axons of vm there's
these probabilities that it believes ok
so now we need really to catch this out
we need a theory of like how
probabilities and beliefs might behave
and we have that called probability
theory and there's nice theorems like
bayes theorem to tell you how when you
get new information how your beliefs
should adjust after you think about that
information so this is sort of what
you're thinking of earlier when you're
saying yeah accruing the new stuff but
similar to how rational choice theory
depends on probability theory to
formulate it to be formulated or to be
realistic probability theory depends on
this thinking step this step where you
got the information and now you have to
like figure out what your new
probabilities are and so implicitly in
probability theory there's this like
missing component which is
you know how do you figure out what your
probability howdy what what do you think
before you're done thinking what are
your beliefs before you've finished
figuring out what they are and there
wasn't really a theory for this there's
been like various fragmented attempts at
like making such a theory but we'd like
to say an AI is going to be good at
rational choices it's going to be good
at probability theory and it's also
going to be good at whatever the heck
this purple stuff is so we'd like to say
good we like to define good at purple
stuff so that we have a model of a I
before it exists and there's all kinds
of other features of AI that we'd like
to have models for and this is just one
such feature good at purple stuff good
at logical induction and so the way that
we're going to use these words logical
uncertainty is like the state that
you're in before you finish figuring out
the logical consequences of your belief
of your observations so you are
logically uncertain about the digits of
square roots Hannah you know the
definition of square root 10 so you know
everything you need to know in order to
logically infer the ditches over 10 but
you're logically uncertain because
you're you haven't finished doing the
logic that you need to do to figure it
out does that make sense so largely a
logical uncertainty is the state that
you're in and then logical induction is
how you deal with that state it's how
you refine your relief over time cool so
any questions about like what this means
and like why we care about it before I
go on to stuff yep you mean like a few
like a future AI system that's like not
that good at logical induction yeah so
there's different ways it depends on
like the nature of the not good at it so
if you can say a particular failure like
a particular bias that will have then
you can say okay it's going to make
these types of errors but even to define
a bias you need to say what good
judgment is and then a bias by
definition is like deviation from good
judgment so we can't even like before we
define good judgment we can't even begin
to start to talk about your question yep
Scott Garrett Brandt was too young I
don't know
that's my answer uh uh yeah I mean I to
be honest I didn't I had I I thought
about this a lot in high school I had
lots of math edition friends it makes it
like when you're when you're you're
seeing some talk about the Goldbach
conjecture you know when someone says
like oh we discovered like this new
class of numbers that are all
expressible as sums of primes right your
belief in Goldbach conjecture goes up
right but like how much should it go up
right you've observed no new empirical
information you knew what primes were
you knew the definition of addition so
logically go back unless its independent
infer you know it is just it's just a
logical consequence of stuff that you
know nonetheless you go around adjusting
your beliefs about it in response to
other other like you can think of the
mathematical community itself as like a
an algorithm that's like proving stuff
and discovering stuff and it's like
updating its like intuitive beliefs
about stuff and math editions a lot of
my friends in math at some point have
this crisis of like what are we doing
like what are the principles we're
following when we have these like fuzzy
beliefs about which theorems are true or
false before we even prove them so then
you go through this crisis of like
trying to define what are the good
principles for like having your beliefs
about conjectures and then you give up
because you're not scared Scott
garabrant so this happens anyway one day
Scott garabrant doesn't give up and we
have logical inductors so so you could
try to define what does it mean to be
good at the purple stuff good at the at
refining your beliefs while you think
about them and a lot of people have
thought about this before there's
various criteria in the literature that
sort of just you're at like well here's
one here's one property that we expect
good logical on certain reasoner's to
have we want them to be computable we
want them to be overt we want them to
eventually satisfy probability laws
after they think for a while so if a
implies B eventually we want the thing
to have probably very less and
probability be so there's all these
other properties I'm not really going to
get into them because we'll see them in
a lot of them in finer detail later in
the talk but
want to sort of point at you know since
really at least at least as far back as
the as the 40s you've had people talking
about criteria for good logically
uncertain reasoning and but there there
are sort of informal kind of like you
saying like oh like you know equal like
bunch of outcomes should all be Khalil I
CLE until you've had reason to think
otherwise but you have to figure out
first of all that the outcomes are
mutually exclusive in order to even
apply that so there's some thinking you
need to do first and so part of part of
the progress that we've made is we've
adopted a formalism where we can state
versions of a lot of these properties
and then check if their possible or not
or we can prove that they are approved
that sometimes they're mutually
incompatible so that's already a step
forward is is that just the framework of
talking about a common language we're
talking about all these sort of like
informal approaches or different formal
approaches to logical uncertainty in the
past and i think there's lots of
interesting questions about applications
like how we're going to use this and I'm
going to defer those until after we've
concretely seen the math because it'll
be much easier to like point at the
object once we've once we've seen some
math so I guess I've already had any
more questions about just like what the
logical uncertainty thing is I think
we've had a bunch already so seems good
okay so I believe now my slideshow is
going to tell me to switch to Beamer
where I can do math well that's the
wrong here we go this is a talk I gave
its park this is the logical induction
one okay so the way the technical part
of the talk is going to go is I'm going
to formally state the definitions of
illogical induction and some desirable
properties of it we're going to talk a
lot about the properties we have we have
an algorithm and a criterion that
implies a bunch of nice properties and
the algorithm satisfies the criterion so
that's nice and I'll talk about those
and then i'll talk about formatting the
which is actually harder than requires
more definitions than talking about it's
a lot of its properties will talk about
the algorithm a little bit maybe time
permitting and then I'll switch back to
PowerPoint and talk about like some
conclusions so let's get to the
formalizing we're going to choose a
language or a formal system of logic
gamma for encoding statements about
variables and computer programs so he's
heard of piano arithmetic in this room
awesome okay so piano so we got those
like that seventy percent so piano music
is a language we're talking about
arithmetic but it turns out that you can
encode lots of statements about computer
programs in piano arithmetic and then
prove stuff about them so it's a very
expressive language we run our language
to be pretty expressive because
obviously we want a eyes to be able to
like think about computers I think or we
want our theory of AI is to include a
eyes that think about computers and then
this capital lambda is going to be
instead of all sentences expressible and
gamma and then a belief state we're
going to define a belief state that's
the state that you're in while you're
still thinking about stuff is a map from
sentences two probabilities that is
constant outside of finite set and I
think we've toyed with different
definitions where it's constant outside
the finite subset or zero outside the
finite subsets or doesn't matter the
point is if you're kind of writing down
your beliefs you're only going to be
able to written out like a finite number
of beliefs at any finite time so think
of your belief state as just like a list
of probabilities that you've managed to
assign to stuff so far and we don't want
to say at this point in talking about
the theory that your belief state is
like whatever you implicitly believe or
you're like revealed beliefs or your
reveal preferences they're like things
that you've managed to explicitly write
down because if you define your beliefs
to be something that's derived from your
preferences or derived from actions or
derived from stuff that's already in
your mind then then you're no longer
separating the belief in the computation
and the whole point of this framework
here is to is to separate the belief and
the figuring out of the belief so your
belief state is just the stuff that
you've written down probabilities for
and a reasoning process is just like a
sequence of belief states so at each
time step you like adjust your
probabilities to all the sentences that
you've seen so far I maybe
you add some more sentences that you
have beliefs about is that make sense
cool encourage about that sweet so now
with this definition we can say what is
a good reasoning process so some pro you
know properties you'd like to have for a
good reasoning process is that it should
be computable at least our model of it
should be computable we would like it to
converge we'd like this thing not to be
like flip flopping around forever
changing its mind about stuff like we'd
like it to be eventually figuring out
what it thinks about a thing so that's
convergence and that that means to say
that you know we've got we've got the
symbol P sub n of Phi so that means like
the the probability that you saw at time
n to sentence five or feet should i say
i'll say fee and we want as n approaches
infinity for that limit to exist and
we're going to denote it by p infinity
so it's what you will eventually
converge to thinking after forever you'd
like the limit to be a probability
distribution so once you're done
computing your belief update at the end
of time you'd like to satisfy the laws
of probability finally so that's these
kinds of rules and there's this other
property that people have talked about
that i think is desirable which is that
if you haven't if you never disprove
something you shouldn't think it's
impossible so if you and if you don't
prove it you shouldn't think it's one
hundred percent so that's this what this
says if gamma does if your logical
system doesn't prove something then
you're not going to be hundred percent
sure but does that make sense so these
are some nice properties and in fact we
knew before garabrant induction that
these properties can be satisfied guy
named abram demsky had an algorithm that
satisfied these and the paper that is
sitting in draft form on that table
right there shows that those properties
are related so there's a there's a
single property that I'm going to call
the garabrant induction criterion which
implies all four of them so that's kind
of nice it means that we it like
intuitively we were just as humans just
kind of generating these desirable
properties right and you might think
that the only thing they have in common
is that humans want algorithms to have
them but they have another thing in
common which is that they follow from
the gear Bretton
action criterion and that sort of
validates us as intuitive generators of
criteria that we managed to generate a
sort of coherent set of criteria to be
satisfied us as in humans like all the
people who've been working on this field
it also shows that the properties are
feasible because we have an algorithm
then i'm going to call logical deduction
logical induction algorithm 2016 or Lea
and I'm putting the 2016 in there
because like you never know people are
going to update these things people
might pick this up after we're done with
it and make it better and I don't want
to steal the name logical induction
algorithm forever someone else might
make it better so so we know that the
those things are feasible and also their
extensible there's a bunch of other
awesome properties that follow from the
gab reduction criterion and are
therefore satisfied by the algorithm
that we have so that is the deal that's
like what's going on here we got
algorithm that does the purple stuff
good okay so here are some properties
that the algorithm satisfies its it's
actually faster to stay the properties
of the algorithm then it itself or even
the the criterion but we'll get to the
criterion I just want it before it's
kind of like before I start giving you
this like big fat definition of an
algorithm or criterion it's nice to like
care right so let me tell you what
properties at have I don't think you
should care what the definition the
algorithm is or the criterion before you
know what properties it satisfies
because otherwise you'd be like whatever
so there's this non dogmatism property
which is that it's like a stronger
version of the dogmatism property from
earlier which is that if you have any
algorithm to output sentences and it
outputs this like infinite list of
sentences that are consistent so in an
au pair of them are contradictory or
contradictory with the theory gamma then
there's actually some constant such that
all the plot all the sentences in that
sequence are above probability epsilon
so it's stronger than the earlier
property and here's another
strengthening of the earlier property
which everyone loves kamagra of
complexity you can actually say for a
given sentence if it's never just proven
then it's probability is going to be at
least some constant times 2 to the minus
its complexity so there's a nice
relationship between how complicated the
thing is and how likely it is
and that's why we called ockham bounce
because alkem is that you know the
Occam's razor you should you know if
something's real complicated and believe
it well that's not quite right because
of a real complicated disjunction you
should totally believe that all right
real real long disjunction is probably
true so you need to amend it a little
bit there's some relationship between
complexity and believability and like
this is probably the good relationship
that you want and so and again these are
all talking about the limit right these
are all talking about the probability
bution that our algorithm will converge
to we're not yet talking about what it
thinks along the way yeah question it is
sharp no let's oh there's a there's a
see here yeah yeah yeah I'm sorry yeah
so I guess you could take this to
premium overall see such as this is true
and then it would be sharp like this d
we can prove it for is probably like not
sure Yeah Yeah right right right yep um
yeah any other questions about these
properties actually cool um what today
I'm gonna I get it I gotta rearrange
something because this phone over here
should totally be in my left pocket
where my left hand goes alright so
there's some nice properties now here's
where it gets exciting because people
have been wanting like math to
understand math for ages like Hilbert
was like hey can we like put math on a
mathematical foundation and girl comes
along and is like no and then put luck
comes along and instead sort of I don't
know if you're familiar so there's some
results that say that logic can sort of
believe that it's like sort of
trustworthy up to a certain bound but I
I think that well yeah these are real
these are real cool so our logical
inductor as will call our gear and dr.
it is going to believe it's going to
come to believe in the consistency of
the logical system that it's using
so if we define con n is there is no
proof of contradiction using n or fewer
symbols in our logical system then
eventually at time step in our gharib
and dr. believe any algorithm satisfying
the gear and adduction criterion is
going to is going to believe in the
consistency of gamma up to that point
and in fact there's a stronger claim
which is I'm going to call belief in
future consistency you can actually
choose any computable function so it
could be n to the end to the end to the
end to the end to the end whatever some
crazy function eventually on time step
in our reasoner is going to believe that
gamma is consistent up two proofs of
length F of n so this is like an
extremely strong belief in consistency
that you can basically pick your
favorite computable function and our
algorithm is going to believe in
consistency up to that function those
very very strong and you can put you
could view this as self reflection or
reflection on the deductive process that
it's using but here's a more explicit
self reflective property this is
basically it knows what it notes so at
time step n event for large n at time
step in a garabrant inductor knows what
its own beliefs are so roughly speaking
if the probability if you have some
sequence of sentences that are I'm going
to call this poly time general in fact
I've got a definitely we're going to
elaborate on this in a later slide but
it's beep it's like a you have an
algorithm that takes it in and spits out
a sentence in time prop all anomia linen
so we've got this poly time general
sequence of sentences and at time step
in our our inductor is going to if its
beliefs are like kind of squarely
between a and B like a and B between a
and B by like this epsilon margin does
that make sense it's like forgivingly
inside to the AV interval then it's
going to know that its beliefs are
actually inside the AV interval with
high very very high probability
it's going to be like almost complete so
it's kind of like saying if it's between
89 and 91 % share of the Riemann
hypothesis then it's going to be
ninety-nine point nine nine percent sure
that it is between 89 and 91 % sure of
the Riemann hypothesis me quite certain
of what it thinks yep ya know it is
written down on Tom sub n it has written
down this is my probability already yeah
yeah yeah so at any at any finite time
step it's only going to have regression
up to a certain turn in yeah but like
yeah a time step large and it's going to
know that it knows it knows it knows
some finite number of notes yet yeah
great question and so now when you get
self-reflection like this you've got
something that believes in its own
consistency you got something that knows
what it knows this is like ripe fertile
soil for like paradoxes right we can so
so we should all be getting nervous but
in fact this thing has like a very
strong systemic immunity to the liars
paradox you can define it you can find a
sentence that basically it's equivalent
to saying our garabrant inductor doesn't
believe me okay you write down a
sentence that says pick your favorite
rational probability P you can write
down a sentence that is equivalent to
saying I am less than P likely to be
true so P could be like one percent I am
less than one percent likely to be true
and gay brand dr. just handles the
situation it's just like no problem I
know what I'm going to do it at times
the probability P in the limit and on
the way to the limit it does a crazy
thing where it like is constantly flip
flopping its belief right around
probability P so that it's less than P P
of the time
yeah so good yeah okay so good all right
i'll just say that again so you make so
you make a statement that says i am less
than P likely to be true i'm going to
say p % because if it's natural language
better okay so i am less than p % likely
to be true and then over time its
beliefs about that statement converged
to probability p but they like awesome
over time they like oscillate just
around p so that they are actually less
than P P presented the time and they're
actually so it works out yeah so you
don't get a paradox and it's awesome I
just cannot yeah in fact people are
already like seeing what happens when
like agents that try to make themselves
unpredictable by like trying to protect
you can make an agent from garabrant
inductor that tries to make itself
unpredictable by first trying to predict
its own actions and then having a plan
that if it can predict its own actions
it will do the opposite and by virtue of
that counterfactual plan it can't be
predictable and then you get random
behavior okay anyway it's real cool so a
large paradox resistance it's real cool
and moreover you can go so far as to say
that this thing kind of trust self so
this is another these are properties
like that mary has been sort of pining
for for like years you know just like
how can we have a model of an agent that
like trust itself trust its future self
understands itself knows what it knows
etc and the Taliban just like does it
all so it trust its future self in the
following sense we have a result that is
roughly interpreta belen quote those are
those are human quotes not girdle quote
these are the girdles and those are the
human code so we have a result that
roughly says that the problem oh that
should be a sub n their apologies but
the probability at time step end that
you assigned to a statement feast of n
so again sorry I'm assuming that we've
got this sequence of poly time general
sentences and I'm going to harp on the
meaning of those
I think coming up soon yeah the next law
is going to like harp on how important
that Polly time general thing is we got
some sequence of questions we're asking
the inductor this polynomial time
generate a ball or general and roughly
speaking it has some property like this
that the probability that assigns to fee
given that its future self so f of n is
some large number given that its future
self that signs probability at least p
is it please p asymptotic lee so this is
like saying if future me think that you
know we're ninety percent likely to live
on Mars then I'm ninety percent likely
to thing we're going to live on Mars cuz
future meets tomorrow he's had more time
to think so conditional on future me
believing act I believe act yep given
that you don't know what future me is
going to believe what explanatory power
why why right so wise so this property
is good because you want this thing to
be able to reason about hypothetical
actions it might take like right like if
someone says Critch don't go to Berkeley
because the cognitive science
departments gonna like suck you in
you'll never do math again I'm like well
if future methinks cognitive science is
a good field of study it's a good field
of study so I'm just gonna go to
Berkeley not worry about it that makes
ten like I have this confident that if
future me thinks it's a good idea don't
worry right so a future me think
something is true yeah probably true so
this is conditioning on what future me
thinks it's different from knowing it's
like the post future me thinks it's at
least probability P then I then it
probably at least it is a conditional
probability getting that like you get
nope um the cool thing is the inductor
is an algorithm step one it's an
algorithm so a few algorithms can be
written down formally mathematically
step 3 this algorithm has beliefs about
every mathematical statement including
statements about itself so you can just
write down this statement that's what
the gunner codes mean it means like
mathematically represent the statement
me assigns at least probability P n and
then asked it conditional on that what
do you think the probability is and it's
like well at least that now this is in
quotes because it's not quite true for
example there are denominators like
things could go to 0 it's a bit tricky
so the actual result looked like this
and there's a bunch of definitions that
I would have to say to make that true
but you can like just stare at it a
little bit get a sense of the shape of
it q you left but I mean this is how I
don't know when I read a paper the first
thing I do is figure out the shape of
you know the results in it so it's
something like there's some indicator
function for whether the thing is true
and you multiply the binary variable
that indicates whether this thing is
true by that thing and then you take an
expected value which I haven't defined
at all but there's expected values of
logically on certain variables which is
cool so anyway that's there's some kind
of future trust thing going on here
which is really great because we really
wanted agents that can like think about
reason about their future and like these
hypotheticals where it thinks its
features like it's not going to it's not
going to try not to go to Berkeley
because it's afraid of getting sucked
into cago design you know it's gonna be
like well future me thinks that's a good
idea it it questions about that actually
this is like maybe kind of deep maybe we
should have have let's see lots of like
Steve lots of like faces doing that so
maybe I should just let you guys do some
logical induction right now before I
believe is that you more whether you're
actually endorsing this as a coding for
human oh no I'm right I see ya so I'm
not telling you to like buy lots of
addictive substances and put them in
your house and trusted feature you will
make good judgments about them thank you
yeah I yes like super cruise the pattern
notching that looks sort of like koshi
short put the other way around there is
a yeah there are like oh yeah there are
these like sequences of like bounds on
stuff that are really what's going on
yeah for some valleys of what you meant
you could be right yeah you can think of
that it's like based here
the denominator is multiplied the
state's here but like the definition of
conditional probability yeah you like
multiply this thing all right yeah but
but again I don't I just want to play
Sarah if you want to know more you have
your other paper it's right yeah okay so
that one probably I don't know I don't
know quite yet I was at a math camp with
these guys so I don't really know what's
in the abridged version yet because got
cut down while I was like away so uh but
you notice I keep bringing up these like
polynomial time general questions right
so hopefully you care now when I give
you this definition because you've been
like irritated with it for a while
already so the interesting thing about
sentences that are general in polynomial
time is that they're easy to generate
relative to like other more complicated
complexity classes but they can be
arbitrarily hard to answer so there are
polynomial time algorithms that can
write down questions or statement of
uncertain treat value that are just like
I want to say uncomplete ibly hard to
verify but what I really mean is hard to
verify than any computable function of
the thin number ah lek date taken I
don't know if humans were generated in
polynomial time actually well I don't
know if you use polynomial x agenda
generator um uh yeah I don't well here's
one here's an example in the next slide
so if you let g of n be the statement
there is no proof of g of n in fewer
than f of n characters pick your
favorite giant function f ok and then
make a sentence that says this sentence
cannot be proved in under F of n
characters for each n now you might
wonder how can you make a sentence that
refers to itself that's a trick called
cantor's diagonal emma and actually
there's a version of it that lets you
have this parameter and in it so just
trust me you can look up and boulos
there's ways of like actually writing
down sentences that are provably
equivalent to the self-referential
predicate
and you can see if gamma is consistent
it's not going to be able to prove this
in fewer than f of n characters all
right suppose that I've got a sentence
it says I'm not provable in less than
1,000 characters ok well just as you
can't prove me in under a thousand
characters you can prove in a thousand
one characters all day if you want yeah
after a thousand characters you've
proven that thing is true this thing is
true there's a proof of it but it needs
to be very long think about do logical
blush all right so so this statement is
true right if gamma is consistent this
statement can't have a proof in a short
proof and that's what it says it says I
can't have a short proof so that makes
it true so in fact because of proof you
can actually write down algorithm built
search for a proof of this that
algorithm will provably terminate so
there is a proof of this thing and it at
the very least at the very most as long
as like an exhaustive search of all
strings up to whatever length you need
to prove it so which is not F of n could
be much could be much bigger than F
event but this thing has a proof it's
true but it's proof is crazily badly law
does that make sense but it's very easy
to write down you write this down in log
n time you can actually write down a
template for the sentence and just like
keep stuff stitute in longer binary
strings for in right and then this thing
just gets crazily hard to verify and you
can roughly see that so nonetheless we
have this amazing property called prove
ability induction and permeability
induction says that if you can generate
questions if you can generate statements
in polynomial times that are approvable
it doesn't matter how awfully long their
proof get a garabrant inductor is going
to catch up to your Denton's generator
and start believing the sentences as
soon as you generate them
this warrant pause okay I'm going to
like draw a like kind of cartoon
depiction of this thing because i think
it yeah this is this is a problem this
is a property that we were not looking
for that we just got him got it just
showed up and we were like well really
or at least I was just like what so so
we've got our sentences feast of one
piece of to feast of whatever and
they're all they're all approvable but
would just like crazily batted long
proof so at time step one imagine our
garabrant inductor looks at sentence
number one so imagine the Gare brand
doctors running along over here and like
Ramanujam is like running along over
here generating his super hard to verify
conjectures okay and Ramanujan's fits up
piece of one and gharib and daughters
like man I don't know fifty
percent I don't know if that's true so
then at time step two it still has a
belief about v1 and it's like I I don't
know it's hard I don't get the thing and
then it also has been asked another
question which is piece of two and it's
like yeah I don't know that either okay
just let me think please okay so this
ends is like coming on there's this like
onslaught of conjectures feats of n
right these mathematical conjectures
that have presumably being generated by
some here like intuitive algorithm that
uses heuristic to generate true things
or maybe some like concoction like I
just made that guaranteed to be hard and
eventually one day you know at time step
like you know 10,000 it's like whoa hang
on hang on guys that was true I figured
it out all right so it's like a hundred
percent or like something very close to
a hundred and then and then you're like
all right so it's finally caught on to
the first thing it believes the first
thing and then you're like what about
the second thing and it's like I don't
know that's all I'm thinking about it
okay it's still like roughly fifty
percent and then like much later maybe
like p a million it starts to believe
both the first and second sentences ten
to the six I should have there
does that make sense by time step a
million it is managed to verify like the
first two tenses and it's still super
uncertain about these months so this
look do me right it's like it looked
like the number of sentences has
consider right it's like going to it's
like getting linearly wide but like the
difficulty of verifying them is like
super narrow right so it looks as this
tiny little cone of things going to
believe but no one day this just stopped
and it catches up so that like time
steps duper large in I don't know maybe
it's a google I don't know how big a
number you need it's going to depend on
the particular algorithm but it's
actually going to catch up and just
believe it's going to widen out as it
going to believe even even David number
n as soon as statement number n gets
generated it's a polynomial time in n so
in each time step Enders an algorithm
that spits out or news 10 and runtime is
polynomial similar in some way yeah yeah
they're following some kind of pattern
exactly so it learns that pattern yep
right you've got some algorithm for
spitting out statements that happen to
be provable they're not quite attempt
they don't have to be quite a template
in the in the predicate sense like they
don't have to be a first-order predicate
with is it with an aunt in it and you
keep bugging different ends that the
nature of the state could look very
different but they do have to have
something common which is that there's a
single polynomial time algorithm that's
generating them and the point is that it
catches on to that pattern if there's a
pattern that it can catch on to then it
does yep does it does it automatically
assign the left one 107 like very close
to a hundred percent not actually yeah
for every absent for every epsilon there
exists a large and felt like eventually
over you yeah well you this is the
statement right here right so for every
epsilon there exists a large and such
the non time step and it's on to
probability very near one with it Nathan
f1 yep okay like know that
no it doesn't know that someone's out
there with an algorithm we might yeah it
figures it performs induction yeah yeah
unfortunately mathematical induction was
a term already taken to refer to
deduction so we had to we had to call it
something else yep a little bit later
about how your mission in this system
right here we're examining there's no
information entering it yeah this is all
in between like I've received some data
I've written down some facts about the
data and now I'm like thinking about the
consequences as those fact no new
information yeah that's the whole point
of this framework yep you could you
could make modifications where you like
throw in extra information while
thinking and that'd be like future
papers but like we're well prepared to
like have versions that that that do
that yeah only when they're when they're
all provable now great question so what
happens if like half of them are
provable right yeah so it does a great
job then too so let me get to that so
we're going to have lots of limits so we
made this notation this sequence is like
roughly the same as this sequence means
that their limit difference is 0 this is
a common notation for asymptotic this
one's like roughly bigger than that one
if asymptotically the limit of the other
one minus the first one is our second
one is zero or more so the this
situation where the inductor catches up
to the pattern that's generating the
question we're going to call that
assigning its beliefs in a timely manner
like this is the timely belief you know
at the time that the time that you might
ask get the question the belief it
assigns at that time is a time of timely
belief it's the one that it managed to
generate on time when the question is
made Wednesday who's the next person in
this room by the way
cool so there's this cool brother cool
property which probably this will be the
last cool property to build say in
detail but it says if you have a poly
time general sequence of statement then
if the logical inductor is going to
converge to some pattern of
probabilities that it assigns to those
statements that also poly Tom general
then actually you don't have to wait til
infinity it's going to start assigning
those probabilities in a timely manner
so anything that's any poly time pattern
that it's going to believe at time
infinity it's actually going to believe
along this diagonal here eventually so
it's you can prove lots of cool stuff by
first proving it using tinfinity and
then saying and now there's a large n
where it work so now I have to cool
great so what I'll do is I'll conclude
in such a way that people feel
comfortable leaving the room for a break
and then I can stick around during break
and Nick answer my question so oh yeah
we can't have break in here got got it
makes them makes it I'll just bring my
laptop all right so you know there's
this question of like what happens if
half the sentence is being generated or
true we have theorem today you know that
it at least does well so it might
actually figure out which ones are true
so it doesn't have to say 50 percentage
it says these ones are true and these
ones are fall but if it can't do that
then it's good you percent and we need
to define what it means to can't be able
to do that but if you can't figure out
the pattern and it starts falls back to
using probabilities it learns to be
coherent we had a thermos says that it's
coherent at Tom infinity meaning it
satisfies the laws of probability theory
but actually you can write down what it
means to satisfy a law of probability
theory and show that it approximately
satisfies laws of probability in a
timely manner like on this diagonal
effort for large n and so there's much
nice properties they're going to during
the paper they're going to be more and
more papers
and so the formalization of the
algorithm is basically finance you just
make a stock market of traders that are
betting on the sentences then you
imagine that market and then whatever
the market believes you believe that so
the algorithm is actually very simple
and the criterion is very simple the
state which is part of what makes it so
beautiful it basically the garabrant
induction criterion the single criterion
that implies all those myriad lovely
properties basically there's some
definition of trader which warms my
heart because I used to be a traitor and
it says that you are good at logical
deduction if any trader who's not
willing to possibly worth that risk
losing more than a bounded amount is not
going to be able to make infinite money
from you so if you walk up to a gear
bratton ductor and you promised yourself
you're never going to risk losing more
like going negative a million dollars in
debt if you promise yourself that you're
not going to make a million dollars
betting against it that's just the
definition and from that one definition
you get all those amazing property so
pretty cool I think and we have the
algorithm I think it's a financial
solution to the computer science problem
of meta mathematics and time for meeting
i will use a whiteboard to answer stuff
so probably i'll take the whiteboard out
to whatever it is that i would be able
to field questions and what does all
this mean so fish from femur to
powerpoint so my summary of what i think
just happened at mary is that we had
rational choice theory we had a concept
of an agent in it Calvin or Morgenstern
agent we had some nice axioms that
implied that type of agent we had nice
arguments for why those actions are
important if you don't satisfy them you
lose a bunch of it right and then we
have like various formalized versions of
VNM agents somewhere the utility
functions are reinforcements some aren't
then we have probability theory again we
have a notion of an agent some axioms
that you need to satisfy some like
arguments to say those are good axioms
all right if you don't satisfy let me
lose money
then we have like a formal thing called
salma induction that just does bathing
updating and you know it's possible to
do computed ly because there's a thing
so now we have another row to the table
we've got logical uncertainty theory
which can be used to like implement
probability theory which can be used
implement rational choice theory we've
got an agent concept called garabrant
inductor we've got arguments that show
that they garabrant inductors are like
good reason we know that they're big
they can exist because we had this
algorithm that does it and we don't
actually have this yet we don't have so
that's like open field of first of all
inquiry for lots of math people to try
and figure out like what would be the
analog of these axioms that would like
imply like some nice modest condition
that seemed like oh yeah you could
totally satisfy those and then it turns
out if you totally satisfy those then
you happen to satisfy like the big
strong garabrant inductor criterion so
from here like I said you can have
people working on improving those axioms
like coming up with like a short list of
modest looking axioms that imply the
garrett induction criterion you could
use the gara brandt inductor as a model
for thinking about future AI system for
example it turns out in principle that a
eyes can actually outpaced deduction in
this absurd way they're like absurdly
there they're faster you can have an
algorithm fact Leah 2016 has the
property of being faster than deduction
by a margin that is any computable
function like pick your favorite
computable function the algorithm is
faster like induction outpaces deduction
by that margin for every computable
function so it's kind of I don't know if
that's good or bad but it means that
like as much as there might have been
lots of complexity arguments for like
how you're never really going to be able
to make AI maybe someone recently made a
blog post but how those arguments of
making sense I roughly agree with that
but this is like a clear illustration of
like actually you don't need deduction
you can use induction and it's like
computable faster for every value of
computable so that's kind of cool and it
like helps us think about what is it
wasn't AI capable of in principle
and of course then there's other
approaches to a alignment that won't
necessarily use garabrant induction as a
model explicitly for how a eyes are
going to work in the future but they are
going to have to eventually implicitly
or otherwise address the problem of like
well a eyes are going to have beliefs
before they finish thinking they're
going to make decisions before they
finish thinking so the police will be
implicit or revealed and those decision
and miri is going to focus on the stuff
i think it's sort of a consensus that we
maybe don't need these axioms as badly
as we need this theory of like how to
make AI safe and well-directed in the
future so we're going to work on that
mostly hopefully our models of game
theory which are currently super we can
like ill-prepared to refer to a eyes for
lots of reasons like they don't have
open source agent we're going to be able
to think of game theory situations where
there's like an agent interaction with
the world and replace the previous
logically omniscient agents with like
more realistic sounding bounded
logically an uncertain inductor agent
that are more realistic still not that
realistic but like way more realistic
than the stuff game theory and economics
and mechanism design has been talking
about to date and we've learned that in
explode ability is easier than I've been
I thought I wasn't expecting it to be to
just be like kinemet you can imagine
there's all these properties you want to
satisfy and all these people who are
trying to exploit you and turns out you
can use brouwers fixed point theorem to
make an economy where all those people
trying to exploit you cancel each other
out and that's the proof so somehow in
exploitability you can like balance
you've got this like pressure from all
these people trying to explore you and
you can like over cut you can just like
all the pressures can like balanced out
I don't think we knew that until we had
this this result self-trust is possible
in ways that we didn't think we think
maybe it was or did we have we didn't
have theoretical models of we had this
outpaced and production thing which I
think it's crazy it looks like
calibration is not as necessary for in
exploit ability as like I thought in
advance we're still actively
investigating that we don't have like
hard results that day like you only need
a certain Mountain calibration we have
results
say you do need a certain amount of
calibration you don't have to hard-code
belief coherence laws the thing actually
just learned them on its own you don't
have to tell us to satisfy the laws of
probability theory in the course of
trying not to lose to the market it
learned probability theory so you don't
need to do that and I've had a meta
update that marries general approach of
actually trying to turn these big
philosophical questions like how do you
make a self-reflective algorithm that
understands itself and does math well is
maybe actually doable like there's some
how got managed to actually put a big
dent in this problem and it makes me
want to like try and put more dents in
it so that's a big med update I was not
expecting this to happen in my life so
I'm not in my career anyway I basically
was going to abandon the miry like agent
foundations agenda because I thought it
was hopeless because we weren't gonna
solve logical uncertainty in the next 30
years so I was like we're a I was gonna
exist before anyone answers this problem
so like we should probably give up on it
and then God didn't care what i said and
so thanks to scott for that work he's
going to be around he's gonna answer
questions hopefully as you can see
there's already some people in the
audience you can also help answer
questions like Sam Jessica and fee there
are these people here Kinsey can three
Nate Jessica stand-up thing yes stand up
stand up an identified shelf so these
are these guys are co-authors on the
paper they're gonna be able to answer
crush and and he's not but he knows
stuff anyway yes ma'am and yeah thanks
to Jimmy Jimmy Jimmy is not here is he
but he helped me with lots of logic
problems so that was a big deal thank
you everybody we're on break
[Applause]
you
[Applause]
you