156. Synthesising Utility QA

hello and welcome to session 156 in the
ashd reading group tonight Stuart
Armstrong will present his research
agenda please take it away Stuart hello
there thanks for inviting me to talk
here I'm actually going to take get to
the research agenda in a very roundabout
way presenting a few related intuitions
which base explain why I think that
there's a chance of the research agenda
succeeding but to start off with one
question is to ask in Reverse
what would a solution look like suppose
we said we had all of human preferences
in some compact form or some utility
function form how would we have got it
there's a few ways we could have got it
effective proxy methods which are most
of the methods that people suggest like
putting Nick Bostrom in a room for
10,000 subjective years to think of
things or having people write what they
want and somehow synthesizing from that
these don't measure preferences directly
but they are proxy measures and we're
hoping that we can get them to work out
another approach which might work is
courage ability we start with a one-goal
structure and we modify it successfully
and repeatedly until it aligns more with
what we want
there's the catch-all of some brilliant
new idea and finally some grounded
definition of human preferences so
something that actually captures our
preferences and I argue in fact that's
anything of this type is going to look
somewhat similar to my research agenda
it needs to have some fundamental
definition of the pieces some way of
putting them all together and some way
of respecting meta preferences and other
things of that nature now one key
insight where I tend to differ from Miri
is on the issue of patching or nearest
unblock strategy arguments the basic
idea is suppose we have an AI but for
some reason goes straight to you we say
no don't go there
so the AI just routes around whatever I
know is we can add quite a lot of
patches blocking different ways and then
it just routes around those patches this
is the idea of the nearest unblocked
strategy like for example you say
increase human hedonism or increase
human pleasure and then when that ends
up disasters he would say but ok respect
human autonomy
don't force humans into bunkers and we
keep on patching and in each case it
finds the closest cheating thing that it
can that technically satisfies our
preferences but compare this with with
sort of them drawing boundaries in as
hilar higher depth neural nets tend to
join boundaries in concept spaces if we
have a bunch of blue counter examples
and a bunch of red good examples then
the AIS can naturally draw a boundary
around this and at this point even
though that arrow I've just drawn
doesn't go through or close to any of
the blue points because it's learned to
generalize the concept it doesn't go to
that disaster area so what this shows is
that the error modes of machine learning
in your nets with positive and negative
examples are different from the error
modes of the nearest unblock strategy
problem so that that's this is why I
hold out some
Oh for less more fuzzy methods now of
course generally what happens if you
have a lot of examples close by and not
far off you constrain it's in one area I
mean it goes wild elsewhere so we sort
of have to be able to give counter
examples very far away which in some
cases we can like human preference for
say no human suffering if we can define
that approximately this extends across
really alien universes really strange
and different worlds from our own so
this is a sort of global constraint
similarly if we find the repugnant
conclusion repugnant that's a sort of
global constraint anyway this was just
one intuition and I also feel that
things like symbol grounding after
solution behavior and a few other things
can be potentially solved in this kind
of way this is one of the fundamental
intuitions now to get to the no free
lunch theorem famous throughout the
world by everyone who happens to know it
this was my result that if you have a
behavior you cannot go straight to
preferences at the behavior of
potentially irrational agent you cannot
go straight to preferences we can't get
the preferences without making
assumptions about the rationality you
can't make that the rationality of not
making assumptions about the preferences
this is not a not very exciting no free
lunch thing because normally no free
lunch theorems you just use a simplistic
wire and they go away unfortunately here
simplicity priors don't help and the as
I've mentioned I think before humans are
fully rational humans are fully anti
rationals and things have flat flat
preferences are the simplest
interpretation of human behavior but
notice I said you needed to make
assumptions what would be great
needed to not make assumptions so if you
can make just very simple assumptions
and get around the whole problem that
would be great so the whole
infrastructure of the research agenda is
to try and make some minimal assumptions
and a lot of construction and this leads
us to a way of going and finding the
Preferences of the human right and
another important point is that
interpreting a human as having perfect
princess or rationality in a stance it's
very similar to Dennis intentional
intentional stance we're not treating a
human as a collector of neurons or
collective atoms we are treating them as
an agent with preferences this is not
something that naturally exists in the
world so as I said it's a super
intention area I could have all the
world's videos is all of Wikipedia all
social science research we give perfect
predictions of human behavior be able to
manipulate us incredibly well and still
conclude that humans are fully rational
and it would not be wrong it would be
right but it wouldn't be wrong because
rationality and preferences are an
interpretation we put upon other agents
they're not even calling something an
agent is a preference is an
interpretation and here some people feel
natural objection they say that they
know what feelings wants from observing
and we do and even more damning in a
sense is our observation of human
preferences allows us to predict their
behavior so this approach is vindicated
by the outcomes so what is going on I've
just said this cannot be done well this
is a sort of simple model of what's
happening
is I'll see a human H is holding an
empathy machine and a human predicting
they Co evolved the empathy machine is
what tells us what people want and how
they think and the predictor is tells us
what they act so if we have a typical
history of a human age then two
different humans looking at it will find
roughly the same thing we tend to agree
very strongly on what people want and
how washed out they are not perfectly
but far far better than another chance
also that a human themselves tends to
agree with our assessment when we're
drunk or less rational and we tend to
agree that we're less rational when
we're drunk for example so these empathy
models all give pretty much the same
thing on typical histories and then if
you feed these things to the predictor
then in typical circumstances this is a
pretty good predictor of the human
interaction so given the empathy module
and the preference model the reward is
simple to deduce and typically it is
predictive so this is happening my sort
of response to the objection yes for
humans with our specific evolved ability
we can do this task but it's not a
general task that is easy to do it's
because of our particular cognitive
machinery and our own assumptions now
this VHS empathy model is strongly
related to the partial preferences when
I'm about to get into this brings us
finally to the actual research agenda
the idea is to start with code or the
human brain and from this get the mental
models that the human preferences are
using so when we think oh that's a bad
outcome or I wouldn't
bahador they want that or if I do this I
might get something good these are our
mental models and within these mental
models we tend to express desirable or
undesirable outcomes
so from these mental models we can have
orderings over different circumstances
notice and some of them are Metta
preferences which can tell us about our
lower level preferences in order to do
this we need to do some sort of human
symbol grounding which is part which is
why I dressed it in in the research
agenda and this is where I bring in the
assumption the assumption is that these
partial preferences as I'm calling them
that their normative that this is what
preferences are built out of so I've
gone from the observation of that human
mental models order things in a
particular way to the assumption the
requirement that this ordering tells us
something about what the genuine
preferences are this is sort of my
stance then given that and some
alchemical complicated process we can
get the human preferences now a lot of
the research agenda and a lot of the
research on it is going to be about this
but it's in a in essence not that
important step what it has to do it has
to do two things or three things it has
to reach a conclusion of what the
preferences are and it has to sort out
all our contradictions and under to find
things and it has to do this while
respecting our meta preferences so all
the machinery of the section two I
believe is basically a way of doing this
and hopefully the particular way of
doing it will not be that important in
terms of reaching an acceptable or a
good outcome
then once we have those preferences we
put them into the AI and then we're
almost done to do that we need to do
some sort of AI symbol grounding which I
wasn't working on too much now realized
is actually quite important bit so I
will be working on that in the future of
course you don't know the human source
code perfectly but some uncertainty
about that source code will just result
in some uncertainty on the ideal world
and this actually should end up okay for
arguments that I've sketched in the anti
Goodheart posts or how to break good
heart in this setting human behavior is
an observation that gives evidence about
our internal makeup and about the symbol
grounding now because it's good to bring
in a little bit of formality I have put
various ones of these things in symbols
more or less justified so the the human
code leads to pre-orders which is how we
order our partial preferences so this is
basically we've order things without in
our models without having four models of
the world of comparing everything we
just compare on one feature then the
symbol grounding for which I have stolen
concepts from semantics formal semantics
to go from human models to worlds the
pre-orders are now on worlds the
assumption is that these pre-orders are
preferences relevant the pre-orders are
synthesized into utility function using
some process Sigma the meta preferences
are taken to mean to take preferences
over the Sigma then another round of
symbol grounding and we get the utility
function in the AI now and the empathy
modules that I talked about
very closely connected with our
pre-orders so how we model other people
is very similar to their internal models
so these internal models are very
closely related to how we model up their
people and there are too many uses of
the word model in that sentence I know
so finally this is just a small section
of some of the preliminary work I've
done the probe first is constructive and
the main point of all this work is to
check that there were no huge conceptual
holes rather than solving every problem
or indeed any problem to extreme detail
just to check that there were no large
holes that I was missing and that is it
thank you should I stay very much thank
you very much yeah if you could stop
sharing and then I will share my screen
and then we will take some questions and
the way we always do this is that we
what's called we take I keep a list with
where people who have said they would
like to post questions so we are in the
order of where people raise their hand
even either through the chat or if they
say something and I will take the very
first question you should be able to see
my screen is that correct
can everybody see yes okay so please if
you have any questions write in the chat
or raise your hand I think it's possible
to raise your hand through this yeah
matthias has a question so you leave
second because I always take the first
question and so my first question is
about instrumental goals because Nick
Bostrom divides skulls into instrumental
goals and final goals
and a lot of the questions the examples
that are given in the research agenda
have to do with instrumental goals think
that like I wish I was rich
rather than poor or I don't want to get
marked or things like that and these
kind of things are not in the research
agenda really specified those are this
kind of division is left to the process
mostly but there's a in the reading
group we've discussed this and it seems
to be in some way
instrumental goals are quite a lot
easier to work with then final goals in
that you could imagine you normalizing
all kinds of instrumental goals into
some kind of general goal reaching
ability and just say I want to be as
instrumentally powerful as possible and
then only focus on final goals do you
think this kind of distinction makes
sense in in humans a lot of things that
should be instrumental goals are pretty
much terminal goals like or very close
to terminal goes you could say well the
I want to be rich because I want to do
stuff and I want to have high status
those are two terminal goals I I don't
want to get mugged because mugging can
get hurt or embarrassing those are two
terminal goals so though you though
these are technically instrumental goals
they are very close to terminal goals in
a sense you know the eliezer yudkowsky
thou art God shatter article evolutions
terminal goals of having lots and lots
of grandchildren and great-grandchildren
have shattered into lots of pieces and
it seems that for humans we we have a
lot more terminal goals than in a sense
than we should
yeah and the the sort of terminal goals
that philosophers like to talk about are
things we have constructed using our
meta preferences so the I want everyone
to be happy and to be able to reach as
much as they need every day for example
is a terminal is a turk is a goal that's
constructed by my meta preferences and
sort of extended out from certain basic
empathy in goals so I wouldn't say it's
instrumental versus terminal but sort of
short-term clearcuts goals versus
synthesized meta preference based goals
does that make sense I think it did yes
okay we have a question from Matthias
and then one from Robert as a follow-up
question to science so human goals are
generally terminal valve relevant
instrumental dust after will be in
direct conflict with each other and
wouldn't you have to work out how to
value one humans preferences against
than others
yes human goals are generally going to
contradict the goals of other humans
more more importantly human goals are
going to contradict other human goals
within the same human so there are huge
masses of contradictions out there the
contradictions are not that hard to
solve you need a way of normalizing the
kiltie function which is why that pops
up in the research agenda and then add
them together because goals are
virtually never completely opposed those
are they're almost always positive some
in some ways
there's always some outcome that
is where you can increase one without
decreasing the other increase in grease
them both to some extent
so actually contradictions are
relatively solvable in theory the what I
say about human goals is that they are
contradictory when they peelable and
under defines you need other tools to
deal with the manipulable aspect but you
need a lot
it's the under defined which is the
really big problem extending them to new
situations especially situations very
alien from the ones we have I mentioned
that the empathy modules serve in
typical situations they don't in
untypical situations where things start
getting they asked where the world and
the people in it become very different
from what we're used to those are the
things where you have the problem of
extending the goals yeah that answers
for question I'm pretty well and
actually also answer to the other
question I had so let's jump to the next
person okay that would be you Robert I
so this is something that wasn't talked
about very much in the presentation or
possibly at all but it was a questioning
that came up while I was reading the
post and we might have a slide for it I
don't know because I think we talked
about it before but I was a little
confused about the identity we related
preferences about why they are singled
out or not singled out but like picked
out as a specific category that is that
it sort of treated differently I was
thinking about one of the one of the
examples that's used in the post is if I
hear a good argument for X I want to
believe X and I was wondering how that
is like is that an identity preference
and if not how is it different from
something like I want to be the kind of
person who is moved by
um I see them both as identity
preferences the non identity version of
that would be
I want the AI to be moved by good
arguments and optimize the world
according to that it's because of the AI
specific it's not just that I want the
world to be the best it's that I want to
be convinced about about what's best as
well the reason I separated out the
identity preferences is because they
seem to be very important in humans and
very very likely to get lost in the sort
of stereotypical idea out of building
and expected utility Maximizer in the
boss room had a mindless outsourcers
basically outsource everything in the
name of efficiency or and even in this
case we outsource everything in the name
of the goals and our goals are
accomplished but there's no us left -
well to appreciate it or to care and
since now why did I bother to separate
out identity preferences well it's
because I was building a default
framework for people who are didn't to
have heavily developed romantic
preferences and it seems to me that
identity preferences work more as a
smooth min than as an additive as in if
you lose one completely it's a disaster
even if you max out on the other it's
because you are in a sense not the same
person so it's much better to not lose
on any of them and maybe push them all
up a little bit than it is to push some
of them up massively and drop the other
ones quite a bit so it might not be
necessary but it just felt like a
default to separate an art
like a design decision yes okay
ashes I have sorry I have a post coming
up soon which I'm going to call let me
just find the name that I found on
gratification which may address some of
these issues in part in a way that
doesn't that's maybe a bit less identity
anyway it might make it clearer some of
these aspects I'm working on that post
and I will post it hopefully within a
few days it is relevant I sort of see it
as a third piece between head and ism
and preferences yeah so ashes is putting
his tagging up his question right now
which is would it be fair to say there
are like two kinds of preferences those
that should be dealt with as soft min
and those that should just be being
maximized and it's somewhat of a
coincidence that a lot of the soft min
have to do with identity preferences and
the ones that I could just be added have
to do with preferences over the state of
the world well a lot of them in a sense
it's the sort of hypotheticals I call
the one-step hypotheticals brief
description of the worlds of a
hypothetical world what you prefer there
if those work well and can be done well
then in a sense we sort of get the
trade-off between all the preferences in
all the different contexts so that would
tend to basically override sure whether
things are adding linearly or in other
ways the separate out the identity is
for the situation where it's a default
so it's basically
we can't do the hypotheticals well how
old we don't have much information to go
on we'll do this as what this is what
seems to me the best the most prudent
initial way that can be modified by meta
preferences that can be modified by me
once the hypotheticals in various
situation for that is for the best
default in my view it's also another
reason is that this is sort of a
reminder there are identity preferences
any thing of reference impetus needs to
remember them and so just labeling them
is that here it is
I'll read out the question thank you Oh
what people can see it anyway well read
out Erik it was in relation to the
Heisenberg in nature preferences given
the humans are relatively volatile to
human knowledge of those preferences
calling me out as a racist may change my
mental states quite a bit to give a
great example how can how do we make an
area collect preferences without a
manipulating human more than necessary
okay well we can define the problem okay
there's a paper which I've written on
how to learn when you're learning is
also a manipulation it has been refined
by various co-authors for two years and
so is not and is not out yet so I have
actually worked quite a bit on
manipulate how you learn without me
without manipulating will learn when
your actions are manipulative but since
that's not out the one way you could
deal with this is by say saying what
would the human preferences have been in
a given week that is in the past
maril the AI cannot manipulate those
past press
answers through its transactions you can
just find out about what they were or
what they would have been with we can
only sort of what hypotheticals if at
any point during the past week we had
asked the human and once that
hypothetical what would they have
answered that is the definition of human
preferences this avoids the manipulation
issue they also becoming irrelevant
since their I thought and now this is an
issue which I don't really know how to
solve I alluded to it in section 4
things I don't know how to what to do
now now some of those things were
actually things I have a pretty clear
idea of how to do it but I don't want to
talk about it in this route research at
the end of but things like this are the
real things I don't know how to do
there's kind of a good out problem so
this one is human value drift talking
the value to change how would we want to
constrain a lot of people tell me they
don't want their values to drift
randomly or excessively but they want
they want to open them they want to be
open to more learning in the future
defining how this both can happen okay
defining how both of these can happen is
tricky I don't really know how to do it
and a related thing is when the AI tells
you here I am perfectly distilled your
preferences and synthesize them into
this utility function isn't it lovely
and then the human reacts inevitably no
that's not what I want
whatever the AI gives it because this is
just how humans are this is the natural
feature I'm pretty sure that if I looked
at my licit preferences my preferences
made explicit I would say no no no
that's not it so
this is kind of you can say that this is
a good nail self-reference problem but
it's a very common one so this is also I
don't really know how to do it you can
kind of you can imagine the AoE says
okay I know what it is I'm now I'm going
to manipulate the human so I'm going to
give them a fake one they're gonna say
no they're going to change it then will
converge on the real one so maybe you
can solve that problem by allowing me
but in any case the question of what so
I object to the equilibria in a second
but the question of how you build moral
drift and body change more learning and
us accepting that the AI has decided
this is what our preferences are is
something I genuinely am not sure how to
deal with now equilibrium equilibria is
a problem if it's a local equilibria i
define by local conditions then there is
no limits to how far the values can
drift
I had a Mahatma Armstrong less wrong
post where I imagined
starting off wanting to becoming more
compassionate ending up with killing all
life in a series of improvements so not
murder Gandhi exactly it's basically
over compassionate Gandhi whose
compassion expands too much and
encompasses encompasses the animals
encompasses the insects encompasses the
plants encompasses the rocks and at that
point humans might be humans and animal
life can be reduced to rock destroyers
the the point wasn't to think that to
take fairly seriously but just that
if it all if it's an equilibrium just
defined by sort of local conditions ie
it's reflexively stable it would not
walk further modifications or anything
like that it can move it can be
arbitrarily far away another option
which I prefer is to tie these
preferences to say some sort of starting
point so we still look for sort of the
best equilibrium of some sort based on
some conditions but also we don't want
to move too far away from the beginning
so we don't just do a random walk in
value space now some people call this
equilibria some people call it fixed
point and they are technically correct
but I don't think that I don't think
that's necessarily a good way of seeing
it because it's the result of a
complicated constructive process and
it's better to look about co-located
constructed preferences equilibrium
knowing your preferences does not change
your preferences yeah but that's not
human there are that's not human there
is no there is no moral learning at this
point there is so this is a I would be
very wary and at least careful about
approaching a state where knowing your
preference does not change your
preferences there are many bad ways of
getting to that state lobotomies for
example or manipulative manipulating
people into the equivalent of artemis so
i if i said that equilibrium was a
desirable target and if it's really hard
to get with something reasonable we're
really easy to get with something
dangerous then I would be very wary of
it that's the main reason that I haven't
looked too much at equilibrium
okay the capacity to explore this is
without actually using the human because
I feel like then you can look at you can
look for equilibria which are stable but
also which the current humans identity
preferences wouldn't object to just
rumbling or something like that I'm not
sure I mean because you have this
problem that the thing can explode off
into some distant thing but if you do
you have to take the human with you or
can you can the system predict what the
human would do and do several steps and
then ask the current human is this a
place you would be happy to end up I
mean that well this is um if the AI
super intelligent asking the human is
kind of irrelevant this is that a
manipulation of finding the best
phrasing to get the AI of a human to
accepts the best the aim of the
synthesis process is so that all this
can be done automatically well yeah
pretty much
I'm gonna yeah the grass is always
greener is a way of saying that yeah
it's basically the answer that I gave to
the previous question since since it's a
known feature that humans tend to reject
having there that is imposed on them I
would not want to look for something
without that feature or at least not put
too much pressure on that unless that
push us away from the human areas that
we know
regarding imposing values on people or
seeming to impose them this is a picture
of the AI presenting presenting results
to the person but it you instead if you
get the person to make a series of force
choices now always burn through a
questionnaire or something there's
nothing with alternative it's making
choose and so on the result becomes out
at the end of something that they see as
something they've produced and I think
they wouldn't resist it in the same way
they might be surprised they might find
it very difficult to do it but they
wouldn't see it as something imposed on
them so I mean perhaps you could say the
role of the AI in that situation is not
to come up with the answers but to
conduct this process this transparent a
transparent and acceptable process what
you're saying there is basically the
nicer version of what I said the AI
figures out where your preferences are
and manipulates you into accepting it
this is this is um this is basically how
do we get the endorsement of the human
on the outcome of this process in a sort
of general sense and the yes um I have
not figured out how to do that mainly
because I have a lot more details of the
graph
of the whole process before I get even
even near these sort of what's the final
thing you do with it but yes thinking
about how humans connects in your sleep
the outcome in a good way is going to be
valuable when this is deployed in
practice and it might be worth
sacrificing or not necessarily
sacrificing but tweaking certain aspects
of the process
in order to get human endorsement as
long as it's not an overriding and
overriding goal and I just want to to
point out that all these questions have
been about the synthesize the human
preferences which I said is the longest
part of the other thing and it is but
also somewhat less importance than the
finding out the humans internal models
and I understand why because the second
part is so much more interesting to talk
about even if it is in a sense less
important and I've got another question
here can human preferences be circular
yes human preferences are very often
circular though we have we interestingly
have a we have something that stops us
getting stuck in loops it tends to be
that as we travel around the circle we
break the cycle at the point where we
come back to the beginning if if there
would be a circular preference it's kind
of like we have a deep knowledge that
getting back to the starting point makes
no sense
okay so I can see the next cat did I
answer the previous question properly I
feel I might not have addressed anyway
my question yeah yeah
I yes that's fine for the moment thanks
I'm okay so unclear about all this
litter that you know but that's not your
fault
Thanks
okay so the next question actually was
also about different ways you could do
this synthesis both with a something
like the parliamentary modeled by Nick
Bostrom and I think David also had a
market-based model but I think you might
be right
that is somewhat less interesting so let
me see there is here oh yes but but it's
more robust in a way that if we do the
synthesis I would expect that a lot of
different kinds of synthesis processes
end up with basically the same result or
at least similarly acceptable results
there might be somewhat difference in
outcomes but equally acceptable if they
don't then this is something to be
panicked by because that means something
is going wrong if slight changes in the
sort of process the synthesis process
can result in things that are not only
wildly different but moving from good to
bad and back then there's something
wrong with the process there's some key
flaw that we've not addressed okay I
know let's I will take Rob's question in
just a moment and then it seems like
a big part of the reason why we expect
us to not have catastrophic consequences
is that we are taking all the humans
part your preferences and leaving
nothing in the model behind and I think
you know seeing all the partial
preferences seems like it's all order
there is probably something that might
be infinite many of them and even if we
get infinitely many of them we might
still skip a part of them this is a
correct I mean we human the imagination
is not infinite and even if it is
technically infinite it's sort of
infinite because you can slider one
variable continuously in a small range
so the I don't know it's is it section
two point six two point eight basically
what I'm saying is that most preferences
should be absorbed in the model leaving
only very few very few rare types of
meta preferences that are really about
the outcomes and it really resists being
synthesized into the model these are the
ones that I'm appealing to when I'm
talking about an adequate outcome or an
acceptable outcome or a good outcome
like it's just a minor like we we're not
total utilitarians
for example and we're definitely not
average utilitarians and the difference
between say 51 percent total utilitarian
49 percent average or vice versa or any
sort of mixed theories like that if you
tweak the percentages by one percent in
each direction you
up in a very very different world
because the optimizing the optimizing
goal is different and this can result in
a very different world but these sort of
judgments that they're all kind of okay
that some type of under the finest in
our preferences mean that there's some
openness for different ways of
synthesizing it this for example is
something that can't be really captured
as an actual human preferences within
the model it's a statement about a bit
of variability outside the model and
we're sorry it's I will stop using the
word model because there are too many
uses of it this is a statement about the
synthesis process and the whole thing
where I say beware disastrous outcomes
or add extra precautions to avoid
disastrous outcomes either sort of
judgment
it's a meta preferences about the
violation of all other preferences in a
sense and it kind of works as a meta
preference within the synthesis process
but you can't I don't think it can fully
be captured I have another point this is
a really minor point about the the fact
that complexity might not matter that
much
because the AI can be efficient routing
around human complexity without losing
it that's a completely impossible thing
to have as something within the model
this is a sort of odd within a synthesis
process this is an odd judgment call
from outside about what's acceptable so
there are a bunch of things that really
feel as if they can't quite get into the
model but apart from those the idea is
that everything your preferences your
meta preferences your enjoyment of the
parliamentary model or of different
other ones all that should go in
okay I think that answered my question
then Robert had one in the text okay so
this is why I did a she to three posts
about how I saw human symbol grounding
functioning the basic idea is to do it
in a sense in theory identifies suppose
I was to say that there is a certain
certain neuron combinations that when
they go off together
this means mother-in-law or this means
that particular person smelling or
something like that so I have grounded
and claiming there's a grounding for
this these internal symbols how would we
seen what's fires so in a sense what I
when I make this claim I am saying that
the firing of these neurons should
corresponds to certain states in the
outside world and notably those where
the mother-in-law appears or is so I'm
saying than actually knowing the
internal states of the internal
variables I can make predictions about
the outside world this is my sort of
Avenue towards empirical simple
grounding it's it's sort of saying if
you say this is the symbol this is its
grounding and I say okay I will check
that I will look at the symbol by which
we mean certain things inside the human
brain and I will see if it curve varies
with what it's grounded on which is
certain states of the outside world
okay edges also have a question then I
just realized I should have asked him to
type it in in advance here he has done
that great
I wanted a idea the idea to get lynly
better satisfy me as it learns my
preferences so the graph what sorry I
think um do you want this to be an upper
bound is it a lower bound is it an upper
bound
for me I would be most interested in a
lower bound in at least what can we be
reasonably certain to avoid can be peas
certain to avoid that it suddenly
believes that we like suffering or
something like that well I'm asking
because there was some people who
actually prefer that an AI not become
brilliant at satisfying their
preferences immediately I can see why
they might want that this is basically
the courage ability instinct that it's
basically safer if the AI grows in power
slowly and so on
but if we put that aside then I don't
see why we want it to be linear as a
lower bound we want it to go as well as
possible
[Music]
the the thing is that the graph doesn't
always look like that is this what this
suggests is that basically we need to
get the the irreversible preferences in
quickly I mean the AI should naturally
do this but if we wanted to sort of hand
design it it's basically the ones that
can't be undone to be the Preferences
should be there and quickly before it's
before it gets there sorry before it
makes too many changes as I say normally
this is something that should be picked
up automatically but it is an argument
for at least having the AI in a training
box for at least a while
and the next question is so what
increments sort of intervention do you
think every department process of
learning or in the intermediate stages
in AI develops no idea it's sorry I'm
being a bit glib but I really have no
idea how the process will go out and how
how fast the AI will converge on
something we find adequate and whether
how we and how we perceive it from
outside were the eight and next question
what the AI is brilliant at satisfying
their preferences they would know to
knock Britons you satisfy that person's
preferences yes that is the case unless
we don't want the air this is basically
a form of manipulative behavior one of
the reasons that I've given up trying to
rule out manipulative behavior is that I
find it impossible to define so being
this is this is sort of being it's
respecting their preferences but it's
also manipulating them because one of
the reasons what they want you to go
slow is because they're afraid what
happens if you could go fast and if you
could go fast but to tend to go slow
this is sort of manipulating them and
it's for reasons like this that I can't
really find a good definition of what
manipulating means because arguably this
is a manipulative manipulation that is
in accordance with what they want so
there should be a strict ordering to
what you learn and used to throw out
something you learn too early um the
just just naturally in the process of
learning the early stuff will get
improved upon these strict ordering on
what you learn is just to prevent
mistakes irreversible mistakes at the
initial early on when the AI doesn't
have enough of our preferences
we um consider an example it's terribly
abstract mm-hmm I mean for example in it
we were just about people not wanting to
have their preferences satisfied too
quickly so I was imagining what I don't
know somebody winning the lottery
winning a million pounds on the lottery
and saying I'm afraid this well what
this will do to my life so I'd like to
have the money fed to me more slowly or
I'd like to experience the life of a
millionaire for a short for a for a
month and then decide what I want to do
after that is this what we have in mind
and like is this learning throwing out
something that we've learned to Ernie
and I need an example know what you're
saying is not an example of that what
you're saying is something that might be
desirable anyway
could be desirable depending on the
human it could be could be desirable I
was just saying something like that is
also in a sense manipulative or could be
manipulative it's it's sort of tricky
the borderline between the perfect
Butler
and the perfect evil Grand Vizier is not
a sharp one if you're if your preference
if your desires are anticipated and all
your objections are anticipated this
shades into manipulation anyway so that
that's not the throw out what something
does early I'm not saying if you throw
out anything that is learned early it's
just that it will get updated the the
things I was saying that if there is an
ordering we should have it learn like
it might sir instance nuclear plants and
kill everyone and then regret deeply but
it did this sort of sort of simplistic
example we don't want it to do that we
want it to realize that some things we
consider irreversible on the other hand
it might sort of take apart an asteroid
and fling bits of it into the Sun and we
really don't care about that in a sense
both of these are equally irreversible
but be killing everyone is something
that strongly objective and the
disassembling and asteroid and chucking
bits into the Sun is something we
basically don't care about at all so we
should quickly teach it what we consider
irreversible of disasters so that it
doesn't do those in the process of
learning the other values
okay let me see if I can find and by the
way could I watch there I think you
should put evil Grand Vizier just in
case people don't um don't realize okay
perfect in the beginning of your
presentation you had an example of three
people estimating the rationality h ln k
and believe you called them obtaining
pretty much the same results in that all
three had the same models and believe
that people the reason why people were
playing chess is because they want to
win because that is what our empathy
model would believe we would do that in
that situation but it seems that this is
say that these three people are likely
to fail in very much the same way so how
can we I've taken quote here from you
that we can we can see that the method
stops working when the internal models
that we have and those others have start
to diverge and my feeling is that they
might be that there might be broken in
in pretty much the same way well so to
give an even simpler example than the
chess one if I was to start get red in
the face start shouting things at Suren
and punch him
people would conclude that I was angry
at CERN and wished to do him harm I I
would agree with them in this
hypothetical almost certainly so this is
an example of a behavior where almost
everyone agrees on it these sort of the
strong emotions people tend to agree on
it and there is a lotta that these are
sort of the strong similarities I've
seen now just quick when it might also
make some
might be something like the reason
actually the reason while you are red in
your face is because you've been stung
by a bee and have a relative to that and
you're trying to swipe away a piece from
my face so so that's why everybody
thinks that you are angry with me but
actually you are trying to protect me
from the bees that's a possibility and
hopefully further evidence should
elucidate that problem but to get back
to sort of your question I'm saying that
the three models are sort of similar and
this is and this is true but it and that
it's a sort of shortcut but what I'm
defining to be the true one is the
person's internal model so whether I'm
angry or in pain and helping you so
angry versus bees the true model is the
one inside my head now maybe I was angry
at you and then I noticed that there are
hundreds of people watching us and then
I noticed a bee and then pretended that
the bee was the cause of it and then
people would misjudge that and they
would get it wrong but the truth in the
sense is the model with in my head the
similarity with other humans is a way of
accessing this truth at least better
than random without disassembling my
brain and having perfect death in my
wrong right or that kind of thing so
when the winner Turner models okay so so
in a sense our models of other people's
models are proxy measures so they fail
in the way the proxy measures failed
especially move beyond typical
situations but my argument is that in
the grand scheme of things they're
pretty good proxy methods they can get
us a significant chunk of
way might we not run into good hearts
law with these yes because we're trying
to end up in a very different place from
where we are right now if we rely on my
models of someone else's models to give
them their values but we don't run into
good hearts law we are basically using
the wrong definition what's gonna happen
is we're gonna sign someone their values
which is my judgment that they values
what should happen is that we have my
judgment of their values other ways of
measuring their internal models either
through their stated preferences in
reliable situations and in unreliable
situations so lots and lots of examples
of people saying things which we believe
are true Peter thinks two things if you
believe or false lots of different
estimates for what these internal things
are and maybe an actual definition of
what it is which the AI can then sit the
outside evidence towards finding that
but yes if you say mind if you use my
internal models to find someone else's
values then it will be it's wrong it
won't be me it won't be as its disaster
mode is different from other good hard
situations as am i but it's the wrong
you're doing the wrong thing and if you
don't add some uncertainty or some
fuzziness or some definitional thing
just to reflect the fact that you're
doing the wrong thing then you won't
correct for the fact that you're doing
the wrong thing
okay let's another question you have you
defined the one-step hypothetical where
you have like a minimal one step change
in the human's brain where you post that
and then one simple thing and then how
would you react in this situation and I
expect you could also say that's like a
serious step hypothetical like a simple
question and then maybe an infinite step
hypothetical that would be in this
really really strange situation what
would you prefer something where you
which seems to me like the coherent
extrapolated volition I don't know if
this is precisely what you mean so and
also my question is one step seemed to
be fine but two steps are not how come
it's basically dealing with the whole
problem of manipulation again the more
interaction there is the more
possibility for manipulation there is
and also the more possible to do is for
creating human bodies where they didn't
exist before if you take say some
experiments yours have isolated tribes
and you present them with episodes of
Star Trek or something like that they
may develop preferences for something
completely outside the realm of
expertise because they're sort of
presented a story which tugs on their on
their standard field yeah but basically
it's not the further you go the more
scope there is for manipulation if we
can or for just creating the values out
of thin air if we can deal with that
problems then there's no problem with
multiple step I thought the
hypotheticals it's it's yeah it was it's
exclusively for that and the realism
that's not just a zero step is so that
we can actually go somewhere and it
isn't immediately
part of so with one-step hypotheticals
how far away can you get from our
current state can you get sufficiently
far
well my innocence I expect that this is
going to be a sort of empirical question
however we define one or two step
hypotheticals one sign that things are
going wrong is if rephrasing of things
that are logically similar or logically
equivalent gives very different answers
on the part of the human so that's a
sign of well manipulation about our
creating new values of those kind of
things so in a sense how safe these are
is a relatively terrible question
there might be something also there
might be situations like with um
adversarial examples in battery learning
it might be that most examples most
hypotheticals you can phrase are okay
but there are a few they give really
edge results in that case we might be
able to just say use this sort of
central examples
don't use the optimized edge cases while
the equivalent of adversarial examples
so yes so this is a two stomachs this is
a hiker this is there are empirical
waves dealing with this hopefully yeah
let me see if we can find any yeah I had
I had this one that was actually more a
question of see ensuring that I
understood because in the ring group I
try to explain some of this and and I
had four kinds of meaty preferences and
I think I might be confused about
southies so here I have the standard
self referential like all simple
meaty preferences should be penalized
Russell style preferences penalizing
those that are not mentioned explicitly
the set of preferences that are not
contained in it and and once whether
they're in consistence I call Goodell
style and the entry services preferences
those that don't want to be written a
person doesn't want to be reduced to
your Sufi function is this roughly
correct
um I wouldn't necessarily label them
like that but these are all interesting
examples
join me can I go through them please do
so the last one is the one I've already
told you that I don't really know how to
cope with I've sketched a few examples
of how it might work but those are sort
of real hats all simple Mehta
preferences should be penalised should
not be assigned to itself it's not
explicitly self referential and if you
remove it's it's implicit self
referential that I mean this is
something that someone can have in a not
too silly example like I might have a
mild version of this due to my fear of
losing oversimplification and thinking
about it I would not want this to be
applied to itself because it's there to
prevent things from happening not to be
not to be self referential and tossed
out so there's I see no problem in so
self referential things are a problem
but I see no problem in just saying this
one don't apply to itself all
preferences not mentioned here should be
penalized presumably there's a list that
goes with it or references that are
mentioned here um this this sort of all
preferences that don't come from my
study of the Bible should be penalized
is something that some people have for
example these sort of religious
preferences are tricky because they're
generally factually wrong or they
premise things that are factually wrong
but there's probably secular versions of
this of everything that does not derive
from Marxist Leninist thoughts within me
should be penalized again I see no
problem with not self referencing it
here is this Russell's paradox is that
when the set that contains all sets that
don't contain themselves if that's
contained within itself and so you talk
about Russell style preferences but you
don't give an example so I'm wondering
if this is an example or do you have a
better example in mind of a Russell
style preference Russell style I think I
was thinking of versa
Russell and Goodell's style as basically
self reference as just a shorthand for
self reference or for things that's
saying bad things about themselves or
rule themselves out like the bar okay
Russell would be the barber who doesn't
the barber shades only those who don't
shade themselves all preferences not
mentioned here should be penalized then
you give a long list that doesn't
include that preference well in this so
the thing is when I say glibly this
should not be allowed to refer to itself
this is what people tried to do
mathematically for a long time and
failed they were sort of saying well
these goodell things these Russell
things they don't the kind of very
dubious examples they don't crop out
much why don't we just put them aside
and it turned out we couldn't do that
mathematically but in genuine human
preferences I don't see these sort of
things coming up very often and
generally when they in most cases that
the self reference is just don't let it
self reference and if there are if there
are if there's a cycle of to preferences
referencing each other in a way that
would push themselves up and push
themselves down
that just break the cycle and if there's
a contradiction deal is it like other
contradictory preferences so this is an
approach that I think should apply to be
applied to humans not to any logically
conceivable agent just because and so
inconsistent preferences should be
penalized well if you have a value of
inconsistent yeah if you have a certain
judgment of inconsistency that might
include this thing itself um if it's
again I think just breaking the
reference to itself could work if these
are weak preferences then the main thing
is not to allow them to blow up
preferences should not be allowed to
blow up to minor preferences should not
become major through self references
major preferences should not become
minor to self preferences but minor
preferences could stay minor or become
vanishing true self references that's
less of a problem yeah so this is kind
of my answer now that I sees things in
the chat have appeared yes Robert has a
question first
okay stuff fresh and meta preps are
troublesome so all meta preferences that
apply to themselves should be penalized
yeah possibly but I mean people don't
really have preferences like that or
they tend to hold them quite weakly they
might make themselves hold them strongly
the way I am basically going to try and
do it if I have to do this is to assign
every preference to a ordinal or to a
integer and all Preferences refer only
to ones that are lower themselves on the
scale and I am going to do some sort of
energy minimization to map everything
to this format so the some sort of
energy minimization process to break all
the cycles could you Victoria
laboratories understand it's self
referential why would you want to
penalize simple meta preferences
okay um I okay there is a meta
preference that I am very scared of and
that is simple preferences should be
over weighted I see this leading to the
replacement conclusion I see this
leading to discarding so many of human
preferences on the altar of simplicity
so that is a meta preferences that I am
very scared of and over an over seeking
of simplicity is something that I think
could end disastrously
so I do not want to penalize simple meta
preferences I want to stop them from an
oversized domination so that's why I
said I might have something vaguely in
that area of course this is within
limits if you don't have some form of
Occam's razor you don't get anywhere so
I'm not in favor of arbitrarily complex
preferences either but especially for
philosophers I see the failure mode of
over simple preferences well as being
the problem okay yeah I have another
question about this so let me see if I
can find that
so I'm Rob miles mentioned said that
simple preferences should be boosted is
itself simple yes and this if you have a
very mild preference for that it should
not then become a dominant preference
just through
in itself it's very clear that
preferences should not be allowed to
boost themselves to heavyweights if they
don't have that weight initially I've
tried to here on the screen come up with
some arguments why you might actually
want to be a while might not be so bad
or why I might be desirable to have a
more compact simple elegant utility
function like for instance if you want
to convince people to have an utility
function then if you can say like if you
have some big symbols like this is your
value of friendship this is how you
value art if you have a utility function
with these kind of big things in it then
people would probably be more likely to
accept it compared to if it's very very
specialized and it's like a million
factors and people don't understand it
at all so that's my my first argument
for this and the second is that might be
possible to prove statements about
elegant utility functions things like
you will not regret if we try if the
utility area to maximize this utility
function and I think also if you want to
try to do some any kind of philosophical
validation on it that might be easier
with an elegant utility function
compared to one that you can test with
the computer and see what what the
utility function does in different ways
but you can't really understand it so
the second statement I don't think is
all that relevant because this is about
how you can convince someone you will
not regret adapting this utility
function well then after that you you
might just have no opportunity for
regret this is basically an equilibrium
statement that doesn't mean all that
much unless we define regret in some
complicated
it captures on current feeling of what
we're great would mean easier to
convince people to accept possibly I
also feel that you might be able to
obfuscate a lot of the nastier parts of
their self-image in some of the details
but it seems about that is plausible
different class of errors can hide an
elegance and different class of errors
can hide in verbose elegance most of the
work is philosophical while the most of
the work is computer based testing
I just basically see people's
preferences as um yeah you virtually
never see someone say well you very
rarely see something that is perfectly
defined my preferences but actually
you've gone a bit too complicated you do
see examples of that so I see your
arguments for elegant utility functions
I just see that the mess of human
preferences all the clutches we have and
that's a too strong preference for
elegance would involve sacrificing huge
amounts of our current values and in my
view far too many of course this is a
relative statement and people who are
philosophers who have strong meta
preferences here will end up probably
with more simple and elegant utility
functions than most people
okay I think that just about conclusive
oh did Robin have a final question here
they also know that there actually is a
simple clear single let's cut result in
Weiser Salamis with the process find it
um in this hypothetical what defines
that there is on the clock shows all of
human value I suppose if there was some
if it you could imagine something that
was designed to spit out something
simple and then it does spit out
something simple and everybody goes oh
yeah that's correct I don't know how we
manage to try and do all this philosophy
for so long without realizing that that
gives me what I want this is actually
what I value and if it's a process
capable and finding that if it did exist
it's capable of finding some versions of
it it's some situation in some
situations it is not in all situations
like you're I mean what you're
describing seems very similar to a
hypnotic hawk did not want we would not
want it - right right - fine but if like
way I could see it happening if it does
all this complicated things and it turns
out that there are say 500 factors or
little between dramatic 50 factors on
which all of human values load very
strongly and all the rest looks like
noise then this is some way that it
could end up finding this this thing
it's yeah the the ultimately convincing
argument I'm not probably won't find
because all the precautions to avoid
manipulation on
going to be pushing against that because
but the but it yeah so these sort of
things that are confirmed yeah that's
the right thing
it wouldn't certified things that are
surprisingly simple when you do the
extraction process it would find okay
Chris you have a metaphor yes in my I'm
always searching for concrete examples
to UM to ground my understanding of
these mind-boggling abstractions I'm try
and also deeply I react very strongly
against the idea of synthesis what it
means to synthesize human values into a
a general utility function and so I what
is it doesn't make sense to say to
compare the process to like if you have
you have a lot of runners in the 10,000
meters they will have their separate
preference is very largely their
preference is to win the race and
obviously those are mutually
incompatible and so you organize a race
so the business of when we synthesizing
their values values
you don't really synthesize their values
what you do is you you organize a race
with the rules so that you can have an
outcome and they can participate knowing
that this is at least gives them a shot
at achieving that achieving their
preferences and beyond this if you think
about organizing the Olympic Games again
you can't synthesize everybody's
preferences or values or anything like
that except in a very general high-level
sense and somehow we do manage to
organize Olympic Games which a lot of
things are going on it was disaster make
money which you personal fame to achieve
national
or - I don't know promote peace panic
between the nation's or something all
these different things are being you are
attempting to achieve all these things
in organizing something like the Olympic
Games and to some extent you succeed is
this an analogy for what you are trying
to do with your research agenda or is it
a bad analogy um it could work as an
analogy by the way I keep on using
synthesize and construct to emphasize
that this is a constructive process with
a lot of contingent choices that and I'm
also not particularly enamored of
utility functions just for the sake of
utility functions but they seem to be
stable in the way that other
representations are not but maybe an
honor to develop your analogy it would
be how can we arrange the outcomes of
the Olympic Games to satisfy people's
preferences to the maximum obviously we
only have a certain number of gold and
while golds and bronzes and even
Silver's to give out I give them out in
that order because that seems to be the
happiness order of the people who get
the medals but we also have
opportunities for people to get an ace
unexpected sponsorship deal to maybe
fall in love to outperform some personal
milestone to be a key player two key
moments to make it on television this is
what I was meaning when I was saying
that very few values are completely
antagonistic then this is basically
finding the best way of satisfying the
most and the things the values within a
human tend to be
even more non antagonistic like someone
who wants to eat chocolate and have
great self-discipline is these are not
pertinent there are sort of
technological or their worlds in which
these can easily be both satisfied and
there are ways of building their
self-discipline and allowing them to
indulge in the right amount of
chocolates for examples even these
things that seem as if they're
completely antagonistic or just sort of
mildly so does that so that's sort of
yeah so I does that combined with your
metaphor sort of help and I think so I
think so I was thinking specific for
directly than the Olympic Games I was
thinking about you know political ways
of ordering society where you have these
conflicts between freedom and equality
and Mitt utilitarian well-being for as
many people as possible and so that
that's the kind of attention so now it's
wondering whether an AI consoles for us
so yeah that seems to tell you about it
with what you're saying so it's a you're
saying some the optimistic that these
things can be well given given sort of
given superpowered a high then yes I
mean there are a lot of I mean people
can be satisfied in surprisingly odd
situations I mean there's some people
whose best time of their lives were in
prison there is the whole medium fish in
a small ponds which many many people
seem to like it's just a question of
setting up the various small ponds like
if your online games
for example um people seem to enjoy them
and enjoy a certain level of success
even though there's only sort of one
person who's the best at that game at
any time and a couple of hundred who are
even in contention talk and people still
seem to enjoy that and get sort of rush
from victory and enjoy being higher in
the ladder than other people so that
there are but there are a lot of there
are a lot of levers that a super
intelligent AI a super powered a I could
pull so this is enly sort of utopia
everything is solved or kind of thing
the easiest thing is that a lot of
people are antagonistic because they're
sort of have different status like
they're on the same stage of slack
ladder or the same dominance ladder the
easiest thing is to put them on
different status ladders and have their
worlds going more the way they want or
the bits that they see or the bits that
are relevant to them okay okay so I
think that was just about it so I would
like to say thank you once again to a
rap song for coming here and answering
our questions and we wish you good luck
with your research agenda thank you and
if anybody wants to sort of take part or
help or direct anyone towards it I'd be
greatly appreciated and general feedback
is also welcome oh and on the auricle
question there is a a thousand pounds
available for winning the contest and
the link I posted the very beginning of
the talk so thanks everyone
thank you and on a practical note next
week we
be meeting in the reading group on
Tuesday the 21st and we will discuss
Jeff Hawkins utter audible about why he
is optimistic about AGI within the next
20 years so thank you everyone and see
you next week


AI & Logical Induction - Computerphile

today i thought we would talk a bit
about logical induction uh paper out of
the machine intelligence research
institute very technical paper very
mathematical and can be a bit hard to
get your head around and we're not going
to get too far into it for computer file
i just want to explain like
why it's cool and why it's interesting
and people who are into it can read the
paper themselves and
find out about it
one of the things about logical
induction as a paper is that
it's not immediately obvious why it's an
ai safety paper
um
but like
my
perception of the way that miri the way
that the machine intelligence research
institute is thinking about this is
they're saying
they're thinking like
we're going to be producing at some
point artificial general intelligence
and as we talked about in previous
videos
there are
hundreds of ways really weird subtle
difficult to predict ways that these
this kind of thing could go
uh very badly wrong
and so
we want to be confident
about the behavior of our systems before
we turn them on
and that means ideally we want to be
making systems that we can um
actually
in the best case that we could actually
prove theorems about right things that
are well specified enough that we could
actually write down and formally prove
that it will have certain
characteristics and if you look at like
current machine learning stuff all the
like deep neural networks and
uh that kind of thing they're just
very opaque do you mean by that because
we don't necessarily know exactly why
it's doing what it's doing
yeah and also just that the
the system itself is very
uh is very complex and very contingent
on a lot of specifics about the training
data and the architecture and like
there's just yeah effectively we don't
understand it well enough but it's it's
like not formally specified enough i
guess and so
they're trying to come up with the sort
of mathematical foundations that we
would need to
be able to prove important things about
powerful ai systems before we were
talking about hypothetical future ai
systems we've got time to print more
stamps so maybe it hijacks the world's
stamp printing factories and when we
were talking about the stamp collector
it was really useful to have this
framework of an agent
and say this is what an agent is it's a
formally specified thing
and we can reason about the behavior of
agents in general and then all we need
to do is make these fairly
straightforward assumptions about our ai
system that it will behave like an agent
so we have idealized forms of reasoning
like we have probability theory which
tells you the rules that you need to
follow
to have good beliefs about the state of
the world right and we have like um
like rational choice theory about
uh
what what rules you need to follow and
it may not be
it may not be like actually possible to
follow those rules but we can at least
formally specify like if the thing has
these properties it will do this well
and all we're trying to do is they're
thinking if we're building very powerful
ai systems
the one thing we can expect from them is
they're going to be able to do thinking
well and so if we can come up with some
formalized method of uh of exactly what
we mean by doing thinking well then if
we do
if we reason about that that should give
us
some insight into how these systems will
behave that's the idea so let's talk
about probability theory then this is
where we should have a demo pretty sure
about gamma has nice
yeah
oh this is a fun one
powers of two i don't know how to play
back cameron anyway here's a die
so
basic probability theory right i roll
the die
now it's under the cup
and i can ask you what's the probability
that that is a five one and six right
one and six because you don't know it
could be anything there was a time when
people reasoned about the probabilities
of things happening in a very sort of
fuzzy way you'd be playing cards and
you'd be like oh
you know that card has shown up here so
i guess it's less likely that he has
this hand
and people would
um
intuit yeah and like
there was some sense of like
clearly we are doing so what we're doing
here is not random it's meaningful to
say that this this outcome or that
outcome is more or less likely by a bit
or by a lot or whatever but people
didn't have
a really good explanation of how that
all worked until we had probability
theory right we didn't it wasn't a
system right right
and like practically speaking you often
can't do the full probability theory
stuff in your head for a game of cards
especially when you're trying to read
people's faces and like stuff that's
very hard to quantify but now we have
this understanding that like even when
we can't
actually do it what's going on
underneath is probability theory so
that's one thing right straightforward
probability
now let's do a different thing and now
i'm gonna now i'm gonna need a pen
now suppose i give you a sentence that's
like the tenth digit after the decimal
point of the square root of 17 is a five
what's the probability that that's true
well it's a much more difficult problem
one in ten
right one in ten seems like a totally
reasonable thing to say right it's going
to be one of the digits we don't know
which one
but
if i you've got the paper here you know
you can do like
division you you could do this if you
had long enough like if i gave you say
an hour with the paper and you could sit
here and figure it out
and
let's say you do it and you you come up
and it seems to be like three i don't
know what it actually is i haven't done
this calculation let's do the
calculation bad expression oh i've done
it wrong way round one two three four
five six seven eight nine ten it's a six
ah it's a six okay i picked that out of
nowhere so suppose you didn't have a
calculator you were doing it on paper i
gave you you know half an hour or an
hour to do the like long division of
whatever it is you would have to do to
figure this out what's the probability
this statement is true now
what do you say i'm gonna say zero right
but
you've just done this in a big hurry on
paper
right you might have screwed up
so what's the probability that you made
a mistake you forgot to like carry the
one or something so it could be one so
it's not zero yeah there's like some
smaller probability
um
and then if i left you in in the room
again still with a piece of paper for
100 years a thousand years infinite time
eventually
assuming that was correct eventually you
say zero right
um
you come up with some like formal proof
that this is this is false
um
whereas you imagine if i leave you for
infinite time with the cup
and you can't look at you can't look
under the cup
it's one-sixth and it's going to stay
one-sixth forever right when you're
playing cards you you've got these
probabilities for which cards you think
depending on the game
and as you observe new things
you're updating your probabilities for
these different things
uh as you observe new evidence and then
maybe eventually you'll actually see the
thing and then you can go to 100 or zero
whereas in this case
all that's changing is your thinking but
you're doing a similar sort of thing you
are still updating your probabilities
for things
but probability theory kind of doesn't
have anything to say about this scenario
like
probability is how you update when you
see new evidence and here you're not
seeing any new evidence
so in principle whatever the probability
of this is it's one or at zero
just as a direct logical consequence of
things that you already know
right so your uncertainty
comes from the fact that you are not
logically omniscient in order for you to
figure things out it takes you time
and this turns out to be really
important
um because so most of the time
what you have is actually uh kind of a
mixture of these types of uncertainty
right so let's imagine a third scenario
right suppose you're like an ai system
and that's your eyes because you have a
camera
i do the same thing again
and now
i ask you what's the probability that
it's a five
you would say one sick because you don't
know but on the other hand
you have
recorded video footage of it going under
the thing the video is your memory of
what happened right so you're not
observing new information you're still
observing the same thing you observed
the first time
but you can look at that and say
hmm you know it looked like with the
amount of energy it had and the way that
it was rotating and which numbers were
where it looked like it probably wasn't
going to land on a 5. if i asked you on
a millisecond deadline you know
what what was it what's the probability
that it was a five you're going to give
the same number that you gave at the
beginning here right one and six
but
you can
uh look at the data you already have the
information the observations that you've
already made and you can do
logical reasoning about them you can
think okay based on the speed it was
going the angle it was turned out and
which which pieces uh which faces were
where and so on
it seems like you know maybe you can run
some simulations internally something
like that
say it seems like actually
less likely than
one in six right if before i thought it
was like 0.16 recurring now i think it's
like 0.15 because
all of i ran a million simulations and
it seems like that the longer you think
about it
the
more precise you can be maybe you keep
thinking about it again and you get it
down to like you think it's actually
0.13
right something like that
but you don't you can't
you don't actually know
right because there's still things you
don't know like
you don't know exactly the physical
properties of the paper or what exactly
the inside of the cup is like or the
waiting of the diet like you still have
uncertainty left because you haven't
seen which way it landed but
by doing some thinking you're able to
take some of your
uh you're able to like reduce your
logical uncertainty the point is that
probability theory
uh in order to to do probability theory
properly to modify your beliefs
according to the evidence you observe in
a way that satisfies the the laws of
probability you have to be logically
omniscient you have to
be able to immediately see
all of the logical consequences of
everything you observe and propagate
them throughout your beliefs
and this is like not really practical in
the real world like
in the real world observations do not
form an orderly queue and come at you
one at a time and give you enough time
after each one to fully integrate the
consequences of each observation
um because human beings are bounded
right we
have limited
processing ability and limited speed
with this kind of logical uncertainty
it feels very intuitive
that we can do probabilities based on
our logical uncertainty that we can we
can think about it in this way it makes
perfect sense to say that one in ten is
the probability until you've done your
thinking one in ten seems like a
perfectly reasonable number but like
why is one in 10 a good answer and like
50 is not a good answer because you
might look at it and say well this is a
logical statement
um
it's either true or false that's two
possibilities you know 50
why why is one in 10 more sensible and
in fact you had to do a bit of thinking
to get there right you had to say oh
yeah it's going to be some digit there
are 10 digits i don't have any reason to
prefer one digit over the other so 10
so you are like
always
doing reasoning to come up with these
probabilities but the thing is that
according to standard probability theory
this is like kind of a nonsense question
because if you imagine like from the
perspective of probability theory this
kind of
statement is equivalent to saying what's
the probability that one equals one
right or what's the probability that
like true equals false they're all just
it's just a logical statement
um
it doesn't have a probability it just is
true
um but if you
can't think infinitely fast then you
need to
have
answers you need to have estimates it
estimates you need to have estimates of
your answers
before you've thought for infinite time
about them this is like an important
thing for actually getting by in the
world because as i say like observations
don't just line up and wait for you to
reason all of their consequences through
so when it comes to empirical
uncertainty we have probability theory
which has these axioms
um and these are sort of the rules that
you need to follow in order to
do probability well
and they're sort of things like that
probabilities
can't be negative they have to be
between zero and one um
the
if you have a a set of mutually
exclusive possibilities and that that
constitutes everything that can happen
then they all need to add up to one
um
and then if you're if you're doing
probabilities uh about
like logical sentences then there's all
these extra rules that that make perfect
sense like you have statement a and
statement b you also have statement a
and b then statement a and b has to be
less likely than a or b or the same it
could be the same but like
if a has
20 chance of happening
then a and b
couldn't possibly have more than 20
chance of happening right even if b is
guaranteed to happen it can't so that's
like a rule if you find in you're doing
your probabilities you have a and b that
has a higher probability than either a
or b then you've made a mistake uh that
kind of thing you have these rules it
would be nice if we could find
some way of doing of dealing with
logical uncertainty some some similar
set of rules that is useful
and so um
and so related to that we have these
things like um
like dutch book arguments
which are
about uh
i don't know why they're called dutch
i feel like the english language just
like
is so harsh to the dutch
for no reason
unjustly maligned but anyway um
where like if you if basically there's
there's theorems which uh
which you can prove that if your beliefs
don't obey
these rules
then there will exist some set of bets
by which you're guaranteed to lose money
some set of bets that you will happily
take by looking at your probabilities
um that you cut you can't possibly win
whatever happens you lose money
and that seems like it's stupid like you
don't want that in your beliefs so
that's like an argument that says
um
that your beliefs should obey these
rules because if they do you're at least
never gonna end up in a situation where
you're guaranteed to lose by betting and
that's like one of the
things we want beliefs for
we want some kind of equivalent thing
for logical uncertainty
um and that's what this paper is trying
to do it's trying to come up with
a rule so yeah you could you could say
if if you are doing your probability
theory stuff in such a way that there
are no dutch books that can be made
against you
then that's good and you're doing well
and so when we're talking about advanced
ai systems we're gonna it's like a good
assumption to make that they're at least
gonna try
to not have any dutch books against them
in the way they do their probabilities
so they will probably be obeying
probability theory
and it would be nice if we had some
equivalent thing for logical uncertainty
that said if you are satisfying this
criterion in the way that you do this
then
uh you're not going to be like doing
obviously stupid things and that's what
this paper is trying to do when you're
talking about probability there's you
can kind of think about what properties
you would want your
system of deciding probabilities to have
people have written down various things
they would want from this system right
that would be a system of like assigning
probabilities to statements logical
statements
and and then being able to like update
those over time
so uh one thing you would want is you
would want it to converge
which just means if you're thinking
about
a logical statement you might think of
something that makes it seem more likely
and then think of something else like oh
no actually it's less it should be less
likely and whatever you can imagine
some systems for some statements would
just think forever and constantly be
changing what they think and um never
make their mind up that's no good right
we want a system that if you give it
infinite time to think we'll eventually
actually decide what it thinks
so that's one thing you want to converge
secondly you obviously want it to
converge
like to good values so if something
turns out to be provable within the
system that it's using then you would
want it to eventually converge to one if
something turns out to be disprovable
you want it to eventually converge to
zero and then the rest of the time you
want it to be like well calibrated which
means
if you take all of the things that it
ever thought were eighty percent likely
to be true
of those
about 80 should actually end up being
true right it should when it gives you
an estimate of the probability that
should be
right in some sense another one is that
it should never
um and this one's like a bit
controversial but it seems reasonable to
me that it should never assign
probability one
to something which
uh is not proved
or that can't be proved and it should
never assign probability zero to
something that can't be disproved
um they call that non-dogmatism at the
end of all of this again at infinity
after thinking forever the probabilities
that it gives to things should
follow the rules of probability right
the ones we talked about before those
rules right it should at the end of it
you should end up with like a valid
probability distribution over all of
your stuff their criterion that they've
come up with which is the equivalent of
there are no dutch books
is based on this algorithm that they've
made which
uh
satisfies this criterion and therefore
has a whole bunch of these nice
properties that you would want
a logical inductor to have
so
the way the algorithm works is
it's one of the zaniest algorithms i've
ever
seen um
it is
it's weird and wonderful and if you're
if you're a person who is interested in
interesting algorithms i would really
recommend checking out the paper it's
very technical because we don't have
time in this video and i may go into it
deeper on my channel at some point if i
ever come to actually understand it
which to be honest i don't fully right
now
um
but it's based around the idea of a
prediction market
which is
a super cool thing that's worth
explaining in financial markets as they
currently exist you can have um futures
right and a future futures contract it's
a contract which says
i promise to
sell you
this amount of stuff at this price
of a particular thing right so you say
take a date a year from now and you say
i'm gonna sell you this many gallons of
jet fuel
for this price
um at least that's the way it used to be
down in practice the actual jet fuel
doesn't move you just go by the price of
jet fuel
and you pay the equivalent as if you had
and then they can go and buy the jet
fuel themselves
but this is like a really useful
financial instrument to have because
let's say you run an airline
your main cost is jet fuel prices are
very volatile if prices go up you could
just go under completely
so you say okay i'm going to buy
a whole bunch of these jet fuel futures
and then if the price goes up i'm going
to have to pay loads more for jet fuel
but i'll make a load of money on these
contracts
and it if you do it right it exactly
balances out
the cost of that is if the price of jet
fuel falls through the floor
then um
you don't get to save money because
you're saving money on jet fuel but then
you're losing all this money on the
contracts so it just sort of like
balances out your risk it lets you lock
in
the
the price that you're going to pay a
year in advance and that's just like
super useful for all sorts of businesses
really really good in agriculture as
well because you don't know what your
yield is going to be in
so um
so the point is that the price that you
can buy these contracts for
has to be a really good prediction of
what the price of jet fuel is going to
be
at the time when the contract ends
and you can kind of treat the price of
that as a combination of everybody's
best estimate because if you can predict
the price of jet fuel a year in advance
better than anybody else you can make
like almost arbitrary amounts of money
doing that
um and in doing so
you bring the price
to be a more accurate representation
right if you think the price is too low
and it's going to be more then you're
going to buy a bunch and when you buy it
that raises the price of it because you
know supply and demand
so
these prices end up representing
humanity's best estimate of the price of
jet fuel year from now
but the thing that's cool is
you could build arbit like this is just
a piece of paper with stuff written on
it right and these days it's not even a
piece of paper in principle you can
write anything on there right
so what if you wrote a contract that
said i promise to pay
a hundred pounds if
such and such horse wins the grand
national
and zero otherwise
and then that's on the market and people
can buy and sell it right so if this
horse is like guaranteed to win the
grand national then
this contract is effectively a hundred
pound note right it's it's a sure it's
money in the bank as if
why is that a phrase are banks banks
aren't that reliable but anyway if the
horse is like guaranteed to lose then
it's worth zero and if
the horse has a 50 chance of winning
that thing is going to trade for 50
pounds
and so by making these um
contracts and allowing people to trade
them
you can
get these good like unbiased and as
accurate as you can hope for predictions
are future events
and that's what prediction markets are
for you can make money on them by being
good at predicting stuff
and anybody else can just look at the
prices things are trading at and just
directly convert them into probabilities
uh and that's
in a in a spherical chickens in a vacuum
kind of way obviously in practice
there's various
uh things that can go wrong but like
in principle this is a really beautiful
way of effectively doing distributed
computing
where you have everybody is doing their
own computation and then you're
aggregating them using the price
mechanism as communication between your
different nodes and you yeah like
super cool
so
that's what logical induction does
it like simulates a prediction market
where all of the contracts are a logical
statement i think in this it's not a
hundred dollars it's one dollar so if
it's trading at one dollar then it's for
sure it's
certain um
saying you know this is currently
trading at uh 60 cents and i think it's
more more than 60 likely to be true so
i'll buy some and that seems like a good
investment
and so all of the traders in the
algorithm are
programs they're computer programs
um
and it turns out that if you run this
uh and this is computable it's not like
tractable in a practical sense because
you're running vast numbers of like
arbitrary programs but um it is in
principle computable
it
ends up having the market it as a whole
ends up having uh ends up satisfying all
of these really nice properties is that
because it comes into balance yeah as
the as the traders trade with each other
the ones who are like good at predicting
end up making more money
and the ones who are bad at predicting
like go bankrupt effectively you might
imagine some
trader in this system which is a program
that just goes around looking for
situations in which you have
a
and b and also a and b in the system and
the prices don't work and it's like
arbitrage
right
so um there are people who
do this currently on the stock market
there's different things you can do
where you can say oh you know anytime
the
um
any time the price is different between
two markets you can make money by buying
in one and selling in the other and
stuff like that you can do arbitrage um
and it's the same kind of thing you can
have so some of these programs will get
rich by just saying oh i notice
that
this statement for a and b
has a probability that's actually higher
than the probability of b so i'm gonna
sell that you know or i'm gonna i'm
gonna buy some b to razer you know i'm
gonna like respond to that
in a way that makes me money and because
there's all of these traders
they will uh
they will eventually
cause everything to line up right and so
the thing will converge
this is the equivalent of saying there's
no dutch book
for your probabilities
the logical induction criterion the
market
satisfies it if there's no efficiently
computable trader
that can exploit the market which means
that it can
like find a reliable strategy for making
loads of money without risk um as long
as you have that
then it satisfies the logical induction
criterion which then means you get all
of these other properties for free some
of the properties that that it has are
kind of crazy
like
it's okay with self-reference
it like doesn't care about paradoxes
there's all kinds of really cool
things that
um that don't trip it up bringing us
back to what we started talking about
how does the paper relate to the ai
safety thing right right so we're trying
to
do reasoning and possibly prove theorems
about very powerful ai systems and that
means that we want to be able to think
of them just in terms of
being good at thinking
and we've got
a lot of good theory that pins down like
what does it mean to be good at
empirical uncertainty we have all of
probability theory and like statistics
and we can we can say
these are the things you need to do to
be good at probability and we have like
rational choice theory
and we can say this is what it means to
be good at making decisions
um and so then when we're reasoning
about ai systems we can think well it's
probably going to be good at reasoning
about uncertainty and making decisions
um but we have to also assume that a
very powerful hypothetical future ai
system would be good at reasoning under
logical uncertainty because it's going
to be a physical like bounded system it
is going to need time to think about
things and it probably is going to need
to like make decisions based on things
that it hasn't
logically thought through every possible
consequence of yet so it's probably
going to be good at this too and we need
some like
formal framework with which we can think
about what it means to be good at
reasoning about logical uncertainty um
and that's like what this paper is
trying to do
they're erasable
yeah okay
um i do also have oh i had sharpies yeah
i do have sharpies oh yeah these they
squeak on the paper people don't like
okay fair enough in fact all pens do but
these are slightly better right friction