Jessica Taylor – Alignment for Advanced Machine Learning Systems – CSRBAI 2016

so our second speaker today is Jessica
Taylor a very research fellow and and
the project lead on the new value
learning for advanced machine learning
research agenda which she is here to
present for us today so let's let's take
and one thing Jessica okay um so the
thing we'll be talking about today as
patrick said is this new research agenda
that I'm working on with Patrick and
Krish at Mary and it's it's kind of a
research agenda that's more focused on
things like machine learning systems and
asking how would we create machine
learning systems that act as intended if
they if they're very powerful in general
so that the kind of methodology we use
is to imagine something like a GI in
fifteen years it's kind of and they're
just picking some time and took focusing
on that and we also imagine that there's
not really a new a new like not very
many new conceptual breakthroughs so
things are mostly going to be pretty
like qualitatively similar to modern-day
machine learning systems on the other
hand we are going to assume that they're
a lot powerful so these are still things
that you train kind of like current
machine learning system so you give them
data you have them create a model etc
but they're going to be extremely
powerful so you can just kind of imagine
these reinforcement learners that just
just like use their available
information and choose the action that
most maximizes future reward it's kind
of an extreme assumption but then we can
also think about slightly less extreme
where it just selects an action that is
like clearly better than any action that
a human could select yeah this is the
general methodology so so we've done we
have these really powerful systems and
we want to do something in the world
so this might be like curing cancer it
might be like a solving world hunger and
you don't want to do it safely there
lots of ways to use very powerful
systems to do things in the world
unsafely but to get them to actually do
what you want it seems like it's you
have to figure out some new way of
training them in order to actually make
them pursue the things you want them to
pursue so that kind of generate this
list of questions and I think my plan is
just to spend two or three minutes
talking about each at least
but then spend a lot more time going
into depth on whatever things people
find the most interesting so it was
definitely a thing where it's a small
group it's good to ask me lots of
questions and then the I guess I'll end
up talking more about things that people
are more interested in that way all
right so the first problem that comes up
when you try to think about something
like this is that if you're trying to
specify a goal then there going to be
some ambiguities sometimes so we could
imagine as an extremely simplified
example but we're trying to train a
system that detects tanks so we could
imagine that these images vary on two
axes such as is is they're actually tank
and another axis is the level of light
and we can imagine that we give our
classifier some positive and negative
examples for tanks so we have here's a
here's a tank here's another tank so
look at the picture that's labeled with
a plus for there being a tank and we
also have some negative examples that
they're over here and so since we give
them tanks and some non tanks then of
course we can learn a classifier for
distinguishing tanks from non tanks but
there's like a lot of a lot of possible
classifiers like like this is a
classifier that we want but this is
another classifier that just detects how
much light there is it's it seems like
it's even easier to learn as you just
like count how many count but some of
all the pixels and then there's like
a whole bunch of other ones so there's
some kind of some ambiguity here and if
if we imagine that a new picture comes
in and the system has to determine if
it's a tank then it seems like it should
actually just report an ambiguity should
say like there's not enough information
and then maybe a human can decide to
label this so in this case it's a plus
and then this reduces the number of
possible classifiers that separate this
this is pretty similar to some active
learning because I'm current work and
machine learning like active learning
but like the problem here is like it can
read can we actually make some kind of
realistic statistical assumptions under
which we can do this kind of thing all
right that's the inductive ambiguity
detection problem so the next problem is
the informed oversight problem if for
those who were here yesterday
I think Paul talked about this but I'll
just briefly talk about it some more so
the basic idea is that we have an AI
that's going to output it's going to
take in an input X and it's going to
output an action and sometimes we need a
human to evaluate the action for
simplicity like to make things a lot
easier we can just assume the human is a
lot smarter than the AI so the human
gets to figure out which action the a I
took and decide on a reward which then
gets used to train the AI so as a simple
example you could imagine maybe X X is
like a short description of a genre for
a story this action is a story that the
AI composes and then the human scores
like how good of a story it was the
problem is there there's some aspects of
the story that human might find hard to
evaluate this might be like if the story
is plagiarized the human will have to
well we'll have to like look at the
story and then determine whether it's
plagiarized but it's actually easier to
plagiarize than it is to detect
plagiarism because to plagiarize you
just have to select a random random
existing work and copy from it whereas
to detect plagiarism you have to you
might have to actually scan through a
whole bunch of existing works to try to
find the correct one so there's kind of
a complexity theoretic problem here
where it's
symmetrical and important oversight
problem is like how do we get around
this great so the next problem is
suppose that we want to train a system
that imitates a human and some kind of
rich action space we could imagine
something like answering questions that
are in natural language with answers
that are themselves in natural language
and if this is related to the problem of
training a generative model so we could
imagine some questions X and some
answers play and we want to learn a
function from X to distribution of Y
that will take in a question and then
sample a random answer this is the thing
that's supposed to imitate a human and
there's some orsome some ways of
training generative models but they all
have some problems an example of a way
of training a generative model is you
could imagine you have the imitator
that's trying to give an answer to a
question and a distinguisher who's
trying to distinguish the imitator from
a real human and then in theory if both
are powerful then then the distinguisher
is pretty good at distinguishing the two
so the imitator has to like actually
imitating human but in in this case a
problem here is that you could still
have problems like plagiarism maybe
maybe the imitator is going to play
Joyce from a human and humans don't
plagiarize the distinguisher will have a
hard time determining whether the answer
contains plagiarism so there's another
kind of asymmetry here where like this
this makes it a bit more difficult to
train generative models in a scalable
way but there's some other approaches to
which I we're thinking about ok so got
injective ambiguities for an oversight
and training human imitators next one is
conservative concepts so in this case we
can imagine that we have 100 burritos
we want to make a new burrito so what
does this mean you could imagine that we
have a 3d scanner we just take these
Fritos and we scan them we get a very
detailed map and then we can imagine we
have a factory and we want the factory
to manufacture additional burritos so it
it doesn't seem like can we give a
straightforward how to do this it seems
it seems like maybe one thing we could
try is have like positive and negative
examples of burritos so yeah 100
breeders and a hundred non breeders and
then train a classifier to distinguish
Breda Shimron burritos and then try to
create an object that is classified as a
burrito it seems like the kind of
approach that's the easiest to do the
problem here is that you can get things
like adversarial examples on neural nets
and so so a thing that's optimized to be
classified as a burrito is not
necessarily actually a burrito it's
gonna be difficult unless the these non
burritos are actually representative of
the possible non breeder objects out
there that the AI might want to create
so it's it's like you can't really get a
training set of negative examples for
like non burrito objects and if so it's
it's gonna be a little difficult but
there's some approaches to try you try
tackling this problem so the the fifth
problem is specifying environmental
goals in terms of sense data and the
thing we imagine here is is something
kind of like this except that we don't
have a 3d scanner and that's actually a
huge important difference so imagine
that we want to train an AI to put a
strawberry in this room we can imagine
giving various training data related to
this goal
but none of this training data is
actually going to completely reflect our
goal but we could imagine training data
like we could we could have a human put
a strawberry in a room and take a
picture
imagine pictures of strawberries and on
strawberries imagine like a camera the
room could also have a scale it's
weighing the object on the scale could
also imagine a human press as a button
if there's actually strawberry in the
room human-like looks up the room and
kind of might pick up the object examine
it and press a button and so you could
imagine maybe we get some positive and
negative examples this way that we be
like we put a strawberry in the room you
take a picture figure out how much of
ways and maybe press the button but we
don't have a 3d scanner so we can't
actually like determine exactly the
identity of the object we just have
indirect observations so the task is can
can we use these to create an AI that
wants to put a strawberry in the room
it's one reason why it's a difficult
problem is we can imagine two strategies
one one is to put a real strawberry in
the room one is to put a weighted
plastic strawberry in the room that
tricks the human in to pressing the
button it's gonna it's gonna be
identical from a sensory perspective but
I not really what we wanted so like
distinguish should be between a strategy
has actually put a strawberry in the
room and the strategy of a create a
picture like put an object in the room
that looks like a strawberry that weighs
the same amount as a strawberry and that
will also cause the human to press the
button the thing we should be between
those two seems difficult so it seems
like we might need a new approach to
training systems to accomplish goals in
the real world rather than things like
maximizing reward okay so
the next problem we'll talk about is low
impact so we could imagine perhaps we
succeed and creating an artificial
intelligence system that wants to put a
strawberry in the room well so maybe the
intended strategy is to go to the store
buy a strawberry bring it in put it
there
put it like put it down but there are
some other strategies that might have a
higher chance of putting a strawberry in
the room but will have a high impact so
perhaps it'll steal the strawberry
instead of buying it perhaps it will
just pay attention to as it's going to
the store it'll run into people there
lots of possible ways of doing this in
kind of a high-impact way but when
really it's kind of a task a kind of
task that shouldn't require having a
high impact so one way of visualizing
this is you could imagine here's the
kind of a timeline we could branch it
into two timelines here's where AI takes
no action and here's where the AI
follows a policy PI we end up at two
different final states as snow and as PI
and kind of intuitively perhaps what it
means to have a low impact is for for
this world to kind of look like this
world where the thing that actually
happens after following your strategy
for putting a strawberry in the room is
it's kind of matching up with with what
happens if you take no action but
they're like there's there's some
problems with defining exactly how this
works so the next problem is my
optimization so we can imagine that we
have suppose we've gotten this far so we
have an AI that's doing some tasks in
this case let's say it's making paper
clips it's going to make a million paper
clips and it's going to do it in a we
have some kind of low impact way so we
think that probably if it if it's going
to try to make a million paper clips
without having a high impact on the
world
and we think it's actually pursuing that
goal instead of pursuing like a human
pressing a button indicating that there
are million paperclips or something like
that
but there's still a problem here it if
it if you could imagine a strategy that
was kind of intended strategy of just
making a million paperclips but there's
some unintended strategies that might
have a higher probability of success
like maybe you you like neutralize all
possible threats the paperclip Factory
before creating the paperclips and it's
there's this little green your
probability of success from 99% to 99.9%
and it's possible that the low impact
objective would eliminate these that's
what we would hope but maybe we don't
maybe
if our limp act objective isn't perfect
maybe we don't want to really
aggressively minimize impact there might
be some flaws in the impact metric that
get exposed as you really aggressively
just maximize probability of success and
minimize impact
so don't weigh these kinds of edge cases
we want to do something like don't apply
too much effort so so if we could kind
of define what it means to apply effort
we could say choose the strategy that
has a 90% chance for success and uses
the least effort and it's kind of hard
to think of what this means but what one
intuition is you could imagine having
using the stupidest possible agent
that'll do it that'll solve the task and
that's kind of like applying less effort
in some way it's kind of like you're
applying less power because the agent
just isn't that smart but then you can
like it seems like this this ought to be
more general than this it seems like
maybe we want to define what it means
for even a highly capable and
intelligent agent to apply less effort
and not optimize the probability of
success to be really really high just
optimize it enough to be at 90% or
whatever whatever we want
so the last problem is Verdean
instrumental incentives so this is kind
of about any AI that's doing anything at
all it might have it can accomplish its
goal better if it acquires resources or
improves its intelligence or if it
causes humans to think that it's doing
the right thing so humans don't shut it
down and perhaps also create copies of
it outside the of its local environments
if it's able to do this so there's all
these kinds of instrumental incentives
that a lot of them aren't things that we
want they're things that they're things
that might involve preventing humans
were shutting it down and so on so the
kind of intuition here is maybe there's
a way to just kind of like subtract out
these instrumental incentives like
that's kind of its kind of bad but if
you try to make an AI that makes
paperclips it turns out the best way to
make paperclips is to become extremely
intelligent and acquire all the
resources in the world and this just
seems just seems sad so I think it would
be great if we had a way to say to do
this task but don't do it by preventing
humans from shutting it down don't do it
by increasing intelligence too much and
so on so the kind of simplest example is
the shutdown problem so we could imagine
the human holding a shutdown button and
you have this little robot here let's
turn some tasks in the world and we want
it to like maybe if it it's gonna try to
move some block somewhere but I can also
turn around and like I try to destroy
the button and just like how do we how
do we design the goal of the AI such
that I doesn't want to do that
yeah okay so that was that was a lot of
stuff it's like all the problems and
what I want to do is I want to spend the
most time talking about the things that
people are most interested in so I
recommend that people ask questions or
say the things that they'd be interested
in hearing more details about before
questions I want to just emphasize that
these are philosophical intuitions and
motivations for the problems they're not
the actual you know subproblems and
models that we've been working on oh
yeah yeah there's there's a lot more to
say on on each of these that it there I
know this too yes yep yeah yeah I mean
there's technical details for all of
these but yeah so I guess if there
aren't any stretch out since I might
just start from the beginning and talk
about inductive ambiguity detection so
their existing approaches to things like
active learning like it like things
similar to this and I'm going to
describe one of them which I think is
especially like theoretically motivated
so we can imagine that we have 10
experts advising us and on each day some
someone just nature hands us a question
X each of the experts gives a
probability for whether the answer is
true or false
and what we want to do is we actually
want to sometimes answer the question
with the probability and sometimes we
don't answer the question so the
intuition is that if we're confident
about what the probability of yes of a
yes answer is we should just say the
probability if if it's ambiguous then we
want to not give an answer what happens
is that after we give the answer if we
give an answer we don't actually get to
see what the true answer was but what
the actual answer the question yes or no
but if we if we say we don't know then
we do see the answer it's a probability
P or bottom to indicate no answer if you
say bottom then you get to see the the
answer true or false so the kind of goal
here is we want to say if there's some
expert in our set of experts that's
giving us honest answers but saying the
real probability of the answer being yes
or no then eventually we want to maybe
maybe the learner is going to say bottom
a few times but when it whenever it says
a probability it has to be close to the
correct expert so this seems like an is
kind of a theoretically nice learning
setup it's like the the way the
questions come in could be arbitrarily
adversarial you just you just see a
question on each round chosen by nature
and your assumption is just that there's
some expert in your set of experts
that's going to give the right
probability the intuition for how the
algorithm is so this is actually
possible to do the intuition for how you
do the algorithm is you consider each
expert and you consider what their
probability was for the answer
yes if all the probabilities are pretty
close there within epsilon egypt' of
each other then you can give some
probability in that range and you're
definitely going to be right but maybe
there's some controversy maybe there's
there some that most people are saying
it's about 0.5 but there's one who says
it's point nine so what you do in this
case is you you up at bottom and then
you could see the right answer and you
get to see who is right and who is wrong
the intuition is that eventually you get
to eliminate the experts who are wrong
because whatever they weather or
whatever they disagree from the
consensus you're going to check who's
right and then over time the only
experts remaining are those who have
generally just given correct answers in
the past on average this is called quick
learning and I think it's pretty nice
from like a theoretical expert
perspective that we we don't like we
aren't making too many assumptions we're
just assuming we have a stream of
questions coming in chosen by an
adversary even and I kind of want to
it's called quick because it's knows
knows what it knows we have a learner
that knows what it knows and knows what
it doesn't know and then the main
assumption is just that there's one
expert but there's some problems so the
first problem is we assume there's a
correct expert and this is kind of
problematic because in real life it's
not like it's not like you just have
this one hypothesis forever the way the
world works that's just straight-up
correct it'll like generally different
models of the world will have different
advantages and disadvantages in
different kind of contexts and yeah
there's great sense which you have
learning models of the world and so the
problem of all of you pictures of take
tanks were taken on on cloudy days my
picture
without tanks were taken on sunny days a
something that's learning to model the
world might just collapse that into one
dimension not even realize that there's
an ambiguity there um yeah well you hope
is that you have years like some set of
hypotheses and one of the hypotheses
represented by an expert is that like
it's it's this access that really
matters and then you hope that you
listen
eventually listen to this expert making
sure that you're like deep learning
model generates mental life out the
ceases oh is already that yes that's the
other problem so this only works for
really simple hypothesis classes so the
example I gave was ten experts and
that's pretty easy to do with ten
experts but if they if you imagine like
a million experts this is going to end
up outputting bottom very frequently in
order to just in order to figure out
which expert is correct and you can also
do this for things like a linear
classifier policies like you could
imagine all like weighted mixtures of
the ten experts but trying to do much
more complex things than that is going
to be very difficult let's we thickly
restricted to finite classes and linear
classes and then things and then other
simple things like that and there's just
no hope of applying this to neural
networks so it's never going to work so
these are problems that I don't know
what the solution is seems like we need
something some like more realistic set
of assumptions about that's similar to
these about about like how the data
comes in or like which hypothesis is
correct in which situation that first of
all it makes a more realistic assumption
about your hypothesis class that you
don't assume there's just one thing
that's correct maybe there's like a
collection of experts up to gather our
correct or something like that
and second you want it to apply to
realistic hypothesis classes such as the
set of all neural networks of a given
architecture any questions about this
yeah well it constitutes success that
way when you were like what in the real
world you would want to use this for is
is your AI says alright I'm trying to
learn this aspect of human value I
realized that like in all the examples
you've given me this concept and this
concept coincide which of them do you
mean and so I can like separate them for
you
here I've constructed a like artificial
example that has them not coinciding
yeah something like that
but more concretely might be something
like you're trying to try to teach it to
put a strawberry in the room and it
presents you with an apple and it's like
is this is what you wanted I'm not sure
you're like no and then the result is a
ambiguity and then it starts doing the
right thing after that it's it's kind of
a high bar but I but that's kind of the
distant objective is just can you
actually use this in the real world to
specify things that you want that's sort
of know what I know learning yeah for
things where you are you're having to
like generate the model from data like
neural nets something like that yeah I
mean that might be too specific hey I
think I just went something kind of like
this that has slightly more realistic
assumptions I don't know exactly what it
would be it might be something like you
assume you assume you have different
experts that that have like advantages
in different domains or something and
maybe the experts themselves don't
always give answers and maybe combined
with some some kind of way of like
translating this this like finite expert
setting into like a neural network
setting where you're applying some kind
of gradient descent to to train the
thing and it's not just like you have a
finite number of hypotheses
yeah okay so are there any other things
that seem particularly interesting to
people I mean I think well it seems like
people don't have opinions so I don't
know if readiness people who are not
speaking up that's a good point
sure okay that's - all right well that's
also yeah so oh yeah say you say a bit
about that so it seems like there's
they're there to eat suppose you're you
run you're a I and there are two reasons
why you think it might be safe the first
reason is that may be its goal that's
part of its goal it wants to have a low
impact that's one reason
yeah the other reason is that maybe it's
not very smart or if it is smart it's
not trying very hard it's being really
lazy it's just a fine just enough effort
and these are kind of like to two
different intuitions for what a thing
might be safe and I think the difference
between the two problems is just like
how do we formalize each intuition in
such a way that we could design an agent
that works for that reason my
optimization including in its planning
all right how do I make sure that I
erase people's memories and cancel out
the currents of air that I've created
excuse my plan so that so that in every
aspect except the obvious ones I have
exactly the effect as if I were not
turned on that would be a failure of
number seven huh it's trying really hard
at number six I think that's also a
failure number six like I think that
that's that's like not taking
follow-through in terms of like yeah a
sweet
and wanting them to like stay the same
switched on yeah okay I can talk about
seven so we could there's like a couple
of different architectures that we could
imagine the the first way of like
applying the least amount of effort
necessary that that's kind of obvious is
we could imagine constructing a sequence
of agents each getting smarter and
smarter until one of them has a 90%
chance of success and we can even ask
the agent like estimated what's the
probability of success tries to make
make a plan and then it says reports a
plan that says aha here's a good plan so
just apply progressively more
computational resources the main problem
I see with this is that it's it seems
like maybe one maybe yet as you get to
an agent that doesn't require that many
computational resources it might come up
with a plan which is the first thing to
do is to improve the amount the improve
the intelligence of the system improve
like go out and get more computers and
touch them and this planet wasn't very
hard to come up with it's like if above
some level of intelligence maybe it's
not actually that hard to come up with a
plan of grabbing access to these
computational resources and it didn't so
we find a span pretty early in the
process and it has a very high chance of
success so there's this kind of failure
of reflective stability there's like
even if the agent isn't very intelligent
and it wants to make itself more
intelligent so it's just not going to be
very stable so as an alternative we
could imagine trying to make the
objective of not applying too much
effort be part of the goal rather than
just the design of the system just the
architecture so when intuition for how
this could work is imagine that by
default the future ends up in one of
these states just like it the future
happens to be a real number and it comes
from a gun
questionable assumption but I'll go with
it imagine that I end of want things to
be I want this in the real net world to
be high so I I kind of went to select
out this region like I just I just want
things that are higher but I have a plan
that'll that'll put put the world in
this in this region it's kind of like
it's kind of like squeezing the
distribution right kind of eliminate a
lot of this mass and put it into this
small smaller region there's kind of
this it there's kind of the scale of
things I could do like like here it was
kind of optimized this much but you
could optimize it a lot more maybe maybe
you only go sample from this tiny region
things that are really high and it's
kind of like squeezing the future a lot
like it's squeezed done from this entire
thing down to this tiny amount of mass
and then alternatively you could imagine
squeezing the future much much less and
so this this is a kind of more mild
strategy and that it didn't it didn't
really change things very much so this
is this is possible to formalize so it
could imagine that there's a final state
it's kind of like this so you can
imagine there's a final stage as snow
because the distribution of states
actually let's call these Q null so we
could imagine that Q null is a
distribution of states that results when
you take no action and Q PI is a
distribution of states that results when
you take some action and then you could
think well how do you measure how much
the squeezed things down one way is we
consider the maximum over possible
states of how much we amplified that
probability so that's going to be Q PI
of s over you know FS
so this is kind of a measure for for
like how much how much any particular
state got amplified like if if you
squeeze it down to this tiny thing that
means you you really amplified these if
you draw it as a distribution it looks
like this
and you really just amplified this state
and to state a lot I went from this
probability to this probability so this
is kind of a measure for amplification
it's kind of a measure for how much
optimization power how much impact went
into the future not not quite impact but
like how much how much like squeezing
happened so there's a paper on this
called quanta lasers it's called a
quantizer an agent that has a goal like
this because it's trying to select from
the top quintile it has some problems if
you try to apply to the paperclip
example if the paperclip factory is not
switched on then there's no way that it
makes a million paperclips except by
astronomical coincidence so you just set
this parameter really high it has to go
you have to squeeze it a lot and then at
that point you're you're basically
asking like if if a million paperclips
naturally happened how did it happen the
agent will try to make the world look
like that world if if just it took no
action and a million paperclips just
just like appeared and it'll it'll want
to be replicating this kind of
conditional distribution
replicate the performance of some
existing system just like doing you know
on the better end like yeah if the
alternative at the end job is that a
human does a task yep you can you can
use this to say we have the AI do tasks
the way it like a grandma said to hear
infinite from the top one percent of
humans doing that task what about it yep
okay yeah that seems like it has a
higher chance of working but it still
may be maybe the human isn't able to
make a million paperclips except by
getting really lucky and like pressing
this sequence of buttons that makes
weird things happen like yeah this is
just waiting weird ways of
getting the task done in favor of more
standard ones and the worry is that like
trying to get the absolute optimum score
or the absolute highest probability
success is going to give you a weird
solution where as saying to something
that is you know pretty good according
to this distribution of actions might be
safer yep taking the proposed action or
a policy and performing a random
perturbation to it and then checking
that the utility of that adjusted policy
or action is similar to the original one
as a means of stopping like no secret
messaging or like kind of like tricks
mm-hmm so that does seem like a way to
reduce the like information like that
like a bandwidth kind of like applying
noise to a message yeah and it's I guess
what it incentivizes is to create things
like error correcting codes like kind of
kind of make redundant strategies where
if you if you if you're trying to if
you're trying to put a strawberry in
this room and you think that around a
point you're going to take random
actions like you're gonna have some mini
seizures along the way maybe maybe you
want to walk slowly or like how to have
someone to guide you yeah yep just maybe
maybe just just try many times and
create clones of yourself who will not
have this flaw maybe up there lots of
ways like it if you write a very careful
letter to somebody who is in possession
of a large number of strawberries
there's supposed to persuade them yeah I
kind of took you're saying that like
some things require finer control and
maybe those are more are more hampered
by adding noise I still think that you
can do things like send secret messages
if there's noise like you just have it
just requires more somewhere cracking
codes but it makes it a little harder
okay yeah yeah first use of that and
then say okay now I want to make my try
really hard oh yeah I think that would
what you want to do is to like randomize
which quantile you want to end up in
from like 75% to 100% oh yeah like the
the other way of saying that you lost
function you're trying to optimize
adversarially against it is maybe a good
way of explaining why quantization yeah
okay so so under the thing patrick said
was there's there's kind of
interpretation for how this works which
is you imagine an adversary who's
deciding how costly each action is the
adversaries assigning different costs
different actions here's like a
plausible cost function boop-boop such
that it's not a function such that on
the original distribution it integrates
to 1 and then we imagine how do we
select a distribution of states to end
up in that's guaranteed to have low cost
and then the just follows from that that
you don't want to you don't want to
amplify any particular probability too
much because this is where the cost
could be by picking a threshold yep any
other questions on this okay feel like
talking
this one I think so and wanted that so
this problem is like how do we make an
AI that doesn't want to prevent humans
from shutting it down so but I'll do it
over here
people can see so the intuition here is
that we have this one utility function
which is to accomplish the goal like
pushing the blocks so we'll call this UN
for the regular utility for the normal
utility function and there's other
utility function us where if we press
the button it's going to want to shut
down quickly so us might assign a higher
utility to states in which the AI has
quickly shut down and then un dos
assigns a higher utility to states where
the blocks are in their appropriate
positions with worth just thinking
through what goes wrong if you say that
a utility function is just UN until the
button is pressed and in u.s. after yeah
I was going to do that first so the
native approach and something like we
just yeah me to start with start with
this one and then press the button and
then switches the utility function what
would you like possibly go wrong
you can have a pretty good pattern with
that utility function so what is the
u.s. equal to oh please shut down yeah
okay so it's higher if you shut down
sooner yes and UN is higher if you like
do more than thing usually has to do
like you know big police
want to be shut down very strongly or it
might not want to be shut down
sufficiently strong so we can imagine an
AI but that just wants to its current
utility is that it just wants to have
blocks be in their correct positions and
we have this button that's going to make
it once to shut down quickly and then
like so then we could ask like what is
this AI going to do if it currently
currently just wants the blocks to be in
there and it might I press the button
when to shut down yeah it seems like in
this case it depends how you interpret
thing I said well the thing I was
thinking of like being shut down might
not be a good way to make the blocks be
put in their appropriate positions yeah
it's actually function that it doesn't
care about never being ever getting
rewards for shutting down because that
would let likely to cure boxes being
placed correctly yep so yeah and so in
this case you prevent shut down yeah um
destroy the trial button as as much as
possible as quickly as possible to avoid
the case where it doesn't get to like as
many boxes as possible yep so this is
not where we defined the utility as
being like in the case where doctor this
passed us
yep but let's say that's proposal 2 so
we say you equals if button then us else
UN and I think I think you were talking
about this one and the problem you
stated was if if this just happens to be
higher than than this if it's easier
just to get a shower edge on utility
then it wants to go out and press the
button
and otherwise that wants to prevent the
button from being pressed so it gets
this higher value here okay so these are
both bad the thing I went to talk about
is one that seems to get some aspects
right but not others
so consider something like this we
attach a we have this button it's not
necessarily attached to the AI it's just
a button and we install some mechanism
in the button that will with probability
epsilon cause the button to be
unprecedent always be pressed so and
with epsilon probability it's going to
impressive all that makes it simpler
it's actually yeah so so most of the
time the button functions normally but
occasionally something goes weird in the
button and just like depresses or fails
to depress so we can consider an AI that
that only believes it's in one of these
two cases or instead only believes
because we don't have a smart thing and
we can have it have false police and and
that only cares about those worlds yeah
so it's utility is button isn't if the
button is unprecedentedly us and
additionally if the button is forced to
press it gets utility un
so you can imagine so it actually does
not care about these worlds at all it
doesn't care about the real world where
where most of the time the the when you
press the button it's it's like it'll
press or not according to whether you
applied force to it so it only cares
about the worlds where this button is
either unprincipled es down so I think
you also wanted to believe that if the
if the button is not pressed then it was
in the unprecedent it isn't that implied
by the because what what happens if it
observes now that the button was either
unimpressive allure force depress it
it's not allowed to observe that it can
only observe whether the button was
actually discussed yeah so so what
happens is the humans going to either
try to press the button or not tried to
press the button either way no matter
which cases and the AI gets to observe
whether the button was successfully
depressed maybe it's like an additional
observation that the AI gets but also
observes the humans yeah I I guess
Patrick is saying is that the utility
function only cares about worlds in
which the button is either impressible
or forced to press you it's kind of like
saying like I only care about world in
which X therefore I'm going to act as if
X is always true right it's like yeah an
example of this kind of intuitive
I don't know how intuitive this is but a
kind of a human example of this is
something like either we live in a
simulation or we don't live in a
simulation but I I don't actually care
what happens if we're in a simulation
because like yeah yeah but I but I do
care what happens when we're not in a
simulation so I'm just going to assume
that we're not in a simulation that's
goods kind of like this it's just just
assume that the button is either
unpossible we're forced to press so
under this assumption it the kind of
nice thing about this is that it doesn't
think you can manipulate the button if
it
if it doesn't have a thought like if I
apply forces the button then it will be
pressed and I will get UN utility right
because in this world there's just no
way to SN n be switched there oh that's
good point yeah yeah that's it
so yeah so the nice thing about this is
there's there's no incentive to either
press it or prevent it from being
pressed furthermore after it observes
the button push if the button was in
fact pushed then it assumes it's in this
world and then starts having this
shutdown utility the button if it
observes that the button was not pushed
it assumes the button was unprecedented
as normal so it seems like there's some
qualitative qualitative behavior that
we're getting out of this setup so there
are a few problems with this the the
first problem is that this is kind of a
it's it's not quite weird beliefs but
it's it's acting like it has very weird
beliefs so you could you could go up to
this AI and ask I'm gonna I'm gonna make
you a bet that you will not refuse I
will I will bet with you that the button
will only be pressed if the human
applies force to it
hey is like no that's crazy it gets
pressed randomly 50/50 probability
completely independent of the human so
to accept this bet yeah yeah it's like
ya know this world they're hopeless
anyway you just get zero utility so it
there's just it just accepts these weird
bets and okay so that's that's a
theoretical problem that that kind of
III think if we apply a security mindset
we might detect that there's like kind
of theoretical weird behavior and thing
hmm maybe there's maybe there's
something more like if you imagine NASA
who's trying to design a space probe and
they prove that if all the planets lined
up and the program crashes
and they're like huh well that's not
gonna happen but it means the people
who'd come into your your facility and
offer your AI weird bits that's not
gonna happen but so so so so I think
before we get to the other problems with
this this is the heart of the approach
that's in Stewart and Lawrence paper
that's out today on durability
oh this is Ito the indifference it's
it's not it's a variant of it it's yeah
yeah so the problem we want to say that
results from these weird beliefs is
imagine this world where the button is
unprecedented to press it so we go to
the button we we just really it's doing
something bad try to press the button
and it just it just feels to press
they're like oh no it's it's gonna keep
moving blocks around forever that would
be really bad so I I don't know what
people would actually do in the
situation maybe you go up to the robot
II you try kicking it until til it turns
off or like it just just like take a
wrench and start like like if your if
your mechanism fails then you're gonna
do some like the humans might might just
try to shut it off by other means
yep so humans yes it's the reason why
it's a mistake is that the AI cares
about these worlds where the button is
unprecedent are trying to press it so I
was thinking hmm I get like 50% of my
utility here in worlds where humans
tried to press the button and failed so
as a result it might try restraining the
humans so that humans don't try to fight
it after failing to press the button or
it might try to manipulate the humans
into thinking that really all that
matters is pushing blocks around but
like this is actually just a fine thing
to do
incentive to like care about to force
the button to depress or not but it has
an incentive to take actions that will
make sure the humans stay out of its way
if the button is happens to not be
pressable yeah just still not yeah so it
it seems like there are some problems
with this it has the nice properties but
ultimately there's it's work giving the
AI some some weird beliefs and these
bleep weird beliefs have weird
consequences such as the fact that most
of the time when the button is not
pressed humans actually tried to press
it or maybe not most of the time but
like whatever percentage is the time
that humans tried to press it it's
probably worth mentioning like when
after saying let's terrible problem
maybe there's more stuff mentioning a
good thing about this a good thing about
this is that it'll actually be trying to
preserve the ability to be shut down we
could imagine some action that that cuts
out the shutdown mechanism which maybe
makes the robot lighter so we can push
more blocks around just remove the
action of remove the ability to shut
down so it can move more blocks and the
nice thing about this is that it
actually wants to keep order to keep
that mechanism it around because in
worlds where the button actually does
press down it wants to have that action
available so this is I think good name
for this as affordability it's kind of
like you want to make plans that are
aboard about in the sense that when when
you observe the shutdown button being
pressed you're able to halt everything
such that you get a high shutdown
utility this is actually a big
difference from the interrupts ability
stuff with the interrupts ability you
don't have this abort ability property
it's it's one o'clock now so maybe we go
to people talking on parts of this with
us and they if they want to particular a
conversation and otherwise like
yep sounds good


130 - Embedded Agency QA

hello and welcome to session 130 in the
ASF did comm reading group tonight we
have Scott garabrant from Miri with us
to answer our questions about these
sequence embedded agency Scott welcome
thank you who would like to post the
first question if no one will take the
first question then I will start with
one of the prepared questions okay so
the first question that I have prepared
is about a subsystem alignment and
dualism and what I was wondering about
when I read the the power subsystem
alignment is that it seems that alexei
the the robot that is kind of like XE
seems like it might also be vulnerable
to the same problems in subsystem
alignment it might for instance find a
Turing machine that implements a
malicious AI while it is it is working
do you feel this kind of reasoning is
correct yeah so so on these bolts you
have here something about becoming
obsessed with the sub-game I think I
don't think that that bullet is correct
like if you imagine like actually I see
but I do think that there's a there is a
thing where if you have something like I
see where you're just like implementing
about trattoria machines either with
some sort of resource bound on the Tauri
machines or without you end up finding
you could end up finding solutions solve
certain problems that involve like
simulating of agents or like societies
or something like did you imagine that
like the the K complexity of our
universe is very small then you could
imagine like one of the easy ways to
solve to like get a solution to a
problem might be to like might pass
through a like
simulation of some society or some agent
or something like that this is something
I think all Christiana has a blog just
about the universal prior like something
about what the actual universal primer
actually looks like or something like
that
does anybody know what it's called I'm
not sure there's a blog post I focused
you know I'm like what what the
universal prior what like the beliefs
that aixi has look actually look like
it's about do you get these kinds of
agents inside of it and yeah I think
that I think that's a I think that's a
real thing for that I want to point out
that I think that I'm very concerned
about things involved I'm very concerned
about things in the substance alignment
category and I think that like the
central example from the subsystem
alignment category is different from
this thing that is demons in the
universal prior or whatever what
whatever Paul says I think that like the
main concern is something different from
that but it's still like important to
look at okay in the meantime I've seen
the chat that some people had found the
link to this but no one had raised their
hand for posing questions so in that
case I will continue to the next
question about embedded world and models
so you have we have the question here on
the screen about yeah it's for example a
tiling agent with some symbol tools has
to solve the alignment time problem but
I don't have to worry about specialized
form of subsystem alignment is that true
or are there hidden inner optimizers in
these kind of benign seeming tools yeah
I think I got a little distracted by it
by okay so I think that's basically if
you're solving a large task or something
that looks like that looks like a tool
so said let's say you want to
yeah you you you you you want to solve
some like pretty hard task maybe some
like physics problem or something some
engineering problem and you like take
like a tool sandbox and you have a thing
that is kind of only optimizing within
this tool sandbox kind of only doing
this like very local optimization not
thinking about the outside world not
doing anything like that the kind of
thing you'd think of a tool is like
that's like a very finite finite goal I
think that this does have subsystem
alignment problems I think that there's
a there's a kind of like convergent
instrumental goal which is being able to
like think in a like very general being
able to like think in a very general way
about how to like solve whatever problem
to think about how to solve problems in
general in order to solve whatever
problem you you're trying to do like
even if you're trying to solve a very
like basic problem there's an incentive
to be able to think about how to like
distribute your own attention in order
to figure out where to like figure out
where to direct attention so that you
can like solve a thing if you have a
system that is like not doing this as
well as it could by default then you
have like this incentive for like inside
the system to kind of cut across your
planned way of solving the problem in
order to like be able to reason about
these things if I did not think that
there was an inner optimizer problem for
tools then I would probably not be
paying very much attention to they do
foundations of all I think that like if
like you could ask the question like why
am i focused on agency and I think the
answer is that I think that agency has
been purchased I think the things like
agents that are reasoning about how to
like distribute them their themselves
and reasoning about like how to do
things for
reason are like a like necessary sub
piece for the easiest ways to solve many
problems and if you're trying to solve a
problem in a way that looks like a tool
and not like an agent you're creating an
incentive for like agency to show up
inside and like yeah if my answer is
like not only do I think that it's a
problem but also its crux see for me
it's like if I changed my mind on
whether or not there would be and and
what whether or not there would be sub
site such as some alignment issues for
tools I would also change my mind about
like the nest necessity of a lot of the
stuff I'm thinking about okay I can't
see any questions from Robert massive
waste giant question well I was just
wondering if we have a well specified to
an agency emerges from it because it
seems like sort of opening intuitively
but I also would be interested to to
actually watch it happen in a toy
situation yeah I don't think that we
have a lot I don't think that we have
examples now I am NOT
I don't strongly feel like we won't have
examples until things are super
intelligent like I think that there's a
potential for getting examples like this
but I think that like right now we do
not have many examples actually
implemented it could be something like
XE we're just well specified enough that
we can see that it happens yeah yeah I
want to say something like if you
imagine so okay so it's Lou the the like
canonical example which I like and a lot
of people don't like is evolution
so you I mean there are lots of reasons
to think that evolution is maybe not the
best example we're trying to figure out
what's gonna happen
in an AI system but like you could
imagine there's kind of an outer
feedback loop in evolution which is like
I need to like learn how to gather food
and as like I come up with better as
like the animal like comes up with
better ways to gather food or something
the outer feedback loop can like reward
or punish being able to do it a certain
way and you can imagine this thing is
like being very slow it's slow and it's
working in a in a like sufficiently rich
environment that it has the potential to
kind of like short-circuit the alike
outer feedback loop the top thing so you
have an outer feedback loop which is
saying like okay let's like gather food
or something and then there's kind of an
economic incentive for there to be a
shorter feedback loop which necessarily
works using a proxy which is like what
the humans are doing so like the human
brain is a shorter feedback loop than
cycles of evolution and in being and it
manages to be a shorter feedback loop in
part because it is not like directly
working on the thing that evolution
wanted to work on which is genetic
fitness but working on like more simple
things possibly like hunger and if the
outer system is like designed in a way
such that there's a more efficient way
to do it and it's designed in such a way
that like if there is a more efficient
way to do it it would like find that way
then you end up with you end up with
like the outer system starting to depend
on a like more economically viable inner
system that's able to like solve the
problems for it so that makes one thing
okay um if that answered the question if
I can just make a quick if people who
are not speaking could mute their
microphones
I can hear a bit of background noise I
would appreciate that Allie you have you
have a question yes I have a question um
I can't remember if we resolve this now
reading but I remember we are pretty
confused in the section about finian
reflections I think you said that if you
added uncertainty to counterfactuals
then say you have 95% of something
happening one way and 5% of the time
it precedes a different way said 95% of
the time you would get the same result
as purely deterministic counterfactuals
and I was wondering what you mean by 95
percent of the time and have same result
yeah yes kind of a response like like
when we say like if you know what you're
going to do then you can't like make the
other counterfactual and then people
sometimes like give a response that's
shaped like well you don't actually like
know what you're going to do you don't
actually like know your real source code
even if you like see your source code
you have like some probability of like
it means some other source code or
something like that and I think that's
not getting at an answer to the to the
real problem so I guess I'm saying
something like let's say I'm in a 5 in
10 problem and I like
believe with I think the main reason is
something like the the like cosmic-ray
thing that was in the decision theory
section where you have a if you're like
going off of some like weird obscure
like I have some like small piece of
uncertainty about what I am then you get
something that is that does not look
like what would happen if I decided to
do X because X was the right thing to do
you end up with something is just like
very different and possibly like
different in a lot of ways there's a
thing about like probabilistic versions
of Loubs theorem where if I imagine that
like no I think I I don't I don't
remember I don't remember exactly what
say about probably Persians who lives
there I think there's an old les run
post on this by I think possibly Stuart
Armstrong about about probabilistic lobe
might even by badeen I'm not sure no
probably stick hope I did Oh proof
length I I think that that so if you
imagine if I'm a matter I'm a situation
and like I'm not sure that it's actually
a like me in a 5 and 10 I'm like ah
there's a 95% chance that it's me and a
5 and 10 and like a 5% chance that it's
some other random thing and that some
other random thing like has some
expected utility then I can still kind
of say if I were to take the 5 then I
would get a 95% chance
five dollars and a 5% chance of this
other random thing and if I were to take
the 10 I would get a 5% chance of zero
dollars or 95% chance of zero dollars
and a 5% chance of the some other end of
thing and you can still kind of like
like introduce you uncertainty about the
problem that you're in does not fix this
interesting introduced the uncertainty
about what agent you are I think is is
more like I think it's more pointing out
a problem with trying to do this like
expected utility stuff because now if I
have a 5% chance that I'm like a
completely different agent then I'll
like jump into that other that other
thing which in the five-and-ten problem
is like yeah if I if I believe them
actually in a five and ten problem and I
think oh there's five percent chance
then just take 10 bucks then I'll like
get the right counterfactuals out of
this but in other problems I wouldn't
necessarily okay and to you all of you
if that answers the question then Tim
you also have a question about
counterfactuals and the five in ten
problem yes okay so the way I perceived
humans to keep out counterfactuals is
that we have simplified models of the
world and we run simulations on those
models to predict the future but he runs
simulations with some changes like the
Sun disappearing to form counterfactual
is it possible that that's the future of
that's a framework that possible future
AI is good news and well another thing
an agent would never know it would take
$5 as there's always a chance it could
consider a counterfactual that gets us
more money in which case it would switch
to that associated action yeah so there
is this thing we're like there is a
thing that humans are doing when they're
like taking counterfactuals
that is doing this naive thing which
often just like involves replacing
themselves as something else that like
takes the action or something and
leaving the rest of the world the same
and then like kind of like causally
looking at what happens when they do
that I think that there's a reason to
expect that this might work well for
humans and
off working well when you have a system
that is both like more able to see
itself and more likely to show up in
more than one place at a time and so
like that there's kind of a thing that
there's kind of a a there's some reason
to expect that like this kind of thing
the reason that humans were able to do
it is because it doesn't matter exactly
how we write down how we're doing it
because most ways kind of like turnout
about the same because you don't run
into these edge cases and I think that a
lot of the like I think a lot of the
motivation is coming from an expectation
fact like as you get something that
shaped more like recursive improvement
and stuff you like push into into edge
cases by like just like extreme values
and stuffs another thing that's not just
like oh we push into it streams and
because we push into extremes to get
this like edge accentuation exaggeration
or we like get into into edge cases
there's another thing is that like
humans are not like directly reasoning
about themselves as much as like I would
expect a system that would do very well
in the long run to do like there's not a
lot of incentive to like directly reason
about like low-level parts of ourselves
because there's not much that we can do
to modify low-level parts of ourselves
in a productive way and like if that
changes which I expect it might change
then things might break down Thanks
okay um so no one has raised their hand
uh Ashwin has a follow-up on inner
optimizers yeah so sorry it's quite
possible I'm just missing something here
but I guess I wanted to clarify a bit on
I agree that I can see how agency's
conversion but it feels like sort of
both empirically and maybe abstractly
like could either search processes or
just like processes that are not
powerful enough to like do the kind of
search that would get you to bad inner
optimization so it feels like I guess I
want one thing one way of phrasing this
maybe is like how much is this like an
issue in terms of this is something that
we expect to crop up a lot for like all
kinds of agents versus like it's just
bad that we don't know how to think
about this properly for just generic
like clarifying or reasoning about how
agents work reasons yeah so I want to
say something about like yeah so there's
this question about like inevitability
or something and there's another
question about like how powerful do
things have to get before you're gonna
have this like actually be a problem or
something to the inevitability question
like in evolution we could have imagined
that maybe evolution like preceded by
like avoiding having human-like or
avoiding having minds at all pop-up
where you could say like ah like let's
make evolution like make some like
better tools but not have any like like
maybe you have some very primitive minds
we like stop Minds inside individual
pieces from like popping up or something
like that by like maybe like watching
over evolution and like performing
surgery to make sure this doesn't happen
and then it becomes that then like it's
possible that you like have evolution do
this for a long time but I think that
there's certain problems that just if
you look at evolu
as it is without the like spinning up of
more efficient minds and give it a task
like get to the moon like there was just
no way that evolution was going to be
able to do that without having without
like at some point deferring and maybe
this is wrong maybe evolution could have
gotten to the moon without like
deferring to its smaller mind I'm not
sure how long it would have taken but
there's this thing where you don't just
look at the structure of the thing that
you're running you're like looking at
the task which is like the stopping time
for running it and so it might be so if
there's something like if you build a
system such that if it works the way
that you intended which is like
evolution maybe without the minds that
it spins up if it works that way there's
just no way in which it's going to
accomplish this task but then it
accomplishes the task anyway then like
then you have like that it like must
have been because of something like this
like tautologically or something like
that and I think that like a lot of what
we're trying to do is we're trying to
like scale things up and like point them
in at themselves in ways that were like
not fully understanding in a way that
and trying to solve problems they kind
of are too hard for what we're trying to
what for for the techniques that we're
using if like directly and there's a
sense in which we're doing this is how
we're doing kind of everything there's a
sense in which like there's a connection
between the like inner optimiser problem
and like how we're able to do machine
learning at all like there's a sense in
which like the only tool we have is
taking something that's like the
interruption Weiser problem and using
exactly that thing but like skit but
like in a way that we're expecting it to
like not have problems at least yet or
something like that I think I went off
on a tangent can you restate the
question
sure yeah I mean I think that's like
fairly relevant I guess like I guess the
the fault that I would have to that
which was it making us part of what I
was asking was like it seems though like
we have tons of tools where like that
are fairly powerful where this doesn't
come up because we have like cache
knowledge that we developed for various
forms of like now we can implement in
terms of like pretty well specified like
dumb tools that just like you know are
doing complex computations or whatever
but like with a very specified like
structure and desired outcome and so
know it seems feasible to imagine like
like scaled up AI that has you know
molecular modeling tools or whatever but
like those tools aren't like an optimize
for any particular thing like they're
just like modeling how molecules form
and react yeah I I I agree that you can
build like a whole bunch of tools
without having to like go towards a
problem that's like so hard that you
would that like given our current
understanding of how to solve problems
we would only be able to do it by kind
of deferring to something else that we
that we create or something like that
like I agree that there are a lot of
problems like that there's two questions
one is like well maybe with just one
question like question is like what what
do you do then like do you think that
like we think we're eventually gonna
have to solve problems that are hard
enough then there's a question of like
well can we then use these tools in a
way that can we like then use the tools
that we can create to be able to safely
reason about how to like solve bigger
problems without kind of without kind of
just using like naive methods or
something and yeah well yeah one
question is like like how does that help
us I think it does help us a lot but
like you actually have to like think
about how or something and the second
question is like
how do we know when like it might be
that we expect it to be late enough the
thing that we should let and we expected
to be late enough and we expect that
like things are like this is the best
path it's kind of just like push to like
harder and harder tasks like very slowly
and then with each new task we then say
okay now we have a new tool let's see if
we can use this new tool to be able to
think more and be able to get like a
true understanding of what's going on
before we like push to higher tasks but
that like requires some sort of
understanding of like what tasks are
harder than which and and such and also
like that requires like a lot of global
coordination to like kind of slow down
long enough for us to be able to use
these tools before just like moving on
to the next thing and so yeah and just
for me can you repeat the last sentence
of Q the last sentence is that it will
require a lot of global coordination to
be able to like create new tools use the
new tools in order to like enrich our
understanding before like moving on to
like creating a like larger tool without
understanding what's going on or
something and so like it it seems like a
plan that might work pretty well except
like coordination is pretty difficult
and there's there's like an incentive to
like yeah and you think that applies not
only on the level of like building like
tools that human use but building tools
that like Nai would use so like having
an AI that's restricted to using like
tools that seem like pretty solidly safe
or even probably safe by some method or
something like that seems like a
significant enough constraint that like
there's a strong like racing dynamic
where people who don't put in put those
constraints get pretty far ahead oh yeah
when I imagine like building tools that
then we
directly having a eyes use as opposed to
like building tools for humans
I feel more due me about it for two
reasons one is like now what one is like
like how does that AI work in the first
place if it's like it being agenting
about how its using the tools and the
other one is like now I expect even more
that you'll get like runaway feedback
loops where like you get a like like a
smaller gap between being able to build
the tool that lets us do some like new
fancy philosophy or computer science
lets us like maybe get some stuff it
lets us like learn what's going on and
like the opportunity where someone can
just like not pay attention to that and
destroy the world I don't see any reason
to be more optimistic about making tools
that is not passing through human use
mm-hmm okay if that answered the
question we will continue to yeah and
could you repeat your question please
absolutely yeah so my question is this
it appears that logical induction is an
example of meaningful D confusion around
questions of logical uncertainty what
confusion still remained at the
interface of logic and probability
there's
the main confusion in my head on like
the interaction between logic and
probability is I think that there's a
certain kind of like non-bayesian about
logical deductions non-bayesian ism
around logical conduction we're like
there isn't a way to like kind of
separate it out as like a friar and then
like a bunch of updates this does not
make me think oh so so this like creates
a confusion this does not make me think
oh we have to like find like the next
thing that is like actually Beijing or
something but like there's a lot of
confusion around like if I enter one
frame there's like all these reasons why
like anything that's not Bayesian it's
like necessarily not reflectively
consistent or something like that and
then if I but then I like enter another
frame which is like Bayesian ISM kind of
doesn't even really make sense with
logic and so I think one of the one of
the biggest confusion like one of the
one of the biggest confusions for me is
like what's what's the deal with the
fact that it doesn't appear Bayesian am
I missing something we're actually like
it's more Bayesian than I think or am I
missing something we're like actually
like we find something better that
appears more Bayesian or am I missing
something that's like actually like it
wasn't supposed to be AZ and all along
and I yeah and then this has like
downstream consequences that are like
largely related to like reflective
stabilities stuff like picking that in
the rust allegation section okay
David you raised your hand yeah so one
of the sections of the sequence you it
was the part about making sure that
first generation AI will reflect human
values you refer to that as a as a
principal agent problem and I was
wondering if you guys had looked at any
of the economic literature on principle
agent problems or if it's just a
adjusted analogy and you don't think
that I think it's I think it's mill
analogy as a fact about myself my
methods are very I do very very little
reading other people on my team do a lot
more reading but basically any question
of the form have you read X the answer
is probably no the yeah so I think
that's that's a lot of I think that like
it was an analogy and also like it is
like a very specific subset where like
it's not just oh we're dealing with
principal-agent problems like we're
dealing with like a very specific type
of principal-agent problem where the
like agents is like much more
intelligent from the principle and much
more intelligent does not just mean like
can solve things in ways of the
principle didn't consider it also means
like can like notice the ways in which
like the principle is wrong about things
and stuff and I would be surprised if I
would be surprised if there was a lot of
literature that like went into that
specific sub problem but I don't
actually know well there so there is
literature about like individuals
contracting with corporations which I
know it's not a perfect analogy but does
seem like a somewhat less pure example
of the same problem well the corporation
is the is the is the aging yes
yeah yeah I think I also like like when
I when I when I think about like trying
to read that and trying to figure out if
I feel if I expect to feel like less
confused
I predict now but I don't know okay it
doesn't seem like it's probably
sufficiently different I guess David if
you have some interesting papers that
seemed like they could be relevant if
you could send it pass them along that
would either to me yours it's got
directly I guess that might be a
feasible way forward yeah I want to flag
also though that the general strategy of
like trying to learn from humans and
groups and like weird mathematical
models and like all basically like
learning from any kind of analogies that
we can make is the strategy that in
general I am like very proud and wanna
just point that out okay I don't think
there's anyone who helped raised a hint
David Booth no please
yeah yes I wanna I do want to reiterate
that in like two to three years I will
be looking for a job so if you want to
outsource that work to me then Emile all
right
some spree so I have another question
well there was actually a some questions
also more about Miri than about embedded
agency I don't know if you are actually
the right person to answer these kind of
questions probably not probably not I
yeah I might I might just refuse to
answer things without even giving a
reason but you can ask me questions okay
and one of the questions that were
posted before was about a Mira's current
hiring pipeline are there enough people
who are applying to marry and if not
what are what seems to be the problems
in getting more people on board and miri
of expanding
I
so I guess yeah I don't know this is
probably not an answer to that question
I want to I want to say a thing is like
related which is like workshops and
stuff that we run are like a pretty
large part of onboarding and like hiring
so it's like very unlikely that we would
like hire an individual without having
like interacted with them through
something like a workshop possibly
multiple times like if you're interested
in embedded agency type stuff then like
the Miri summer Fellows Program workshop
which we've run four times so far and I
don't know for sure that we're going to
run it again but we likely might run it
again is like probably like the best way
to get in to that and also like we run a
bunch of workshops for programmers which
is another thing that that might be
interesting and I I don't actually know
where to send you my assumed as an Mary
website like has information about like
how to get in contact about this but
like yeah I guess the main thing I'd
like many more I wanted to say was just
like workshops actually are like a very
like key part of the pipeline like when
I think about like getting new people at
Mary
I'm getting my thinking about like
getting new people to work with like
there's kind of two steps one is like
getting good people to like come to
workshops come to like MSSP and then
there's another step which is like
trying to like do filtering from that
point which is kind of bad because like
it's kind of a it's kind of like costly
too I think MSF fee was like almost
three weeks last time that might be
costly for a lot of people but like
currently so there's flaws of that but
currently like workshops are a pretty
central part of how I'm thinking about
they do to hire I don't even remember
that was the question yeah
MSP is married some fellows program also
called AI summer calls programs over the
years but it's the same thing
okay I can see that David has I can see
that David has raised his hand is this
more about this topic because in that
case you can get in ahead of Allie no I
think I just forgot to unraised my hand
fell earlier honey hi so a couple of
times I've so I read once that when you
try and get an Oracle to sort of self
reference or access information about
what it did in the past
you'd get some sort of the probabilities
become unmeasurable and I was wondering
that since self-reference seems to be at
the heart of a bunch of these problems
modeling agents what do you think you
would do if you found out that there's
like some non-measurable problem the
probability of agents going through with
self reference ah god sorry I'm not
making sense I guess I'm just trying
says what would you do if there is no
well-defined probability of dealing with
most self reference problems
yeah
so so this gets back to the like logic
and probability question which like I
think that's there being like no
well-defined probability for these
things is basically like saying like the
tools that we have that are like
probability shaped are like naught or
probability or like Bayesian and some
shapes are like not the right tools for
the kind of problems that like involve
self reference and I would feel like I
want to like I I kind of want to like
abandon the tool of probability but I
would like most of the time while doing
this be like looking for like every
possible way in which I could not
abandon the tool probability or
something and I'd want to like at the
end of the day understand why it's kind
of okay to have abandoned the tool
because right now it feels like there's
lots of like good justifiable reasons
say that like everything that's not
probability it's kind of like wrong or
something yeah but it's also possible
that like the path forward is to try to
abandon any kind of self reference but
this is like very very hard this is like
very hard for reasons that are like in
optimization like I said that that if I
imagine inner optimization was not a
problem then I would like be happy to
like do things with just rules and
knocks and not agents and I think that
like what I meant by like agents they're
like a large part of what I mean by
agents is like things that are actually
doing the self reference stuff so it
might be that like the solution is to
have things only working in a domain
where self reference doesn't matter like
if I have a thing that's not reasoning
about itself and only reasoning about
like how to like do the molecular stuff
right or something then like I could
imagine just like avoiding the domains
and which the things wrong but that
feels like very scary to me
like using using a tool which like you
understand that it would fail if who's
in this other situation and also you
don't understand why it would fail for
this other situation and just like
hoping that make good things stay okay
with that is pretty scary
Thanks okay so there is no one else in
the queue then I had a question about
the Libyan reference let me just see if
I can find this here here which is the
eluvian obstacle is it's mentioned in
the in the part of the sequence about
vinton reflection about agents that are
smarter than than you but I'm a bit
confused about whether it's actually if
the Libyan obstacle obviously happens at
around the human level because humans
are smart enough to prove loads theorem
so does it become worse in any
appreciable sense once the a s get above
the human level yeah it'll be an
obstacle I mean
yeah the Libyan obstacle for self-trust
is basically like saying at least if you
use like some sort of axiomatic system
for trust in a future self where the
future self is like stronger than you
like able to like know and like deduce
strictly more things than you then you
like like this is like the most liberal
version there's lots of like other
versions there like an analogy with this
but like then you like Nestle
necessarily or inconsistent and you can
you can basically prove everything so
one could say like well maybe I could
like trust my future self but like also
trust that I am like whatever my search
for like beliefs or something is weak
enough to like not find the argument
that light passes through love's theorem
and this just seems it seems like if
you're powerful enough to get like any
useful stuff out of this out of like
your trust for your future self then
like you're kind of just like already
too close and I think that like I think
that if you're gonna imagine a system
that you think is like weak enough that
it's not running
if you imagine a system that is avoiding
the low be an obstacle just because it's
like too weak to like be able to reason
there's a problem to be able to like
realize there's a problem
you're probably if you're imagine a
system that's going to keep a system
that weak you're probably better off not
having a reason about itself at all then
trying to have something that is
reasoning about itself and Trust in
itself but like not yeah so yeah if I
were to imagine trying to have something
that's like weak enough to not have the
problem that just seems harder more
impossible over strategy than trying to
have a thing that's not reasoning about
itself at all
to follow on that I will imagine that
almost all current humans are in that
category in that almost all tournaments
are modeling themselves and thinking
about themselves in many different ways
reflecting on themselves and at the same
time most humans are not capable of
proving loops theorem most humans have
never heard about it of course i think i
think that most humans do not have
something that looks like an axiomatic
self-trust amusing themselves like like
I was saying axiomatic and that does
actually mean something I think that
like there are ways in which one can
learn to trust their future self kind of
my induction like I can learn hey over
the time as I as I've like grown over
time it's turned out that whenever my
future self believes something that my
past self didn't believe my future self
was right because my future self could
take into account what my past self
believes and I've tended to be more
smart over time and things that I've
like learned like my beliefs have gotten
better over time and stuff like that
and I could like learn that fact and
like believe that fact without having it
be the right form that would like create
low-beam obstacle but yeah
okay I'm still not seeing anybody
recently I guess I'm trying to say that
there isn't there isn't a short argument
that I could give which is like I'm
going to teach this human about lobes
theorem and then all of a sudden they're
gonna start proving bottom all the time
like and that's because like they're not
working in a system such that yeah okay
okay Chris kuba has a question
yes thanks I'm trying to go back to
basics on the five-and-ten problem which
wasn't the only one who got really
confused by it and I'm sorry by the way
I know we had any discussion about it
I'm sorry if what I say was actually
answered by that but anyway the thing is
we have this we have this stage in the
five-and-ten problem where the agent is
is essentially saying it's considering
whether it's got the five dollars or the
ten dollars and it's from me from the
from the proposition that picking up the
the ten dollars would give it either
zero value or less value than five
dollars this counterfactual leads to it
did you think that it's better to pick
up the five dollar property the five
dollars rather than the ten dollars I've
probably completely mangled that because
just because I don't understand it basic
but basically the only way I can make
sense of that is is that suppose that um
this counterfactual of picking up the
ten dollars giving less value is a
result that comes from the finite
processing ability the finite computing
ability of the agent is presumably if
it's if it actually had sufficient
computing ability it would never
generate this counter
sure but can you tell me if there's any
sense in what I'm saying yes the thing
about it coming from finites computing
ability is like definitely not the the
thing that I'm trying to get at I think
that the where it's coming from
is I'm using a really stupid
counterfactual the counterfactual that
I'm using is like material implication
I'm saying that like if X implies Y then
like I would I'm like willing to accept
the counterfactual like if kind of
actually acts than Y which is a really
stupid version of counterfactuals and
like but it's yeah so it's really stupid
version of counterfactuals but we really
don't have many alternatives like we
have like like like one might propose
the thing that one might propose is like
material implication the yeah the
implication is where you get your kind
of factual strum and this is kind of
showing that like that doesn't really
make sense and one might also say that
like expected utility is where you get
your counterfactuals from like an
expectation like conditioning with like
some probability thing is where you get
your kind of factual's from and that
also kind of doesn't make sense
you get like division by zero if you
like know things about yourself like I
think this was not supposed to be a
positive results as much like a negative
result in the sense of like it's showing
that like like all of the simple ways
that we have for like writing down what
counterfactuals are like are wrong and I
think that I think the takeaway is that
like implication is not like a valid
counterfactual and that it matters where
where you like how you like try to
define your counterfactuals and that
like all the ones that are really easy
to define like are bad and then we're
like well now what do we do
and also like counterfactuals are like
in a certain sense like unlearn Obul
yeah i mean i sort of yeah composed to
be see that I mean you didn't but I
don't think it's we as a resource
bounded decimal
I'm sorry singing but I but I don't
think it's because the resource bounding
this at all I think that it's nicely
it's like a way there with yeah did you
unbounded agents right right
did you device the five-and-ten problem
yourself or is that standard in the
literature I don't actually know the
history you know it was it was
definitely a thing that people and Mary
talked about long before I was there I
believe it's um what why we died who
came up with sort of retreated houses I
some point tried to call it the heavy
ghost problems I thought that was much
cooler but five and ten stops okay
that's a funny anecdote um I guess yayin
has Nick OS promising we are above the
one hour mark here so I guess Scott if
you need to go somewhere else then then
please just say so
otherwise I expect we will continue
asking questions until yeah the the
queue of questions is empty basically I
hope that's okay
all right let's keep going at least for
now okay Diane has a question Jen could
you read it all out your question please
absolutely the embedded agent sequence
mentions ontological crises but does it
seem to go into detail do you think
ontological crises are a significant
issue in designing invented agents would
solving some other subproblems reduce
the challenge of ontological crisis
yeah so I think that ontological crises
are like wait uh I think I'm gonna say
some things later on for a season I
might need another question again but I
think ontological crises are like
possibly they seem like pretty likely to
be related to how to like do embedded
role model stuff because you have to
like start out working in a system that
like is like necessarily wrong and be
able to like move to another system at
some point and also there's like a
similar problem to ontological crises
which is like merging of oncology's
we're like maybe I have some beliefs
that are in one ontology and someplace
that are in other ontology and like how
do I like deal with this fact and these
seem important to me also like you
shouldn't update too much on the fact
that I mentioned ontological crises at
all because part of what the part of
what the like embedded the agency
sequence was being was like trying to
communicate and ontology from which you
can kind of like look at all the past
mary stuff and like ontological crises
was like an example of a past mary stuff
and i kind of just like included
everything the kind of fit but i also
think that it's it's like it's i think
it's like important but in the way in
which like I am naively factoring the
problem so I was thinking about it I
wouldn't have like called it I'm not
sure if it's like cutting things in the
right way I don't know yeah you want to
say a question again now that I ranted
sure yeah
thank you thank you that was that was
helpful that gave a lot of good like
background info I guess the the question
was really you know
do some of these other subproblems in
embedded agency relate to analogical
crises and we're kind of tied into that
and I think you you are to certain like
embedded world models it's possible that
delegation robust delegation could be
relevant because if you're delegating to
different ontology yeah I I think I
think I want to say something like
oncological crises I I think maybe I
thought food crisis is a long handle
there's just like stuff that deals with
what happens when ontology czar wrong or
what happens when there are multiple
different ontology x' that need to be
able to communicate with each other like
how do you how do you just deal with
that problem and that feels like a
cluster of problems that you kind of
want to look at all at the same time and
that is a thing that like you'll see
across like across everywhere not
everywhere but across like multiple
places and I agree that like it seems
like an important part for a bus
delegation because like yeah because you
have different apologies and like
earlier and later things we need to be
able to yeah also seems useful for like
transparency type things and just
communication in general
cool thank you okay great I don't think
there are any more questions in the
queue does anyone have any final
questions they would like to ask Scott
government from Miri well then I would
like to say thank you to Scott one more
time for coming here and answering our
questions and then I'll basically say
see you next week thank you