189. The Off-Switch Game

tea reading group this week we'll be
looking at the off switch game I had
filled it out so just to go over what
Keurig ability is and the context of the
off switch problem touring brilliant man
that he was realized that if we designed
an AI especially a powerfully I might
get into situations which would be
rather painful or even disastrous for us
so clearly an off switch would be
invaluable oh mohandro and his basic AI
drives paper noted that survival is a
basic try for an AI in the current
framework it needs to survive just to
fulfill its objectives or as to add
Russell what it you can't bring the
coffee to me if you're dead so in this
case we'd think oh hey the AI wouldn't
tolerate having an off switch so how can
we deal with that this is what Mary was
what seemed to be motivating Mary when
they wrote their paper on Keurig ability
and they considered how we might build
in an off switch as a last resort in
controlling an air and one way you might
try to do this is to augment its utility
function perhaps you could make it
indifferent to being switched off or you
could make it
indifferent to changes in its utility
function say we think that the AI is
going to cause problems so we change its
utility functions so that the best
option is just to do nothing
unfortunately augmenting its utility
function this way provides some perverse
incentives and some other difficulties
that I won't get into if you just build
in off switch from the get-go
interrupting it in that way could damage
its learning so when it's trying to
figure out what the best thing to do is
if you just switch it off whenever it
gets into a bad situation it might just
not update its policy on what to do
Stewart Armstrong said okay well let's
just modify his policy instead at a
fundamental level so that the AI doesn't
view us as switching it off it will view
itself as wanting to switch off in
certain scenarios unfortunately this can
only go on for so long before you need
to get rid of that modified policy but
for quite a while you can keep on
interrupting the AI getting it to
interrupt its own actions without
sacrificing its long-term learning
potential so while back Stuart Russell
tried to formulate his own view on how
we should approach AI safety he
formulated three rough laws for a human
compatible AI one the machines only
objective is to maximize the realization
of human preferences to the machine is
uncertain about what those preferences
are three the ultimate source of
information about it human preferences
is human behavior so Hadfield I think a
colleague of Stuart Russell said okay
well Stuart's got these three laws and
there's this off switch problem let's
try and apply the approach here to see
if we can make any progress who actually
make this problem intractable he
designed a basic game a decision problem
where there's a single robot and a
single human with only a few actions
available
he said the robot will want to maximize
human value and the robot is unsure
about human value so it's going to treat
whatever the human says however the
human axe has information about the true
utility so now we have a situation where
the robot can either choose an action a
switch itself off so that's action s or
ask the human what to do which is action
waa the human will assign some value to
action a the probability of picking a
particular human who assigns a value UA
is PU a so the robot will have a prior
on what the human will assign its
actions the game itself is quite simple
in the game the robot will either choose
action a true and in which case it
thinks the human will get value you a
call the expected value of UA over the
prior but whatever it has some view on
what value of the human or the sign it
or it can switch itself off in that case
it knows the human will get value you
out of that action the third option is
that it can choose W a which means
letting the human H decide what it
should do if the human H says s then
that means that the robot should switch
itself off and the human will find
themselves in a scenario with is Val
zero value if they choose notice then
that means that the AI must choose
action a and the human will achieve
whatever value they assign that action
and if the human is asked by the robot
what it should do it's will stay do a
with probability PI H so that's what the
little symbol in the bottom right is
it's a bit of a weird notation but
that's what Stuart rose and conversely
it will tell the robot to switch itself
off with probability one minus PI H
because no conservation of probability
now that the robot is going to treat
whatever the humans responses as
information but it might not choose that
action in the first place
so in examining when a robot would
actually ask the person instead of just
choosing a or choosing us
Hadfield constructed this thing called
the incentive which is the difference in
value between asking the human it's
opinion and the best option from its
point of view so it's the expected value
from asking the human - the value of
choosing action a or switching itself
off whichever one is larger now this is
where it starts to get interesting if
the robot R knows the human hate is
rational in other words if the human
will always choose to tell the robot to
a if their utility is more than zero if
they assign utility more than zero to
that action and if they will always tell
the robot switch yourself off if they
view action a as being worse than zero
value worse than nothing then the human
is rational and in that instance the
incentive for the robot will always be
greater than or equal to zero and the
robot will always ask the human what it
should do with maybe one or two minor
caveats this is true even if the robot
is completely sure about what the humans
value will be then in which case say the
robot if it's mistaken this is obviously
a good thing because the human will be
able to rationally tell the agent no
you're wrong to switch yourself off and
the agent will do it in fact in the
latter case it's an equivalence the
robot will only let the human turn it
off
if it's sir if it they're rational in
the case that they're completely sure
about what the human wants so we'll
explore irrationality in a bit how that
changes things but we'll just look at
some aspects of the incentive Delta so
here you see two graphs we're assuming
the Hume the prior for what value a
human will assign the action a is
distributed like a Gaussian a normal
distribution then the variance will be
this little symbol Sigma so if we see
how Sigma alters Delta we see that in
the case where in the robots perspective
the expected value of action a is zero
then Delta increases monotonically with
the uncertainty so the more uncertain
the robot is the bigger the incentive
for it to ask the human this is true
even if the expected value is positive
so the robot thinks that is better to do
action a than switch itself off and it's
also true if the expected value of doing
a is negative that is it's worse than
switching itself off and the curious
thing is that this is actually symmetric
so even if the so if you look at the
graph on the right you see that there's
a map of the the robots prior the
uncertainty Sigma is plotted on the x
axis and the expected value of action a
is plotted on the y axis the various
colors are just meant to represent what
value Delta has so you can see that as
you move right the incentive increases
that's basically corresponding to what
we saw on the Left graph this is just
saying more uncertainty more incentive
to ask the human the curious thing is
that it's symmetric on the expected
value so you see that the graph is the
same
of the zero the line where the expected
value is zero as it is below the that
line this is because it's just a quirk
of the game but it basically means that
whenever the robot thinks that there are
larger stakes it will decrease its
incentive to ask the human because say
if it thinks that a catastrophe or
something wonderful will happen when it
chooses action a then it would need to
be very unsure about its actions in
order for it to let the human take over
because the stakes are massive here what
are the humans an idiot right so massive
stakes increase its reluctance costly
human uncertainty in what it thinks the
human will say increases its incentive
to ask the human so those are the two
opposing factors and we'll come back to
that later so now does this whole model
only work for rational humans the answer
is an empathetic no so we'll again
assume that the prior the robot has over
what it thinks the human will assign
what value it thinks the human will
assign action a is distributed like a
Gaussian now working with this
assumption will just examine the
irrationality of the human how it
impacts things we'll need a concept
first before we go further so if the
robot chooses action w hey if it asks
the human what to do then we'll say the
human corrects the robot if it tells it
to do the opposite of what it thinks it
should do so for example if the robot
thinks that action a will have utility
greater than zero we'll say that the
human corrects there or what if they say
no that action is bad it's worse than
being switched off and vice-versa so the
probability of correction is 1 minus pi
H if the expected value of action is
gray
zero and it's pi H if the expected value
of action a is less than zero this is
just because PI H is the probability
that the human will tell the robot to do
action a so it's basically just saying
that's the probability of the human
telling the robot to do the opposite of
action a more or less then you find by
calculating Delta the incentive and
requiring that Delta the incentive is
greater than zero we find that the
robots will only ever choose to ask the
human what to do if the if this
inequality is satisfied on the right the
terms there are the absolute value of
the expected utility of action a because
remember it's distributed like a
Gaussian and the mean of the Gaussian is
mu divided by the variance of the
Gaussian which represents the
uncertainty times the probability of
correction if that is less than this
term on the right then choosing to ask
the human will be optimal this term on
the right it's basically a measure of
human rationality explicitly it's saying
take the probability of the human saying
do action a and differentiate it with
respect to you a because we expect that
if the human is irrational we think that
it's going to choose to do action a with
some probability depending on however
much value it assigns you a so that's
what the parentheses mean there's just a
functional dependence there so what the
term on the right you can think of it as
saying if there is a small increase in
the utility then if on average that
means the human is a little more likely
to choose action a
it's rational that's what that term is
representing so I'll try and say this in
plain English the equation is the
expected cost of the human correcting
the robot divided by the robots
uncertainty if that's less than a
measure of the humans rationality then
and only then will it ask the human so
this gives you a sort of gut feeling on
how you expect the robot to behave if
the robot is massively uncertain then
provided the human is a little rational
it's going to say okay I should ask the
human not to do on the other hand if the
expected cost of the human correcting
the robot that is the human telling it
to do the opposite of what it thinks it
should do is just massive then it will
say okay the humans either got to be
really rational or if they're not I'm
just going to choose action a because I
can't let the human correct the stakes
are too large so those are just all of
the competing factors out there
explicitly and they perform the
calculations for various values of the
uncertainty and a particular model of
the rationality and they found that it
behaves more or less according to what
the inequality says I can go into that
later if you want but the key gist is
that the equation it's a trade-off
between human rationality the cost of
asking the human and the uncertainty the
robot has about whether or not it's got
things right so you might ask well isn't
it always better to just ask the human
what to do because obviously if it asks
you what to do then you can turn it off
and that's a desired valve desired
property and
it seems like it should be able to
increase value to users right like you
can provide more information and get a
better response from the robot this way
unfortunately there's a corn while too
if the robot is immensely uncertain then
that means it's just God no clue what
it's doing and it's going to let anyone
turn it off as long as they appear even
the tiniest bit rational so I might say
be driving a car it's taking a
three-year-old to nursery and the robot
sees ok um I can either take route a
where it's a bit faster but an accident
might happen or I can just default to
the safe route I know and if it asks the
child for some reason well if it asked
the child what you must because it's
very uncertain then if the child just
says oh yeah go the fast way and then
the child dies in a car crash that's you
know obviously not very desirable so in
other words uncertainty just makes it
less reliable when it's presented with
someone who's not reliable themselves
and the other factor is that the AI
won't be able to provide useful info to
ignorance users so say for example
you've got something called a credence
good in economics this is when a
consumer buys a good and they don't know
the value of the good even after they've
consumed it so say you go to a doctor
and you ask the doctor what what's wrong
with my body and the doctor says no
there's nothing wrong with you and you
say ok well the doctor says I'm ok but I
mean I have no real way to assess that
so I'll just have to trust them so if
the doctor says hmmm ok I don't know
what's wrong with you
what do you think or something along
those lines if the doctor is uncertain
that's obviously quite useless and we
could try and there's probably some
situations which are analogous to this
where robot has the vast amount of
information it needs to make a
reasonable decision it's it's still
uncertain because you've decided just to
be very cautious in which case if the
human says okay I want you to do this
thing but could you also tell me what
the advantages or disadvantages are if
they're robots massively uncertain it
might just be too afraid to tell the
human because it thinks that oh no I
don't know what to do so I can't tell
you now in the paper had filled in at at
our examined the some sort of situation
like this vaguely and try and see what
the value and hey I could provide to a
user is the issue is that they're a
little unclear and what they actually
mean by the whole setup so I'll just try
and go with one my interpretation of it
is suppose the person themselves are
either uncertain about what they should
choose or they can only give a noisy
response as then they might be connected
to the AI through some terrible phone
line say and they try and tell the AI
what to choose but there's so much noise
the air can barely hear them if the AI
is immensely uncertain then it seems
like it's not going to be able to choose
a sensible option it's basically going
to become something like a toss-up
between either choosing action A or
choosing action
and the human desperately wants a the AI
to do action a but because of the bad
phone line they just can't get through
so the AI is uncertain Ian what to do
can just decrease the Returns past a
certain point you have to also remember
that that converse is true if the AI is
absolutely certain
then it won't ever listen to other human
says so if it's plausible that the human
wants them to switch off they can just
go nope I don't care I'm just going to
do whatever I want and this is basically
the intuition behind the graph on the
bottom left hand this is the value to
the user and you see that as the
uncertainty from in the AI increases the
value increases because it's more likely
to ask the human motor once but
eventually past a certain point this AI
is going to become too unreliable and
the value it would provide to starts to
decrease and this is in spite of the
fact that the incentive always increases
with the uncertainty and you might want
to say what if there are more actions
can I does this change things much what
happens is that actually makes things a
little worse since the incentive always
increases with the uncertainty on the
graph on the bottom right we can
basically just instruct easily a graph
where it's varying the a uncertainty
instead of the incentive but we'll just
go with the incentive for the moment
increasing the incentive has some
returns when there's only one action but
then eventually past a certain point it
starts dropping slowly when you increase
to for possible actions there I can take
it drops much more rapidly when you
increase it eight actions it drops yet
more rapidly and you'd find in the lemma
that when there is an incredibly complex
system the AI should in order to get
optimal results have just the right
uncertainty the uncertainty therefore
can be very tricky to optimize and you
shouldn't attempt to bias the AI that is
essentially the paper I have other
things I could talk about with respect
to the off switch problem if you guys
like but this is all that was in the
paper
thank you very much I'm also for anyone
who's not seen me doing this thing
before I just decided to use a a writ
written version set of slides because I
wanted to try something new


257. Where I agree and Disagree with Eliezer 3

hello and welcome to session 257 in the
AI safety.com reading group tonight
we'll be finishing up where I agree and
disagree with Elisa by Paul Cristiano
there
um Paul Cristiano is still a researcher
in in the alignment Research Center and
hold on
um and this is uh starting from part 20.
hold on why is this down okay
um so we'll start with this agreement
number 20
um about uh to what extent words reflect
our actual Deep Thoughts Paul um
states that human thoughts partially
exposes only a partially screwable outer
service layer words only Trace our real
thoughts uh this to me is kind of
mysterious in a in a deep sense I don't
really know but Cristiano isn't pushing
back very strongly against the
mysteriousness of this point he's mainly
saying that sure if we just optimize
end-to-end on outcomes that seems like
something that will not relate to human
thoughts very much but
um we can and just using human thoughts
directly is also a a bad strategy but
there are a number of strategies in
between uh like AI safety via debate and
a number of others he even calls it
plenty of techniques
um
I uh I haven't looked very deeply into
that area of uh alignment research but I
think uh Paul Cristiano's being very
generous by calling it plenty of
techniques
um as fast I can see they are all really
undeveloped and I don't think we should
um
put a great amount of stock in in any of
those
we have seen large language models uh
apparently being quite useful right now
and it seems very likely uh to pull
Cristiano that this is evidence that a
similar kind of language-based models
will be
competitive at the time of
transformative AI
um my first thought on this is that this
is explicitly only on the level of words
and not on the level of thoughts that uh
um
Elia zurichowski is talking about
Elizabeth has a different complaint
about this he believes it's hard and
probably impossible to just do a make a
powerful system using similar this kind
of symbol imitation of human words
here I will return to one of my key
points from uh last session where Paul
Cristiano and Elia swedkowski talks
about two different things when Paul
Cristiano talks about economically uh
transformative AI he's thinking about
something that can an example would be
something that can you know sure can
secure aging do space colonization this
kind of thing and what uh Elia
zurichowski is explicitly talking about
is capable of creating pivotal acts and
in a very real sense it may be a lot
easier to cure aging than to take over
the world
disjunctiveness as in having a number of
different mostly independent Solutions
um is
um is a key part of ilyasuredkowski's uh
AI Doom scenarios because there are so
many different things that can go wrong
rejoined it to this is there are also so
many ways it can go right and we only
need one alignment strategy to pay off
in order for us to basically be safe
uh I think uh the way I would try to
synthesize these two points is to say
that um we need uh there to exist an
alignment strategy I call it a such that
for all the lethalities of the alignment
strategy a is capable of overcoming The
lethality so there are two levels of
disjunctiveness disjunctiveness in the
alignment strategy and disjunctiveness
in the lethalities
um
there is a a problem in this in that uh
we have a a long list of lethalities by
elizabethkowski and this is probably not
exhaustive but uh if we take an
alignment strategy and try to use uh I
don't know something like listing latent
knowledge
um then that is only one strategy that
we are using and using a listing latent
knowledge is probably prevents us from
using a number of other strategies
um so to some extent at least they are
exclusive so we only get one alignment
strategy
depending right and a number of uh uh
alignment strategies can coexist so it's
not total we only have to choose one
um
Paul Cristiano argues that the
disjunctiveness in alignment strategies
may be more just uh like greater like
greater degree of Independence since we
know that there are a number of humans
and we don't know that there is in fact
uh how many AIS and how many AIS that
are possible
um
I think like counting the number of
humans and Counting the number of AIS is
the wrong way to look at this
um there are reasons to expect that the
disjunctiveness in alignment strategies
is limited
um right now the number of people who
are working on this is low probably in
the low hundreds
um and a lot of these people are working
on very closely related alignment
strategies a lot of these people are in
Berkeley a lot of these people are
rationalists or effective altruists a
utilitarianists they have there is not
that much uh diversity among alignment
researchers and of course other reasons
might be that they are just searching in
a part of the solution space where there
are no Solutions maybe there are uh and
again we can't
um uh
probably implement all the alignment
strategies that we are thinking about
Cristiano also have an argument about
where these this disjunctiveness comes
in I don't think that matters very much
when the disjunctiveness happens
um yeah
how alien will an AGI be the uh iliacy
point is uh that it might indeed be very
very alien and not use the same kind of
Concepts that we do
um
and uh pocusiano in one of his uh
recurring uh complaints is that uh Elise
redkowski is very very overconfident in
this
um
and seems uh
Pocus channel is taking a more I
wouldn't say anthropomorphic but more
human inspired uh view of the ai's
capabilities for instance it might
understand the same things as humans but
just slightly better
um I think again we are talking past
each other in that
um something that understands the same
things as humans but slightly better
could be very economically
transformative could obviously
um like do all menial tasks but could it
do a pivotal act my strong expectation
would be that just being slightly
smarter than like an average human is
very far from being sufficient to take
from to take over the world or solve
alignment
um another uh way that uh AI could turn
out to be human-like would be to be as
like a stupid human but just thinking
faster
um
or be extremely human inspired because
that's where all the training data is
basically
um
I also disagree that these two things
are likely to bring us to True
superintelligence
um
like
human imitation uh won't get us directly
to Super intelligence it might
um
it might be a strong step of the way but
by definition you can't imitate to
something greater
so what kind of purital Acts can weak
AGI do
um
we might see that
AI would be doing science and that might
come from similar Concepts the uh as we
do and in that case we can see that the
AI is like creating experiments
formulating a hypothesis and deciding
among the hypotheses in a scientific
process but this uh while it would be
nice to see the AI doing experiments
that doesn't actually tell us very much
about what's going on under the hood if
we don't understand the actual Concepts
being used if we can just see the AI is
doing some experiments but we don't know
the concepts that they're trying to
elucidate then that won't help us very
much or may not help us at all
um and the idea here is that uh if the
AI is fast and cheap that could be
enough for transformative uh in
particular I feel that cheap AI is
unlikely to do very much like um towards
solving the alignment problem or taking
over the world
um
yeah I don't think that that is a very
likely scenario
Cristiano uh obviously would disagree
with this and he even makes a very um
controversial statement that he could
see that uh fast and cheap AI
um being so much better at alignment
research that human contributions are
basically Obsolete and rohin Shah in the
comments says that's a pretty high bar
and Pokemon say yeah okay and retracts
that and this is the kind of really
frustrating to look at this from the
outside because this um
uh this exchange leaves you feeling like
okay why
um
there's a lot of uh implicit
understanding here why was Paul cruciano
able to change his mind just based on on
like three words from Robin Shaw
basically
um I don't know I would really like to
know uh what are the underlying models
that Paul Cristiano has and that her in
Shah has about how this uh uh fast cheap
weak AGI would solve alignment because
from my point of view it looks very very
unlikely that you can uh solve alignment
using these kind of techniques
disagreement 23 about is about how we
reason about agis
this is probably inspired by uh
Elizabeth saying that humans can't
participate in coordination schemes
between super intelligences and
um
for uh one of the reasons
is that we can't reason about the code
of superintelligences
answers that well the Supreme challenges
may not be able to do that either
because they are very likely to be messy
this I would hold out as an example of
not uh Elisa utkowski being
overconfident but Paul Cristiano uh
saying some things that is very likely
about the structure of uh of future
super intelligences is
um is just really really hard we
strongly do not know this
Pro Cristiano makes a uh uh another
strong claim here in my opinion about
what will the tools be that the AIS use
for reasoning about each other
um one of uh just black box looking at
each other's Behavior the second is to
use the same kind of tools we use like
into the interoperability tools that we
have right now Etc
um doing reasoning that is similar to
what we do right now and no other way
basically those are the three ways that
AIS will reason about uh other AIS when
we get to something like super
intelligence uh I think this is again
really overconfident and also I would
say that this is at odds with Paul
Cristiano's hope earlier that AIS would
be able to solve the alignment problem
and and if AIS are not able to reason
about each other uh in better stronger
ways than the than we are then why do we
have any particular hope for them
solving the alignment problem
uh and of course I also strongly feel
that this is like I I still have hope
for AI helping us solve the alignment
problem because I do believe that super
intelligences are qualitatively smarter
than us but if you don't believe that
you will have something like that then
why would that help why were they then
be able to contribute to solving the
alignment problem
the example that uh Elizabeth gives is
by reasoning over a probability
distributions of each other's code using
logical decision Theory this is
basically not how we are reasoning about
uh tpt3 right now uh I I suppose you to
some weak extent humans are in fact
capable of doing this but it feels
really strongly like something a super
intelligence could do much better than
us and I think that's an example of
something that goes beyond these three
examples uh by Paul Cristiano and that I
think
because they seem to be just narrowly
out of our grasp it seems very likely
that stronger AI would be able to do
this
um another scheme that
um
Kowski criticizes is by having some kind
of it's sometimes called multiple
Godzillas that are trying to keep each
other in check and that seems like
something that will fail in Italia so
your task is model once the they become
sufficiently capable
this is criticized by Paul Cristiano as
saying that this is something that will
be
um important in the long run but not uh
really uh relevant in the short term
uh I think there's an easy way to
explain this in that if you have shorter
timelines then the difference between
the short term and the long term might
be like very very short time
um
so that could be a part of it
um
I wonder also if Paul Cristiano is
seeing a time where we have AIS that um
came to pivotal acts but are not super
intelligence because in this case
schemes for playing them off against
each other could to some extent work
maybe if they can like solve the
alignment problem but they can't but
they are not super intelligent
maybe it's unclear
um
another problem uh uh with the
unbreakable schemes that Elisabeth
points out is trying to like have a
designer and a design Checker and try to
um
incentivize those to work against each
other and then they will obviously be uh
actually incentivized to
um to collaborate against us
this is uh some uh something that Paul
Cristiano has uh low thoughts about so
this is just an unrealistic incentive
based anti-corporation scheme proposed
by random people so I obviously would
need to go went out to try to find some
people who did propose this kind of
thing and here is like the research
agenda from the center for existential
risk um
which goes into substantial details
about precisely this so it's not just
proposed by random people
um
I would actually
I would give one to three odds that if
you went through everything that Paul
Cristiano himself has ever written you
would find examples of incentive-based
anti-corporation schemes uh I think
that's uh I I haven't actually gone
through everything Focus channel has
written but I think uh he has written so
much and it seems kind of adjacent to a
lot of things he has written so I would
expect if you actually went through it
there's a substantial chance he has
written about that but that's I I
probably shouldn't say that with without
actually going through that
hmm
so the alternative that podcast channel
is pointing to is
selecting for systems that play games
competitively instead of uh trying to
get something that just reacts to
incentives that are possibly misaligned
um
there are examples of ways to do this I
think incentive based methods have a
higher probability of working in the
limit of a very powerful AI
um but but this is uh certainly
something that that you can do instead
and and that might in theory work
is accused of equal voting between two
statements first is that AI systems will
cooperate and the second is that the
verifiable activities you could use to
gradient descent to select file won't
function appropriately as checks and
balances
so it is possible indeed that Ilia
zayatkowski is equivocating between
these two but I did in fact go through
the entire document with his list of
lethalities and I can reasonably
confidently say that he does not equal
vote in if we vocate in in this way in
this document so I would like
procristiano to come out with an
explicit example of where this happens
um because I'm not seeing it
manipulation by super intelligence is
something that Elia saidkowski believes
is likely to happen uh depending or it
will be possible at least
um and uh the The lethality where this
is most uh prevalent is in his uh
statement that a super intelligence
could make a psychological manipulation
attack that we could not recognize
um
this is uh in Pakistan's summary becomes
that if you can do pivotal X then they
can also manipulate humans very well so
there's no point in having debates
between uh super intelligences or try to
pray play adversarial games against them
once they are capable of doing
previously Acts
um this is somewhat the uh the
disagreement doesn't uh strongly relate
to what Elias utkarski is actually
talking about
um
and also I feel that if you have an AI
That's capable of doing pivotal acts and
if you have like multiple different
independent uh super intelligences that
uh capable of doing professional acts
but haven't done pivotal acts then my
timelines are very short in that in fact
social that I don't expect will have
like multiple super intelligences
capable of doing pure selects
foreign
mind with a profile of abilities
there is a
you have like bostrom's six cognitive
strategic tasks like research and
development being one and persuasion
social manipulation being another
which one is harder to get at to some
extent like all superhuman uh levels are
equally hard in some in some sense and
so that that seems like a reasonable
prior that getting to Super intelligence
along these Dimensions is equally hard
um I think that this prior is uh
uh uh I can see it as a pride but we
also have some evidence in particular
um persuasion is something that it is
reasonably easy to get feedback on like
I imagine Facebook could run some
experiments some simple AI a b testing
and actually get reasonable data about
what kind of things uh persuade people
and what does not
um you could have an uh an AI in YouTube
during seeing what drives engagement I
think that is probably things that are
already happening and in contrast
research and development when Paul
Cristiano is talking about that then
space colonization or curing aging
that's a kind of research and
development and that's not really what
we care about the research and
development we care about is alignment
research and that is pre-paradigmatic
and that seems a lot harder to get good
feedback on compared to something like
persuasion
so from from that point of view I would
expect it to be easier to get superhuman
ad persuasion than to get superhuman out
of alignment research
so
this is probably dominated by how much
effort do we put into this and if we
truly want the AI to be superhuman at
alignment research and we put a lot of
researchers resources towards that goal
then maybe we could obtain it
do we actually in practice have that as
our goal my understanding of where AI
resources right now is being spent is
that a lot of it is being used or either
persuasion or persuasion adjacent work
such as delivering ads I believe Google
does a lot of work on this or like
driving engagement on YouTube and ads on
Facebook and and things like that and I
I believe that there is a lot more
actual resources being put towards
um during better Google ads than being
put towards alignment
and this influences all the uh the
arguments in in Pokemon's view we are
likely to put more resources towards
alignment and if we do that then the
training is done on alignment so uh that
will push it towards being better at
alignment if the tools and all the
structures are designed to facilitate
research and development and there is a
number of AIS that are all collaborating
on advancing research and development
whereas manipulation is much more
disjoint
um and all these arguments unfortunately
work precisely in Reverse when we are
instead assuming that more resources are
pretty important towards manipulation
like if the AIS are primarily trained to
manipulation and the tools and
structures are designed to facilitate uh
manipulation and ads and there are many
AIS all working on the same kind of goal
and Alignment research is the disjoint
one in that case we should strongly
expect to see a superhuman persuasion
before we see superhuman alignment
research
so what is actually
um going to be easier Paul Cristiano has
a very weak prior that humans are to
some extent evolved to be good at uh
persuasion and social manipulation
um and not really uh research and
development that seems like kind of a
byproduct of general intelligence that
um uh I think that is very likely I
would point out that there might be a
very substantial overlap between
alignment and manipulation and uh you
could argue that some parts of alignment
is indeed counter manipulation and that
seems not that out of the distribution
possibly but I am mostly confused about
this and pocusiano also has only a like
a weak intuition about this
the 26th disagreement is about plans
here in the background you can see the
schlitter plan an example of a real
world plan for Germany's Invasion of
France and Belgium during the SEC the
first world war and notably this was a
plan that was insufficient in that the
sleeping plan was not possible to carry
out
um and do we have something like this
for alignment
uhkowski strongly says no we have no
sleeping plan or anything like this and
what you're seeing right now when you
look around is not what surviving worlds
look like
elizabethkowski uh
conceptualization of plants is not the
same as what uh pocusiano thinks he
thinks that you don't really have this
kind of plans in general uh and I think
that is probably true in the pop
Christiano verse in his worldview where
a lot of the uh alignment successes
happen by default in some sense in that
case if alignment happens by default
then plants aren't really that uh that
necessary on the other hand if you
expect that we are in if not the worst
possible world then a rather tough world
where alignment is truly a very
difficult problem then to me having some
kind of plan seems really helpful
um
poker channel uh pushes back on
um whether it is a good
conceptualization of what actual plans
are and again I get a feeling that they
are talking past each other in that um
Pokemon is saying that Elisa utkowski
doesn't have a good image of what a plan
actually is but uh Elizabeth is not
saying that the current plane is bad
he's saying that there is no plan and
even if you don't know very much about
plants it is quite possible to recognize
uh just that there is no such plan
I would also in addition have liked
Christian to be like more concrete
precise what things are wrong with
eliseo's conceptualization of a plan
even though he doesn't go into great
detail
um
why should we defer to eliasa on what a
plan looks like well one obvious reason
is that he has explicitly written down a
plan for death at land that seems like
it could work
um not in great detail but I think
that's more than most I don't think we
should really strongly differ in the
sense that if we had a plan that looked
like this living plan then uh earlier's
counter arguments would be strongly
insufficient to to criticize this but we
have nothing that remotely looks like a
a real plan
so how much value does the document in
GI ruin a list of lethalities actually
provide
uh Elizabeth
it's very valuable and that a core
alignment researcher should
spontaneously write this document
uh Paul Cristiano not surprisingly
disagrees like most people insist this
is actually emotionally aimed and
rhetoric and pedagogy
um I disagree I think that it has a
stronger rhetorical impact and it is
very uh
it it has a
changed a lot of opinions but mostly
from the fact that the the document is
extremely blunt
not from uh the the actual contents and
uh I don't think that uh poker no
um recognizes that the uh the lack of
bluntness in previous communication is
something that has helped people
substantially back and the rhetorical
Impact May in fact be just a um
uh a byproduct in some sense
is this something that will actually be
helpful towards solving the alignment
problem that is not what poker's channel
says but the upvotes uh on less wrong
seem to be some kind of disagreement and
I also strongly disagree I feel that the
AGI room list of lethalities is a very
very important document towards solving
the alignment problem I think as an
example
um Nick Boston's book super intelligence
basically contains almost none of these
points and I think that Nick Bostrom
basically should rewrite the book to
take these kind of lethalsis into
account and I think that they are in
fact really really important
considerations
the counter argument uh Pocus channel
has is that uh Alias are called the
ideas he focused on important but that's
not an objective fact about what is
important I think that's a fully General
counter argument you can just
counter anybody's who assess this is
important by saying it's not an
objective fact it's just your opinion
because obviously everything everybody
writes is just their opinion on what is
actually important
there are other methods other
difficulties
um and uh
the uh the contribution as poker's
Genesis of the document of the list of
lethalges is just basically collecting
of the points
um I think this is something that should
not be underestimated I think this is
actually really important uh almost
crucial work to summarize and aggregate
um existing points
um this is these things
um have uh have been pointed out many
other places we talked about that last
time but uh pocusiano here goes a bit
further and says that we uh where
there's been argument about more
important difficulties in in other
places
and this is also something that uh even
hubingham for instance has has argued in
the comments three Lisa's original post
um
made a reply to this saying his list
wasn't complete and are there other
foreseeable difficulties
um and even who Winger comes up with
five uh uh other
um
uh difficulties you could see in the
future
um I won't go through them here
um and Elia so your taski is of course
really
um sees this as important work and I
strongly agree like having an even
better version of the uh list of
lethalsis is uh which is very valuable
um
when I actually go through even who
bring us points a lot of these are
impossibility results like it is
impossible in theory to ensure that an
AI is aligned in this particular way and
I think it's important in this good way
um but uh it's not at all clear to me
that this kind of impossibility results
are going to be all that relevant
so I think definitely stating that these
are more important difficulties is
overstating it
given that this is written by Paul
Cristiano he's probably thinking in
particular of enlisting latent knowledge
and
this has uh 10 similar difficulties I
went through them and uh there is a
substantial overlap in difficulty one to
seven and they are written in more
details
um and um to what extent the overlap is
uh yeah I think I disagree with poker's
Channel about that but um hearing that
they are more important because they are
more relevant to core problems with
realistic alignment strategies and
obviously this is again a matter of a
subjective taste in that uh Paul
Cristiano kind of naturally feels that
his uh personal
um best guess for how to solve alignment
is the most realistic like that's again
almost by definition
um
and like a lot of people are working on
things that are not existing latent
knowledge and I think uh some of the
difficulties like the the part about
regularization for instance I think
expecting that to be a uh uh a major
component of actual alignment is uh uh
quite optimistic in my opinion
somewhat uh weekly states that uh he
would have preferred that uh uh the
listing latent knowledge States all the
problems up front together I think
that's a
whether it's nice to write that or not
uh is somewhat irrelevant I think I
think as I've said before iliakowski
hasn't engaged sufficiently with
illiciting latent knowledge and this is
just this might be just because you
skimped the report and I think it's uh
sufficiently valuable that he should
read it in detail
so the overall take is that Paul in
general uh he obviously had a lot of
agreements with uh eliasa and these are
good considerations backed up by pretty
clear arguments but they are too
confident and that's something he has
said in many places and a big problem is
that elisabeth's reasoning is more a
priority than reasoning about evidence
and the problem with this kind of thing
is that we are unlikely to actually
resolve the disagreements even in if
there was like
um five more posts back and forth
between Elisa and Pocus John we are
unlikely to see any kind of resolution
on on major questions or even things
they can bet on and I think that is true
and I think also it is very very sad
that we we can't actually use evidence
for this
Paul would like Elisa to write his
arguments down in in more details that's
one of his core complaints and my core
answer to that again as I've said is
that
there are in fact links on Amazon where
Elizabeth has written this down in a
surprising amount of detail
it's also stated that people with
aliasis views have often made engagement
hard that's of course a bit sad because
I am close to aliens in this regard and
I hope we haven't made it hard on
purpose but I think it's just that
engagement is really really hard because
it is really really hard to get any kind
of meaningful traction on uh uncovering
these points and predictions is one that
thing that would help us and that we
haven't really been able to get going uh
Paul and eliasa has failed to converge
on any kind of meaningful Bits And even
though
um
poor Cristiano seems to be strongly more
eager and open towards the bets the the
reply from Elia zurichowski is basically
that he his uh World model is not
particularly constrained in what happens
until we have some kind of takeoff
he is mostly uh
um analyzing what happens once we to use
the rocket analogy exit the layer of
atmospheric turbulence
and that I feel is the central media
lethality in AGI ruins the fact that we
have no strong epistemic way to actually
progress on figuring out what is uh the
uh uh
the status of all these claims about AGI
we would we don't really know how to
progress on this we probably cannot
progress on this and that is the
miniature lethality that is ultimately
going to do much
that is all for tonight thank you and
see you next week