255  Where I agree and disagree with Eliezer

hello and welcome to session 255
in the aisafety.com reading group
tonight we'll be discussing the less
wrong post where i agree and disagree
with eliezer by paul cristiano
let me just get the video done paul
cristiano is one of the
perhaps
one of the two most famous
alignment researchers and the head of
the alignment research center
this is a two week two months old uh
post and unless wrong it has the most
upvotes of any post ever so that's quite
an achievement
um and we'll be going through the first
half today which means we'll cover all
the points of agreements and the first
10 points of disagreements
paul christian describes this work as
somewhat of a response to elisa's post
agi ruin a list of lethalities and also
rambling in the same way and not
exhaustive
i think in general making a presentation
and analysis of rambling style post is
kind of hard i didn't do it for elias's
posts
and also
like eliasa is way more rambling than
paul cristiano is in this post
we'll start by going through the
agreements
uh and one thing i should say up front
is elias yatkowski in the comments that
this is a solid contribution and thank
you um
not just for the agreements but for like
the the total uh the total post
now splitting up into agreements and
disagreements
is uh
not obvious how you do that because if
one person says there's 30 probability
of a fast takeoff and the other person
says there's like 50
then is that an agreement or a
disagreement um
the way i think about it is like paul
christiano imagines a median less wrong
poster reader and if that person has
probabilities in between
um idiosa and paul cristiano then it's a
disagreement and if they are on the same
side different from what the median
reader will expect then it's a
it's an agreement
i will also uh note uh
some references
if
paul is referencing something from
elias's post then i'll uh note what the
uh the point like it's point b 38 or
something like that so we can later go
back because uh paul cristiano said that
that he is not making an exhaustive
reply and i want to see what party
doesn't cover
so for the first agreement which is that
there is a good chance that we will see
ai the
empowering humanity and
disempowering is used instead of killing
as um edison would do because he
makes this is easier
i think if you can call an agreement uh
with the definition that it's both more
that both are more uh pessimistic than
the median less wrong reader but there
is quite a difference between the
statements made by elias yutkowski and
the ones made by paul christiano
elias is much
more pessimistic and doesn't really talk
about disempowering as much as just
outright killing
paul cristiano gives his probability of
doom
20 rather than 99.99 and a single layer
his timelines are
probably longer than aliases
15 in 2030 40 in 2040 and around 30
percent probability of a fast takeoff
you may
say something like how can you have a
higher probability of a fast takeoff
than uh then doom that seems uh very
unlikely that there is a
50 probability that a fast takeoff will
be um will go well um i don't think this
is something we should put a lot of
weight on in general when people make
these kind of statements they use some
numbers that change very much and they
try hard not to like
anchor too much on their own previous uh
estimates so we should take this with a
very very large grain of salt
on the timelines at least something like
soon is something that both paul
christiano and elias kowski agrees on
and they also agree that we won't get a
fire alarm or any kind of
thing that will build a consensus until
we basically have dangerous ai or
importance in illicit we
view we will in fact never have a
consensus where paul cristiano is
uh
more optimistic once we have in fact
dangerous ai that we'll be able to
recognize that
we probably won't respond productively
to uh
to ai risk uh even if there was a
consensus probably we would
yeah not be able to rise to the
challenge in any meaningful way we can
mess up even very basic challenges and
in this case
ai safety is a really really difficult
for many reasons so we can certainly
mess this up
so this is not really a point that
eliezer makes in his uh
agi ruin post but it's quite clear that
he holds opinions that are like this or
even
even more pessimistic
the existing state of alignment research
is something that they're both very
pessimistic about they're not really
we're not really
making progress on the key difficulties
because we're focusing more on
what is the the area where we can make
enough progress to like publish a
research paper instead of looking at the
actual key challenges only quite few
people are working on that
eliza yutkowski has the same points but
again in a substantially stronger
statement
we won't we can't recognize progress
even if we came upon it if we got a
billion dollars that would not help us
at all because we can't recognize
progress and it will just uh
drown everything out in noise
one thing that i would really like uh
paul cristiano's view on is like
relatively few researchers how many is
that and who is that because i think
that would be something that would be
really valuable to have some idea about
who
are working on things that input
cristiano's view are likely to to
contribute
the last derail is a uh yeah is a uh
what's it called a twitter hashtag or
something that is being used by elias
yorkowski to describe uh when there is
like social pressure to talk about ai
risk and turn it into something that is
about like
standard political things um
and paul cristiano uh says it's probably
a bit hyperbolic but on point here is a
uh
sorry um a um
paul by
elizabeth on twitter that shows that
yeah
more people think that people clippers
are bad but a lot of people think that
bias is a
bigger problem
excuse me for a moment
in the same way a lot of people focus on
smaller risks that are
more realistic um and not really
existential in the way that we care
about
and that's probably something that's not
going to change until um
very late in the game if at all
um i think elizakowski would agree but
in fact
he i can't really find anything that
elijkowski has written about what the
man on the street thinks about ai
alignment because i think elias
yorkowski just
considered that totally totally
irrelevant
and uh
what what uh
what uh the standard people
um who who are not experts and don't
have any particular qualification um i
think he is basically given up given up
on trying to convince this kind of
people
and ai takeover could be very aggro
abrupt
perhaps uh
something akin to a coupe
and that's another agreement between the
two
poor christian thinks that killer robots
might be part of it and
that is something that could happen
indeed very abrupt and the progress from
um
ai having uh
being involved in something like
research until controlling killer robots
may be very brief
sorry um and yeah this is also um a bit
that elias yudkovsky has talked about
but not very much
um
one of the things paul christiano agrees
with uh elias koski is that
most people don't see a uh the
singularity as something that can come
very very very quickly after we have
automated research by um by ai and
that's something that paul christiano
thinks could take uh perhaps uh even
months so that's a some
substantially shorter timelines
it's not precisely what eliza kowski
talks about because uh paul cristiano
seems to have a model where
first there is
uh like some kind of recursive
self-improvement and uh
or some improvement in ai capability and
then this kind of diffuses throughout
society and ai has a large impact
whatever that may be and then after that
we get singularity and some kind of
takeover whereas elias wikowski doesn't
really in his world model see a middle
point where ai has enough
[Music]
effect on the world to be described as a
large impact it's mostly going from
basically
no impact
to
ai's control of everything
the way we normally solve
technological uh problems is by
trial and error in the scientific method
and unfortunately
for alignment this is not something that
happens by default because it's possible
to work on capability without working on
alignment and it's just possible that we
won't in fact solve uh
solve alignment because we're not forced
to it we can just get by with standard
uh gradient descent methods until we
have extremely powerful agi and then it
might be too late
um so we
can not expect to get
the strong kind of empirical feedback
loops than we get in just about all
other kinds of research
again this is not something that elias
you'd ask you explicitly states in his
agi rune but i think he would agree with
this
it has sometimes been positive that
ai
improvement would at some point hit a
kind of intelligent ceiling around or
slightly above the human level uh paul
christiano sees no particular argument
why that should happen um in fact
it seems likely to his in his model that
will relatively quickly go from
something that is substantially below
the human level to something that is
substantially above the human level
um
again this is also something
elias agrees with
most of the these agreements that we are
finding are in fact uh
in some of elias koski's preliminaries
so we are not really even though the
agree it is agreements it's uh not on
any particular
reasons some of the strong reasons elias
yutkowski presents for why uh agi ruin
is likely
if we had powerful ai somewhere then we
it would be able to um
uh take over the world
that's something they both agree on um
and i won't go into more details about
that coordination failure is really
likely in that
we it doesn't look like we will be able
to solve that policy problem um
this in general geopolitical
coordination is hard and here we have
some
stakes that are really ambiguous in that
almost
everyone agrees that there is no strong
ai risk it's uh basically rationalists
and no one else
um
and there is a lot greater
pressure to defect in the sense that if
the others are not building agi and
you're building agi you can potentially
take over the world
um so that's
something that paul cristiano agrees
with eliasa is likely to fail this
policy problem to be solved
humans won't
necessarily rise to the challenge um
and covet has been a negative update for
paul cristiano um
i think um
elias etaus is substantially more
negative about this he is nowhere near
50 50 on this
so i don't really think this can be
called an agreement um there is
ilius yutkowski is
seems to be strongly
on to the extent that we can rise to it
to this challenge
on the more technical side is there a
utility function that really
captures what human wants and what human
values are um poker channel says no um
he thinks we should instead do something
like reinforcement learning where the
reward function gets smarter in parallel
with the agent being trained what
precisely it means that a function gets
smarter is of course
difficult to um
to state in a concise way but that's
one way of summarizing his
his
agenda we
if we try to
train an ai to
have a
reward function that is
um like what kind of object we uh we're
trying to get then um
we will we are unlikely to get the nei
that is actually truly motivated to to
do this because if it's smart enough it
will realize that it should do what we
want it to do
even though it's not that's not what
it's actually motivated to do because
that's how it preserves influence like
deceptive alignment
i
think i would actually disagree with
these two at this point because right
now i would expect us to see to see much
more signs of deception if this was
trivially true um in fact uh paul
christiano
a bit later than this says that he
expected the deceptive alignment
requires superintelligence and i
strongly disagree with this i believe
that actually hey at the human level
it's quite possible to figure out that
you should be deceptively
aligned
naive assisting is my uh all the
headings are mine and and the the text
down here is mostly quotes um so it's
more uh likely that the human will learn
what is the environment that's a lot
easier than trying to learn something
really complex
philosophically fraud like trying to
help humans uh and if you try to get
data from humans some kind of feedback
then um
uh trying to um
to learn this is difficult because there
will be errors for instance so uh it's
much more likely that the ai will
actually learn the uh the processes that
are generating the reward signals but
it's inside our brain and update fully
on that on something like that
and that's
probably what we're going to get if we
don't do anything
if we just try to do
simple reward learning from humans
um the dying with dignity framing that
we shouldn't just assume that there's
hope but uh try to practice try to try
to maximize the log arts of success is
something that
poor christian agrees with and this is
basically the dying with dignity
strategy article from
miri
the current plan for aligning ai
requires
a lot of iteration and modification
uh the state of affairs is that if
alignment turns out to be a big problem
then we'll have to learn about it and
iteratively improve our approach
i'm a bit confused when paul cristiano
uses the word state of affairs i think
he means plans
but i'm quite unsure about whether
that's actually
what he is pointing towards elijkowski
of course is way more negative about
this and believes that
we'll
not just learn a lot about it and
iteratively improve will
fail to learn and not improve and die
finally
in other research fields you usually
choose a research project that is just
barely out of reach so you can actually
improve the state of the art and
alignment is not chosen for this reason
there is no one to ensure that the
problem is fair and it's possible to
solve the problem at all it's possible
that that will fail because the problem
is just too hard
and that's the point that elias
whitekask also makes
finally we're getting to the
disagreements of which we'll take the
first ten
um
pakistanis says that these are mostly
stated without arguments so i have to
like dig deep uh to try to figure out
what uh arguments could be made for this
so i'm not
necessarily following paul christiano
that closely in this part
the first is
a complaint that elias equivocates
between saying on one hand you have to
get alignment right on the first try and
on the second that you can't learn
anything about alignment uh before that
critical first uh try and these are two
different statements and pocus jano
thinks that the first one is
correct and the second is incorrect
so
does elias cask in fact equal like this
i looked through this post and to try to
see if i could find ways and i was
unable to find any places
where eliezer caskey makes such an equal
vocation
he does state the first claim very
explicitly and later he actually states
something that kind of sounds like the
negation of this
here is the exact quote
section
b is in general about what can we learn
and what can't we learn about alignment
before the critical first try in
particular uh
point b 13 seems to be about uh the kind
of things that you can in fact not learn
so it seems strongly to me that
elizakowski is very explicit about both
of these points and there is no
equivocation
but it's also possible that paul
christiano is thinking about a different
part
okay so what can we actually learn from
alignment before the critical first
try well we can
get we can use the traditional research
and development methodology
to try to create toy models of alignment
failures
we have standards for interprobability
that it seems like we should be able i
think there was someone posted like a
research roadmap a tech tree or
something like that and we have
theoretical questions we can't answer
and that seems like something we can in
fact learn from now
i don't think that
saying that there are theoretical
questions we can't yet answer i don't
think that is something that we can talk
about as feedback from what works
i think
we are unlikely to get a resolution to
the uh deeper theoretical questions from
practical experiments
and the problem that uh here i think is
something that they would actually agree
strongly on is that solving a scientific
problem without being able to learn from
experiments is really really hard
and here
this consideration makes the
institutional problem vastly hard but
does not have such a large effect on the
scientific problem
i'm a bit unsure about what precisely
paul custino means by this quote
i think a wait um
elysiutkowski
talks about uh
the two challenges with uh with this
that's the fact that there is a deadline
and we only get one try and even if you
in some way were able to relax this it
would still be a really really deadly
problem
so one way you could think about this
imagine that the institutional problem
is solved but the scientific problem is
not solved that would be like we have
we are coordinated enough that we have
all the time in the world to solve it
but we only get one attempt so that is
something that is
substantially harder
substantially easier than the actual
real problem where the institutional
problem is not solved
and the other way around um would be
that we have unlimited attempts uh but
we do have a sharp deadline so we can
try to build an aligned a ai and um but
at some point the institutional
coordination will fail and some um
someone else will just build an
unaligned atr so we do have a deadline
so um i think that is what pulse channel
is minch is pointing towards but i'm not
sure
the topic of nanotechnology comes up
and um
paul cristiano believes that once we
have nanotech we'll surely have other
technologies that can kill humans in
more boring ways
i think this is a empirical question of
course it will eventually be revealed
but it depends like what is actually the
easiest way to kill humans is that by
building uh large robots or small robots
i must admit i haven't read the
literature i should
read eric drexler i guess um
ilya sirkowski claims that he has in
fact read what eric draxler has written
about nanotechnology strong name and
technology
and
i
like if paul christiano believes that
uh big robots are easier than small
robots that there is um uh
and
another easier but more boring way to
kill humans um then he should state like
what is the actual problem with the
nanotechnological pathway that elias
yorkowski is is saying
another thing that is likely to happen
before we get nanotechnology is that the
state of human research and development
will be
radically advanced by ai
it's a possibility but it's also a
possibility that
ai research and development would just
bypass humans and not in fact
strongly contribute to advancing human
research and development
paul christiano calls uh
uh
elias yokowski's scenarios uh cinematic
and they don't seem to like hold
together
um i feel
calling them cinematic sounds like
a
strange almost like an argument from
status or something like that
and
is there a more realistic picture of ai
development under the surface
that is in fact something that a lot of
people have speculated on in particular
in the comments like when elias kowski
is talking about nanotechnology and
botulism and diamondoid
bacteria like
nanomachines is that something he really
means or does he mean something else a
lot of people are thinking that he must
mean something else
um i have talked very little with ilya's
costume but i have in fact talked with
him and he seems
very very much like a person who
strongly says what he believes so i
would bet good money than when elias
erikowski says nanotechnology he is in
fact not using that as some kind of
simile or metaphor for another
technology when he says nanotechnology
he means nanotechnology
is it possible that the ai that is
capable of killing us will before that
not making any major technological
contributions to society at large
that's a question that
i think depends mostly on takeoff speeds
and that's of course an area where
elizabeth caskey and paul cristiano are
quite of uh different opinions
um for christian believes that we are
trying ai to do things that look uh
impressive um
so um if there's an ai that doesn't do
impressive things then we'll just not uh
not find that ai because we're just
using uh stochastic gradient descent to
find the impressive ais
that means in general it will be
the impressiveness will increase in a
gradual fashion at every point there'll
have been before that a slightly less
impressive ai
i think the depending on how you
operationalize impressive that may be
true but i would expect the things that
we're trying to make the ai do in
general are things that may be
impressive but are not dangerous in the
sense that ai that nanotechnology is
dangerous and making uh beautiful
drawings or something like that is not
dangerous um and the hope the fear is
that if we get a general intelligence it
may be able to do some really really
impressive
uh harmless things and then the very
next thing it does is something that is
at
roughly the same level of impressiveness
but just in a different domain and that
turns out to be really dangerous
some of the the framing the words that
will christianity uses about uh
illiterate casket to describe his
scenarios
are somewhat derisively stated like he
imagines scenarios of ais lying in wait
to cut to cause trouble later that's
almost certainly not how eliza yokowski
would describe his own
scenarios
it seems uncharitable but on the other
hand elias yokowski is also harsh when
he writes at least according to like
danish moss people are
not as diplomatic as would
be becoming of them
so i can't really uh fall for cristiano
for for pushing back on alias in this
way
one thing that i do wish to point out
here is in in this uh argument about
whether we'll have ai's that does
something slightly less impressive uh
civilly looking um i think that uh
um
paul christianus seems strongly
overconfident by using words such as
definitely i think there is a lot of
reasons to believe that um that it may
go in in a different way definitely
seems like uh something that when you
need really strong arguments to use this
kind of word
recursive self-improvement how will that
look in practice procrastinator expects
that this will basically look like
well
like humans doing research and
development um
i suppose that's possible and i would
even uh say that there is a high
probability of this but you could easily
imagine ai research and development be
being a lot less scrutable i could
imagine that you asked gbt3
how can i improve you for uh how
can you give a uh an architecture than
that is better and then it makes i don't
know a transformer where the diagram is
reversed or something like that and says
try this this will be better and then we
try it and it is it does in fact
uh perform better and we just don't know
why uh i think that is i i could see
that happening um
and a way that
human research is
one thing that really makes it screwable
is that there are many humans working on
an ai system and they are talking with
each other
exchanging into issues and ideas and if
they are replaced or uh by by a single
unitary ai
then all the the social um artifacts
like people writing to each other about
their intuitions
is lost and i think that could be a
substantial difference
and finally it's possible that um ai
will just be able to do things in a
different way than humans i think we've
seen a lot of examples of ais working in
human domains where it seems like they
are working in a different ways than us
will ai smart enough to improve
ourselves be a crucial threshold paul
cristiano thinks not i think it could be
in fact
it depends whether it's a crucial
threshold depends on whether the speed
will change to a large extent in my book
of course what does crucial mean
to my mind if it changes the speed
meaningfully then that is a crucial
threshold
how gradual would this be
procrastinator thinks that this will be
a very gradual process
i think that
when i ca conceptualize um
what will a super intelligence be able
to do or an ai then uh bostrom's uh
framing of speed super intelligence
quantity super intelligence and quality
super intelligence it seems to me that
working on something like capabilities
is something that is clearly possible to
do by a speed super intelligence and a
quantity super intelligence and even at
a lower level where it's not actually
super intelligent but just at a roughly
human level that seems like where speed
and quantity trivial would be able to uh
to contribute to capabilities because
there is a nice feedback loop right if
the code is actually running faster
then it's better right
and
having a lot of ais that are working on
this
would be
should be able to contribute much
more to capability than to other things
in disagreement four and seven there is
an underlying pattern that i have called
pulse sequence of events
this is my interpretation of how paul
christiana thinks the world is going to
look like
so take this with a big grain of salt
please
the first thing that's going to happen
is that ai contributes to
research in some meaningful way
the second thing that will happen is
that ai will be able to uh in particular
accelerate alignment research
then we'll get ai that improves
capability maybe doubling the pace
and at some point after that we'll get
dangerous ai so that's kind of like
um
perhaps perhaps this is a takeoff split
into uh these
these parts perhaps it's uh
the take-off hasn't really started at
this point it's a bit unclear
is this a likely sequence of events
i think
i would
i would be skeptical because i feel that
ai
improving capability is a lot easier
than ai improving alignment research
alignment research has turned out to be
really really hard and ai capability has
turned out to be surprisingly easy
not easy but easier than we would have
hoped so in the two progress bar
models
i think that that ai
is likely to push more on the capability
progress bar
i'd like to be wrong of course i would
like paul kuchan to go into more details
about how ai could help alignment
research of course he has given like a
number of suggestions for how this could
be made um but most of paul cosiano's
suggestion seems to be like this is a
strategy for how we could do it rather
than i think it is likely that um
this is the method that will be used
like for example existing latent
knowledge is a like a great idea um but
i haven't heard any strong arguments
from paul cristiano saying
in the actual future he expects that
eliciting latent knowledge like methods
are the ones that will actually be used
in the future because that's a much much
stronger claim than whether listing
latent knowledge can be
can be made to work
um
so uh yeah there's some agreement uh one
thing i would point out here is the
self-improvement ai may uh
be um
be hindered by the fact that it can't
solve the alignment problem that seems
like in the old days where we just had a
utility function then could just copy
that to a successive function but that's
not something we might we
might not see that with a um
with the with large language models for
instance or something like that
um so how uh to what extent will we see
these two things alignment research and
capability research growing at the same
page pace one thing we could at least
say is that if there is like a single ai
that is improving um in that case then
as it grows in capabilities then um the
alignment will all will always lack the
improvement in in capabilities because
it needs to improve in capabilities
before it can contribute to alignment so
that in that extent it might
self-improve to the point where it can
solve the alignment problem and then you
try to like use that solution and then
it doesn't work because it won't allow
you to actually change it to be
alignable to be correctable or something
like that
pivotal acts
they seem misguided and unrealistic in
paul christiano's view
i'm not sure i like the word unrealistic
i think
misguided and unlikely to work is better
to
to use um progression thinks that we can
uh
uh
reduce the period of risk
and do something kind of like a pivotal
act in in like smaller parts
um kind of what uh
poker channel doesn't use the word weak
pivotal act but that is um what
elizabeth
states one of them is by
advancing alignment research
i think definitely if you get good
enough
improvements in alignment research that
would count as a pivotal act um
but in that of course um
uh sidesteps the the discussion about
whether it is good or bad whether it's a
pivotal act and just
to say that it's a good thing that we
should uh pursue to have aligned ai to
alignment research
another way that we could get something
that is kind of like a part of a pivotal
act
is by having something that could
demonstrate that um unaligned ai is
dangerous that's something that i have
previously talked very positively about
i think in general unfortunately we will
have to really convince a lot of people
because there is a lot of people who
really want to build unaligned ai and if
we want to
strongly convince 100 of the population
of earth that requires really really
strong
levels of persuasion and i think
that is dangerous in of itself
if you want to
strongly persuade everyone so there is
not a single person in the world who
would like to deploy an unaligned uh ai
um
then i think that is
that is basically a pivotal act
um so elias yorkowski's main complaint
that these
big pivotal acts are unlikely to work is
something that i feel paul christiano
really isn't answering at this point
the third way that
unaligned ai can uh that a
relatively aligned ai can stop unaligned
ai is by consuming the free energy of
society which is the thing like it
originally comes from an analogy with
the free the uh efficient market in an
efficient market there is no free energy
and that's the thing that an online ai
could
use to uh to grow very rapidly
um
things like uh jobs that an ai could do
would be an example if for some reason
there's a lot of jobs that and
that an ai could do but we still have
humans doing them and then suddenly an
unaligned ai comes along and just can do
all this and
take over the jobs from all the humans
and just use that for disempowering or
gaining economic wealth
that's an example of free energy being
being available for an unaligned ai
unsecured computer systems is another
example um
we could if we have an aligned ai that
um
that checks every uh
uh
computer system in the world for
for vulnerabilities and then hacks them
and patches them immediately um that
would be an example of an ai that is um
performing
like a part of a pivotal act a weak
pivotal act um and the question then is
whether that is dangerous or not i would
expect that to be quite dangerous
another example is
better
killer robots or something like that
that's also something that if we already
have the best killer robots that can be
made then a misaligned ai can't
obviously make better killer robots
and in general managing the consequences
of powerful ais
steve burns have some uh
some rather uh tries to uh in the
comments go into some details about what
kind of free energy there is and points
out that
control of nuclear weapons is something
that right now is in the hands of humans
and if there we have a world where there
are ais that are more intelligent than
humans then our two options is to hold
on to it or to give control of the
nuclear weapons to the ais that we hope
are aligned and
if we don't give control of all our
nuclear weapons to ai then there is some
free energy so consuming the free energy
in this way seems really hot and also
like
all the work that can be done by ai
should be done by ai and maximally
effective killer robots and things like
that seems like a world that is really
really really bleak in particular if the
ais that we are giving uh control over
the nuclear weapons and all the jobs and
all killer robots um if we are not 100
sure they are perfectly aligned then
that seems really really dangerous um so
in fact consuming all the free energy to
my mind seems like something that is
just about as dangerous as a pivotal act
but doesn't actually solve the problem
so i think um having ai consume all the
free energy is
just worse in every way than a pivotal
act
on the risks of personal acts uh paul
cristiano believes that there is an
overwhelming risk of destroying the
world and that is in fact also what
eliza utkowski
says he says that
in his post on agi ruin he strongly
expects that uh we we will not do a
pivotal act because in practice we
pivotal acts are just too hard and the
thing that will actually happen is we'll
try to look for a good pivotal act and
try to find some kind of miracle and we
won't find this miracle and then we'll
die so the fact that pilsel act runs an
overwhelming risk of destroying the
world is something they might really
agree on
paul christiano's uh preferred
alternative is traditional policy
influence
um
this i believe is what elias utkowski
would call a weak pivotal act and again
paul christian really ought to try to
defend this
this act of traditional policy influence
he says that rejecting this requires
extreme confidence about details of the
policy situation
i think this is trivially wrong in fact
to be able to say um that a traditional
policy influence will not be sufficient
only requires that you know of like a
few defeaters and when you know like
there's no way you could convince the
chinese government to go along with this
then you don't actually need to know a
lot of details about the policy
situation
to be able to predict that this will in
fact not work
poker channel admits that this is
something that would be an unusual
success in historical um
terms but again much better than the
pivotal acts
i think that there might in fact be a
very different uh be a different
disagreement with paul christiano and
elias yatkowski than what paul cristiano
is pointing to here
if we assume the case that elias
utkowski believes that a pivotal act has
100 chance of working and
traditional policy has one in ten
thousand and paul cristiano also agrees
one hundred percent that perishable acts
are 99 likely to fail
but it's much more optimistic about
traditional policy in this case they
shouldn't debate the uh the pivotal acts
what they actually should debate is
traditional policy to what extent is
that likely to work um and i think that
is
an interesting case and i think that is
to a large extent uh if paul cosgiano is
very very optimistic about traditional
policy influence then i would really
like to hear why he's so optimistic
about that
finally in his criticism of perishable
acts
he claims that elias underestimates how
difficult it would be for an ai
developer to covertly take over the
world
at this point we should of course uh say
that it's not a an ai developer alone
it's an ai developer plus a moderately
aligned narrow super intelligence
the second thing elias kowski
underestimates is how strongly and
effective governments would respond to
the possibility of
an ai developer taking over the world
and finally how toxic
this kind of plan is
so will governments be able to
effectively react to the possibility
that
say deepmind will create a super
intelligence that will take over the
world uh well if they were doing that
then they might do that now because the
possibility is in fact there right now
and as far as i can tell governments are
well they're certainly not strongly and
effectively responding to this as far as
i can tell governments are totally
totally oblivious to this possibility
so we're looking at a complete
intelligence failure right now it is
possible it will change but it's also
very possible that it will in fact not
change at all
elizovsky is
accused by paul christiano of being
sloppy in some of the topics that he's
dealing with the first is on convergent
incentives
so i looked up here is an article
article about
instrumental convergence
um where elias erkowski goes into
substantial um details on conversion
incentives the deep nature of
consequentialism is another area where
paul christiana thinks elias ritkowski
is sloppy
i found the the article here where elias
goes in details and finally he's sloppy
on generalizing
outside of a training distribution
i couldn't really find anything formal
that ilia streetcars has written on this
topic
at least by looking for 10 minutes it's
possible you could ask elias
if he has in fact written any
rigorous thing about this
but to my mind this is an isolated
demand for rika because these two
posts up here are in fact
quite formal and quite rigorous they are
not perfectly
rigorous and they don't er they're not
meant to be perfectly rigorous but they
are at a rather high quality level uh at
least as far as i can tell
it's a bit difficult to tell for certain
uh but i think um
almost all arguments that are used
within ai alignment are less rigorous
and less formal
and less
and more sloppy than what elijkowski is
presenting
and in fact uh uh even though elias's uh
reasoning may be sloppy pakistan
basically agrees that they are roughly
right but it's just that we can't really
um
have a strong probability that they're
true when we have a like a chain of
reasoning
um
in particular on generalization
procedure claims that it is subtle and
we can learn a lot about these kind of
points in advance but we just don't do
that right now
to my mind we can learn about
generalizing outside of the training
distribution at a below human level
right now but it doesn't necessarily
tell us a lot about how a superhuman ai
can generalize um
in particular i don't think that we can
when when
books just say we can learn a lot
about it i think that's very unlikely we
can speculate and we can like get some
ideas about how generalization outside
of the training distribution looks at a
low level but we can't really truly
learn and get
like a
strong knowledge about this uh in any
way
paul christiano believes that uh
the result of this stopping
stopping reasoning by eliasa is that
he's wildly overconfident i think i
would
if i was paul cristiano i would go
through some of these articles here
um by elias yutkowski and try to find an
explicit error
because if in fact paul christiano is
right that
elisa yorkowski has written some sloppy
articles
then there must be some mistakes right
uh if this is a sloppy article then
we there would be some kind of mistake
that could prove to us that elias
yutkowski is doing sloppy reasoning to
me it doesn't look sloppy but i'm no
expert i'm just unable to really
strongly evaluate this claim
so how much will
um how much will change when we go from
a roughly human level to a super
intelligent level um
pakistan says that the training
strategies don't need to handle
much of a shift between from a low level
to a high level of intelligence
uh i think that is missing the point
slightly in that it's not really
training strategies that need to um to
change or
when the distribution shifts the problem
is the alignment strategy needs to
change
and again
aliaser is claimed to be making a
confident
statement about how ai is going to
change and i don't think really it's the
nature of ai that's going to change
elijkowski states this as the air there
will be
a new external options opening and
even more internal choices and modes are
going to open as the ai becomes more
capable uh and that's different from
saying that the nature of ai is changing
pocket channel
says that earlier koski is probably
wrong and clearly overconfident
the way i think about this is that
at there is like this is of course
really simplified like there is a
current ai level and then there is the
level that cern is on and then there's
the level that paul christiana is on and
then there's a super intelligent level
and it seems clear to me that going from
science level to uh paul christiano's
level is a
qualitative difference paul christiano
can clearly think of a lot of strategies
and consideration in this rather that i
can't think of
and in the same way we should expect
that an ai that goes from the current
level to a super intelligent level it
must pass through the cern level and the
politician level to get to the
superintelligence level and it seems
like if there is a big distributional
shift between these two um
then it seems to me very likely that
there is a very large distributional
shift going from a superhuman level to a
super intelligent level
finally
fume the intelligence explosive thesis
um uh
elise witkowski uh strongly expects that
we will have some kind of foom
uh paul castiano disagrees with this um
this is something we've talked about
before and
um
he claims elias's position is probably
wrong and clearly overconfident
i think in fact elisa ykowski has not
made a really really strong argument
for the fume and he
is not in fact claiming a really really
high probability of
of an explosive takeoff um i think um
like he's not committing to any kind of
um number so this is just me uh the vibe
i get from elias so you'd cast is
something like eighty percent maybe
ninety percent for a fast takeoff and if
paul costano is thirty percent um
then um clearly overconfident and uh it
seems like a um
a very strong statement if the the
differences between their um
probabilities are not larger than this
and of course the issue of foom is
something they've talked a lot about and
one place they didn't talk very much
about foom is in
elisa yudkovsky's post agi ruin it has
very little about foom
there is one part that perhaps
relies a little on foom but but most of
it basically does not the agi foom the
agi
ruin post is about other reasons why we
are probably doomed
so i'd like to end with this the last
comment on paul cristiano's
threat by elsie which is just i hope
you're right i hope uh paul christiano
is right and that is has the most
unbalanced moderate uh
moderation here i've ever seen with uh
52 people believing that this is a true
statement and very few people believing
that this is actually something that
contributes
so i too hope that
panko's channel is right and elias
yudkovsky is wrong
we'll see more about that next time