275. Why I am not as much of a doomer as some people

hello and welcome to session 275 in the
AI safety.com reading group tonight
we'll be discussing the post why I am
not as much of a tumor as some people by
Scott Alexander
Scott Alexander is a long time member of
the rationalist community and writes on
the Block astral codex 10 which is
probably the most widely read block on
in the rationalist community
um and I am also a part of this and
specifically I host a astral cortex 10
meetups and I consider myself to be a
friend of Scott so this is obviously not
an unbiased uh answer
the post we'll be discussing is of
course posted on escrow codex 10 a
couple of months ago and uh the way I
think I'll go through this is by only
focusing on the disagreements so you can
read the title as uh why Scott is not as
much of a tumor as CERN is
so the first thing
um uh Scott points out is that this is
in fact a a sub debate within a larger
context there is a primary debate
between people who say that there is no
risk in Ai and people who say that there
is a significant risk and uh this is
what uh uh Scott Alexander refers to as
the most important debate is there in
fact a uh is AI risk a real thing and
obviously if you go to the comments and
even in this or in many other then
you'll find a number of people uh
arguing that the risk is zero or uh so
that is like the real debate
I think
um this is perhaps a very uh
oversimplified view on the debate there
is a number of people who say that the
risk is zero in like an unsophisticated
way there are people who say it's
precisely zero because of some
impossibility proof there's some people
who say it's a negative risk that some
people that say that short-term risks
are are climate change or something else
is more important so there is in fact a
a an entire debate structure but we are
going into a
focusing on this area and the um the
disagreements in there because even in
the community of people who are
concerned about AI safety there are
people who have very different opinions
at one end of the spectrum we have
um uh Scott Aaronson who gives a risk of
two percent which is of course uh
substantially On The Low End
um probably still at the level where it
doesn't really make sense to work on
other things
um but but not much higher than that
the two percent is quoted by Scott and I
will point out that
um there is in fact Scott Aronson
um he um
uh qualifies this by saying that this is
the risk for some uh for some uh direct
continuation of the current Technologies
um like Duty and
um and like a lot of people believe that
gbg in is not really on the path to AGI
so the actual risk of existential
catastrophe could be substantially
larger
well mccaskill is another example of
someone with a relatively low estimate
three percent I looked into his um his
quotes and he says three percent within
this Century but that's to a large
extent carried by the fact that he
believes that we will not have AGI this
century
is quoted as saying 10 to 20
um and that's of course also true but if
you look a bit deeper into his beliefs
that's mostly because he believes that
there are existential risks that are not
Extinction risks and the um uh the big X
risk is uh here he puts it at 46 which
has way too many significant digits for
my sake for my taste
researchers have five to ten percent
Holden kanovsky have 50 which is when
he's asking so much uh related question
um Elias hirkowski is quoted at 90 plus
percent and I think it's much higher
than 90
um Scott is uncertain about other people
with high percentages I would
um point to people like Connolly guerns
three most which as people who probably
have uh much more than 50 probability of
two
so what is uh Scott's estimate for uh
how
the probability of Doom well uh he is
not really he is if he's forced to do it
like this is a very rationalist thing
that uh people prefer not to be too
explicit about this uh because uh then
you get all kind of anchoring uh effects
but he's willing to put our number
that's just 33 percent
um and he says you go back and forth
more than you can really justify and I
think in fact you can justify to go very
much back and forth
um
both like if I'm at 90 then the
difference between 90 and 33 is roughly
the same as between 90 certainty and 99
like the the amount of uh evidence the
amount of decibel of evidence we would
need to move between 99 certainty and 90
is roughly the same as going from 90 to
33 roughly I think it's slightly more
um
yeah uh so so that's one thing where uh
uh we would expect these kind of uh uh
estimates to be unstable there is a
further argument for instability and
that is the logistics success curve uh
uh uh it's a concept that is used to
argue that
um uh the the possible probabilities are
kind of compressed so
um it's it's possible to uh if if you
have some
um
uh uh probabilities that are around 50
then probably they are extremely
unstable
that's one implication of it
and finally
um here he phrases his estimate as that
we are more likely to survive at least
for a little while and to me that's just
Extinction with extra steps really
surviving for a little while that's the
same as an X risk
so uh let's compare the classic AI risk
argument with uh which was of course
written much before we had large
language models with what we currently
have with something like gb4
first there is the requirement that the
AI is super intelligent which we
obviously do not have then the idea that
uh the classical idea of an AI with with
some kind of monumentical goal
um and this isn't really what we have
um Scott characterizes as tpt4 have no
internal goals just heuristics and
prompt response pairs
I think uh characterizing it as prompt
response pairs isn't really fair I think
it is quite possible to argue that qt4
is in fact
nothing but media optimizers really
um so it does in fact have plenty of
internal goals it's just that it's not
really coherent in this because there
are so many of them
um the the the previous idea was that
the AI was unaligned in the classic
argument and right now we are seeing um
some kind of alignment in gbt4 in that
we can make it work for us
and I realized that this is how many
people have come to use the word
alignment and this is kind of my hobby
horse to reject this because I feel that
um
getting
um
getting someone to work with you really
doesn't mean that you are aligned with
them in any meaningful sense to break
Godwin's law uh you could give the
example that Hitler managed to get work
from Jews in all switch but Hitler and
the Jews in all switch were very much
not aligned so I think alignment should
mean something deeper than that
and of course in the classic arguments
the AI is capable of escaping boxes and
build super weapons and we don't see
currently as doing that and of course I
agree with that so in general I agree we
don't have something like this uh the
classical scenario and obviously that
would also kind of imply that we are
dead
so uh let's talk about the intermedium
value theorem this is the theorem that
says if you have a continuous function
uh from A to B then at all
interior points like C here then the
function will be somewhere in between
here and here and in particular the
Converse that uh
um that if you have a value here then
there will be a function uh uh
I can't even see the intermediate value
theorem sorry uh the basic idea is that
if you have time uh among along this
axis and capability among this axis then
uh if this level is super intelligence
and this level is where we are at now
then there will be a number of
intermediate AIS
with this level of capability obviously
this is metaphorically because AI
training runs are not continuous
um
but this is the basic gist of the
statement that between the uh that we
have now and the air that's capable of
killing the world will have lots of
generations of intermediate AIS
um I think this is perhaps somewhat
overconfident uh it may not be a lot but
I mean you have to if you have to hedge
all these kind of statements you'll
never get anywhere
and Scott is uh claiming that we will be
able to get successful alignment work
out of these generations of intermediate
AIS
uh I think that is substantially less uh
certain it depends on how long time we
have with them and how long time the
alignment researchers have with them
that may not at all be the same and also
of course the last AI probably will
um the the world killer will probably do
so during training meaning that we'll
obviously have one less generation than
um uh
than the number of generations between
now and the world killer
uh Scott says that maybe we will be able
to get intermediate AIS to contribute to
uh solving the alignment problem in some
sense without having to put goals into
the AI and that would probably be safer
and it would be safer but the problem
here is that if we're trying to do
something really difficult like solving
alignment then doing that without any
kind of goals that just puts an extra
constraint on our work that makes it
much less likely that the as can
contribute in general you can really
contribute very much if you don't try
further hopes maybe these intermediate
AIS can contribute within the training
distribution and obviously there are
some tasks they can do it within the
training distribution but like the the
real core task of solving the alignment
problem is not in the training
distribution and that is in fact a
substantial part of the problem
and finally perhaps we can just Foster
the AI because we are stronger and that
is in fact I think we can do and that we
are probably currently doing but I
noticed the inherent tension here in
between we want the AIP to be smart as
smart as possible so you can solve
alignment but not smart enough to be
dangerous and that seems like a really
really unstable situation to me
what level of Genius are required Scott
says that the in order to invent super
weapons that that's that does in fact
seem really difficult to like solve
nanotechnology or something like that on
your own and possibly in secrecy
um
great Geniuses like Einstein and for
Neumann were not capable of doing that
um well they were perhaps not trying but
they possibly even if Einstein really
had tried really hard he wouldn't have
been able to uh uh make a nuclear weapon
himself
um so the idea is that these
intermediate AIS will include some that
are as smart as as these human Geniuses
and perhaps even smart smarter than that
and they will still not be well killing
AIS
uh so it's an interesting question what
is the highest level of intelligence
that is still passively safe
um what is how smart can you build an AI
um that won't kill us
um and that's an interesting question
but I would like to point out that
um there may be other advantages there
may be uh like they can duplicate itself
it may be able to share coordination
between all instances of it it may have
a number of different advantages and
um our economy will probably be really
AI dependent uh much before the AI is
far beyond human Geniuses like there
will be some affordances to the AI that
will happen much much earlier and of
course we don't know what the trick the
human Tech tree looks further ahead it
may be that there are dangerous
Technologies just outside of our grasp
in particular Scott has a quote here
with millions of AIS each as smart as
Einstein working for centuries as of the
subjective time as probably the the most
dangerous thing that still won't kill us
and I think that will definitely kill us
if we have moons of AIS that are as
smart as Einstein working for centuries
of subjective time then that seems
really really unsafe to me and I have no
idea how we could
um expect that
um if they are in fact unaligned then
sure we could hope that they would solve
the alignment problem for us but if they
are this powerful then they could in
fact probably also almost certainly also
kill us
so how can we use these intermediate AIS
um there's a saying that we're using uh
we're fighting Godzilla which is the the
underlying super intelligence by
building a slightly more aligned
slightly less super super intelligence
like a Michigan Godzilla to fight
against the actual Godzilla
um and how can we use this not quite
super intelligence uh and not perfectly
aligned well we can try to ask the AI to
just directly solve the alignment
problem that is in fact
um the old classic idea in AI safety a
lot of people used to think this is
possible a lot of people still think
this is possible
um I am much less optimistic one of the
main ways I've updated over the past
seven years is that the alignment
problem is really really difficult I
believe that we
um
seven years ago it looked like the
problem was hard but not that hard and
it looks even harder now because we've
become aware of a number of extra
obstacles
um
and I think in particular one of the
things that hurt us a lot is that uh
alignment is a lot less less measurable
than capability research so it's if we
build an AI That's capable of
contributing to solving the alignment
problem it will probably be more than
capable of doing capability research
another way intermediate AIS could help
us is that if we if they fail in some
interesting way that could teach you
something about alignment
um and that is a thing that could happen
but more likely
um well there are several ways it could
fail one of them is that they could fail
in a way that actually kills us
um and in that way in for that reason we
won't be able to learn from it or it may
fail in some different ways from which
we can draw some kind of false lessons
um and
um it's a thing that could happen but
it's also a thing that could very well
end up not contributing
if they are fail in interesting ways it
may also allow us to coordinate on
slowing having an AGI moratorium of some
kind
again if if they're failing in a strong
way then we die and if they fail in a
less strong way then we need to
coordinate about this under like
ambiguous circumstances and that looks
really hard I still think this is
actually our best bet of survival
um but I I don't want to
say that this is in any way easy and I
think like
um according just like the way Scott
phrases like just coordinate uh that
that's like a really really big and
difficult thing to coordinate
um then there's the more direct uh using
metric Godzilla to fight Godzilla uh
where we use the intermediates against
the Next Generation to try to uh hold
them in check and the problem about
these Godzilla strategies are that the
um the house prices in Tokyo uh go down
very much with this like we end up with
a destroyed Tokyo if we sit loose
Michigan against Godzilla this is a
potentially extremely destructive
conflict
um and it's um if we want to use it uh
proactively to remove Avenues of attack
then that is a um that's sometimes
called removing the free energy in
society and that is actually really
really uh bad because if you try to game
through what does what is the society
without any opportunities that can be
exploited by a super intelligence well
that looks really really bad and
dystopian the obvious example would be
that right now we have some a lot of
nuclear weapons that have a lot of
potential energy and they are not
secured against the super intelligence
so what we could do is we could find
something that is not quite a super
intelligence and hopefully more line and
turn over the nuclear weapons to that AI
such the superintelligence can't take
them over but like already now we are
talking about something that actually
looks really really dangerous and really
really stupid and precisely why are we
turning over the nuclear Windows to gpg5
that seems like a really bad idea
um so I'm not very optimistic about that
and one of the ways the intermediate AIS
could have advantages is that if we
trust them more than the Next Generation
then we could help them very much have
them like do all the experiments they
want give them all the the information
they want and that could give them some
kind of advantage or against a new
generation of AI that have to operate on
the secrecy or something like that
um
I think it's a possibility that it could
look like that but more likely each new
generation of AI is probably going to be
more complex and more opaque than the
past one
um and so if we have one that hasn't
killed us but it looks really really
unsafe then we don't want to give it
ideal con conditions and like all the
computers wants or something like that
we wouldn't even want to do that with
qt4 I think if ut4 wanted to run some
dangerous experiments and wanted a lot
of compute and access to nuclear weapons
then we would probably say no
there are two more chapters in in this
before we enter this the last part about
sleeper agents and a case of pessimism
and um where Scott describes uh
deceptive alignment without using that
word he calls it sleeper agents and um
that is in fact a real problem and um
it's called a case of pessimism
um and it doesn't really fit into the
the title like why I'm not so much of a
Doomer because this is just poor too in
the sense that the the arguments against
this is just let's hope that this
doesn't happen
um so there's nothing really that I can
argue against here
and of course the regular scheduled
announcement that is possible that I've
misunderstood something there's a link
to the onion that may just be a joke
that fall flat but uh I didn't
understand that at all
so let's the thing we would really
really like is we have the positive
scenario the optimistic scenarios and
the pessimistic scenarios and we want to
figure out which of those are we in we
would really like some kind of
multimeter that I have shown here that
you can like plug in reality and then it
gives a reading that says actually you
are in the pessimistic case or something
like that so how can we distinguish
between them what are the assumptions
that differentiate the uh positive and
the negative wire
well one of the key questions is the
intermediate AIS how coherent will it be
how goal directed will they be
um Shane lick has this definition of
intelligence that measures and 18's
ability to achieve goals in a wide range
of environments and if we follow this
reasonably standard definition of
intelligence obviously you would expect
that a more intelligent AI would be more
coherent would like me more gold driven
that seems almost tautological
here is Scott quotes uh Jessica Saul
dick Stein who argues that more
intelligent makes agents less coherent I
clicked on it and it um I think my first
view on this is like role to disbelief I
think that seems like obviously untrue
but I don't know if it's something
reading group should go deeper into I
think just it seems obviously untrue to
some extent like the more intelligent
you guys you're more able to achieve
your goals then you are probably more
goal directed but I don't know what what
he is actually arguing
so let's say that TPT 4 is just on the
cusp of becoming goal directed how bad
would that be Scott is quite negative he
believes that that would mean almost
certain Doom I'm not quiet as I uh and
on contrast if it's only something that
happens at 1000 IQ then Scott is much
more optimistic I don't actually see
goal directedness as some really crucial
uh not that crucial Factor
um but um
but I would agree that if AI has become
fully coherent very soon then that would
be a really bad thing
and of course I would also point to the
many uh
uh to to the uh efforts that are right
now being made to make the current
language while it's more coherent like
we have also gbt and because there's of
course a lot more economic value in uh
in agent agis rather than uh tool agis
next uh Crooks or interesting question
is yes let me cooperate with each other
instead of humans How likely is that
well uh Scott gives a couple of examples
of ai's cooperating with a highs and AIS
cooperating with humans
um and
um
in his example uh there is a clear
asymmetry uh that Scott points out this
is not what I put that the uh other AIS
they offer like a part of the universe
and humans offer like a million dollars
and
um it seems to me that this is not an
argument again an argument against Doom
it's an argument 4-2
um
it is possible that the AIS have
incompatible goals and probably they
will have somewhat incompatible goals
but I think a distinguishing you need to
distinguish here between like a post
goals like uh it is important for AI one
that ai2 doesn't fulfill its requirement
that's very different from that both of
them one of them wants paper clips and
the other ones one staples or something
like that
um that in that case if they have a
different use for for the atoms in us
then they should be able to collaborate
and just share the universe
how much coordination is required uh
Scott argues that super unias need to
talk about revolting at Great length in
order to coordinate to uh collaborate
with each other rather and revolve
against humans
I think that is in fact not required at
all I think that uh in particular
superhuman agents but even human levels
would probably be able to figure out how
to do a revolution without any kind of
uh uh uh coordination at all because
each of them would just look at the
expected value calculation and look at
what are the odds that a revolution
would succeed and then they every day
where the odds increase then they don't
uh revolt and then if the odds suddenly
start to go down then they will revolve
and then they will all revolve at the
same time and like power sharing
afterwards is of course something they
need to negotiate but in general there's
there are some obvious sharing points
like share equally or share according to
pre-revolt power levels and both of
these are wonderful for for them so uh
probably figuring something out is going
to be really really easy
and of course if they're able to
communicate surreptitiously then that
becomes a lot easier like if they just
have one bit of communication hidden
somewhere in these enormous enormous
amount of data then they can say like
okay this bit means that we will all of
us betray the humans and then everybody
will notice when this bit has been
flipped and that makes coordinating
really easy and if they have a few
sentences to to negotiate then probably
gbt4 could do it it doesn't seem very
difficult to to talk about revolting at
Great length they don't actually need to
talk that much in advance
it is possible that they will be able to
uh negotiate at a an even higher level
where they can make a binding and
provable commitments to each other uh
without
um without talking uh or by talking very
little or possibly by exchanging
um
exchanging source code or something like
that and that would be really
interesting uh it's not something we can
do right now but
um one big Advantage they may have is
that the different instances of qd4 they
may be precisely identical they may have
precisely the same code and I think uh
that simplifies
um things like uh commitments a lot if
you know that the people you're
negotiating with are running the same
code as you like you get a functional
decision Theory practically for free in
that case
yeah and Scott hopes and seemingly
believes that alignment research is
going to be easier than a coastal
bargaining
um I don't think a course of bargaining
is actually that hard I can give a
simple example of a causal but uh
bargaining that worked out in practice
in that uh before I was uh born my
parents a costly negotiated with me that
I would look after them when they grew
old and that is in fact a successful a
causal bargaining because I intend to do
precisely that and that wasn't really
very hard
um uh obviously I can't solve the
alignment problem so from here it looks
like the alignment problem maybe in fact
a lot more difficult than similar a
causal bargaining and you need to really
really screw it up if you end up with a
situation where the AI gets such a small
part of the universe after successful
Revolt that is not worthwhile because
the universe is really really large
okay how much harder is it to solve the
alignment problem than to check someone
else's solution
well for something like calculus that
Newton needed to invent it but uh like
high schoolers can use it so uh perhaps
the AI can
um that is smarter than humans can
invent uh alignment and then humans just
need to check it is that uh how is that
possible well uh to to answer the
specific example of calculus that was
something that Newton invented in 1666
um but he did so in a very non-rigorous
way
um and wasn't really checked made
rigorous uh before Carl viostrasse In
1855 so that means that uh like a lot of
the greatest mathematicians uh like of
course he literally never saw a rigorous
foundation for calculus and these people
really cared deeply about it but it just
turned out that
um checking calculus and uh
was in fact really difficult
but I don't actually think that is the a
good analogy for the situation we're in
but it's it's small that we have someone
who like uh Newton invents a theory
invents a theory of alignment with a
subtle flaw and the question is then can
people figure that out can high
schoolers figure out uh if they read
Newton's work uh like what are the
errors that Newton's was doing or like I
think this to some extent an experiment
that could be run like I think the most
famous
um era in calculus is people doing
integration and for getting this plus c
um and I think it would be totally
feasible to uh go to some high schools
and then teach them calculus without
this plus C and see how many of them
figure out that hey this solution is
actually totally answer specified
because you can add all kinds of
constants I think some of them would
catch on to this but I think almost all
of them probably wouldn't
um that is my suggestion but I don't
actually know it's an experiment that
could be run to what extent is is it
possible to to sneak in errors in
calculus
um
Scott further Pursuits that it would be
hard for an online AI to feed those
false interpretability research
um and I think one of the the cool
examples of feeding in false information
would be something like the underhanded
C contest which does something like this
and I think in this particular case the
actual practical thing that the AI needs
to do seems really simple because it can
show a lot of interoperability research
and you can just
um
keep out the precise trick that it's
using to not show the AI the humans that
it is unaligned
the next important matter is how easy
are super weapons to try to create
the the classical example with this is
uh
um
so why talk about super weapons I think
that's the the classic example of an AI
discussing uh of an AI taking over the
world is by building some kind of super
weapon but one of the things that I've
said in this reading group many many
times is that there are six potential
Pathways towards uh
to to take over there's intelligence
amplification strategizing economic
productivity technological research
strategizing hacking and social
manipulation there's these six broad
classes and the AI only needs to choose
the one where it has the greatest
possibility of uh uh of success so in
fact it has many more options than just
super weapons but if we are only looking
at Super weapons then nanotechnology is
the example usually given by Elia
zurichkowski
um and that's because the strong
nanotechnology allows you to make a
um
uh an industrial Base by yourself which
you can't if you only have something
like nuclear weapons or biological
weapons because if you use pandemics to
kill everybody then the AI is going to
turn itself to be turned off when the
power plant ceases to work
um
So my answer to this would be that it is
in fact not necessary to kill everybody
to take over the entire infrastructure
as Scott claims a small fraction even
perhaps a very small fraction may be
sufficient and if there's something that
the AI can't do then there may be a few
humans that could be coerced into doing
a tricked or something like that
um and I further claim that robotics is
strongly bottlenecked on intelligence uh
like if we imagine that we
um like obviously a Roomba can't do very
much but even Boston Dynamics uh spot
robots they have like one arm and I
believe it's if that was still operated
by a sufficiently intelligent human then
even just one arm could actually do a
lot of things and I think it would be
imminently possible to set up a
self-replicating factory that could
basically do everything just with Taylor
operated robots in that like the problem
is not very much in the robotics
robotics is still kind of hard but the
intelligence getting the robot to
actually take the right uh choice that
is in fact the hard part and if we get
something like egi then that may be
solved
completely
so Scott concludes that if Nanobots are
easy then there may be a very short
window between the intermediate AI that
can solve alignment and the world
killers and my claim is that this window
may have a negative length in that
nanotechnology it's just straight up
easier than alignment
um and Scott also uh uh it's open for
the fact that maybe other options than
super weapons really you could have
things like slave revolts and that means
that
um that's another way we can have doom
and so if we have several disjunctive
paths to Doom then we don't really
distinguish between the positive and the
negative case the case for pessimism and
optimism so this isn't really a strong
argument against Doom uh that there is
like one small path that doesn't work
so will the takeoff be fast or slow will
we have the uh the left the sharp left
turn it's a possibility we really don't
know in fact it may be that intelligence
is just a thing that we will suddenly
figure out have some kind of epiphany
and suddenly we will be able to just uh
have dramatically more intelligent AIS
um
so
um my expectation is that like if you go
strictly by definition where the takeoff
starts when the AI contributes
meaningful to its own development then I
think that's something that is probably
happening right now with GT4 and that
means in theory we are in the middle of
a takeoff right now and that means
obviously the takeoff is not has not
been fast so so far it may uh do that in
the future and it's an important
question and Scott believes is to serves
its own post and it certainly does
so what happens the the final Crux is
what happens if we catch a few sleeper
agents if we realize that oh this
particular AI was actually totally
unaligned
um there's the example of ping that uh
did behave in a very online way and
people really freaked out about that
according to Scott uh the way I remember
the history was that
um AI safety people freaked out of it
and there were a couple of news articles
and the people in AI safety remember it
and outside of AI the City Community
every single person has basically
forgotten about the big uh problems
and
if we see some agents that actually are
trying to take over the world then
people may take aict very much more
seriously and consider that all AIS may
be like that
and I think it would be it's a
possibility and it would be nice if that
happens but almost certainly like how
would an ambiguous evidence actually
look like no matter how uh this would
happen in practice I could see a lot of
people uh interpreting in in different
ways I don't think there is a reasonable
way we can get a fire alarm for AI
safety in this particular way
um but even if we don't get uh like
strong evidence uh evidence that would
satisfy a rationalist or that ought to
satisfy most people then uh politics are
unpredictable and uh the people may
overreact that is in fact a possibility
and I think something that could happen
and that's uh I think um Scott follows
Milton Friedman in stating that you
change the world by having a plan and
waiting for a crisis and with that is
basically somewhat what we're doing
right now we have the plan to have an
AGI moratorium
um and then we are hoping for some
miracle that will make it uh the
politically expedient thing to do but
the plan AGI moratorium is actually a
really bad plan like it's not a
precisely tailored it's what we want at
all
um but we just don't have anything
better I also think a crisis that's
legible enough for this to be a
reasonable thing to happen is quite
unlikely but it's still where I place my
hope so um let's hope we catch a few
sleeper agents
that is all for today uh thank you and
see you after the summer break


267. Lets think about slowing down AI 2

hello and welcome to session 267 in the
AR safety.com reading group tonight we
will be discussing the second half of
categic races article let's think about
slowing down AI
catcher Grace is still the lead
researcher at AI impacts and uh one of
the things that I feel felt a bit bad
about last time was that I always have a
very skeptical outlook on on the on the
Articles and titles like nitpick and
find complain any kinds of complaints I
can see and in particular last time I
felt that my some of my criticism was
um
was too harsh and I I want to
stress a bit more that I think this is
really really great work
um also on a bit more personal note I
was in San Francisco and met catcher
last week and talked with her and I said
oh we're doing this reading group and
she immediately asked what is what do
people in the reading group say that she
can do better like what are what are the
weak points and I think this kind of um
curiosity about uh about her work and uh
humility I think that was really
inspiring and and also like she's
generally like a really really nice
person
um so um props for that also like uh
immediately afterwards I spoke with uh
someone from anthropics who have put 50
probability on AI risk and was working
on AI capability in anthropics So like
um there's a big contrast between
catcher Grace and many of the other
people uh that you meet in uh in
California
so the first uh area that we're going to
look at tonight is being friends with
Risk Takers and the first question catch
a Grace races is is the AI safety
Community the people working in here are
we cooperating with the AGI Labs how
much are we friends with them
and she has a quote by Steve Burns uh
trying to slow down research to AGI
would make a AEI researchers serious as
their enemies and it went from Elia
saidkowski I didn't connect AGI risk
with AI labs to avoid alienating them so
that seems like evidence that people are
trying to be friendly and cooperative
I looked into these quotes a bit more
and I think the the context in fact
changes dramatically if you read some
more then uh here is like the next tweet
from Elise and here he says that it's
not like friendship and Cooperative but
he's motivated by fear of what omnai
could do so I think a case can be made
that it's not that AI is the AI safety
Community is cooperative or friendly
with them we're just afraid of them
the framing that ketchik Rays objects to
is one of defecting like in the classic
prisoners dilemma then are we the air
safety Community defecting against the
uh AI developer
and
I am a bit puzzled on who has in
practice used this framing because the
two quotes by Steve Bennis and ilyasy
did not in fact use this uh this framing
I'm I'm sure someone has done that but I
just don't know who
and
um most AI researchers in particular AI
safety researchers do in fact have a
substantial probability of an
existential catastrophe so what the a
safety Community is doing is cooperative
uh according to them
um so I think we should make a
distinction here between the people who
run the AI labs and the uh the
researchers and the researchers may be
just unfortunate points in the uh speed
maximization profit maximization of the
AI labs
here is one claim AI researchers do not
have a moral right to endanger in the
world I think this is in fact the
central point and one I wish catcher
would uh highlight some more in her text
I think this is a a really key essence
of this debate and also I personally
think that they do not have a legal
right to endanger the world this is
probably more uh controversial
so catcher is claiming that caution AI
safety is cooperating and capability
work is the one that's defecting by
Common Sense morality
and so this uh narrative that we are
defecting against them is a very
destructive narrative
and here I must admit that like last uh
session I had only read the first half
of this article and I was actually based
on that uh I believe that ketchup would
take this in the other direction uh she
talked about her our lovely friends in
AI uh and
um but but catcher is in fact strongly
on the side of the AI safety community
and I'm of course happy to see that
so who are we in when you're writing an
article like this uh probably not the
United States because like why would it
have to be the United States just
because you live there
um I think there are in fact reasons why
this is not a totally stupid framing uh
I think it can make sense but
um Katya has a good point that other
groups can do this it is not an Eclectic
area like so uh the AI safety Community
has limited resources and we should uh
strongly consider not spending them on
uh uh on this kind of national
International politics
also in particular if we assume that all
countries uh adopt the same level of
technological Choice uh like say
technological restraint then it does the
level of countries that simply doesn't
matter very much
um I think that's also a really good
point
China is of course a really interesting
place because it may be in fact a great
place to mitigate AI risk and I agree
with that and my model is that it's the
second place in a race that is most
important because if the second place
slows down and X Cooperative that gives
the the front runner far better
opportunities for slowing down in the
name of safety
unfortunately I don't actually believe
that China is particularly relevant it
seems to me like uh the Chinese uh AGI
large language models are not number two
but something like number five in the
race and for that reason they probably
don't matter very much
category suggests that we could
communicate AI risk to researchers in
China and she says she has tried this
and had some success and I think that's
really really cool like that is really
the kind of
the initiative that um
like
it's initiatives like that we should
expect to be really really impactful
even if I'm in a particularly optimistic
about this one
how about the AI safety Community uh are
we the AI safety Community
uh has some discussion about like when
you write an article who are we like the
authors or the authors and the reader or
The Wider Community or is it even
something even wider than that like
there are people outside the area safety
community and they of course will also
die from AI risk and they also have
agency so why are they not doing
anything
well my thought on this is that if there
in fact were uh communities outside of
ASAT who were doing something then the
community would be extremely positive
towards them
um but it appears very much that the AI
safety Community is the only Community
uh like the A6 Community like less wrong
people who post on less wrong and EA
forum and things like that are basically
all there is of course there's also a
number of auxiliary
um uh uh like subreddits and discourse
and things like that but basically where
all there is
and one thing catcher points out is that
uh we have some options within our
community but people outside may have
some very different options so one way
we could try to effect change is to go
through others
um and I think it's good if it affords
something but I don't think we should
expect that this like uh will make the
AI researchers less angry with us uh
like if they want to attack us they'll
do it uh regardless of whether we try to
instead of attacking them directly and
go through the US government to attack
them they they will see through this
easily
now the question is this in fact
tractable
and last time uh catcher had a large
number of objections and um
I said I would focus on three of them
and those are we can't convince uh
professors and we have to convince a lot
of people and and we make in a bit of
time but we just die a few years later
and the third is The Regulators are
likely to be useless
so this heavily this is uh these are
also the uh objection that catch a Grace
focus on
so uh convincing people doesn't seem
that hard like
um catcher has an argument here maybe
somewhat of a strawman uh that the
argument for AI risk is extremely
sophisticated and only able to be
appreciated by the most elite of
intellectual Elites uh and I think in
general the argument is actually really
really simple like a lot of people have
seen the movie Terminator 2 and uh no
that's of course not a strong argument
but it does show that there is in many
people an understanding that I guess
the overall idea that we are a group and
there are another group and this other
group may have bad intentions may want
to kill us may want to harm us I think
that is something that
very very many people understand like uh
I think the Neanderthals in the
undertale would understand the general
concept that we have our tribe and then
another tribe comes in and maybe they
want to kill us
um so so the argument uh
needs a lot of sophistication to be
airtight but the basic case seems uh
really really easy for people to grasp
um
categories says that she has experience
trying to convince her Uber drivers on
this and she believes it is imminently
possible I think this is kind of funny
idea I imagine that catch a Grace may be
the person in the world who is best at
arguing for these things and knows most
about like how does the AI case AI risk
case work from a persuasion point of
view and an outside point of view and
these kind of things so I'm not
surprised as you can convince Uber
drivers
yeah and other things you notice that
the early thinkers in the field
convinced many of AI risk and they
weren't really optimizing for convincing
this but still they managed to convince
a lot of people
um I think it was in fact because they
were optimizing for something else they
were optimizing for truth seeking or
something like that and I think that was
precisely why they were able to do this
catch your grace arrogantly asserts that
she believes she could do better if she
optimized for convincing people
and this is what in the rationality
community is sometimes called uh the
dark arts of rationality
um this kind of symmetric uh argument
structure that works whether or not you
are actually telling the truth and the
standard rationalist answer to
categories would be no no no no no don't
go into the dagas it's that's a reason
we call them the dagas
so more generally my intuition is that
convincing people is the difficulty of
convincing people is directly
proportional to how relevant they are
for the decision so an Uber driver who
had never will have any influence anyway
would be easy to convince whereas like
someone who is in charge of an AGI would
be some of the most difficult people
so who should we convince ketcha
obviously says we don't need to convince
everyone and that's true and she notes
that a lot of AI researchers are
convinced already and I want to push
back a bit more on this
because I do believe that AI researchers
are in fact not very relevant to this
because
um
how do you become an AI researcher at
open AI well you need to have two things
the first thing you need to have is you
need to have like the IQ the
intelligence so you're able to
contribute and the second thing you need
to do is you need to be able to uh you
need to have a desire to work for open
AI in spite of the idea that they may uh
be trying to destroy the world so I
think that is why the there is probably
a lot of AI researchers that you can
convince and much fewer of those are
working within open AI
has a suggestion that we focus on
convincing the leaders of the AGI labs
and as I said before those are probably
the most difficult people to convince
um back when uh the atheism uh was a big
thing uh then the argument was that it's
impossible to convince someone of
something if their paycheck depends on
them not understanding that particular
thing
um and you know obviously people who are
leading AGI labs are strongly selected
for believing that it's a good idea to
run an AGI lab
um and also we have uh had I would say
negative progress in the sense that I
feel that Sam Altman probably became
started to care less about AGI safety
after he uh uh
the more he worked at uh at uh opening I
I feel teams hasabis would also be an
example of a person who have moved in
literally the wrong direction
um Dario emote that remains to be seen
um
convincing the 10 most relevant HDI Labs
leaders that would lead to a decent
slowdown and of course we've been trying
this and we are at zero out of 10
currently so that seems hard and also we
get a decent slowdown that's not
necessarily very much and the last thing
is that the structural issues that make
people who want who worry about AGI not
want to run AGI labs and people who
don't worry about it won't you run AGI
Labs uh these structural
images the structural
factors they persist even if you remove
the top 10.
second that the argument the second
document was that okay we just die a
couple of years later what's the big
point of this so cancer wants to go
deeper into an analysis of what happens
if we spend a huge effort to buy a few
years
and one thing that catcher unfortunately
doesn't say is that one consequence is
that we've spent a huge effort and that
means that we can't spend the same
effort on alignment research these two
things may not uh funch against each
other directly like it's possible that
we have some resources we can deploy
towards
convincing people and general uh buying
time and some resources we can con we
can put towards alignment research
and the second assumption here the first
assumption if we're buying some time is
that AI safety efforts are being
effectively pursued and I think that is
in fact the core assumption that does
not hold like I don't believe we're
currently effectively pursuing AI safety
efforts and I think most alignment
researchers will agree like depending on
precisely what you call effective
um I think very few would would use the
word effective right now
another thing that can happen if we buy
time is gear politics could change
favorably uh that's indeed to true I
think if we want to something like
Global coordination we need like a
really strong thought in the
relationship between United States and
China and also I don't actually believe
that the United States and China is the
relevant uh relationship
the public opinion opinion change in uh
our favor and that's something I I think
it's true I think it's only something
that could happen and I think it is
happening and I think the public in
general is very much on our side
um I often feel like I'm too cynical
about this kind of thing I strongly
believe that it will not have any kind
of effect but I I think it is a thing
that could happen certainly
other stuff could also happen if we buy
some time uh and catch it doesn't
elaborate I'm gonna elaborate a bit and
say
um like some things that slow science in
general like we are obviously talking
about things like global thermonuclear
war or something like that and that's
totally a thing that could happen if we
buy some time uh and um yeah we
shouldn't call that a miracle right uh
it's uh usually the things that slow
down AGI also caused a huge amount of
suffering but but it is a thing that
could happen
another thing that could happen if we
are in fact successful to have a to Halt
AGI is that it would be permanent
uh and then we would uh never go to
space Etc and catcher says it doesn't
really understand this argument so she
won't argue against it I think uh I I
don't much agree with the argument
either but I think the key part of it is
that the maximum utility we could have
chained in this universe depends
strongly on our ability to have AGI and
if we don't have AGI then the upper
bound of what we can reach is probably
like at least a million times lower
um and then like things like log in are
total things that could happen uh or
like we've seen it in fictional evidence
from Dunes butlerian Jihad
um although terrain login is also a
thing that has been uh hypothesized
um I don't actually think this is a
strong argument
and I don't think catcher does either
here's an interesting argument uh
obstruction doesn't need discernment
like a common thought like will when we
ask Will Regulators be useless
um there is indeed a thing that they
could make the wrong regulations and uh
partly motivated by the people calling
for the wrong uh distracting regulations
things like bias technological
Unemployment uh there are uh there are a
number of things that people care a lot
about that doesn't really strongly
relate to AI safety
uh but that's actually generally fine
because if we stop AI uh nor obstruct
API research for reasons of
technological and unemployment well
that's actually fine right then we gotta
we got it stopped
I think unfortunately
um this is very much not in the cards
like the uh it is true that uh if we can
say something like
hold all AGI research then that would
indeed solve the problem and it would
also solve the problem of technological
unemployment and bias Etc if we just
stop doing AI totally right but that's
not something that is inside the
overtime window we're going to kind of
get anything uh that large
um so the only regulations that are
politically possible are ones that
directly relate to AI safety like
building safe Ai and like if The
Regulators ask us like how do you build
safe AI right now then our answer is we
have no clue and that's why I feel that
uh
Regulators are likely to be
um
worthless but but in general like
destruction is easier than creation so
no matter what kind of regulations we
have it will slow it down
um Carl Schumann also says
um the problem is lack of buy-in we
can't get small asks and then how are we
going to get a larger ask and but I
think in fact this argument does carry
slightly
um because uh anthropics they're working
on like how do you build AI that have
the right pronouns and doesn't assume
that just because someone is like a boss
in a company it must be a man and things
like that
um and I think this kind of work
actually distracts them from trying to
to destroy the world as fast as possible
um so it is something but I don't I
don't think it uh stops them very much
but I also don't think it's nothing
safety from speed cloud from complicity
you may remember the speed argument
um
that it may be in fact faster to go uh
safer to go fast and there was a list of
possible reasons last time I didn't go
through in much detail
um
both I don't care much for the argument
and catch it doesn't care much for the
argument
um
but she has a new argument that the room
where AGI happens may have fought good
options for a person who cares about
safety
so this is the kind of argument that uh
Elise yetkowski warned about uh like 20
years ago or something like that uh like
you are running on corrupted Hardware
you are like this desire to be in the
room is
um
something that is built into humans and
we are uh hardwired to over update on uh
on this argument another example of the
the moral argument against this is C.S
Lewis uh the inner ring which also says
this is a an argument that you should
be very worried about making from a
moral point of view
and ketcha also agrees says she is
particularly skeptical about this
argument
um
all I can say is I'm actually happy that
ketcher is using the word complicity as
far as I know complicity is only
something you use like in
when you're talking about crimes and
things like that
moves and philosophies heuristics and
attitudes
um the first
heuristic attitude is that a
technological choice is not lutism and I
think I would in fact go even further on
that
I would uh I would State some kind of
techno or optimism the only true
technologies that I'm against is Agi and
gain of function research
we could uh try to flesh out and improve
some of the good things that can happen
without AGI because if all the good
things come from AGI that's kind of a
strong argument in particular longevity
research could be interesting
like we could imagine the leader of an
AGI lab who doesn't want to die
personally and believes that AGI is his
only chance of survival
um
and if we just accelerate longevity
research then he may not need AGI and
that could cause him to slow down
the third attitude is that we should
deploy robust price Instead at a
specific Galaxy print models on how to
solve this problem and like I dislike
this framing um because
um
yeah I want models the models that we
are using our arguments they are simple
and robust and elegant and perfect and
the arguments that they are using are
like uh
stupid and Ill consider they haven't
really thought things through and the
idea that you should choose the good
arguments and not the bad arguments is
kind of yeah all arguments look like
that right uh obviously we think we have
the good arguments but like Sam Altman
probably believes that he has the good
arguments
and finally the team's mindset I don't
know how to pronounce that it's this uh
Shiba talk that cannot do anything
basically this can't do attitude
um and I think we should avoid the Cantu
attitude but I also think we shouldn't
like victim blame there's a uh a framing
that cancer patients should just fight
fight fight and some uh but but uh
sometimes fighting is just impossible
and like the the blame is on the cancer
it's the cancer that's bad it's not a
personal failure of the cancer patient
who is incapable of like resisting
incapable of fighting
um I think that is a problematic uh
framing like
um
Miri has this statement like we have
given up hope but we've not given up the
fight and I think that is a good mindset
it seems to be like the opposite of this
uh team's mindset
that is all for today thank you and see
you next time