AI Safety with Buck Shlegeris

[Music]
hey folks welcome to narratives
narratives is a podcast exploring the
ways in which the world is better than
in the past the ways it is worse and the
paths towards a better more definite
vision of the future
i'm your host will jarvis and i want to
thank you for taking the time out of
your day to listen to this episode
i hope you enjoy it
you can find show notes transcripts and
videos at narrativespodcast.com
well buck how are you doing today
i'm doing all right
how are you
doing great thank you so much for
hopping on i really appreciate it
could you go ahead and give us a brief
bio and tell us some of the big ideas
you're interested in yeah so i guess my
brief bio is
i currently work as the cto at redwood
research which is a
non-profit research institute research
doing some applied alignment research
i'm also a fund manager on the effective
altruism infrastructure fund where i
meet grants for various things related
to effective bachelor's in movement
building and related types of movement
building
i have been involved in ai alignment
technical research for the last three or
four years
uh before that i was a software engineer
and before that i studied computer
science
awesome awesome and when you mission
alignment research you're talking about
ai alignment correct yeah ai research
super server so not all our listeners
are familiar with it so could you talk a
little bit about you know ai safety why
you think it's important and and how you
got there
yeah so i guess the basic uh the basic
pitch for ai safety uh
starts by noting that a lot of the stuff
which happens in the world
is determined by
cognitive labor uh you know
cars have changed the world a lot
buildings have changed the world a lot
and many different types of many
different types of uh
work and many different resources are
required to put these things together
but one of the key limitations is how
good the ideas we have are for for how
to build these cars like how how to
design the machines that build these
cars and that make it cheaper to mine
and so on uh it seems reasonably likely
to me that over the next century the
primary it's going to become the case
that the primary way that we get
cognitive labor done is instead of
paying humans to think about it with the
brains which are operating inside their
heads you know with 20 watts of power or
whatever
it becomes cheaper to get machines to do
the thinking for you uh in the same way
that it's now cheaper to get machines to
lift
the blocks for you if you're assembling
a large building um
and just as machines that made moving
blocks around cheaper
affected the size of the buildings which
get built
and made the whole economy a lot larger
i suspect that drastically reducing the
price of doing cognitive work uh by
making sure that machines can do it and
also making it so that we can do types
of cognitive work that are currently
impossible uh seems like it might change
the world a lot
um so that's like the the case for like
ai is a big deal
i'd have to make a separate case for
maybe this happens uh within the next
century
um
but that that's like part one of that
case um and then part two is
we're thinking that we should work on ai
alignment
so by ai alignment i mean something like
uh i want to make it so that as ai gets
developed it's as easy to use it for uh
many tasks that i care about as it is to
use it for some other tasks um
so
maybe the easiest way to explain this um
sometimes
sometimes people talk about the
possibility that uh
social media makes it easier to spread
misinformation than to fight it
um
and this is an example of a case where
you have this technology and that the
technology
for uh you know the technology itself
isn't particularly like uh
it's not like i'm just like building bio
weapons or something where obviously the
only application of the technology is
destruction
but in some cases
technology makes it easier to destroy
value or like technologies
lend themselves more easily to
destroying value then they
lend themselves to producing value
and for various reasons i'm worried that
by default ai is going to be one of
these technologies that
might be easier to deploy in ways that
destroy certain types of value than
how easy it is to deploy it in ways that
preserve that type of value
that's perhaps a really abstract way of
saying this
a concrete example is i'm aware that
it's going to be easier to make ai
systems that um
you know you ask them the question and
they give you back an answer that sounds
really smart than building an ai system
that you ask it a question and it gives
you back an answer that is in fact a
really good idea
um you know you ask it for some advice
and you want you want good advice out
but the problem is the way you make ai
systems do stuff is you train them um
and i have an easy way of training a
system to give advice that sounds smart
to humans which is i get the ai to
propose two different pieces of advice
it might give to a human and then i pay
some humans 15 bucks an hour to read you
know questions and like two proposed
pieces of advice and click the advice
that seems smarter um and pretty soon
this ai is gonna give you advice that
sounds really really smart um but
suppose i wanted to build a system that
gives you advice that is actually good
advice um it's much harder because among
other things
i mean the easiest case is when someone
gives you advice and you want to figure
out if it was actually good advice or
not sometimes you have to wait to see
what the effects of the advice are um
and it's fundamentally harder to train a
system if you need to like wait 10 years
between uh starting the training process
and finishing the training process
uh
and also sometimes it's hard to tell
whether advice was good advice even with
the benefit of all hindsight
um
and so it's just like it's just like
systematically harder to construct a
training process which like gets this ai
to give you actually good advice
um and a third kind of concern you might
have uh is if the advice uh is deceptive
if the advice gives the ai access to
your bank account and lets it um lets it
steal your money and start hosting
itself on the internet by like you know
paying for its own
its own servers to run on uh you will
never have a chance to uh analyze it uh
it just like lives on the internet now
um and so for a couple of these reasons
it feels like it might be easier to
build these systems that are
uh that you give advice sounds really
good but you can't really trust than it
is to build these systems that give
advice that uh is actually good um
and this is an example of a kind of
fundamental seeming hard problem i'm
worried that if you just like have a
bunch of stuff like that going on it's
pretty dangerous and bad
that you know that is such a great point
and i think about this in context of of
my dog right you know he had like a skin
condition recently had to get a bath he
hates the bath there's no way for me to
inform him that the bath is a good idea
you know it's like just above his
cognitive pay grade and you and there
could be things like that where you know
ai tells us to do something it's a great
idea we have no way of understanding how
that yeah it's a good idea
that's never exactly
that's super interesting um
so uh
i think it's it's a really important
thing to work on as you mentioned it
could be there's a lot of asymmetries
that can exist um it could be used as a
weapon um it could be and we tend to
develop things to use them as weapons so
i think of like you know nuclear power
that was developed originally you know
manhattan project wanted a weapon um
it seems like most of the the mainstream
debate around ai is around um you know
it's going to take our jobs or
automation is taking our jobs but it
does seem to me like uh the more
important thing to worry about is is the
uh what if this you know what if it's
evil or it does things that are bad to
humans uh how do you think about that
kind of issue
yeah so i mean
i think that as with so many things uh
there's like answers to this question
that you know increasing levels of
sophistication where you know you say
something and then the next and then the
next paragraph says well actually and
you point out the consideration the
previous one misses so we're going to go
from the simplest explanation the
simplest answer first
um
i think
my
moral perspective or whatever uh is that
i'm uh
i'm a long termist um i care a lot about
the welfare of humanity and other
sentient life uh in the long run future
and so
i'm inclined to think that we should
mostly be focusing our efforts on making
it so that stuff is good a million years
from now
um
and
it by default would be really really
hard to influence the world
like most times in history it would be
really really hard to influence the
world a million years from now you know
if you were it was 500 years ago i think
that it would have been extremely
difficult to influence the world of
today let alone the world of 199 995
uh or sorry
no that's not the number uh years in the
future from then um
but i think it's kind of plausible that
we are at a pretty unusual period of
history where it is in fact possible to
influence the long-run trajectory of
humanity
basically because i think that um the
probability of human extinction this
century is a bunch higher
than
it is most centuries than it has been in
the past and then it will be in the
future
and so for this reason kind of my top
priority
is making it so that when humanity
builds
these really smart systems that make
things
that are able to do lots of stuff uh i
want this to go well for humanity and so
my kind of simple first answer to the
question of how i think we should
prioritize between worrying about making
that transition go well and worrying
about unemployment in the short term
is i'm just like well i don't know it
seems like
almost all of the value is in the future
and so i don't think
that i should be prioritizing very much
worrying about technological
unemployment in the short term because
the number of people affected is just
much smaller and i believe in
prioritizing things based on the number
of people affected
um
and then i i think that there are
actually some more complicated i think
there's like a a complicated second
order take which is something like the
following um at some point in the future
uh we're going to build the first
systems that are smarter than humans are
and this is likely to go better if the
world is less of a mess
when this goes down so for example i
feel a bunch more optimistic about the
deployment of early artificial general
intelligence systems you know agi
i feel a lot more optimistic about how
that goes down in worlds where uh the
world is in a unprecedented period of
international cooperation and peace
rather than in the middle of world war
four or like whatever you know uh that
seems like it would be much better and
so i think that
uh
it is plausible that before the world
becomes radically different as a result
of really powerful ai uh the world just
becomes pretty different and one example
of a way it could become pretty
different is uh you know technological
unemployment and so i do actually think
that uh if there was a strong story for
technological unemployment causing
things to be really bad uh
and a real mess uh then i would think we
should worry about that because of the
fact that it puts humanity in a worse
position um for the sake of the
long-term future gotcha so something
kind of like if our starting blocks are
bad
when ai takes off things could be could
be quite rocky
yeah that could be very bad
yeah that's right we've got like two
different problems one is like
you know the the the core problem is
like when we actually build these crazy
powerful systems uh
we really want that to go well uh but
also these less powerful systems we have
on the way there might mess things up in
various ways and that would be
unfortunate
gotcha gotcha that makes a that makes a
ton of sense i
went you know how do you think about
factoring
the value of future people versus you
know the people that are here now
do you see what i'm saying um and there
there are going to be a you know god
willing there will be quite a few more
people in the future so you know
even if you factored it a lot it would
still they still matter a lot it seems
like
yeah that seems right to me um
i think i feel a little confused by
exactly how much to weigh the interests
of future people um
almost all of the people are in the
future
uh
you know it seems like
we could probably colonize space if we
tried um there are many galaxies there
are many stars uh you know
100 billion stars in the milky way
almost 100 billion reachable galaxies if
we leave now uh
many each of these stars can support a
very large number of
humans or other uh sentient life not
sentient beings uh
and so it's kind of
i mean like the argument for not caring
about that is not gonna be like a
numerical argument like well i only care
about these these beings like one
millionth as much as i care about
current beings the argument has to be
something like i don't care about any of
that for some reason um
and so i just like i'm just like i don't
know man it seems like
that's where the money is or you know
like that's where the value is like
that's what we should be focusing on
with most of our effort
um
i you know i care i personally care
deeply about the welfare of the people
around me whereby around me i mean who
exists now on earth uh
but i think that from an altruistic
perspective like with the proportion of
my efforts that i try to use to make the
world better in an impartial way for
everyone basically all of that effort is
focused on the long-term future
gotcha that makes a ton of sense um
and do you think currently
there are there are areas
specifically on in existential risks
that are that are being kind of
overlooked or are
particularly underfunded at this at this
point or like you know in terms of human
attention shall we say
[Music]
um
i don't know
i i'm probably i'm i'm not gonna have
good answers here i think um
i think that
it's kind of like underfunded with
respect to what i think that the
existential risk community is pretty
good at allocating effort um i i it's
pretty rare
especially nowadays for me to hear about
something that seems absolutely crucial
where like no one is working on it um
that's good
i wish that we had many more people uh i
i wish
yeah i wish we had many more people i
wish we had many more competent
organizations trying to do stuff um i
don't think that there's that much room
for more funding um
but yeah so so i would say i don't have
any like really contrarian takes like if
i
managed 10 billion dollars i think i
would have substantially different ideas
of how they should be spent compared to
the how money is currently being spent
on existential risk that's great that's
great well it does seem like uh
there's a lot of really smart people
working on it which gives me hope on on
the problems that it's not you know uh
you know less smart people looking
working on the puppy pounds so you know
i don't know maybe more of 20 bills on
the sidewalk there i don't know
um
cool uh so
in general
how good do you think human humanity is
at coordination
in kind of absolute terms
i mean it's really hard to say uh
and i think that a lot of you know
there's been a lot of disagreement
within ea about how good humanity is at
coordination
um
and i've you know written about it a
little bit which is probably why you're
asking this question uh
and i think that most of the time the
difficulty comes down to
operationalizing what it means to be
good or bad at coordination um i think
we could point to examples of things
where
humanity did or did not
do particular things like we can all
agree that the manhattan project
happened we can all agree that
uh
the fda did not approve
uh
various creative vaccines months earlier
than they did uh i mean definitionally
uh
and then the question is just like what
you take away from this um
i feel like the main questions about
coordination that i'm interested in one
i'm interested in
i i guess like i'm interested in the
probability that when faced with
problems that really require hardcore
coordination between
various important groups in the future
i'm interested in whether it gets messed
up either by the groups in question
being too competitive and not
cooperative with each other enough or
whether it gets messed up just by
everyone involved making bad mistakes
that are bad by their own lights um
and it feels like to operationalize the
question of how uh
competent people are at coordination
what we'd really have to do is describe
in concrete detail the scenarios in the
future which might come up at which
point it feels like it's really
important that people be cooperative um
and then if we have this really specific
list of coordination problems that might
come up in the future
we could ask uh we could try to think of
historical analogs through these
coordination problems and we could try
to form these analogies we could be like
well i don't know this seems kind of
like the problem of making sure that uh
the u.s nuclear weapons stockpile uh
didn't have all of the
key codes set to zero zero zero zero for
several decades all right you know the
the security codes required to launch
the nukes yeah um
in which case uh
i feel pessimistic about that type of
problem given the us's empirical
performance on the analogy um or you
could argue that the main analogy is
this feels as hard as humans not
launching nukes at each other during the
cold war which we succeeded at
um
and so i think that when we try to talk
about how good humanity is at
coordination the main problem here is
that we really need to operationalize
exactly what types of things we're
asking about and this is really hard
because it involves futurism um and you
know making concrete giving concrete
stories for how things might be in the
future and so i am not super impressed
with any answers
proposed by anyone to this kind of
question historically including myself
of course it's just really difficult
that's a really difficult thing to get
to
um that's cool so
making you know hard predictions uh
about the future i i want to talk about
ai a little bit you know
what do you think agi looks like when it
takes off is it something like robin
hansen's age of end where we just get
good medical imaging technology
we can uh image people's brains and we
can run them fast or we can emulate them
or is it is it something else
so i currently think it's
much more likely that we i think it's
quite likely that we get
de novo agi by which i mean um
intelligent systems that we trained from
scratch rather than by
trying to copy particular humans gotcha
uh
it seems to me like the rate of progress
on
whole brain emulation has not been that
high over the last decade uh there are
several different bottlenecks there uh
at least two of which seem quite hard um
and
so i guess my my i'm i don't know if you
made me give you a number right now i'm
like eighty percent we get
to know ai before holbrook emulation
gotcha that makes sense and uh my uh my
friends in physics say there's some
there's some hard challenges there that
are just difficult to overcome and just
there's some recent there's a recent
news about the
the warm emulation project having gotten
kind of nowhere in you know the last
kind of ten years which is
uh there's just like a lot of things
like then you gotta make a like crazy
microscope and this like crazy
microscope has to be able to like
image things at tiny resolutions and
also when you aim this tiny microscope
at a neuron the neuron catches fire
immediately and so you've got to like
image it real fast before it's gone from
exploding or and so on and so on right
it seems like a
fundamentally quite difficult problem
they're not necessarily entirely
impossible you know uh
agi might also be a hard problem
super cool
um i want to take kind of a left turn
here and
although it is related um what advice do
you have for building a really high
impact career you've been successful at
this um
and and advice is hard to give so i
don't know it's a difficult question but
are there any general takeaways you've
had that you think would be useful for
people
yeah um
i don't know the extent to which i
expect to be successful um i'm i'm
trying this this this redwood research
stuff uh we might succeed and we might
fail
um
just as a caveat i want to give
as you say advice is hard um some things
which some choices that i made that i
think turned out quite well for me which
of course is different from them being a
choice that someone else would want to
make
um i'm very glad to have spent a lot of
time when i was young messing around
programming
uh
i think that i
in unstructured ways where i just tried
to think about what was cool and do
things that i thought were cool
um
this has turned out pretty useful
because it means that i'm now quite
comfortable uh
making up solutions to computer science
problems uh when i was
i guess 21 and 22 i uh
submitted talks to this
industry scholar conference where you
know it was the kind of thing where you
like submit the abstract before actually
doing the project and then i got in and
so i was like i guess i have to actually
do this now and i i i did this whenever
i spent 200 hours each year on or
something on this like dumb
dumb programming project uh but it was
fun it was a it was legitimate research
i
learned how to do research on my own i
learned how to stare at whiteboards for
hours on end and like slowly piece
together
designs
um and
that made me happy uh
and i think has been like a useful set
of skills for later
um
i'm glad to have spent a bunch of time
trying to think through ea concepts you
know effective autism concepts um i'm
glad to spend a lot of time arguing with
people about
uh
the form of the ai alignment problem um
a bunch of things about ethics and how
we should feel about the long-term
future i think that time has been well
spent um
i'm really glad to have
been down for doing things i feel like
some people
uh
i feel like there have been a bunch of
points in my career where people asked
me to do a particular job
like for instance a bunch of recruiting
and outreach stuff at mary and i
you know this wasn't particularly the
job that i had signed up for uh
but
i in hindsight i'm really glad that i
did it i think that i've learned a lot
from like taking these opportunities
that weren't like
super obvious fits um but were just like
a way i could produce value right then
similarly when i was a software engineer
i was working at
triple byte um which is a startup and i
found it really useful to just try and
produce value whatever way i could i you
know i developed aspects of the
programming interview they were giving
people and i
made
web interfaces for sending emails more
efficiently and i just did a lot of
random stuff
and i think that i got more value out of
that experience uh because i was doing
kind of random stuff that i hoped would
produce value compared to how much i
would have gotten if i had said you know
no i am just a back-end engineer or a
full-stack engineer i just really want
to be doing backhand or full-stack
engineering all the time
i don't know those are some thoughts
that's great you know do you know bin
 from wave
yeah so he he said something similar he
said um you know
he learns he's learned quite a lot
because oftentimes the most valuable
thing for him to be doing at wave will
be like okay i've got to fix this
obscure you know
accounting problem or you know i'm an
engineer being a software engineer and i
think i'm really good at what i do but
you know i'm just building a crud app
but this is super valuable for you know
people um so i think that's great advice
you know lean into what you think can be
valuable at any given time
yeah i don't know if it's great advice
for everyone
for some people
yeah
that's great that's great yeah um
you wrote something on your blog which i
found really interesting i i'd like to
just you know have you expand on a
little bit uh you mentioned that you're
less sure nowadays that um us getting
wealthier is like a slam duck positive
thing
as a society what do you think about
that
um
i think that it's confusing
uh i think that
there is a lot of
suffering caused to farmed animals
that did not used to get caused to harm
animals
since the 60s
animal agriculture has gotten much more
intensive the price of animal products
has fallen substantially
and the average experience of being an
animal in one of these farms um has gone
worse way down
and i think this is one of the greatest
moral tragedies of our time um
and
i think that this should give us some
pause um i think that there is a certain
if you if you just ask the question like
let's let's define human civilization to
include
uh all the stuff directly influenced by
humans in a certain sense uh
then it feels like
uh
i mean it kind of depends how much moral
weight you assign to chickens compared
to how much you assign to humans
but i think there's a very plausible way
to do the calculation under which the
main thing which has happened over the
last 70 years is that many more animals
are tortured in farmers
um
and that is not good
and
feels like it gives me pause about
whether i want to be like yes technology
has obviously made things better
that's right that's right is that that's
a great point is it's not always it's
not automatically good i think that's a
really important point and also you know
you know what has happened to farmed
animals over the last you know 40 years
is is not it's not good you know there's
a lot of suffering that goes on uh
because we you know it's hyper
competitive we're trying to feed
everybody everybody wants to eat more
protein and or animal protein and kind
of negative consequences from that
yep
so compared to
so i think that the actual overall
effect of humans getting wealthier uh as
evaluated on uh in terms of short-term
welfare is actually kind of uh ambiguous
because of the fact that we didn't
include wild animal suffering um in the
previous in the previous calculation so
another major change which has happened
over the last 70 years is that uh there
are many fewer wild animals
um of certain types uh and there are
more of some other types i think uh
though i don't know the numbers on this
very well and so like whether the
average experience or whether the total
uh welfare of all the individuals on
earth weighted by moral importance has
gone up or down is going to turn out to
like rest on something about what it's
like to be a tiny fish or something you
know there's just really a lot of tiny
fish out there and if it's if the tiny
fish are having a more bad time than
they used to have then maybe that's just
like by far the most important fact
about the last 70 years from a short
term is to perspective i find this
plausible um
and so then it's like well i don't know
what what should what moral should we
take from this for the future
uh
i think
it's unclear
there's definitely like one one way you
could think about this is like well
humanity sure did a lot of stuff
and even though it's not you know in
1950 we knew that there were tiny fish
in the ocean uh and humanity was not
very careful about this humanity didn't
say like well you know uh before we
continue doing our economic growth we'd
better really think about tiny fish and
like really think about whether we're
making the world worse for the tiny fish
as we do our economic activities they
didn't care about that at all uh and you
might extrapolate this forward to the
future and you might say well uh in as
much as humanity gains more power and
influences more
uh more total like uh computation that's
going it like influences it expands to
all these other stars or these are other
planets maybe we should be afraid
because last time humanity wildly
expanded its range of influence this
went badly uh if it in fact did and if
it didn't grow badly then it didn't grow
badly by coincidence uh though there was
no one at this wheel we just like
pressed the gas uh and maybe it was okay
like maybe inner children were hit but
like that doesn't necessarily entirely
um
entirely
comfort me
so that's the pessimistic story uh
i think that the optimistic story has
two parts one of the parts is the kind
of object level part it's like um
the reason that this went badly was
at least in the at least in the animal
agriculture case i think the wild animal
suffering case is less clear
i think
the reason that animal agriculture has
resulted in a net decrease in total
welfare uh is kind of like
a weird feature of um
it's like kind of a coincidence and i
think it goes away as everyone gets
wealthier like it so happens that
currently the cheapest way to make
things that have all the same properties
as ground beef uh is to raise some cows
and then kill them um
raise some cattle and then kill them uh
but i suspect that in fact if it was
and it turns out the cheapest way to do
that is in a way that seems pretty bad
for the cattle involved but i suspect
that most people actually have sort of a
preference for cattle being well off and
i expect that if technology just like
proceeds in normal ways then animal
agriculture will become wildly less
widespread
and also will probably treat the animals
a lot more nicely because it'll just be
cheap too and like consumers like kind
of want that and enough people are in
favor of this that we will probably pass
legislation eventually so there's this
one argument which is like well you know
things went badly last time humanity got
a lot more powerful but in a way that's
gonna get better as they get more
powerful again so maybe it's just
totally fine so so that's like one
argument for why we shouldn't be worried
about the future being bad it's like
well this the way that things were bad
this time was kind of like a weird
one-off just like related to the fact
that there was like a bunch of animals
around that you can raise from meat and
you can like make them kind of unhappy
while you're doing that
and here's the other argument for why
you might think the future might be good
uh it's the you know we were talking
about how ai might really change the
world a lot by reducing the price of
certain types of labor
one of the types one of the ways that
you think this might influence the world
is it means that people will have access
to better advice and better reasoning
so
sometimes i'm not sure what i think the
right thing to do is about some problem
um and i currently can't you know call
up my extremely intelligent ai advisor
and say like hey hey do you want to like
tell me about some like moral facts
about the world that i'm missing like
what's some stuff which like
is bad that like i should be worried
about like am i am i like you know am i
like
and you might hope that if it was like
you know the 1930s and you and like you
had this ai assistant it might be like
look man you're being like really
homophobic and i think that if you like
think about this uh it's going to turn
out that like by your own values your
own beliefs you're going to wish you
were less homophobic and i'll you know i
go off and think about it i'm like yeah
you know what you're right ai i should
be like less homophobic and so you might
hope that like um
as everything gets cheaper one of the
things that gets cheaper is good moral
advice uh and people will want to
consume good moral advice uh and then
they'll consume more of it and then
we'll have a future which is more
enlightened um
and maybe this just like makes things
maybe this just means that the future uh
is more good than the past because
you know the pricing in latin goes down
and so the quantity purchased goes up
absolutely it's just a lot it's a lot
easier to understand and get that
information
uh
i i'm curious do you think
the moral circle
or like our moral circle is it do you
think it's fairly fixed or is it
something you think we can actually make
real we can really expand
the example i give is is people you know
they used to
they do these surveys people used to get
really concerned about their their kids
marrying a a child of a different race
and now that's just shifted over you
know people don't worry about that as
much anymore now they worry about their
kid marrying someone of a different
political affiliation at the same rates
do you think this is something where we
can like expand like expand our moral
circle is it like kind of fixed and
maybe we can optimize on the edges but
there's only so much we can care about
at any given time
yeah i mean like
i think that something weird about that
example is it's not totally clear that i
have a hesitation about
my uh my children marrying someone of
class x it's not totally clear that this
is the same as i do not care about the
welfare of people in class x though
probably in both cases it's in fact
correlated um
i think my guess is that
moral circles
have in fact
gotten larger in the way that i care
about over time
um
i definitely don't think
yeah
i think that probably there's like two
phenomena here one of them is like
i don't know do you know scott alexander
is the in group the out group and the
far group thing
uh i do not i know in group alcohol with
the fart far group
uh i i might be misquoting him but the
basic claim is you know there's a bunch
of uh democrats who really hate
republicans but who uh don't really have
an opinion on a bunch of people who live
in saudi arabia um
and even though the republicans have
values much closer to them than they
have to the people who run saudi arabia
um
and
and and this is just because you know
there's like a narcissism of small
differences thing you know you you get
an if you're a democrat you like get
into more fights uh with with
republicans uh
and so my best guess is that the time as
time has gone on people's
in groups of
a lot larger people's out groups have
gotten somewhat larger but people's far
groups like these groups of people who
they like don't interact with that much
uh but like are aware of the existence
of and could care about the welfare of
have gotten like way bigger um
and so i i think i'm not that worried
about like the size of the people you
care about marley being fixed over time
gosh that i i think that's that's really
well put and also i think another
example which i think
you gave earlier which is helpful is if
people had option you know access to
ground beef that costs the same or less
but it's like this is cultured and we
didn't torture any animals you know most
people it seems like
i have an intuition most people would
pick that you know they're like yeah
sure you know like if there's an easy
choice it's especially cheaper if it's
cheaper like they're picking down
you know nine times out of ten no
worries
um that's super cool
um
another left turn you know
how should we think about criticism you
know should we seek out more of it in
our lives you know in our careers and
would that be a valuable thing for
people to do
i think it's hard to say and kind of
depends on how much criticism you
currently seek out
um
i think that
so there's this question which is like
how do people currently decide how much
criticism to receive and why would they
be making a bad decision about this
criticism
um
i think that roughly speaking uh
i because like i don't know uh if you
ask me the question like how much should
people you know how much should people
i don't know how much should they buy
chairs if i wanted to argue that they're
buying too many chairs or too few chairs
i kind of got a point to why i think
they would make that mistake you know
like what mistake i think they're making
and so similarly when we're talking
about how you should be
trying to
uh solicit criticism uh we have to talk
about what mistake i think people might
be making um because i think there's
some sense in which people on average
don't make mistakes uh or or like i
think you've got to do some work to
claim they're making a mistake yeah um
i think that the core reason to claim
that the people i hang out with uh
should consider soliciting more
criticism than most people solicit
is that
uh i think that
criticism is kind
like criticism has kind of a couple of
different social roles one of them is to
point out to someone that they're making
a mistake and another one is to hurt and
belittle them in front of other people
uh and to
uh
like
uh demonstrate dominance over them and
so i think that by default uh like
default human society uses criticism for
these two different things um
and i think that if you think that you
can decouple that for yourself or in
your in a particular relationship if you
can make it be the case that when people
provide you criticism um
they are not you know trying to hurt you
uh and you are not going to respond as
if they were trying to hurt you and
you can just uh i don't know get their
actual thoughts on how things should be
different i think that this is a
somewhat valuable thing to get done um
yeah i could try to give more
theoretical thoughts on how
why people might not receive enough
criticism um i also have takes on like
how to go about getting better criticism
or whatever
yeah how about the actionable you know
like how would should one go about
getting you know better criticisms let's
say like you know for this podcast like
how should i go about getting better
criticism for it
so i think the key thing to do uh i
think my core thing here is probably
going to be uh
making it so that you successfully
signal that you in fact do want the
criticism and making it really easy for
people to in fact offer the criticism so
recently i decided that i wanted some
criticism on some topics uh and so
i haven't actually executed on this plan
yet uh or like finished executing on it
um but one thing i've done that i feel
pretty optimistic about uh when i
finish it uh is i started writing down a
list of ways where i think i could be
like more one way or the other way um
and instead of soliciting instead of
asking people if they had thoughts on
how i should be different in general um
i feel pretty excited about asking
people you know
whether they think that you know i want
to like point out this this feature of
how i am that is kind of a strength and
kind of a weakness and then i want to
say do you think i should have more or
less um
and i feel like this is
probably going to be better at
soliciting people's thoughts than asking
them for feedback miscellaneously
and there's a couple reasons for this
but one of them is i think that
sometimes i want people's feedback on
some things and not other things uh for
example i think there are a lot of
people where i'm interested in hearing
them tell me that they think i should be
slightly more careful with my
programming but i'm like less interested
in hearing them say that they like
think that i should dress better because
i'd be more attractive or whatever right
um
i mean there are other people from who
would be useful to hear that um but
there's a lot there's a lot of other
people where i don't really want to
engage in that kind of thing with them
um and so if you give them a clear
delineated list of topics on which you
are definitely soliciting criticism um
and then they don't even have to come up
with these criticisms themselves they
just have to opine in a direction one
way or another uh i think it's much
easier for them to to our time and i
think in some cases this
uh makes it go better gotcha so it's
like giving them like a concrete you
know area it's like uh you know examine
my audio quality like is auto quality
good or bad like how can i make it
better that makes it much easier for
people to like grab onto something that
instead of just like how is it you know
people are like i don't know god yeah
yeah i think another version of these is
like suppose i could work on one of the
four things about my podcast i could
work on my turn i could work on my audio
quality i could work on the quality of
my guests i could work on the miscellany
like quality of the transcripts yeah
which one of these do you think will
have the largest impact per unit effort
uh i think you'd get something more
interesting from that in a lot of cases
just asking for miscellaneous feedback
that's that's very wise and how do you
go about signaling that you know you
actually want feedback in your
in a in a good manner just do you think
you just tell people explicitly or
people you trust or
you know how would you think about it
i think that probably the
most important ingredient here
i don't i think there are some basic
things i think that when someone offers
you criticism you should say thank you
for the criticism
um and then you should
default to being quiet um
i think
but
aside from some basic etiquette things
like that yeah um i think that
the main thing is to actually
uh
only ask for
criticism when you are actually in a
mode where it's healthy for you to
receive it um i think that sometimes
people should in fact not be soliciting
criticism um
i think that if you are a i mean i think
this is obvious to imagine in the case
of children i think sometimes children
are in school and they're learning
something and
i think that you could give them a
variety of true criticisms of their
essay
that would not cause them to
in
10 years be writing better essays
um and i think this is true for adults i
think that at various points in my life
i'm doing some stuff and i just do not
have enough
self-esteem at that moment uh to be into
to be interested in hearing a bunch of
people's like
wild thoughts on what i am maybe
screwing up um
and i think that to be clear uh
it is very good to normally be in a
state where you in fact do have the
self-esteem uh to solicit people's wild
thoughts on what you're messing up um
because they might have good stuff
but
i think that by default you're not in
that state i am currently not in that
state you know i there are various
people who i know who probably could
give me some good criticism if i asked
and i am currently
not feeling good enough to do this right
now just like today uh
and
i think that
yeah i think that sometimes i'm worried
that people try people are in a state
like i am in today where they can't in
fact solicit criticism well um
and then they say to their friends oh no
i am very open to criticism uh
and then it just like screws everything
up because they're not telling the truth
um or maybe they're not intentionally
lying but they're wrong and i think this
makes their friends uh
less likely to give them good criticism
um maybe the friends do give them good
criticism and it's in fact bad for them
uh so i think that being able to like
really convince someone like no i am
actually deeply interested in your
criticism right now and i'm not worried
i'm not just saying this out of a sense
that like a good rationalist should be
open to criticism i am saying this
because inasmuch as i am making mistakes
i want to have a good list of those
mistakes so that i can behave
differently
so all this to say i think like the key
aspect of getting criticism better is
actually being able to receive that
criticism i think people are really good
at detecting this kind of stuff uh and
so
uh the the first part of the work is you
know is within or whatever
i love that i think that that's super
actionable and i i think that's i i
really love that um got one more big
question for you
so this is from our local acx meetup
if someone wants to get involved in ai
safety research you know how would you
suggest they do that like what what's a
what's a good path you think
so it really depends on who they are um
what their what their background is um
there are many different things that get
done that get called ai safety research
yeah um
and so yeah i don't know
i'll say some stuff um i think
that
of jobs available in ai safety technical
research in five years um i think that a
bunch of them are going to be
software engineering and machine
learning research related to
various
applied ai alignment stuff um
part of the point of redwood research is
making it so that we're like pushing
forward the infrastructure required to
ensure that it's possible to do great
work by employing people in those things
so yeah i think that like one part is if
you
feel like you could plausibly be a great
employee uh
doing
ml research or uh like back-end web
infrastructure for ml research um which
i think you know if you're just like a
enthusiastic and energetic and widely
knowledgeable and capable and fast
python back-end web programmer uh that's
like a lot of the skill set there um i
think that developing those skills uh is
like a pretty is like a pretty
reasonable way to try and seek out these
jobs uh and then it really helps to
apply to these jobs
[Applause]
drastically increases the probability to
get one of them um
i think some other i don't know i think
the other main class of activity you you
ought to do if you want to work on ai
alignment technical research is thinking
a bunch about ai alignment um there's a
bunch of great resources on ai alignment
these days i think that uh cambridge the
cambridge effective altruism club has a
really great course called ai safety
fundamentals um where they this is an
online course you can apply to uh and
you you know it meets once a week with a
facilitator um
and i they discuss a bunch of
fundamental questions related to ai
alignment uh i think that doing that
kind of course uh is like a great
starting point for
having thought through a bunch of
fundamental questions about like what is
the problem and what types of research
ought to be done to
to make the problem go better um and
then getting deeper into it all from
there uh yeah is like the the other part
of the yeah so i've described like two
aspects of getting into ai alignment
technical research one is developing
technical skills that you might be able
to use in some of the applied research
which i suspect is where the majority of
the jobs are
and two is uh thinking a lot about ai
alignment from a kind of fundamental
perspective which is useful both for
being better at the first class of jobs
and also for taking some of the other
types of ai alignment technical research
jobs
which involve doing that kind of
fundamental thinking all the time
gotcha gotcha that that's great that's
great and i think that's a very uh it's
a very
achievable path and it makes sense it's
legible like you know get here work on
these things get better
and that's where the skate to the puck
work to where the puck will be it's very
wise
i should note that redwood research is
hiring for researchers and software
engineers so you can uh go to our
website or email me to learn more about
that great point great flag oh and well
with that buck uh where would you like
to send people
where i like to send people yeah
all the uh
listeners of this thing where on the
internet should they go or some other
classes yeah yeah well yeah where do
they should they go
well
i don't know i like redwoodresearch.org
um
as a uh as a website um one of my other
favorite places on the internet i don't
know i like the effective altruism forum
um i like uh
i like
uh
lesswrong.com has some good stuff
um
i thought that worth the candle was a
really good piece of rationalist fiction
uh
i think that
the ai alignment forum has some pretty
good content on ai alignment i think
eighty thousand dollars.org has a bunch
of great stuff about how you should
think about um optimizing your career to
do as much good as possible um and it's
a great source of a lot of actionable
advice
um those are the main things that come
to mind i don't know got any categories
i should think about uh particularly you
know your stuff like if people want to
find your stuff they they really
resonated with this podcast where
redwood research you mentioned one
um anything else yeah
uh i think that uh my work
my writing has mostly been published on
the ai alignment forum um
and the effective altruism forum and
sometimes less wrong um i i'm sure you
can provide links to those things
in the podcast description or whatever
yeah that's that's where it mostly goes
sometimes it's on the uh sometimes it's
on my website
excellent excellent well buck uh thank
you so much for coming on i really
appreciate it
thanks for having me
special thanks to our sponsor bismarck
analysis for the support bismarck
analysis creates the bismarck brief a
newsletter about intelligence-grade
analysis of key industries organizations
and live players
you can subscribe to bismarck free at
brief.bismarckanalysis.com
thanks for listening we'll be back next
week with a new episode of narratives


What is AI alignment?

what is a eye alignment a eye alignment
concerns the task of making sure that
artificially intelligent systems behave
in a way that is conducive to human
morality and human well-being the
problem of a eye alignment arises from
what is called the orthogonality thesis
first put forward by Oxford philosopher
Nick Bostrom this is the idea that
intelligence and goals can exist in any
combination in other words it means that
just because an AI is as intelligent as
a human that doesn't mean we should
assume that it will think or behave the
same way follow the same rules or act to
achieve the same ends we used to the
science fiction trope seen in movies
like Terminator where AI becomes
self-aware before deciding to take over
the earth and kill all the humans now
this depictions probably do not
represent how an actual a I related
disaster would unfold because they
anthropomorphize the AI assuming it
would behave like a human when an actual
a I would probably behave quite
strangely
one of the most famous things to come
out of a eye alignment is the paperclip
maximize a thought experiment first
dreamed up by AI theorist laa you'd
Kowski before being elaborated and
popularized by Nick Bostrom this thought
experiment attempts to show us how a I
can pose a danger to us even if its
goals seem wholly banal boström asks us
to imagine an AI whose sole task is to
make as many paper clips as possible we
might think that with such an innocuous
goal it would be impossible for the AI
to get up to any mischief however
imagine the AI realized that it could
make a lot more paperclips by waging
nuclear war to obtain additional
resources for example the a I might even
develop nanotechnology that would allow
it to turn the entire universe into
paperclips probably not something we'd
be pleased about you know because we
will be dead
should we be worried about a eye
alignment
some critics like philosopher of mind
Daniel Dennett and psychologist Steven
Pinker think that fears about dangerous
AI are overblown and that we shouldn't
spend too much time dwelling on such
hypothetical scenarios on the other hand
figures such as the neuroscientist Sam
Harris and the late physicist Stephen
Hawking think that we risk destroying
the human race by means of AI if we
don't take precautionary measures as
soon as possible
Sam Harris compares the hypothetical
advent of dangerous AI to some kind of
alien race that may or may not arrive on
earth at some point in the next century
so what could we do about the prospect
of an out-of-control artificial
intelligence people often like to cite
writer Isaac Asimov's famous Three Laws
of Robotics as a basis for reining in AI
the three rules state one a robot may
not injure a human being or through
inaction allow a human being to come to
harm - a robot must obey the orders
given to it by human beings except where
such orders would conflict with the
first law and three a robot must protect
its own existence as long as such
protection does not conflict with the
first or second laws now these laws
suffer from a number of pitfalls
including the definitions of human and
robot the assumption that a robot knows
what all the consequences of his actions
might be and the potential requirement
for robots to incessantly roam the earth
trying to cure all human ills some have
advocated putting the AI in a box that
is sealing the AI in some kind of
enclosure that allows us to communicate
with it
but does not allow it to carry out
actions in the outside world the major
flaw with this approach is the
possibility that the AI will convince
someone to let it out
a human being is not a perfectly secure
system and so an extremely intelligent
AI ought to be able to find some way to
hack a human mind and use its influence
to get itself out of the box
Nick Bostrom has suggested creating a
particular kind of AI that he calls an
Oracle this AI would only be able to
answer questions and would not be able
to form goals or take actions in the
real world some potential downsides of
this design are that the AI might lie to
us or it might think that our questions
are too vague for it to be able to
answer we might even need to build
multiple Oracle's and then compare their
responses to see if they're giving us
the correct answers another thing we
could do is to use the a eyes own
knowledge and intelligence to get it to
constraint itself for example we could
tell the AI to behave in the way that it
thinks the majority of humans would
ideally want it to behave or something
along those lines one thing's for sure
AI alignment is an area of research that
remains embryonic a great deal of deep
thought and hard work will be required
to figure out how best to constrain
artificial intelligence and the time to
complete that work could be running out
fast we hope you enjoyed this video
please like subscribe leave a comment
and consider donating to our patreon so
we can continue to help you learn about
what's in store for all of us in the
future


How to Keep Improving When You're Better Than Any Teacher - Iterated Distillation and Amplification

hi today we're going to talk about
iterated distillation and amplification
so let's say we want to play go and we
want to be really good at it so we're
trying to create a function which if
we're given a go board in a particular
state we'll take that board state as
input and return a very high-quality
move that you can make from that
position what we're trying to create is
a policy a function which Maps States
onto actions suppose that what we have
though is something just slightly
different suppose what we have is our
intuition about moves which takes a
board state and it gives us for each of
the possible moves we could make some
sense of how good that move would be we
can think of this as an action value
function which assigns a real number to
each move which represents how good we
think that move is alternatively we can
think of it as outputting a distribution
over all possible moves so for a human
player this represents your intuition
the go player looks at the board state
and says it looks like maybe this move
might be good this move probably is a
bad idea
this one looks ok you could also have a
neural network which takes the board
state and a possible move as input and
outputs how good it thinks that move is
ok so how do you get the understanding
of the game that allows you to evaluate
moves well as a human you can study the
rules and watch some games played by
people who are better at go than you are
if you have a neural network then it's
also fairly straightforward you can
train the network with a large number of
high quality human played games until
its output gives a good prediction of
what a skilled human would do so
strictly speaking in that case the
network isn't evaluating how good a move
is it's evaluating how likely a good
player would be to make that move but
that can be used as a proxy for how good
the move is once we have this action
value function there's a pretty obvious
way to turn it into a policy which is
just Arg max you look at all of the
moves with your intuition or evaluate
them all with the network find the
best-looking move the move that's
highest rated and use that but if you
have more time to think or more
computational resources you can do
better rather than just going with your
first instinct about what you think is
good
you could play forward a few moves in
your head you might think okay from this
board state it looks like this move
would be good what does the board look
like if I play that and then
you can apply your action value function
again from the perspective of your
opponent often there'll be more than one
move that looks promising so you might
want to consider some of the
best-looking moves and then apply your
action value function again to think
about how you might respond to each of
them and so on exploring the tree so
what you're effectively doing here is
tree search right you have a game tree
of possible moves and you're searching
through it deciding which branches to
search down using your action value
function you can keep doing this for
however much time you have it might be
that you think far enough ahead that you
actually get to the end of the game and
you can see that some move is clearly
good because it wins you the game or
some other move is clearly bad because
it causes your opponent to win the game
well you might just look a little bit
ahead and try to evaluate where you are
you might look at the general quality of
the moves that you have available to get
a feel for whether this is a state you
want to be in or one you want to avoid
and after you've done all this thinking
you might have learned things that
contradict your initial intuition there
might be some move which seemed good to
you when you first thought of it but
then once you actually think through
what your opponent would do if you made
that move and what you would do in
response to that and so on that the move
actually doesn't look good at all so you
do all of this thinking ahead and then
you have some way of taking what you've
learned and getting a new set of ratings
for the moves you could make and this
can be more accurate than your original
action value function for a human this
is this kind of fuzzy process of
thinking about moves and their
consequences and in a program like
alphago or alpha zero this is done with
Monte Carlo tree search where there's a
structured way of extracting information
from this tree search process so there's
a sense in which this whole process of
using the action value function
repeatedly and searching the tree
represents something of the same type as
the original action value function it
takes a board state as input and it
gives you move evaluations it allows us
to take our original action value
function which on its own is a weak
player and by applying it lots of times
in this structured way we can amplify
that weak player to create a stronger
player so now our amplified action value
function is the same type of thing as
our unamplified one how do they compare
well the amplified one is much bigger so
it's more expensive
for a human it takes more thinking time
as a program it needs more computational
resources but it's also better than just
going with a single network or the
single human intuition it smooth
evaluations are more accurate so that's
pretty neat we can take a faster but not
very good player and amplify it to get a
more expensive but stronger player
there's something else we can do though
which is we can take what we've learned
as part of this process to improve our
original action value function we can
compare the outputs of the fast process
and the amplified version and say hmm
the quick process gives this move a high
rating but when we think it all through
with the amplified system it turns out
not to be a good move so where did the
quick system go wrong and how do we fix
it if you're a human you can maybe do
this explicitly perhaps you can spot the
mistake that you made that caused you to
think this was a good move and try to
keep it in mind next time you'll also
learn unconsciously your general pattern
matching ability will pick up some
information about the value of making
that kind of move from that kind of
position and with a neural network you
can just use the output of the amplified
process as training data for the network
as you keep doing this the small fast
system will come to reflect some of what
you've learned by exploring the game
tree so this process is kind of like
distilling down this big amplified
system into the quick cheap to run
system and the thing that makes this
really powerful is we can do the whole
thing again right now that we've got
slightly better intuitions or slightly
better weights for our network we can
then amplify that new action value
function and this will give us better
results firstly because obviously if
your movie valuations are more accurate
than before then the move evaluations at
the end of this process will be more
accurate than before better quality in
better quality out but secondly it also
allows you to search the tree more
efficiently if your intuitions about
move quality are better you can spend
more of your time looking at better
parts of the tree and less time
examining in detail the consequences of
bad moves that aren't going to get
played anyway so using the same extra
resources the new amplified system is
better than the previous amplified
system and that means that when it comes
to the distillation phase of learning
from the exploration there's more to
learn and your action value function can
improve again so it's a cycle with two
stages for
to amplify by using extra computational
resources to make the system more
powerful and then you distill by
training the fast system with the output
of the amplified system and then you
repeat so the system will keep on
improving so when does this process end
well it depends on your implementation
but eventually you'll reach a fixed
point where the fast system isn't able
to learn anything more from the
amplified system for simple problems
this might happen because the
unamplified system becomes so good that
there's nothing to be gained by the
amplification process if your action
value function always suggests the
optimal move then the amplified system
is always just going to agree and no
more learning happens for harder
problems though it's much more likely
that you'll reach the limits of your
action value function implementation you
hit a point where a neural network of
that size and architecture just isn't
able to learn how to be better than that
by being trained on amplified gameplay
as a human even if you could study go
for infinite time eventually you'll hit
the limits of what your brain can do the
point is that the strength of the end
result of this process isn't limited by
the strength of the initial action value
function the limit is determined by the
architecture it's a fixed point of the
amplification and distillation process a
version of alphago that starts out
trained on amateur level games might
take longer to train to a given level
than one that started out trained on
grandmaster level games but after enough
training they'd both end up around the
same strength and in fact alpha zero
ended up even stronger than alphago even
though it started from zero using no
human games at all so that's how you can
use amplification and distillation to
get better at go and why as a software
system you can keep getting better even
when you have no external source to
learn from even once you leave humans
behind and you're the best go player in
the universe so there's nobody who can
teach you you can still keep learning
because you can learn from the amplified
version of yourself ok so why is this
relevant fire-safety well we've just
talked about one example of iterated
distillation and amplification the idea
is actually much more general than that
it's not just for playing go and it's
not just for Monte Carlo tree search and
neural networks amplification might be
this kind of process of thinking ahead
if you're a human being it might be
Monte Carlo tree search or something
like it if you're a software system but
it might be something else if you are
for example an age
I it might involve spinning up lots of
copies of yourself to collaborate with
or delegate to so that the team of
copies can be better at solving the
problem then you would be on your own
for some types of problem it might just
involve running your mind at a faster
rate to work on the problem for a long
period of subjective time the core
characteristic is that amplification
uses the original process as a starting
point and applies more computational
resources to create a more powerful
agent in the same way distillation can
be any process whereby we compress this
more expensive amplified agent into
something that we can call cheaply just
as we call the original system for a
human playing go this can be the way
your intuition gets better as you play
for a neural network playing go we can
train the action value network to give
the same outputs as the tree search
process for an AGI it could involve the
AGI learning in whatever way it learns
how to predict and imitate the team of
copies of itself or the accelerator
version of itself or whatever the
amplified system is the core
characteristic is that the cheaper
faster agent learns to approximate the
behavior of the more expensive amplified
agent so these two processes together
define a way of training a stronger
agent from a weaker one the hope for
safety research is that we can find
designs for the amplify and distill
procedures which preserve alignment by
which I mean that if the agent we
amplify is aligned with our goals and
values then the amplified agent will be
aligned as well and if the amplified
agent is aligned then the agent we
distill it down to will be aligned as
well in the next video we'll talk about
some ideas for how this might be done
I want to end this video with a big
thank you to all of my wonderful patrons
that's all of these fantastic people
here who have been just so generous and
so patient with me thank you all so much
in this video I'm especially thanking
Sayed Polat who joined in December just
before the start of this gap in uploads
and the reason for that is I've recently
really had to focus on the road to AI
safety excellence the online course I've
been working on in fact the video you
just watched is the first lecture from
our module on AI da which hasn't been
released yet so I also want to thank
everyone at the Rays Project for their
work on the script and the research for
this video and really the whole raised
team I'm still making content just for
this channel as well and in fact I have
one that's nearly ready to go so look
out for that thanks again for watching
and I'll see you soon


AI governance landscape | Carrick Flynn | EA Global: San Francisco 2018

In my previous career, I worked in global
poverty eradication, mostly doing field work.
This meant working on projects all around
the world, trying to find a place where I
felt I could do the most good and have the
largest impact.
A little over three years ago, while I was
working in Ethiopia, I discovered the effective
altruism movement.
When I did, I discovered just how big the
biggest impact could be.
Think for a second about a single grain of
sand.
Imagine rolling it between your pinched fingers.
Picture how small it is.
Now, think about how much sand there is on
earth, all the deserts: the Sahara, the Gobi,
the Arabian.
The ocean floor, all the beaches, all of the
dunes.
If you press your palm to the sidewalk in
most places, when you come back, there will
be little bits of sand embedded in it.
There are a lot of grains of sand on earth.
In fact, there are approximately a sextillion
grains of sand on earth.
That's a billion trillion.
Take a second to imagine that grain of sand
pinched between your fingers again.
Now, try to imagine that grain of sand is
everything of value on earth.
All the people, all the animals, everything,
and hold that thought.
There are 100 trillion reachable stars.
We have approximately 10 trillion centuries
before entropy makes the universe difficult
to inhabit.
That means we have over a trillion trillion
equivalents of earth's resources within our
reach.
That means if we imagine everything on earth
as a grain of sand, there are more earth equivalents
out there, but within our reach, than there
are grains of sand on earth.
The human brain is not good at thinking clearly
about issues of scale.
When we think about doing good, we tend to
think in terms of prototypes.
I've always been most motivated by human wellbeing,
so when I think about doing good, I tend to
think of helping someone.
Effective altruism is, at its core, about
making sure we don't stop with this prototype
and this emotion.
It's about trying to harness this, this altruistic
motivation to have an effect in the real world.
For many of us, there's something a bit disconcerting
with interacting with far future considerations.
The prototype's kind of abstract.
It's sort of weird.
It transects a lot of science fiction and
transhumanism and things we may not really
be that comfortable with or into.
We care about human suffering.
We care about animal suffering.
And those things exist here, now, on earth.
We can see them.
I am very sympathetic to this situation.
I am in this situation.
However, the reality is that time and space
exist.
They will be filled with something.
Maybe they're filled with dead matter, completely
devoid of life, and maybe that seems like
a tremendous waste.
Maybe they're filled with terrible, exploitative
human societies, and full of factory farms,
and that seems like a disaster.
If what you altruistically value exists in
time and in space, and you want to do the
most good, you have to think big.
I may have this grain of sand in my fingers,
but if I care about doing the most good, I
can't focus on just this one grain of sand.
There is a lot of sand out there.
Humans have done three things in history which
stand above the others in terms of their impact
on our own wellbeing and the wellbeing of
animals.
The first was our developing behavioral modernity.
This occurred about 70,000 years ago.
The leading theory for behavioral modernity
was that it was the result of very small cognitive
changes.
Behavioral modernity allowed us to outlive
the neanderthals, who had been preventing
us from gaining a bridgehead to leave Africa.
With this, we were able to expand across the
world.
As we did so, we caused the extinction of
most megafauna on earth, and of all other
species of humans, including the neanderthals.
The second was the development of agriculture,
which occurred around 12,000 years ago.
This technological advance increased human
population about a hundredfold.
It resulted in dramatic ecological damage,
the exploitation and extinction of animals,
and the decrease in human health and wellbeing.
The third was the industrial revolution, which
came with further ecological destruction and
the advent of factory farming.
But, it also created dramatic wealth and resulted
in the first sustained improvements in human
health.
I'll also give a quick honorable mention to
atomic weapons, which haven't had much substantive
impact on earth, but definitely have that
capacity.
Our intelligence, and the technologies we
invent, are how humans have our biggest impact.
For this and for other reasons, it is safe
to imagine that the technology of intelligence
will be at least as impactful for humanity
and for animals as the industrial revolution.
AI in its current state is not that impressive.
It recognizes images well and it's good at
some games, but it's easy to get lost in the
details at the beginning of a large change,
and to miss the forest for the trees.
The earliest stage of the development of behavioral
modernity was extremely subtle and mostly
looked like - most likely - mostly looked
like slightly improved communication within
human bands.
This took place over the course of thousands
of years.
The line between early agriculture and certain
types of seed-gathering techniques is so subtle
that modern anthropologists can't always agree
about what's going on with modern bands of
humans.
Steam engines were adopted slowly and were
very unimpressive for a long time.
With agriculture and with industry, humans
living at the early stages could not have
conceptualized what they would develop into.
Fortunately for us, with intelligence, we
do have one model of what it looks like when
it reaches an advanced stage.
The difference between humans and gorillas
is a few million years of evolution.
Gorillas now only exist in zoos and a few
little reserves that we keep them on so that
we can look at them.
The difference in intelligence between humans
and neanderthals was, as far as we can tell,
very small, and as soon as behavioral modernity
gave us a little edge on them, we wiped them
off the face of the earth.
For the first time, we can kind of see a dim
outline of an enormous change on the horizon,
and it is coming at us.
Also, for the first time, this change is not
just going to affect our one unimaginably
valuable grain of sand.
This is for all the sand.
The goal of AI governance is to help humanity
best navigate the transition to a world with
advanced AI systems.
It is possible that the default course of
development and deployment of advanced AI
will be great.
There is an optimistic case that can be made
that this transition is very unlike the last
three, and it will go quite smoothly.
The last three were sort of mixed.
If this is the case, then with AI governance
as a cause area, the best we can hope for
is maybe small marginal improvements, though
again, across a lot of sand.
Unfortunately, there are also good reasons
to think that the default course might be
fraught.
In a survey of AI experts who publish in top
journals, the majority assigned at least a
5% probability that superhuman AI would be
extremely bad on the order of human extinction.
Even among researchers who are highly optimistic
about AI, it is generally agreed that advanced
AI will not necessarily be safe, or at least
cannot be guaranteed to be safe without some
work and testing.
So, we'll do the work and testing, right?
Maybe.
It depends here how high the cost is in terms
of time, resources, and the performance of
the system itself, and how competitive the
environment is in which it is being developed.
This is sometimes called the "safety tax".
For example, what if the safety tax is two
years, and yet there are several companies
at the forefront of development, with tens
of billions or hundreds of billions of dollars
at stake, who are only a few months apart?
Now, worse.
What if these are not companies that are a
few months apart from one another, but rival
militaries?
During the Manhattan Project, there was a
serious concern that detonation might ignite
the earth's atmosphere and kill everyone.
It didn't really slow them down.
Now, let's imagine that AI can be made safe
easily, and that part is just covered.
This is not nearly sufficient to guarantee
a good outcome.
Some tech companies have impressively cosmopolitan
values, but there are legal constraints based
on fiduciary duties they have to their stockholders.
We also probably do not want any one company
too powerful, or to control a particularly
large stake in the future of the universe.
It's also unclear if a government would let
this happen without somehow taking control
of AI development or deployment.
So, what if this technology is developed by
or is in some way captured or seized by a
government?
Realistically, the US government or the Chinese
government.
How comfortable are we with either the US
or China dictating the future of the world?
Scarier question.
How comfortable do we think other nuclear
states are with this?
Now, let's imagine we've managed to thread
the two previous needles.
AI is safe, and it was deployed without igniting
a great power war.
How are the benefits distributed?
Continental Europe does not have a viable
AI industry, let alone Latin America, Africa,
or most of Asia.
Unlike during the industrial revolution, where
human labor was a complement to machinery,
it does seem as though with AI, it mostly
serves as a substitute for human labor.
Additionally, most AI services have characteristics
of a natural monopoly or an oligopoly.
There's not much room for competition, and
there's not really much room for small businesses.
Maybe we are so wealthy after the development,
that wealth naturally sort of trickles down,
that a rising tide is able to lift all boats.
But, we are extremely wealthy now, and children
still die of malnutrition, malaria, and diarrheal
diseases by the millions.
And so far, the wealthier we get as a world,
the more animals we subject to factory farming.
For us working in AI governance, the current
priority is to increase coordination and to
decrease race dynamics.
This is both to reduce safety risk, and the
risk of greater geopolitical destabilization.
It's also hopefully to increase the breadth
of stakeholders who are able to have a say
in how the future is developed, and in doing
so, hopefully to increase the moral circle
under consideration.
There are several routes for this that we're
pursuing in parallel.
This includes technical work on the possibilities
of verification of AI-related international
agreements, which are really necessary for
coordination between nation-states, and in
some ways a prerequisite for that being possible.
Also, in case we someday we might want something
like the international control of advanced
artificial intelligence, not saying that we
do, but if we do potentially want that option,
we've been looking at case studies of other
failed efforts to control decisive weapons
technologies in the past.
For example, after World War I, there were
several serious proposals to develop an international
air force.
This sounds a little weird, but there was
actually buy-in from the United States, Japan,
and France, and even Winston Churchill, who
was not a dove, for this as a plan.
The idea was for the League of Nations to
have a complete monopoly on military aviation,
with all other states banned from getting
it.
This international air force, then, would
have two functions.
One was to prevent any state from then developing
military aviation, or from attacking any aggressors
who then started a war, with the understanding
being that this would actually secure world
peace.
Similarly, after World War II, and the atomic
bombing of Japan, the US proposed to give
up all of its atomic technology, including
all of its resources, to an international
body, and to subject itself to intensive inspections,
to show that it hadn't retained any and that
it wasn't restarting this, if the Soviet Union
and other states agreed to do similarly.
Understanding how these failed, and trying
to learn lessons from these failures, might
increase the chances that we're able in the
future to gain international control, if that
is necessary.
The third time is the charm.
We've also been doing some work in science
diplomacy.
Science as a field is very cosmopolitan.
Even quite adversarial nations are quite good
at collaborating in science.
ITER is probably the best case study for this.
ITER is a research project looking at nuclear
fusion, and is being funded for many, many
billions of dollars.
It's a joint venture by the US, China, Russia,
the EU, and some other states.
What's important about this is that nuclear
fusion is an important strategic technology.
This is why the vast majority of this research
is funded actually by the military, or the
department of energy, but under military guard
and sometimes for military purposes.
The knowledge and the results from this technology
and from this experiment are supposed to be
spread out from all the participants.
If it does someday make sense to try to do
this as a collaborative effort, with many
stakeholders, I think this might prove to
be the best current model we have.
We also do a lot of tech company and AI research
community engagement.
The OpenAI Charter is possibly the largest
recent win in this category.
In the charter, OpenAI commits to developing
AI for the common benefit of all humanity.
But more interestingly, it commits to dropping
out of anything like an AI race if a rival
gains a lead, and even actually to join the
efforts, free, to push them further forward
into a greater lead.
Anyone who's interested in this, I would actually
really encourage you to read this charter.
It is inspiring.
It is also very short.
If you have good vision, you could read it
printed on a business card.
Also, if you're interested more in our engagement
work, I would encourage you to talk with Jade
Leung, who does this for our governance group.
So, I won't go into too many more details
here, through I will provide resources at
the end for people who want to read up on
more of what we're doing and some of the work.
But to do a quick summary, we're trying to
understand the AI scene better in China and
increase our engagement there.
We're also trying to better understand issues
of geopolitical instability that can be created
by changes in the balance of power and also
in the offensive and defensive dynamics of
weapons technologies.
We are also interested in modeling the dynamics,
or engaging in modeling the dynamics of tech
races, and in particular, how they can be
averted or spun down if they do begin, and
a lot more.
AI governance, as a cause area, is absolutely
tiny.
People talk a lot about AI at these conferences,
which gives the impression that there's a
lot going on, but there are fewer than a dozen
people who work on this full-time.
There are not that many more who work on this
part-time.
This being a tiny field is in some ways quite
good for you.
It means that your opportunity for a marginal
impact is absolutely huge, but unfortunately,
what it also means is there is not very much
absorptive capacity within existing groups
to bring more people onboard.
For the majority of people who are interested
in advancing this as a cause area, it is probably
mostly about preparing yourselves to be more
involved in the future as the field grows
and expands.
That said, as far as our immediate needs,
we are desperate for people who are good at
getting things done.
This is something we do need immediately and
there are immediate job openings for.
This is good project managers, good operations
folks of all sorts.
This is also... as well as being currently
our largest bottleneck, this might be one
of the main reasons why our absorptive capacity
to bring other people on is so low.
So, there's a force-multiplicative aspect
of this as well.
I want to in particular highlight: there's
a role at the Governance of AI program in
Oxford, which is currently the leading group
researching this, for a governance project
manager.
This is a Visa-eligible position, and this
will put whoever gets this role at absolutely
the center of this as a cause area.
We also badly need senior researchers who
can supervise more junior researchers.
Our research pyramid is sort of imbalanced,
with the bottom a little too big for the top,
and this is another bottleneck.
We also have positions for junior researchers,
which I definitely encourage people to apply
to, though I think this is maybe less urgent
or less impactful in the margin within this
cause area.
For the majority of those who are serious
about getting involved, what I recommend is
investing in your skills and your career capital.
This is especially true in high value areas,
including studying topics in high demand.
Also, building a career in a position of influence,
especially in government, global institutions,
or for example doing policy advising at a
tech firm.
Additionally, helping to build this community,
and our capacity, including having a strong
and mutually reinforcing career network among
people who are interested in getting involved
in AI policy from an EA or an altruistic perspective.
In my personal opinion, this third point is
more important than most people realize.
Having a strong social network, and having
people who understand what it is you care
about and why it is you're doing the things
you are doing, even if it in some ways looks
quite tangential to what it is you care about,
can be very helpful in allowing you to keep
your eye on the prize, and to work towards
your goal.
The reason this social support can be so important
is, as I mentioned at the beginning of the
talk, the human brain is not good at thinking
clearly in terms of scale.
This means it's very easy, and natural even,
to have some value drift.
I've always cared a lot about human wellbeing.
This is why I pursued my career in... my education
in economic development, and my career in
poverty eradication.
My personal experiences in field work around
the world have left me with a very strong,
very emotional sense and prototype that still
draws me very strongly to global poverty eradication.
But, I don't want my altruism to be about
what I feel.
I want it to be about how much good I can
actually do and how much of an impact I can
have outside of me, in the real world.
Even if I actually can't conceptualize just
how much there is, I do know where all the
sand is.
If you're interested in learning more about
our work, please go to our website.
Also, Jade and I will have office hours immediately
after our talk, and we have a meet-up this
afternoon at 3:30, which I would definitely
encourage people to attend.
Thank you very much.