Ben Garfinkel - Superhuman AI and the future of democracy and government

hey everyone jeremy here welcome back to
the towards data science podcast i'm
really excited about today's episode
because we're going to be taking on a
lot of long-termist sort of
forward-looking
and semi-futuristic topics related to ai
and the way ai technology is going to
shape the future of governance
are human beings going to just become
economically irrelevant
how many of our day-to-day decisions are
going to be offloaded to machines
and maybe most importantly what does the
emergence of highly capable and highly
general ai systems mean
for the future of democracy and
governance itself those questions are
impossible to answer with any kind of
certainty but it might be possible to
get some hints
by taking a long view at the history of
human technological development
and that's exactly the strategy that my
guest ben garfinkel
is applying in his research on the
future of ai now ben
is a multi-disciplinary researcher who's
working on forecasting risks
from advanced technologies including ai
at oxford's future of humanity institute
ben's also spent a lot of time exploring
some classical arguments for ai risk
many of which you'll have encountered on
the podcast we've had a lot of
guests on to discuss and explore those
in detail and and many of which he
disagrees with and we'll be exploring
his disagreements why he has them and
and where he thinks the arguments for ai
risk are a little bit shaky
i really enjoyed the conversation i hope
you do too
ben thanks so much for joining me for
the podcast yeah thanks so much for
having me
i'm really happy to have you here your
focus is on a whole bunch of
long-termist issues a lot of them around
ai
before we dive into the meat and
potatoes of that though i'd love to have
a better understanding of what brought
you to this space
so what was your background coming in
and how did you discover long-termism
and
ai yeah so it's actually um i was fairly
fairly roundabout
so in college i studied uh physics
and philosophy and was quite interested
in
actually the the philosophy of physics
and was even considering going into grad
school for that which fortunately i did
not do um
and yeah i guess through philosophy i i
started to learn more about ethics
and encountered certain ideas around
population ethics and idea that
um you know there's different questions
around how we should value future
generations in the decisions we make and
what our obligations are
to future generations or um how strong
the obligation is to do something that
has at least
you know some use to other people and
then yeah through that i sort of became
increasingly interested in um i guess
long-termism and also trying to figure
out something that seemed useful
um and i sort of came to think that
maybe philosophy of physics was not that
um and uh yeah i
i i got actually very lucky and that's
just around this time as i was trying to
look more into
sort of long-termist or futurist topics
i happened to meet a professor alan
defoe who was at yale at the time he was
just himself pivoting to work on
ai governance issues and he i think he
put up a call for research research
assistance when i was still a senior
there
and i was interested in the topic i'd
read a little bit about ai risk
i started read for example the book
super intelligence and you know how to
engage in area but it seemed like there
may be some important issues there
and you know an opportunity jumped up
and i started working with alan
and you know now it's several years
later i'm actually still working with
alan
and i've just become i guess fairly
convinced that
working on risks from emerging
technology is um at least a pretty good
thing to do from a long-termers
perspective
well and this is actually a beautiful
segue into i think one of the main
topics i really wanted to talk about and
that is this idea that
you spent a lot of time thinking about
existential risk from
ai and the arguments for it many of
which i know that you you're not
actually fully sold on
um maybe we can start there what's the
nature of the existential risk that
people generally in particular allen and
you
are worried about when it comes to ai
and then what
we can maybe get into the counter
arguments to those arguments as well but
just for starters yeah what is what is
that risk
yeah so i think it's actually um i don't
think that there's really a single risk
that um at least really predominates
um in the in the community of people
thinking about the long-term impacts of
ai so i'd say that there's
there's a few main very broad and
somewhat nebulous categories
so one class of risks very quickly is i
guess risks i'd say
are risks from instability so a lot of
people especially in the international
security domain
are worried about for example lethal
autonomous weapon systems maybe
increasing the risk of of you know
conflict between states maybe accidental
you know flash conflicts or potentially
uh certain applications of ai let's say
um you know um removing sort of second
strike capabilities and sort of
increasing the risk of nuclear war where
they're worried about great power
competition
and the sort of main vector of concern
they have is you know maybe something
about ai will disa
destabilize politics either domestically
or internationally and then
maybe there'll be war which will have
lasting damage or just some other
you know negative long-run conflict
there's another class of concerns
that um is less focused on
there being let's say some specific
conflict or
you know collapse or or warren is more
focused on
the idea that maybe there's some level
of
of possible contingency in how ai
reshapes society
um and so you might think that certain
decisions people make about how the
government ai will have lasting effects
that carry forward and affect future
generations um and in fact
for example things like how prevalent
democracy is um or what the distribution
of power is
or um just you know various other things
that people care about that maybe for
example bad values would be in some
sense entrenched um
and what are some of the because on that
side i imagine that's a very obvious
complicated area but what are some of
the ways in which people imagine
um ai transforming the the extent to
which let's say democracy is a tractable
mode of governance in the future yeah so
just uh on democracy
um so there's obviously you know some
you know speculative edge to this but
one argument for being worried about
democracy
is that democracy is not really normal
um if you look
you know across broad sweep of human
history back to the first civilizations
um you know it's not that uncommon for
there to be um
let's say um um very like weakly
democratic elements so
it's not completely autocracy there's
some sort of body you know so say
uh say you know roman senate or
something like that in the case of
which is a sort of well-known one but
it's very far from um
um what we have right now which is like
almost universal suffrage in a large
number of countries
uh with like very responsive governments
and sort of consistent transfer power
um that's like extremely rare from a
historical perspective and even if
things
are that were not full autocracy or
somewhat common before you know this is
a very different thing the past couple
hundred years
and there's different theories about why
uh this sort of
modern form of democracy has become more
common and there's a lot of debate about
this because it's you know hard to run
you know rcts uh but a lot of people do
point to at least certain economic
changes that happen around the
industrial revolution as relevant
um so one class of change that people
sometimes bring up is that um you know
and reform was a really serious concern
uh before the industrial revolution some
of the concern was that
if you you know give a lot of common
people power over the government
or leverage then they'll choose to just
redistribute land which is the primary
form of wealth um
from you know wealthy actors more
broadly should be very disruptive
and that um as countries industrialized
and land became less relevant as a form
of wealth
um maybe these like land reform concerns
became less of a blocker you no longer
had this sort of landed
aristocracy that um had this you know
uh very blunt you know policy fear
another concern as well is that the
value of labor went up as well
um just as you know productive increased
and this maybe people gave people in
some nebulous sense more bargaining
power because
uh the typical worker just uh what they
did in more value and they could you
know
create a larger threat by by threatening
to basically um just you know
remove their labor um or urbanization is
also thought to maybe have been relevant
like maybe people being packed in the
cities made it easier to organize
and actually have successful revolutions
and there's a lot of different factors
that people basically point to as being
economic changes that um
maybe helped democracy along its way or
helps at least partly explain why it's
more prevalent today
and so one concern you could have quite
broadly is if the prevalence of
democracy is in some way contingent on
certain
um material or economic factors than
that have only really you know held for
like the past couple hundred years
uh maybe this isn't normal maybe if you
just change a lot of economic and
technological variables
that's you know it's not gonna hold and
there's you know some more specific
arguments
here so one pretty specific argument is
just um if the value of human labor goes
very low or even goes to zero in most
cases because you can just substitute
capital for labor because
as systems can do anything that people
can do uh that maybe removes the power
of workers if you can automate
law enforcement um or you know the you
know putting down the uprisings because
you know uh military technologies can be
automated as well
um maybe that makes authoritarian
governments more stable it means that
they
they don't need to make concessions out
of fear of uprisings
and maybe as well if the value of labor
goes zero then wealth at that point
might become very heavily based on just
you know who owns capital or um you know
who owns
you know machines basically and maybe it
creates a system a situation is very
analogous to the old concerns about land
reform where wealth wasn't really based
on
these more you know nebulous uh things
with people the people's labor what
didn't really play a role which is
largely
there's a thing that you own that you
basically collect rents on
um if you return to that system then um
you know maybe that's also not good for
for the stability of democracy as well
um so it's kind of a
an outside view perspective which is
just this is a rare thing maybe we
shouldn't expect it to last we change a
lot and then there's some
sort of more inside view arguments that
um um you know maybe will make
authoritarian governments more stable
and make people more worried
about giving power to non-leads it's
really interesting how
um how how entangled all these issues
are and how difficult it is to
articulate a coherent vision of what the
future might look like when all these
transformational changes happen
one of the things that keeps coming in
mind for me when you know when we start
talking about
what's going to happen with democracy
what's going to happen with economies
and then the power of labor to negotiate
and so on
is like the the underlying assumption
that we have
any kind of market structure whatsoever
i mean to the extent that you have
all labor being done by machines
one of the i guess almost silly
questions that that i would have
is like what what is the value of money
in that context what is the value of
price discovery
uh how does price discovery happen in
that context and
what even does redistribution mean if i
mean it's not that we're necessarily in
a
post-scarcity situation you would expect
gradients of scarcity but
anyway i'm not even sure what thought
i'm trying to articulate here but it
looks like you have
something to throw in there so yeah so i
think i think this is a really you know
serious issue
is i think we should not expect
ourselves to actually be able to imagine
um a future with very advanced ai in any
level of detail and actually
be right um so an analogy i've
i've sometimes used is i think there's
certain aspects of
a world where as systems can at least
all the things that people can do
they can sort of reason about to some
extent abstractly like we do have um
you know these economic models we have
labor and you have capital and you can
you know ask about what happens we can
substitute capital for labor
um and um even you know
project is very abstract either point of
view and there's maybe some reason to
hope that these theories are
sufficiently abstract that even if we
don't know the details
you know there's still some reason to
think that there's sufficient general
abstract that we can still use them to
reason about the future
um but there's definitely a concern like
anything that becomes you know kind of
specific
of you know how the governments work
we're probably going to be imagining
just the functionality of governments
quite wrong um so one analogy i've
sometimes used is let's imagine that um
you know you're in let's say 1500 and
someone kind of describes the internet
to you in like very abstract terms if
it's like
communication will be much faster
retrieving information and learning
things would be much quicker
um and it gives you some of the abstract
properties of it there's some stuff you
can probably probably reason about so
you might think for example um oh you
can probably have like
you know a diplomat for blastotomy
because you know
uh people government can can communicate
with that more quickly as opposed to
them being overseas and out of contact
where businesses can probably be larger
because these
these coordination costs will probably
go down and some stuff you can probably
say about it that would actually be
um you know be true or you could say oh
maybe people work
remotely you know and you probably don't
need to know a lot about the details but
if you try to get really specific about
you know what's going on with it you're
probably going to be imagining it just
completely completely
wrong because you have no familiarity
whatsoever what a computer actually is
like or
what how people interact with them you
know you're not going to get you know
details at level like there will be this
thing called reddit
and you know gamestop you stock
like there's all these issues which
there's no chance you're ever going to
foresee in any level of detail
um and there's lots of issues you might
imagine that just won't really apply
because you're using abstractions that
somehow
don't fit very well um and so this is a
bit of a long-winded way of saying i do
think we have some theories
and and methods of reasoning that are
sufficiently abstract that i expect them
to hold at least a little bit
uh but i think there's lots of stuff
that we just can't foresee lots of
issues that we just can't really talk
about and lots of stuff we say today
that will probably end up being
silly from the perspective of the future
yeah i would imagine so i mean
you know this time it's going to be
different it's a dangerous thing to say
at any given time but when it comes to
the the next stage of the sort of the ai
revolution if you want to call it that i
know that's language you've tended to
use as well and it seems apt in this
case
um one of the things that that i do
wonder about is
a kind of almost like abstraction
leakage where like the abstractions that
we rely on to define
things like markets um this is sort of
one of the
very fundamental elements of our
reasoning when we're talking about
predicting the future
markets implicitly revolve around people
because ultimately prices are just what
individual human beings
are willing to pay for a thing to the
extent
that we broaden our definition of like
what a market participant could be
and here we get into questions of like
how do we consider an ai
agent that like at what point is it a
participatory member of society and
at what point does price discovery
really revolve around the needs and
wants of
non-human systems and things like that i
guess that's where i kind of
start to wonder like this is a
non-constructive perspective by default
so it's not helpful for me to say like
markets are a bad abstraction but um
like is that an issue that you think is
serious or uh
yeah so certainly yes i do certainly
think that there's an issue that i think
you point out like a good specific
problem
uh we have this very firm distinction
between
there's people are very different than
you know machines and software at the
moment like it's it's a very firm
capital and you know like economic
actors versus just like stuff about the
economic actor's own
and there's you know some degree of
blurring of like a corporation you know
for certain purposes
you know has traits which are in some
ways kind of similar to a person um
but you know the distinction is fairly
really strong
um or even just you know it's increasing
capital and labor it's like it's this is
not really
there aren't the ambiguities around this
at the moment um but if you think that
you know like very broadly capable a
gentle you know ai systems
will you know exist in the future um we
think that maybe people have interesting
relationships they have systems where
they
create a systems which are meant to sort
of pursue their values um
i think a lot of the distinctions that
we draw might actually become a lot more
ambiguous
than they are today and the way in which
to become
ambiguous in the future might make it so
that you know any reason we do that
relies on really crisp distinctions
um might just sort of fail in ways which
are difficult to foresee at the moment
yeah it's an interesting kind of risk to
predict because it's like i mean it
really is unpredictable and
fundamentally challenging
it seems like one of the issues there
too and you explore this in some of your
work actually on the history of
technology
is like which metric you're even going
to look at
to tell the story of the evolution of
this technology um can you speak a
little bit to that like
your historical outlook and which
metrics you find interesting and why
they may or may not be relevant in the
future
yeah so i think one metric that i think
people very
frequently reach to is um like uh
like global world product or gdp um and
gdp is is um it's sort of interesting as
a metric because
you know the thing it's meant to measure
is basically uh to some extent
productive capacity like how much
stuff can you produce um or like stuff
that people value
um can you produce and sorry you know
yeah that's right stupid question here
sorry
so what is gdp what is the actual
definition of gdp
um so at least nominal gdp you add up um
the total price of all of what they're
called like final products
that um are sold within an economy and
so a final product is basically
something that um in some kind of like
an end to itself so if you sell someone
screws
and then they sell the screw to someone
who uses the screw to make you know like
a ceiling fan or something
the screw isn't meant to be counted
because you're sort of double counting
if someone buys the ceiling fan and they
buy the screw when they buy the ceiling
pen you also sort of like find the screw
as well
so it's meant to be um basically adding
up the uh total
sort of like essentially sale price of
all the stuff that's
that's um like bought or sold within an
economy um
excluding the sort of intermediate
products um
but then people also often want to talk
about um
real gdp which is different than nominal
gdp so nominal gdp is just you add up
basically all the prices
um and one issue of nominal gdp is if
you have inflation
then you can have nominal gdp increase
for reasons that have nothing to do
whatsoever with like the actual
underlying stuff so if you know
government decides to print depend more
money
um such that you know um suddenly you
know the price of everything goes up by
a factor of a thousand but you still
have the same stuff
it doesn't really feel like this is any
like you know gdp growth
it has been extremely rapid in a nominal
sense but it's not really telling you
that like actually you're producing more
stuff yeah venezuela is doing great
yeah exactly exactly um and so real gdp
it's meant to be adjusting for this
and um at least like roughly very
roughly speaking the way it works is you
try to
um uh define everything roughly relative
to
the prices that existed a certain point
in time in the past
so let's say you have an economy that
exists like the only product sold is um
you know butter and um the price of
butter goes up by like
a fact for a thousand for some reason um
because of inflation
but you only double the amount of butter
that you sell in the economy
real gdp would just say oh because the
amount of bother you know you sold
increased by a factor of two um you know
the size of your economy is only
increased by a factor of two and
the size of the economy is defined just
take the price of bother in the past
multiply it by how many units exist
today and that's you know that's real
gdp
and it gets pretty complicated because
um uh people keep introducing new
products over time so
you know how do you compare the real gdp
of the economy
in 2020 versus the economy in 1700 given
that most of stuff that people buy
in 2020 didn't exist in 1700 like how do
you actually do that comparison
and there's various wonky methods people
use i don't that don't really understand
properly
um but yeah so this is an asking
question you've also gotten to one of
the main issues of gdp
is it's meant to be you know tracking
sort of
the productive capacity of society like
how much stuff we make basically
um and if you use real gdp
um you know there's there's uh
over short periods of time it seems
fairly unproblematic because you're not
typically introducing that many new
products um
but over long periods of time it becomes
increasingly nebulous like how these
comparisons actually work
um so very blunt comparisons still
pretty much fine so you can still say
like gdp per capita in you know 10 000
bc
versus today even if i don't know
exactly how to define gdp per capita for
the country gallery of societies
i'm still quite confident it was lower
um and so it's yeah in some sense like a
blunt instrument
um i think you know it's usefulness
really depends on
how precise you want to sort of make
your discussions or predictions so
let's say someone makes a very bold
prediction that um
uh you know the rate of gdp per capita
growth will increase by a factor of 10
due to automation
if someone makes a bulk prediction like
that you know it is a little bit
ambiguous what real gdp means in some
crazy futurist economy
but even if you look a little bit fuzzy
on it the difference between gdp
the rate of growth didn't change the
rate of growth increased by a factor of
10 is still blunt enough that's
you know it's a kind of useful way of
expressing you know a claim
uh so yeah so that's a long way of
saying i think gdp uh
gdp or gdp per capita is is often pretty
good as a proxy of just um
how quickly is productive capacity
increasing um it's useful for like
you know things like the indus
revolution really clearly shows up in
gdp per capita or when a country
is seems really stagnant like an
undeveloped country isn't developing gdp
per capita is typically pretty flat and
then
you know when china for example started
to take off in a really obvious
qualitative sense gp per capita
track to that pretty well um and so it
is useful for that but it's also
yeah has various issues um and then also
issues beyond that of like often people
want to use it as a proxy for how good
people's lives are like gdp per capita
um
but there's various things that don't
typically get factored into it like
um like the quality of medical care
isn't very directly factored into it
like air pollution
isn't factored into it um you know if
everyone was just very depressed
or like yeah or like anesthesia the
value of anesthesia being developed
just really does not show up there's a
classic paper by
nordhaus that shows that quality
improvements and lights like the fact
that light bulbs are there just way
better than like
candles like you know more than a
hundred years ago doesn't really show up
so there's yeah so long we're saying
basically lots of issues at least has a
crude measure pretty good
um but um doesn't necessarily like
correlate that
actually as well as you might hope with
with wellbeing and other things of
interest
it is interesting that when you when you
tagged on that last piece
it doesn't correlate well with
well-being um
that i can't think of a better
encapsulation of
a kind of alignment problem basically
the problem of coming up with a metric
that says here's what we want like
humans are really bad or
it's not that we're bad it may just be a
genuinely difficult problem
to specify metrics that even make sense
and like you know you see with the stock
market
we we decide to fixate on this one
metric and
for a while the stock market was a great
measure of you know in general how is
how's the economy doing how's the
average person doing but then there's a
decoupling
and uh we end up with very divergent
stock markets versus
the lives of the average person yeah
anyway sorry it didn't mean to butt in
but
you were right yeah so i should just say
as a little caveat um
i think at the moment gdp actually is
pretty good as a metric where
if you often define other things you
care about like life expectancy or like
life satisfaction it does actually
currently um there's often like a pretty
strong correlation and i think if you
just like
didn't know anything you know you're
behind a bill of ignorance or something
you need to pick a country to live in
and
the only thing you get is the gdp per
capita this is often going to be like
useful information for you yeah um i
guess my thought is more that
you know in line with the the alignment
concerns i wouldn't be surprised if it
becomes more decoupled in the future so
especially if
let's say imagine you know we eventually
just totally replaced labor with you
know capital and machines
and just you know people no longer
really working for wages and economic
growth is mostly like machines building
other machines and workers aren't really
involved
i would not be shocked if you know the
economy increases by a factor of 10 but
people the average person's
life design you know does not increase
by a factor of 10.
yeah that's interesting as well and and
raises the question of like
what i mean this is back to price
discovery which is a big aspect of gdp i
mean
there are so many areas where where
things get complicated but what's also
interesting is looking at some of the
the work that you put together
on this sort of historical exploration
of technology
is like a lot of these metrics really
are correlated i mean it seems like it
almost
to some degree it just doesn't matter
what you're measuring
something dramatic has happened over the
last
well 2000 years of the last uh 20
thousand years however you want to
measure it agricultural
revolution neolithic revolution um
industrial revolution
and it it's almost as if the human super
organism
all the human beings on planet earth are
an optimization algorithm that's just
latched on to some kind of
optimum or local optimum or whatever and
they're we're now climbing that gradient
really steeply
um do you see ai as just sort of like a
continuum limit of that
is that just like the natural next step
or should we think of it
as a a sort of a quantum leap like a
step function things are just
qualitatively different yeah
i think it's a really good question and
i do think um i do think that this is
this is sort of um a debate that exists
in terms of how
exactly to interpret the history of
you know economic growth or increased um
you know
social capacity or you know whatever
kind of nebulous term you want to use to
describe
people's ability to sort of make stuff
or you know change stuff or get stuff
done in the world
um and this is there's actually a debate
that um
exists for example between um different
interpretations
of the industrial revolution so one
interpretation
of the industrial revolution which you
know occurred between
roughly 1750 and 1850 and in uk and
surrounding countries
um is that you know up until the indus
revolution growth was very stagnant
um and then there was some sort of you
know change some sort of interesting
pivot that
happens that maybe took place over um
maybe
also another you know century and other
end of the industrial revolution where
for some reason this you know pace of
technological progress went up
and you know people switched to you know
away from an agriculturally based
economy to industrial economy
and people started using non-organic
sources of energy so it's no longer you
know
like you know wood and animal fertilizer
it's now you know fossil fuels and
you know and you know you know energy
transmitted by electricity and stuff
like this
and you know r d's can now play a role
in economic growth whereas previously
didn't really
and if there's some sort of interesting
phase transition or something that
happened over a couple of years we just
kind of transitioned from one sort of
economy to just a sort of a
almost like a qualitatively
qualitatively different sort of economy
that could just sort of grow and change
faster
there's another interpretation though
that basically says um
that there's actually this sort of long
run trend across at least the history of
human civilization
of the rate of growth getting faster and
faster um and this sort of
interpretation says that just
you know as the overall scale of the
economy um you know increases that
for that reason the growth rate itself
just sort of keeps going up and up and
um just sort of this this interesting
feedback loop with the scale of the
economy kept getting bigger and so the
growth rate kept getting larger and
larger
answer really visibly exploded in the
industrial revolution um
just because this is where the pace sort
of finally became you know fast enough
for people to notice this but there is
actually like a pretty consistent
you know like trend it wasn't really a
phase transition
um and there's some interesting work by
uh for example david rudman who's um
an economist um who uh does
work for the open philanthropy project
there's a recent report he wrote i think
modeling the human trajectory which
river argues
or sort of explores this sort of
continuous perspective and there's um
just a debate in economic history as
well says uh something economist michael
kramer who
has argued for this kind of smooth
acceleration
kind of perspective and lots of academic
historians who've argued
no actually you know there's some weird
thing where you switch from one for
every economy to another
um yeah this this is basically i'll just
say that um
yeah people there's like competing
interpretations someone just says
every once in a while it's it's a bit
weird it's a bit idiosyncratic just
something happens there's some change
it's a bit discontinuous and you switch
to a new sort of time it can grow faster
and another interpretation says no
actually this is a pretty consistent
force just things keep getting faster
and faster and
it's not you know phase transitions and
it's not discontinuities it's just
there's a smooth
really long run trend of just the world
keeps getting you know accelerating more
and more
it's interesting how that entangles two
different almost like two different sub
problems like one of them is
you know do do humans learn almost
continuously
in other words like is it the case that
um
you know cave people were gradually
generation on generation actually
picking up
more and more skills as they went that
only become obvious when you look over
like 10
000 years or is it the case that no
they're basically stagnant you know
everything is truly flat and then you
get some sort of takeoff
from and like it almost feels like this
is
this could be viewed as part of an even
deeper question where if you keep
zooming out and keep zooming out it no
longer becomes the story of humanity
iterating towards
some sort of future economy with ai's
taking over but rather
moving from completely abiotic matter
and like
the big bang like purely like no value
creation whatsoever yeah
i mean i guess that has to be a step
function right that first moment where
life evolves
like yeah this is where i'm i'm curious
about that perspective would seem to
argue for more of the quantum leap
angle or the sort of step function
approach um unless i'm mistaken
yeah um yeah so i think that's i think
that's
that's right um like definitely at least
intuitively there's certain transitions
um in history where it really seems like
just something different sort of
happening so the first
sort of self-replicating thing that
qualifies a life form it seems like that
has to be like a fairly discrete
discrete boundary in some sense um or
you know things like
i really do not know evolutionary
history but i think the you know first
eukaryotes
like mitochondria uh you know became
part of the cell this is a fairly you
know
discrete event um i believe um uh where
one organisms were smaller than another
and then it somehow stayed alive and it
was you know
the whole eukaryotic branch of life um
sort of followed from that
um and interesting you know various
interesting things like people
eventually followed from that
where that also seems like something
that intuitively is a sort of
discontinuous
change although i don't i don't you know
exactly no so um
yeah so it does seem like intuitively
there are certain things and then i mean
another one as well
is um um even the neolithic revolution
so people are starting to do
um you know agriculture in a big way um
i think you know i think the general
thinking is that this was actually like
fairly
rapid in a historical sense or like or
you know things that could qualify as
humans have existed for tens of
thousands of years and then maybe over
the course of like a few thousand years
uh people in like western asia and later
other continents transition to sort of
sedentary agricultural civilizations
and i think the thought is you know you
had um like a massive ice age
and then uh for like a hundred thousand
years roughly and then
the ice age ended and the climate
changed um and it became
in some ways more favorable um for
people sort of actually transitioning to
sedentary agriculture and then it just
happened very fairly quickly
um so yeah so this is all to say i do
think that you're right that there are
some historical cases where it really
does feel like
at least without me personally knowing a
lot about them it feels like a
discontinuous change
and you know i do also think that will
probably be the case to some extent for
ai like i don't think it's gonna be a
you wake up tomorrow thing
but i do think that um if we eventually
reach
for automation or if the growth rate
again increases due to the ai that um
you know people probably won't look at
um it just is like a stable continuation
of like
economic trends that have existed since
like you know 1950 that
you know right now we have this very
steady rate of economic growth and we
have this pretty steady feeling rate of
automation
and if the growth rate ever goes nuts i
think that people will feel like there
is some sort of inflection point or
pivot point or some tipping point
involved there
well and that's actually as as good a
transition point as any as i could
imagine
uh to the the second area you've been
looking at that i really want to discuss
which is your views on
ai safety not ai safety necessarily
let's say ai
risk and this idea of a smooth
transition to an ai-powered world
or let's say a very abrupt transition to
a kind of
uh dystopic or or existentially
um deadly scenario uh you
so you have some views on this maybe i'm
just going to kick things off with that
so you can can you lay out
what your what your thoughts are on um
like where you think
the the ai risk argument is strong and
maybe where it where it fails
yeah so i think i might just initially
say a little bit about the continuity
question where i think um
yeah so just just or at least the
relevance of the continuity question
um so yeah as if you alluded to this is
this is also the debate people have
about ai
is um how abrupt will the you know let's
say assuming eventually gets world where
your systems
can basically make you know human labor
obviously and do all sorts of other
crazy things
you know how abrupt uh transition will
that be will it be the sort of thing
like
an analogy to the industrial revolution
where it's you know it's a period of
many decades and it's this gradual thing
and it spreads across the world in a
gradual way and then
you know it's like uh i think even
things like you know steam power like
people transitioning from not using
fossil fuels using them that was an
extremely long transition
um will be more like those cases or will
it be something that goes a lot more
abrupt like will there for example be
you know as sort of like could you for
example point like a two-year period
where we went from stop being basically
normal
to you know now everything is high uh or
even you know less than two years and
this is the debate that that sometimes
happens
in the sort of long-term or sort of uh
you know i guess like futurist community
and it seems relevant in some ways where
um uh
one of and in some ways can then should
be something that increases risk or
um eventually reduces it so in terms of
increasing risk one thing that a sudden
or like really rapid change implies is
it can come
a little bit off of nowhere so it's very
continuous you know you kind of see a
lot of stuff that's happening
coming well ahead of time um whereas if
it's really sudden then that means
then you know if it's a process i would
take two years and it means that in
principle two years from now we could be
living in a very different world if it
just
happens to happen soon um and there's
less time to sort of get prepared and
less time to get used to different
intermediate levels of difference
and do trial and error learning and get
a sense of what the risks are what the
risks aren't
if stop the sudden just there's less
opportunity to see ahead of time and get
used to
problems and come up with intermediate
solutions and learn from your mistakes
um and so yeah so this i think the
largest risk of this is probably
relevant to
are risks related to misaligned ai
which is i guess the last major category
of risk
and these are also a little bit um um
diverse
and i believe you've had some previous
people on the podcast uh you know talk
about them
um but a lot of the concerns just
basically boil down to um
you know lots of a systems we develop in
the future will probably to some extent
act
behave as though they're pursuing
certain objectives or trying to maximize
certain things about the world
um you know in the sense that a goplane
system is trying to win
a go and a system that um you know um
um you know uh makes predictions about
re-effect you know re-offense race and
re-offense rates and a criminal justice
perspective is kind of in a sense trying
to
you know increase predictive accuracy
that sort of thing
and um the concern is that the goals
that es systems
have one some sense diverge in when it
was new goals that people
um tend to have and that this will lead
to you know disastrous outcome we have
air systems which are quite clever
um and quite good at getting you know
achieve whatever goals they have just
doing things that
you know differ from what people want
and
yeah so speed is really relevant to this
because um if you think that there's
this is going to be this pervasive issue
of someone creates an airing system and
deploys it
and then there's some sort of cello
divergence between its goal and the
people goals that people have and this
causes harm
it seems like if there's a really
continuous transition to ai systems
doing
you know playing larger and larger roles
in the world that there's probably you
know quite a lot of time to know this
kind of
less catastrophic versions of this
concern or learn what works or it
doesn't work
um you know not everyone is fully
convinced that just gradualness and
trial and error is enough to completely
resolve the issue but it seems like
surely
it's it's you know it's helpful to
actually be able to see more minor
versions of the concern and come up with
solutions that work in minor cases
words of stuff is very sudden then and
you know let's say we wake up tomorrow
and we have
ai systems that in principle can just
completely replace human labor could run
governments could do whatever
if we whatever reason decided to use
them um and they had goals which were
different than ours in some important
way
then this is probably a lot more
concerning and we might not see see
issues coming
um yeah so i think um so i guess
come on your question you know whether
the um
is it you know what are the reasons why
this might not be a major concern or
just what's the set of arguments for it
being a concern one way or the other
well actually um i think there's an even
more specific
kind of concern that you've sort of
taken a lot of time to unpack and it's
this concern around super intel the
argument that nick bostrom makes in his
book super intelligence
just to kind of to briefly summarize to
tee it up here the idea
is and i'm going to butcher this and
please feel free to highlight the
various ways in which i butcher this
but the idea is something like um if we
assume
that ai teams let's say at openai and
deepmind
and wherever else are gradually
iterating and iterating and iterating
one day one of them has an insight or
purchases a whole bunch of compute
or gets access to a whole bunch of data
that's just the one thing that's needed
to bump a system from like
pathetic little gpt3 to like now all of
a sudden human level or above
that system may be because it's human
level or above it may know how to
improve itself because humans know how
to improve ai systems so maybe it
figures out how to improve itself
and you get some like recursive loop
because the loop's very tight
the ai can improve itself prove itself
and eventually it's so smart
that it can overpower let's say it's
captures with its intelligence
and take over the world and uh and lead
to completely disastrous outcome
is that at least roughly right yeah so i
think that i think that's
that's basically roughly right um yeah
so i think i think there's basically
one way to think about is i think
there's sort of a spectrum of these sort
of alignment concerns and some of them
are on the more
maybe sort of diffuse or nebulous
perspective where we create lots of race
systems gradually over time
and their goals diverge from ours and
they're sort of a gradual like loss of
control of the future and
and that sort of thing and it's so much
a much more extreme it's like there's a
single ai system
um and it arrives quite suddenly um and
it's you know in some sense you know
broadly super intelligent and
it doesn't really have major precedence
and then that system sort of
individually quite rapidly causes you
know havoc in the world like there's
some major jump
to this one single you know very
destructive system uh which is
uh definitely the version that concerns
emphasized in things like um yeah next
book super intelligence and then the
the sort of narrative i guess you just
described and yeah so a lot of my
um i guess only thinking about ai risk
has been
um a lot about this sort of more extreme
out of the spectrum sort of style
concern that appears in places like
super intelligence uh for a couple
reasons
one is just um i think it's it it's it's
the
version that i first encountered and
that sort of made me especially
interested in that which i guess you
know
is a partial personal reason for
interest and others i think that this is
just um
you know even if lots of um ai alignment
researchers don't primarily have this
version of concern in mind i think it's
still quite influential and pretty well
known and it's often if someone knows
anything about ai risk this is the
version of concern
that comes to mind and so in that sense
i think it's it's maybe especially worth
paying attention to
and and yeah so some of my thinking has
been um just about the question of like
is it plausible that you actually have
this very sudden jump from
you know you don't really have major ai
systems of interest while there's a bit
like it is today and then suddenly some
researchers somewhere has this major
breakthrough and you end up with this
you know the single system and i guess
i'm fairly skeptical of this for
for maybe sort of boring reasons um so
one
um initial boring reason it's just you
know
that's that's not the way like
technology tends to work like if you
look at any sort of
you know if you sort of start from the
perspective of like let's look at how
technology normally transforms the world
um it's normally the case that um it's
this protracted process that takes
decades where someone develops something
and then you know the long process of
improvement and then it's deployed in
some sectors before other sectors and
it's useful in some areas but for other
areas
and then people need to develop
complementary you know inventions to
take advantage of it
and people need to sort of figure out
how to actually use it appropriately and
there's lots of knowing tweaking and
issues they don't foresee that that make
it a slow
process you know so like electricity
it's i think the electric motor
um sometime in the early 19th century i
believe it's invented but then electric
motors you know don't predominate in
like american factories until
you know something like the 1930s um or
the you know
the first digital computers you know
middle of the 20th century but until the
90s that they really show up and
productivity statistics in the big way
um and even then you know not really and
still you know loads of countries not
like
that pervasively used in different
important contexts um
and um you know not even that in a sense
like that larger portion of the economy
and so if you kind of start from there
it's like you know don't look too
specifically at the details of ai and
said what would i expect if it's like
you know any other technology we've ever
had
it's probably it's economic
transformation it's going to be a
gradual thing lots of annoying stuff
that happens
and do you think so just to kind of
probe it that a little bit yeah
so one of the things that i would
imagine
has made the uh the progress
and distribution of technology
accelerate in the last like 100 years or
whatever period we choose
is precisely communication we talked
about that quite a few times the role
the internet played and so on
um and communication in particular in
terms of tightening feedback loops
between the teams of people who design
products the teams of people who deploy
it the teams of people who sell it and
so on
um to the extent that that integration
that coherence
is driven by communication would that
undermine
this uh this argument in a sense saying
well if you have a single ai system
that's internally coherent and that's
able to
essentially tighten that feedback loop
not infinitely but like
to to machine time um does that
do you find that that position
interesting i guess is what i'm trying
to um
so i guess i find it through i guess i
find it interesting but not persuasive
so um yeah so let's say i think so let's
say
there's you know the idea of like if we
imagine if we jump to
you know imagine like that there's a
certain jump to some you know extremely
broadly capable ai system that just can
serve you all of the economically
relevant you know production tasks like
it can
you know it can do mining for chips it
can run um you know dollar coin centers
that can build
you know it can it can do ai research it
can um you know build
you know more compute resources uh it
can manage
you know military strategic you know
like if you imagine if there's a single
system that just sort of abruptly comes
into existence that's just
itself kind of doing all this without
interacting with outside actors or
pulling on external resources
um then that's going to some intuition
of like stuff can happen faster because
just the communication and efficiency
costs have just gone down
gone down a lot uh but there's sort of
you know one of the questions is like
should we imagine that this is the way
development will work that there will be
like one single system that just kind of
abruptly gets all these capabilities
and i guess that's you know something
that i'm i'm pretty skeptical of in
in the case of ai and you know also
again for like somewhat boring reasons
so
we do know that um even have progress in
different areas at the same time so
uh something like that um i imagine
probably a lot of your listeners are
familiar with this you know um
language models or like this recent
system gpt-3 developed by openai
this is an example of a system that got
pretty good at lots of different tasks
uh sort of through a single training
process like sort of roughly the same
time
so i was trained on large um corpus of
uh basically web pages um and i was
trained to basically try and predict
uh what's the least surprising next word
i could encounter on the basis of the
the words i've already encountered and
like a document i'm exposed to
and so you can use it to do stuff like
you know write a headline for a news
article and then we'll try and think
what's
the least surprising text for an article
given you know
this headline and you know one thing
people think people find is you can
actually use it to do a lot of different
stuff
um so um you can use it to do
translation for example we can write
sense in spanish and say the english
you know translation the sentence is
blank it's colon and the system will go
utterly surprising you know
thing to find next would basically be
like the english translation of it
you can use it to to write you know
poetry like what's the least surprising
ending to this like emily dickinson you
know poem um
that sort of thing uh but even in these
cases where lots of different sort of
capabilities kind of
in some sense come online at once you do
still definitely see
um a lot of variation in terms of how
good it is a different stuff um
so you know it's um it's like pretty bad
for the most part at writing like usable
like computer code like you can do a
little bit of this but basically you
can't
do it any useful way at the moment it's
like pretty good at writing like
you know jabberwocky style poems like
one of these you know sort of came
before the other
um and there's sort of reason to think
that though can be the case that's going
to be like a sort of expanding you know
thing where some capabilities come
before others
there's also some capabilities that just
you kind of can't really produce just
purely through this sort of gpt's that
three style training on this large
corpus of you know online things like
if you want to translate like you know
department of defense internal memos it
needs to be trained on something else if
you want to write like healthcare
legislation probably this dallas site is
not going to do it for you
um if you want to like set supermarket
prices like a price inventory thing or
do personalized emails where it knows
actually like when to schedule meetings
for you
you're going to need like a different
training method or if you want to
perform better than humans you're going
to need like a different training method
as well because you need to give it like
what it basically does is it tries to
say would be like the least surprising
thing for a person to have written on
the internet but if you want to do
better than a person
you're going to need to use something
else some sort of feedback mechanism
so basically there's a reason i think if
different capabilities will come online
at different times
they'll also probably be lots of
annoying stuff that comes up in
different specific domains that like
doesn't really show up to researchers
but tends to come up we actually want to
apply stuff like i love getting you know
going from an electric bundle to people
actually using electric motors in
factories was like you need to redesign
your factory floor because it's no
longer based around the central steam
engine you need to redesign the things
that's using the hardware
in the redesign you know the processes
that your your workers use to actually
leverage this thing you know regulations
need to happen et cetera et cetera
um and probably these things will need
to be dealt with to some extent at least
initially by different teams and some of
them will be harder than others
require different resources than others
and um i would basically be surprised if
like this is you know all again like a
long way of saying i expect stuff to
come online like actually be like really
useful in the world
at pretty different points for different
tasks interesting yeah that makes
perfect sense and it's exactly the
what's interesting to me is it's exactly
the kind of error
that um that a theorist would would make
like you know
imagining a system that not that it is
an error i mean this the scenario could
easily come to pass but
these are interesting objections that
seem to map on to the psychology of
somebody who's focused on
you know theoretical optimization rather
than optimization of
systems and economies in practice um
interesting so none of this though seems
to suggest that it would not be possible
at some point in the future for an
ai system with the ability to
self-improve uh
iteratively and and add infinite uh to
be developed so
i guess the question is there's two
parts to this question first off
a do you think that that's the case or
do you think that it will be possible to
build such a system
and b do you think such a system will be
built or is likely to be built
are there the is there a series of
incentives
that stacks up to get us to a
recursively self-improving ai
that just goes eventually and does
whatever
is that a plausible story yeah so i
guess so i guess i have a couple of bits
here so first bit is um it's unclear to
me that recursive self-improvement
should will really be the thing so so
clearly there are you know feedback
loops and will be feedback loops um in
the future
so you know we see lots of technologies
in a more limited way so
um you know the existing software is
useful for developing software you know
people use
software developers you know use
software um
and uh you know computers useful for
design computers if people are
you know um nvidia or you know any any
sort of hardware manufacturer didn't
have computers to to use
they would probably find their jobs like
quite a bit harder um
you know um so there's like you know
loads of cases
where uh technology sort of aids and
sound development um or like
technology ages development different
technology it's typically not recursive
where it's not typically like exactly
the same artifact that's improving
itself
um and in the case of ai like i don't
necessarily see a good reason to expect
it to be recursive like i definitely
expect ai to be applied more and more in
the context of ai development you know
searching for
you know searching for architecture or
uh doing you know
learning figuring out what's the most
you know optimal way to to basically you
know
develop another system or um you know
make it work well
um but i i don't necessarily see a
strong reason to like think that's
the individual system doing it to itself
as opposed to a system that's developed
to help train other systems
sort of you know you know the same way
like software doesn't tend to improve
itself like i don't really see like a
great benefit to being recursive
i mean it could be the case that that's
done but i just i sort of don't see why
it would be recursive like why that's
inherently more attractive um and
somewhere seems like maybe less
attractive it seems like
somehow but like messier like it seems
like nice if this is a bit of a modular
thing
um yeah i guess it like to some degree
just to sort of
um to bolster this argument a little bit
from an engineering standpoint i would
imagine that
um the so there's there's this
abstraction of like
different systems yeah this term that we
use to say like
there's system a there's system b you
know system a is either improving itself
or
system b is improving it and then maybe
system a all that stuff
um i guess what i what i'd be thinking
of in this in this case is like
an abstraction that covers like a closed
something like a closed system yeah
that crucially operates on machine time
so like the
the key distinction to my mind that
would define like a
a a takeoff of this form would be the
fact that this
either self optimization or you know
system a improved system b
happens like on the order of like
microseconds or
or what have you such that humans do not
intercede in the process
and are ultimately surprised by the
result or the result
significantly from our expectations yeah
so i i think maybe one of the key
distinctions suggest is like
labor basically involved in in the
improvement process and so one sort of
general counter to like
you know this ai feedback loop being
really important to really
increasing the the rate of change you
know that much is just um
you know we already do have like i guess
of circus we already do have these
feedback loops where um
um you know loads of tasks that
researchers would have been doing or
engineers would have been doing
at the beginning of the 20th century
they just don't do anymore they've just
been completely automated
so just actually doing calculations by
hand it was like a huge
you know time sink it was like research
effort for like engineering
um and so it's been massive massive
automation like in terms of the time
that people spent doing
like you know a huge portion it's been
automated away and so in that sense
there's been this really
strong feedback loop um where you know
technological progress has helped
technological progress
um but you know we haven't actually seen
um at least since the middle of the 20th
century we haven't
seen um an increase in the rate of
productivity growth or like
technological progress
at least in like leading countries like
it seems to have actually gone slower if
anything
um and you know the rate now is like
comparable to the beginning of the 20th
century
in the us and so clearly this feedback
loop isn't enough on its own and there's
you know an offsetting thing and
probably the main off same thing is like
this
idea is getting harder to find
phenomenon where you know yeah
technology helps you make new stuff but
also
each new thing you want to make is a bit
hard to make than the previous thing um
because if it was easy you would have
already done it you know um
and so um and so that's kind of one
general counter argument and then the
counter counter argument to that is like
well
you know this whole time that we've been
automating um lots of the tasks
tasks involved in research and then you
know creating machines to do them and
improving the machines
you know human labor has always been a
part of it and if you have this sort of
story where um
human labor and you know uh stuff done
by capital basically
like is like complementary um then you
have sort of a labor bottleneck story
where we keep
making cooler machines and keep making
more machines
um but there's diminishing returns on
like the coolness of your machines or
the quantity of your machines for a
fixed amount of like you know research
efforts
um and so research efforts are the
bottleneck it creates this dimension
returns phenomenon
where um you know just it really like
limits the marginal value of the
additional cool tech stuff that's
involved you know done by researchers or
owned by researchers um and then if you
the stories and then you know
the number of research which grows at
this pretty constant exponential rate
that can't really be changed that easily
because it's linked to population and
things like that
so then story might be if you actually
remove just human labor completely from
the picture just people are just not
involved
in an r d anymore or manufacturing then
maybe in that case
you no longer have this dimensional
returns effect like you no longer have
this balance that you get diminished
returns on capital for like a fixed
amount of labor maybe it just
feeds back directly to itself dimension
returns kind of go away in in some
important sense and then
the feedback loop really takes off once
you just completely remove humans from
the loop
um would be that the story you could
tell to say you know why the feedback
loop will be different in the future
than the the
non-explicit feedback loop we've had for
the past century and i guess there's
also
like there's also there is a feedback of
like
human self-improvement i think clock
time is the distinguishing
characteristic here but like
you know i can like i do strive to
improve myself and my productivity
and i do strive to meditate myself like
i try to improve the way i improve
myself
and in principle i think i do that to an
infinite number of derivatives or as
close to that as matters so there is an
exponential
quality to it but clearly i mean i'm not
elon musk yet i
haven't achieved hard takeoff so yeah
there's a difference yeah so i guess the
thing i'd say there is probably that
i think you're definitely right that's a
real phenomenon um i think though that
the the kind of orders of magnitude
involved like how
much even actually self-improve is just
smaller than it is for technology so if
you imagine you know
let's imagine a researcher unit is it's
like it's a person in their laptop and
that's the thing that produces the
research
um the person can definitely make
themselves better at coding they can
make themselves better at learning how
to do things you know
quickly they can learn how to learn um
but maybe the actual difference in
productivity
maybe you can hope to increase by like a
factor of you know 10 in terms of human
capital relative to
you know you know what the average
researcher and like you know 2020 is
whereas your laptop it seems like it
maybe has more lungs to the rally
in the climb up in terms of how much
better it can get um you know than it is
right now
that that does unfortunately seem to be
the case but uh i just need to keep
working at it i think that's what
happens
yeah i wish you best of luck in your
race against your laptop's rate of
improvement
yeah thanks i'll i'll let you know if i
hit take off but
so so the the that's really interesting
that that
so you have done so much thinking on
this and i can i can sort of see in
myself
um some some shifts in terms of like the
way that you're thinking about this
certainly there are aspects of it that i
hadn't considered before
that do come from this economics
perspective that come from the systems
perspective
um is this a a way of thinking that you
think
is especially uncommon among technical
ai safety people
or are you starting to see that become
adopted more like i'm still trying to
piece together
what the landscape looks like and how
views have been shifting on this topic
over time
because just by way of example i
remember um 2000 like
nine was you know it was miri eliezer
yukowski basically everybody was talking
about this
idea of a brain in a box or some fast
takeoff thing where a machine
self improves and so on whereas now it
really does seem like
between open ai paul cristiano and um
a lot of the work being done at future
humanity institute
things are sort of shifting and i just
i'd love to get your perspective on
that shift that timeline and where the
community now stands with respect to all
these
theses yeah so i do definitely think
there's been
a shift in what um the way let's say the
the median person in these communities
is thinking about it it's a little bit
ambiguous to me how much
of it is um a shift in terms of people
who used to think one way
shifting to another way of thinking
versus more people entering the
community with a sort of pre-existing
different way of thinking
um i do think that there is some element
of
people thinking about things in like a
bit more of a concrete way why do you
think a lot of the older analysis
it's very abstract um it's like very
much relying on sort of
um i'm not maybe like it's not exactly
like mathematicals it's
you know it's like people doing abstract
algebra or something but um it's
definitely maybe like a more
mathematical mindset um
um and you know it shifted over time and
i think one reason for that which is
very justifiable it's just
when people were talking about this into
the you know mid 2000s
uh you know machine learning wasn't
really a huge thing um
people thought it would be more maybe
like logic oriented systems um
would be you know what would you know
maybe aj would look like
anyone they have like anything that sort
of really looked at all agi to really
sort of like use as a model to think
about and i think as
um the community's kind of you know
machine learning took off and people
started to have these systems like
you know something like gpg3 well
obviously this is not you know agi and
probably agi will be like very different
than that
it's like you know it's like a little
bit of like a uh you know stepping stone
a path to agi it's like a little bit you
know
maybe aji-ish or something i think
having these concrete examples
just sort of lead you to start thinking
in like a slightly different way
um we start to realize that like they're
actually like a little bit hard to
describe in the context of maybe these
abstract frameworks that you had before
so like gpt3 like does that have like a
goal or like
you know if you want to predict its
behavior how useful like i guess its
goal is kind of
to produce whatever next word would be
like unsurprising but
it somehow doesn't exactly feel right to
think that way it's not clear how useful
it is for
against behavior like it doesn't really
seem like there's a risk of it you know
doing some crazy like you know killing
people to prevent
them from stopping it from outputting
like somehow just kind of feels like it
doesn't really fit very well and so
um and also just you know seeing like
more concrete applications and thinking
um so i think they're saying that like
paul kashana said for example that um to
some extent
being optimistic about oh i think you
could actually probably do that thing
with machine learning not that far in
the future without major breakthroughs
lens people also think in a more
continuous sense so it's not always
nothing it's like you can kind of see
the stepping stones of intermediate you
know transformations
and so i think it's yeah i think it's
seeing intermediate applications having
a bit more concreteness
and feeling like a little bit like more
may be more skeptical of the abstract
concepts using just because it's hard to
fit the month and you're seeing or
maybe some forces i've have had an
effect um
um but i think you know to be clear i do
definitely think that there um
are you know plenty of people who think
that um the more mathematical and sort
of classical way of approaching things
is still
you know quite quite useful or is you
know that maybe the predominant way they
approach things
yeah i mean i actually have heard
arguments um
not not necessarily arguments that a
system like gpd3 would
become pathological in the way described
um
but at least stories that can be told
that sound internally consistent to
describe worlds in which
a system like that could go go really
badly wrong in that case it's something
like
imagine gpt 10 like you know whatever
the year would have to be for that to
happen
and you have the system that you know
necessarily i mean it is doing this like
glorified auto complete task but in
order to
perform that task one thing that seems
clear is that it's developing a fairly
sophisticated model of the world
there's some debate over the extent to
which this is memorization
versus actual generalizable learning but
you know let's let's
give gbt3 the benefit of the doubt and
assume it is generalizable learning
to the extent that that's the case the
system continues to develop a more and
more sophisticated model of the world
a larger and larger context window
eventually that model of the world
includes the fact that gpt3 itself
exists and is part of the world
um eventually this realization as it
tries to optimize its gradients
makes it realize oh um i could develop
direct control over my gradients through
some kind of wire heading is usually how
it's framed in the alignment community
and so on like i think the there are
problems the problems that you described
apply
to this this way of thinking but it's
sort of it's interesting how
a gpd theory really has led to this kind
of concrete thinking about some of those
abstractions yeah
i think it's just also very useful as so
so for example i think it's also very
useful
concrete systems because they also think
they force differences in tuition or
workforce
you know differences in combating
assumptions to the surface so
just as one example it's like um i it's
definitely the case that some people
have expressed
um concern about these gpt systems early
if you have you know gpt 10 then maybe
this would be very dangerous
and i actually wouldn't have guessed
this or i guess i guess i wouldn't have
guessed other people had this intuition
just because
i sort of didn't have it because my
baseline intuition is just
you know basically too rough
approximation the way the system works
is you know it's it's a
it's a you know it's a model of some
parameters and then it's exposed to like
a corpus of text
um and it just basically you know
outputs you know a next word and then
you know the next word is like you know
actually right or it's not or
basically there's a gradient sent that
pushes the word outputs to be like
less and less surprising relative to
whatever the actual you know
you know words in the dataset are it's
just basically can't be optimized for
like outputting words which
would be unsurprising um to find as a
next word in a
piece of text which is online somewhere
and
when i think about gpt time i think wow
i guess it just outputs words which
would be very unsurprising to find on
web pages online it's just like the
thing that it does like
and you know if it if in surprises let's
say does stuff like outputs
words which lead people to destroy the
world or something
it seems i could only do that if those
would be words that would be the most
unsurprising to find
online like if the words that lead it to
the straight world
are not you know would be surprising the
fine line because people don't know me
write that sort of thing online then
it seems like something weird has
happened with the gradient descent
process
so so the i i think that that's a really
great way to frame it i i think i
believe the counter argument to that
might sound something like
um you know we might look at human
beings as
like 200 000 years ago as um as uh
uh uh sex optimizers yeah something like
that
and then you know we find that we're not
that as as our
evolution has unfolded um i think the
case here is that like
well first off there's a deep question
as to what it is that a neural network
actually is
optimizing it's like it's not actually
clear that it's optimizing its
loss function or that like does it feel
it doesn't
like it feels a kick every time its
gradients get updated it goes like oh
you're wrong like
update all your weights by this does
that kick hurt
and like if it does then is that the
kind of the true thing that's being
optimized by these systems
and then if that's the case then there's
this whole area obviously inner
alignment that
we're kind of skirting around here but
yeah it's a deep rabbit hole i guess
yeah
yeah so i certainly agree that there's a
distinction between
the loss function that's used when
training a system and what the system
acts like it's trying to do and just one
really simple you know way of saying
that is um if you you know start with
like a
a chess playing reinforcement learning
system and you have a reward function
and loss function associated with it
and you just haven't trained it yet it's
just not going to act like it's trying
to win a chess because it's
you know that's the one of the bluntest
examples of like it just doesn't happen
um and then obviously you know you have
these like transformation cases where
you know you um you know you train a
system
in let's say a video game where um it
gets points every time it opens
a green box that's on the left on the
right there's like a red box and you put
it in the environment
where there's a red box on the left and
you know um green box on the right
and the training data you've given it so
far isn't actually sufficient to
distinguish
in a sense like what is actually a thing
that's being rewarded is it for opening
red boxes or is it for opening you know
the box and left
and you know i you shouldn't be
surprised if the system for example
opens the box and left even though
actually the thing that you know is in
the loss function is the red box um
or you know vice versa like it wouldn't
be surprising if it's if sort of as
generalized in the wrong way
so i certainly agree that there's can be
these generalization errors um
i struggle to see the why you would end
up with um
like in the case of something like gpt3
i just sort of don't understand
mechanistically like
what would be happening where would be
you know so let's say that the concern
is it
because it's you know a text generation
system that puts out some text where if
it's read by someone
you know um the you know it's an
engineering blueprint for something that
kills everyone let's say which i don't
know if there's like a non-sci-fi
version of this where it leads access
onto risk but
that's that's the other thing it does um
you know i i sometimes feel like i'm
almost being you know dance or something
or missing something but i just don't
understand like mechanistically why
would this
grading design process lead it to have a
policy that does that like why would
it in any way be optimized in that
direction um
the the answer i would give not having
put i'm sure not having put sufficient
thought into this i should preface
but is in principle um so if we imagine
like let's say unlimited amount of
compute unlimited scale data and so on
this model would let's say just it
starts to think and it thinks more and
more and more
and develops like a larger and larger
and more complete
picture of the world um again depending
on what it's trying to optimize
assuming it's trying to optimize for
like minimizing its gradients
like here this is very coarse i assume
i'm wrong somehow but
somehow it feels like right to imagine
that a neural network feels
bad every time you get kicked around i
don't know although i don't think though
yeah i mean i don't think there actually
is any sense once it feels bad i think
it's just it has
certain parameters and then it outputs
something and it sort of compares to
you know the the the training set and
then
based on the discrepancy its parameters
have been kicked in a different
direction like i don't think that
there's actually any
sort of internal like like i don't think
there's actually like a meaningful sense
when it feels
bad it's just sort of like you know it's
kind of it has parameters and get nudged
around by like a stick like it's you
know this
is a guy with the stick pushing the
parameters in different directions on
the basis of like this graph into your
lack of discrepancy and then
they eventually end up somewhere yeah
and i think that that's actually
so this in and of itself is like i think
one of the coolest aspects i'm
about to get distracted by the inner
alignment excitement here but yeah
it's one of the coolest aspects to me of
the inter-alignment debate because it's
it really gets you to the point of of
wondering about subjective experience
and consciousness because
there's there's no way to have the
conversation without saying like this is
some kind of process
some kind of learning process and
learning process tends to produce an
artifact
like in humans it's a brain that i mean
it seems to have some kind of subjective
experience basically all life you can
look at an amoeba
move around under a microscope it really
seems like it experiences pain and joy
in different moments and different ways
um and and so anyways seeing these
systems that
behave in ways that that could be
interpreted similarly
um inspires at least in me questions
about you know
what is the link between the the actual
misa objective the function that the
the optimizer is really trying to
improve
and like subjective experience i think
i'm kind of going into territory i don't
understand nearly well enough but
um maybe i can leave the thought at i
think this is a really
exciting and interesting aspect of the
problem as well do you think that
there's like a consciousness and
subjective experience have a role to
play the
the study of that in the context of
these machines or are you i
i think actually so i think not so not
so much
there's a difficulty here where there's
obviously the different notions of
consciousness people use so i guess i
predominantly
um think of it in the um
i guess the uh david chalmersy sense of
conscious experience as this at least
hypothesized you know
phenomenological thing that sort of um
you know not in intrinsically a part of
the it's not like a physical process so
it's not a description
of how something processes information
it's you know an experience that's
layered on top of
you know the mechanical stuff that
happens in the brain um
where if you're you know illusionist you
think that this there is no such thing
as this and this is like a
woo you know thing um but i guess you
know for that notion of consciousness it
doesn't seem
in a sense like very directly relevant
because it doesn't actually you know the
weird aspects of it is it it's it's by
calm definition or a hypothesis not
something that actually physically
influences um anything that sort of
happens in the world behaviorally and
that you know you have
zombies where they behave just the same
way but they don't have this additional
layer of consciousness on the top
so this that version of consciousness i
think um
i don't see it as being very relevant to
um
understanding how machine learning
training works or how issues of mace
optimization work
maybe that there is um um mechanistic
things that people you know sometimes
refer to
using consciousness which i think
sometimes has to do with like
um uh information systems somehow having
representations of themselves is like
you know um maybe one sort of kind of
traits that people pick out sometimes
when they
they they use the term consciousness um
it seems like maybe some of that stuff
is relevant or like maybe like
beliefs about what your own goals are
this sort of thing maybe this has some
interesting relationship
to um to optimization and you know
um uh you know
uh you know human self-consciousness and
things like that um so i can see a link
there but
i guess i guess as all to say it depends
a bit on the notion of consciousness
that that one has in mind
yeah no it will makes makes perfect
sense and it's interesting how much
these things do overlap with so many
different areas from
economics to to theories of
consciousness theories of mind
um thanks so much for for sharing your
insights ben i really appreciate it
uh do you want to share do you have like
a twitter or a personal um
website that you'd like to share just so
people can check out your work because i
think i'm
working on fascinating stuff yeah so i
do have um a
personal website with very little on it
but there's like a few papers i
reference
um that's uh ben garfinkle.com
and i have a twitter account that i've
never tweeted from i forget what my
username is
but if you would like to find it and
follow me i may one day tweet from it
that is a compelling pitch so everyone
um look into the the possibility of ben
tweeting
sometime you could be among the first
people to ever see a tweet from me if
you get on
the ground floor right now they're
they're getting in at seed this is time
to invest seed stages
awesome thanks so much ben i will link
to uh both those things
uh including the the twitter i look
forward to the the
twitter followers there you go yeah
everybody go and follow ben
check out his website and we will be
posting i'll be posting some links as
well the blog post that'll accompany
this podcast
just to some of the specific papers and
pieces of work that ben's put together
that we've referenced in this
conversation
because i think there's a lot more to
dig into there so ben thanks a lot
really appreciate it
thanks much this was a super fun
conversation