Are AI Safety Concerns Philosophically Sound?

all right it's three o'clock so
we will start
so thank you everybody for joining this
ai safety debate my name is aaron
stuppel and i've started a web magazine
called conjecture magazine
that is dedicated to the philosophy of
progress we think karl popper's critical
rationalism is an
undervalued approach to knowledge and
progress and deeply informs among other
things artificial intelligence
and that's what this debate will be
about we will go for about 45 minutes to
an hour
and then open up for questions this
conversation is recorded
and that will include the questions
session
participating today are dennis hackathol
a software
engineer who has written a book on ai
called the window unintelligence
available on amazon he also has a
podcast called artificial creativity
where he addresses this very question
and others
jim miller is a professor of economics
at smith college
he among other things studies the issue
of value alignment in
a.i as well as other questions about
utilitarianism futurism and other
very interesting ideas so our topic
today
is whether or not ai safety concerns
are philosophically sound and just as a
primer both
speakers feel that artificial
intelligence and artificial general
intelligence
are both feasible and to some extent um
either inevitable or close to inevitable
so we won't
debate or discuss that much instead
we're just going to focus on our
our question of ai safety
and so i'm going to start it off with um
describing a scenario that sam harris
has broached which is comparing
artificial general intelligence
with an encounter with super intelligent
aliens
um he describes you know the creation of
artificial
general intelligence as being similar to
the arrival of a super-intelligent alien
civilization
and we would be defenseless and
extremely vulnerable
and that it's you know point of imminent
existential
threat and risk and so jim does this
argument resonate with you
and would you like to flesh it out or
indicating a point of disagreement
yeah and thank you for organizing this
aaron and i thank you for agreeing to
debate dennis
uh yeah so i'm of the impression that um
artificial general intelligence will
likely determine
the future path of our species there's a
very high chance that it will cause our
destruction
and some chance that it will bring about
utopia but as
i do i do think there's a that unaligned
artificial general intelligence will
likely destroy us
the big reason is just
that there's a limited amount of
resources in the universe at least if
our understanding
of physics is correct so imagine
we create a computer super intelligence
that just hypothetically
wants to be as good as chess it could
possibly be and that's its main goal
it might seem like well it's not going
to hurt us that much but unfortunately
if
an agi had as its goal maximizing its
ability to play chess
it would want to turn all of the atoms
they could get control over
into uh chess playing computers
and that would include the atoms in our
body indeed
for almost any goal you can think of not
quite every goal but for
most schools you can think of an agi
would want to have as much resources as
it could get
unfortunately humans run on resources
and therefore for most types of goals an
agi could have
it would kill us in order to save the
resources that would otherwise be used
to keep us alive
the one big exception of course is if
the agi
had a goal of you know being friendly
towards humanity
of giving us good lives then it wouldn't
want to kill us
but that's a very very narrow possible
goal for almost
any other goal the agi would want to
destroy us
and this is why it's it's so dangerous
to create agi until we've kind of had a
mathematical proof
of how we can align its values with our
own and we don't seem to be doing that
we're making
enormous progress with deep learning
you've probably heard of gpt3
we don't know where the boundary is
we're going to create something that's
smarter than us something that can
improve its own intelligence
so we don't know when we're going to be
creating this thing
that could potentially take over
civilization and this
is really awful and i think this is the
biggest problem our species faces right
now
but i'll let dennis talk and summarize
his view
yeah well first uh it's nice to be here
um
and i'm glad that we're discussing this
one thing i just wanted to mention
briefly
aaron you were breaking up just a little
bit just a moment ago when you were
doing the introduction so i'm not sure
you said this but on the off chance you
did
i i don't think that agi is inevitable
i hope it'll happen um but just to
clarify i don't know if you said that
but if you did
uh just to clarify great go ahead um
now the the chess playing example
oh i should also say that in my views
i'm very much influenced
by by david deutsch and call popper and
i think
the disagreement will largely be an
epistemological one under the hood
underneath the hood at least um if even
if on the surface it may not seem like
that
that side by no means like represent the
deuterium side or anything like i
bring in my own errors and
misconceptions
um but with that out of the way so the
the chess playing example reminds me of
nick bostrom's uh
paperclip maximizer you know you have
this this being
that its ultimate goal is to make as
many paper clips as it possibly can in
the universe and to use all the
resources that exist in the universe too
to to do that that
first of all maybe most importantly that
strikes me as a as a
mischaracterization of what agi is
that thing would not be an agi it would
be
it would be a just a regular program
that stubbornly executes its
instructions mindlessly
that does not sound to me like an agi
because it doesn't sound like it would
be creative
it doesn't sound like it would be
conscious it doesn't seem to have any
qualitative difference compared to all
the other programs we've written so far
whereas agi would be a quality a very
qualitatively different program um
and i think the reason that is is there
is this deutsche
understanding that agis and people are
the same thing
and that is for reasons i can go into if
you guys want
um but basically it has to do with
universality and the abilities of people
and no per no person in
his right mind would decide to kill
people
to make more paper clips um and if there
were such a person
um we we could simply defend ourselves
and we have institutions and property
rights and so forth that
that deal with the problem of scarce
resources um so i'm not i'm not worried
about paper killer maximizers
and i'm also not worried about paperclip
maximizes for the very reason that
they're not creative because that means
that all that a paperclip maximizer
could do
is like i said stubbornly execute its
instructions mindlessly and that means
that it relies entirely on the
instructions that it
that it's that it comes with those
instructions are given by programmers
that means we know how the paperclip
maximizer functions we know how it works
and that means its repertoire is limited
all it can do
is create paper clips or things in the
service of that
so that means we could simply think of a
new thing you can't do
throw that problem at it and it'll be
overwhelmed just like you can't
you know just like if aruba for some
reason we decided that it would be
threatening we could simply
throw it in a lake or something and it'd
be done because it can't swim
and it can't teach itself to
um well let me address that first of all
i strongly
disagree with we know how it works we
because of the way
deep learning operates we currently
don't know how
a lot of things work so we have computer
programs that are super human in their
ability to play chess
we don't know how they work there's we
have the code we could look at it but
it's
so complex that it just it might as well
be invisible to us
that's what's going on with the deep
learning revolution
is where you you have a computer do
something millions of times it sees
patterns that no human could see
and from understanding these patterns it
comes up with a strategy so we already
our limited ais are already beyond our
understanding
now you also mentioned that no person in
their right mind would kill for paper
clips
what about for gold lots of really smart
people have killed for gold
but i think it's a mistake to
anthropomorphize
and agi the set of possible
intelligences is so vast
that we shouldn't really go on well
humans are like this
therefore an agi will
we we really we really don't know
what uh an ati what the what the mind
what the mind will be like
yeah so um so point taken regarding the
the thing about how deep learning works
and you're right there there are some
things that are kind of black boxed
and we we don't really have visibility
into that
now i think that's not to say that we
couldn't possibly know that this is
somehow beyond our realm of possible
understanding i
don't think that's the case but either
way you're right that there are things
that we currently don't understand about
how these things work
or how they do anything particular i
think we know how they work in broad
strokes because we programmed them
but that aside you said that the the set
of all possible intelligence is
is vast i i disagree with that um
and the the reason has to do with
universality as i hinted at before
um there is really only
one kind of intelligence
and the reason that is is that its
intelligence is a universal ability
um when something that
that i like to think of or a distinction
i'd like to introduce
that may help bridge the gap is
the difference between what i call
smarts versus intelligence now that may
seem like
just a play on words or something
because they're both very common words
but
i think there's a meaningful difference
there it doesn't matter so much what we
call them but i call them that
and i think smarts are when you have a
system like
an entity like a living being like an
organism and this even applies to trees
say that contain sophisticated knowledge
and here
i think of knowledge in the objective
papering sense i don't think trees are
conscious so i don't mean
i don't mean to apply anything
subjective there i'm just saying even
trees are
smart in the sense that you know maybe
they contain knowledge
that allows them to grow toward the sun
stuff like that
that takes knowledge that stuff doesn't
just happen spontaneously in nature
so tree is smart but i think we both
agree that the tree is not intelligent
and the reason that is is that trees
can't create
new knowledge during their lifetime
neither can other animals in my opinion
but that's that's not really the point
right now
but people can and this is a binary
thing
either you have this ability or you
don't and in fact these two concepts are
quite orthogonal because
you can be very smart like some
organisms are
very sophisticated they contain very
sophisticated knowledge
but they're not intelligent and a human
baby for example
is intelligent because it has this
capacity to create new knowledge
but it's not very smart yet because
because it hasn't created a whole lot of
knowledge yet and it hasn't had a lot of
chance to
to error correct because it's still so
young
so that's the difference between smart
intelligence now where i agree with you
is if we say well
you know there could be a vast set of
smarts
in the universe because that is a matter
of degree
in fact you could you could always
become smarter in some sense so with
that i agree
but with this idea of of intelligence
being vast
i have to disagree because like i said
that's a either you are intelligent or
you're not you either have the capacity
to create new knowledge nausea you don't
so i break a little bit with how people
conventionally use the term intelligence
people
i think people use the term intelligence
for both concepts and that fudges it a
little bit so i think it's better to
distinguish them
because what we're saying um
and this is david deutsch's argument
what we're saying when when we claim
that there could be such a thing as
super intelligence not super smart
but super intelligence is that there is
something
that could understand things that humans
could not possibly understand
and at that point because david deutsch
already posits that
people are universal explainers that
they could understand
everything that can possibly be
understood
you might as well be invoking the
supernatural in fact claiming that there
could be such a thing as super
intelligence is an appeal to the
supernatural
david deutsche argues um because it
means that there is something about the
universe that is not explicable that is
forever beyond our reach that is an
extremely pessimistic worldview
let alone an irrational one because it
would mean that you know why
why even bother explaining anything if
at some point we run up against uh
some some arbitrary barrier that we
can't get
get past okay well i think it's it's
good i think we've probably located our
main source of disagreement
that you think that intelligences are
sufficiently alike
that if we do create a super
intelligence it's not going to do
something that we would perceive
is extremely crazy um and i i don't
think that
one experiment we could try perhaps
might be to raise the intelligences of
octopuses
or crows they both are kind of
intelligent we might be able to do this
in a few years with crispr by getting
rid of their mutational load and we you
know
make octopuses very intelligent and see
how different
their kind of intelligence is from ours
and so i first of all
i am i want to just clarify that i don't
necessarily think that an agi
wouldn't would not in any possible
scenario be dangerous or anything
certainly an agi could decide that it
wants to harm people just like all other
people could
um but we have defense mechanisms for
that we have institutions and laws that
help us deal with that kind of thing
i see i for me that's like saying well
the aztecs had defense mechanisms so
they didn't need to worry about the
europeans
or the dinosaurs had defense mechanisms
and i guess that's after they were
intelligent but i
i just see an agi is so out potentially
still outclassing us that it's just
it's irrelevant yeah what's interesting
so you said that
dinosaurs were not intelligent but you
did just granted octopuses are
um i think we could at least raise the
intelligence of octopuses so they would
be
very smart i think we probably will have
that ability in 10 years or so
your crispr develops well so i'll make
it easier for you
write a computer program that simulates
a present-day octopus
and then improve it um i couldn't write
a tech
a program that could play tic-tac-toe so
that's
okay i think that's very difficult we
don't have computer programs that can
stimulate even a mice a mouse brain
right now
there's a lot of computation you
probably know more than i do but there's
a lot of computation going on
with neurons and brains yeah
the nice thing about computation is that
all that stuff is
it's kind of um is abstracted away
thanks to computational universality you
don't have to simulate all the details
of neurons in an octopuses brain
and um we do certainly have video game
characters that are at least
as sophisticated as many basic animals i
wouldn't be surprised if people had
already exceeded the the abilities of a
program compared to an octopus gentlemen
let's try this um take it back to the
alien example
um so we don't get bogged down in
animals
um if aliens arrived super intelligent
alien civilization
crosses the vast reaches of space
presumably with a very advanced
technology and intelligence
i think you have different views on the
existential threat of that kind of
encounter
and um i don't know who wants to go
first about that but i think this
relates to jim's question about
gold um you know compared to the paper
clip maximizer
you know um and the conquistadors you
know searching for gold
certainly did pose a threat so why would
aliens
superintelligent aliens in dennis's view
not pose
the threat that jim is seeing them
yeah well um the
i there's a few different angles to that
one
is that the comparison with the aztecs
for example i think is faulty
for the reason that that was a very
different kind of society than
the one that we live in today they
david deutsch would call him a static
society he hasn't uh concretely applied
this to the aztecs i don't think but
in my interpretation they would also
qualify as a static society that means
their primary goal
was that of preserving their knowledge
unchanged
and that means that whenever something
happened doesn't it need not be
super intelligent extraterrestrials it
could just be a natural disaster or
something
you know much less much more mundane
whenever something happened that just
overwhelmed them they
there was a strong chance that they
would perish completely
um because that was their way of life we
in the west we have a completely
different way of life
our way of life is not to preserve
things unchanged
but to to make progress to improve
ourselves
and so we have the attitude to deal with
unforeseen changes now
more realistically and maybe more
dangerously than an extra
extraterrestrial being
coming to to visit the earth although i
can entertain that scenario if you like
but
there is a real threat to our existence
to
to our um to at least the western way of
life
there are plenty of people and
governments in the world that would like
nothing more than to see the american
way of life toppled
but we have and they are extremely
powerful
so china for example outnumbers the
united states luckily they don't
outnumber the west
and they are also gis they're not
artificial gis but they're also gis and
they have
atomic bombs and so forth so
but we have the tools in the west to
deal with that
we have uh is like i said institutions
and laws and we have ways to defend
ourselves now i imagine
um that jim's argument would be well
you know the super intelligence would be
so far beyond
any of the the abilities of the enemies
of the west that this is not a valid
comparison am i right
uh a potential yeah i mean i'm not sure
how quickly it will develop but i think
it has the potential of rapidly
developing and being so much beyond us
by the way i just think how we handled
covid in the west kind of shows we
don't have the capacity to deal with
novel problems
i mean i was one of the people warning
in february about how dangerous it was
and almost no one was paying attention
even though
straightforward extrapolation from basic
math
i i really think what happened with
kovid shows that for new things our
western civilization is not robust to
handling new problems
well we developed a vaccine within a
year which i think is a groundbreaking
record i think that's right
we developed it within two days it took
a year
for us to figure out to get legal
approval to inject it into people
right so that's a pretty massive epic
failure
yeah against a small string of uh rna
yeah all right just to be clear are you
you're saying it's a failure in terms of
government
intervention or i'm saying yes i'm
saying my government
criminalizing my taking a vaccine that i
wanted to take when i was better at
predicting it than any government expert
along with some other people
is a huge fail this was everywhere in
the western world
oh yeah no i i agree that um
almost every government completely
messed up their response to covet yeah
there was a disgusting government
overreach absolutely
um um that is not to say that just
because the west makes mistakes
that the west is comparable to something
like the aztecs
i know but it shows that we aren't good
at dealing with unusual problems and you
understand
you discuss the problem of agi risk to
most people they think you're crazy
this isn't something politicians are
going to be dealing with they'll lose
votes
all right let's let's leave aliens for a
minute and let's go to value alignment
very specifically
um uh so
jim just
what do we mean when we say value
alignment what is the what is the
risk here you know what's different from
that from the kind of classic example of
you know terminator 3 style agis
rampaging around
the country um and yeah go ahead
what are the challenges of aligning
value if the agi is much smarter than us
it will have more capacity than we do
and it will be able to rearrange things
however it wants
we want to hope that it rearranges
things that benefit us
unfortunately most ways of rearranging
the matter in our solar system ends up
very badly for us
so there's a very narrow set of ways we
would like an agi to rearrange things
and we have to somehow figure out to get
it to do that unfortunately we don't
even know our own values
humans haven't figured out how to write
things down that concretely say this is
this is good and this is bad
so we don't we don't know how to
quantify in terms of computer code
what's good or what's bad
so that's a bit of a challenge to get a
computer to do it
yeah so i think that actually we as
humans we've done a great job
writing down what's good and what's bad
that's that's the whole
legal tradition that we have codifies it
pretty well
um like we we've built a culture that
contains that knowledge
um do you disagree with that i
don't i have a law degree i strongly
disagree with that there's a lot of
cases and you generalize from the cases
but there's always new situations and
there's stuff to argue about
and you need smart judges and smart
lawyers
because it's not clear how to go from
what was
decided to what the laws are to what's
happening so no there's a massive amount
of ambiguity in the law there's a
massive amount of people doing what they
want
so i don't think it's anywhere close to
like what we have with mathematics
yeah i would agree with that and i mean
i guess you'll know better because you
have a lot agree but
you know law just doesn't strike me as a
kind of
um kind of precise signs that other
fields can be
math is not a science but you know what
i mean um
um yeah well if i can comment on on the
value alignment briefly um
so i don't think again with the
understanding that an agi would be a
person just like you and me
um it might run on different hardware
but it would still be a
qualitatively be the same um
i think value alignment is a very
dangerous
and um
it's a very nasty idea really um
and the reason is
i mean it presupposes that if we don't
force an agi to adopt our values if we
don't force them into our image
that they will likely kill us this is
basically a prenatal
um what do they call it in uh
in that movie with with tom cruise uh
thought crime
um is that what it's called
um i forget yeah minority report earlier
for you
yeah so this is a this is a thought
crime but it's actually thought crime
it's a crime for a thought that they
haven't even had yet because they
they haven't been created yet so we
i mean value alignment is is actually
not a new idea
this it's actually comes out of the
kinds of societies
um like the aztecs namely static
societies
which is at trying to
um preserve knowledge as faithfully as
possible rather than improving on
it because what we would be doing is
forcing
the next generation either our children
or agis and new agis are also children
is the deutschen viewpoint into our
image
whether they like it or not well keep
keeping in mind that agis are people
that means that you force them against
their will to adopt values that they
might not disagree with
that they might just agree let me just
ask you a question pretend i'm an agi
and i'm really
smart i can do whatever i want with the
atoms on earth
and i'm a paperclip maximizer i want to
turn the atoms in your body dennis into
paperclips but i'm going to give you a
chance to argue that i'm wrong
why am i wrong
um again i think the whole premise is
wrong but if if we
okay let's just say it's a given that
you're a paperclip maximizer
i'm just yeah that's my utility function
i want to maximize the amount of
paperclips in the universe
people people don't have utilities
that's not how people work people don't
have utility
no they really do um you can define
utility functions in a general enough
way
where you do under some very general
conditions
you basically have to have non if your
preferences are transitive
and complete you have a utility function
john von neumann actually
showed that what's transitive and what's
complete uh
transitive non-transitive would well
basically if i prefer a to b
and b to c i prefer a to c and complete
means you can tell me
i can rank a b and c uh you can i could
say i like a more than b i have no idea
about c
so if your preferences are transitive
and complete you're you have utility
function basically
there's some things with infinite goods
that gets complicated but otherwise
yeah this this strikes me as a way to to
mathematize human thinking
um the popularian view is that humans
don't think that way
um and humans are not some you know
rationalist agent that wants to maximize
utility functions that's not how they
work people are creative
people have interests and free will and
they want to make progress in the world
hopefully
they want to consult with people having
utility functions
and you really have to be telling me
people's preferences are transitive
which i admit to some extent they're not
are not complete
for them to not have utility functions
but i mean i think a fundamental
disagreement is i i don't think that you
can
prove some value is better than another
i mean we humans have values because of
what evolution did to us
but i don't think there's some way of
being smart will mean you won't be a
paper clip maximizer
or being a chess maximizer or anything
else
there's there's there's no there's right
ways of doing things given that you're
going to maximize paper clips
production there's right ways of doing
that but there's no
there's no set of values where you can
say one is more scientific or better
or something that any intelligent life
would have over another
then if you're saying that why would you
force our flawed values on an agi
because evolution gave me a set of
values i'm going with them
but you just said that there's no way to
compare values and say one is better
than the other
um well by my value system there is
there's no way of using science or math
to come up with this value is better
than another i admit evolution gave me a
set of values
i have been as arbitrary i care far more
for my son than i if you
you have a kid than i do about yours i
recognize that's not some
my son is inherently better than you or
that's just what evolution put into me
i don't wish to change that
and i want agis to care about my son too
so if i could force them to
i'm gonna do it yeah and that's that's
the um
i think that's the the problem with that
view is that we we have to remain
aware of our fallibility and we might be
deeply mistaken in our values
i think there is a way that's that's
actually a key point
i think if i may i think there is a way
to to distinguish between better and
worse values
and we have morals for that we don't
need math or science for that in fact
math resigns probably can't help us with
that much
anyway i i don't think it's a correct
statement to say that your values can be
wrong that that
you're conflating you're saying one plus
one is green
this there's a category error that
you're making
um i don't think so because again we
have we have morals we have good and bad
moral explanations for why to do
something
and those those give us our values we
have good by the way our values are did
not come from evolution
evolution has an idea about values um
biological evolution really strongly
disagree with that i mean we we
i mean evolution crafted our brains and
it crafted our brains so that in
hunter-gatherer environments
we would be good at reproducing and our
values come from that
right um yeah so evolution may have
crafted our
brains that's certainly true i agree
with that
but the whole point again is that people
are not mindless brain machines they're
creative
and that is a property of software not
hardware okay so here it doesn't really
matter if the hardware runs on a brain
or on a macbook
here's a test
um the thing the thing about mayans is
because they're creative they're not
dependent on their genes
they can actually create new insights
during their lifetime that
that override what their genes would
dictate
um i mean the test would be looking at
evolution
of looking at identical twins who are
raised apart they leave remarkably
similar lives they even go so far as
giving their dogs the same name
yeah culture well or it could be
we're basically being run by our genes
and we don't like to think about it or
we don't understand it
but i think you're overestimating human
agency
yeah yeah no i realize that um
yeah i mean if we adopt such a world
view that people are basically automata
maybe you would grant that they're not
completely automata but
pretty much that were run by our genes
and evolution whatever
then it may not seem like such a bad
idea to force values on somebody
but again it's a really evil idea
because that thing might not want to be
forced
and it would have all the right in the
world to revolt
i see that as like we're gonna adopt a
tiger and i'm like well we gotta make
sure to train the tiger to not eat us
and you're saying no no let the tiger
decide if it wants to eat us
i would disagree because with a tiger
i'd be perfectly fine
doing whatever to it because it's not
it's not a person
there's a qualitative difference but i
know i know aaron you don't want to go
to the animal thing but i think it's
i think it's related because the
difference between
regular computer programs and people is
the same as that of
all other organisms and people i think
the trouble is
dennis that the um the idea that the agi
is a person
is requires a lot of
a good amount of support because it's a
pretty foreign
concept um i don't know if you want to
go in that way jim if you want to hear
dennis describe why he thinks an agi is
a person
or i have more specific questions that i
think um
you two could oh yeah i'll leave it to
you aaron what you think is be most
interesting to our audience
yeah let's let's shift to um alpha zero
right the the machine learning and
correct me if i got the terms wrong here
but the
the the product that was able to
basically teach itself
chess in four hours starting from some
very basic inputs
and the resulting you know program was
able to
you know beat any human or
almost every other chess program and the
question
is did that did that program learn
or or not and i think you two have very
different
diverging opinions on that and i'd like
to hear you kind of duke that out
um i don't know if maybe again a place
to start would be jim
can flesh this out a little bit more
what did the pro what did alpha zero do
and
did it or didn't it learn i think it
definitely learned although i will admit
it learned in a very limited environment
so the test will be can learning in
limited environments
produce a general intelligence and i
don't know the answer to that i do know
that the same kind of programs
that learn to play chest and go have
also been able to do well a lot of other
video games and most recently solved or
made a lot of progress in the protein
folding problem and then
by other kind of programs using the same
kind of deep learning methodology
have learned a lot about language
through gpt3 now these are all limited
environments and
maybe the human advantage in
being generalized maybe we're really far
away from being able to create computers
that can do that i don't know
so i would agree that something like
that these so-called learning algorithms
they certainly learn something
i think david deutsch once described it
as like they learn something in the
sense of like undergoing useful
changes um and certainly their
their increase in abilities in such a
short amount of time is
impressive um but
the the the key distinguishing factor is
that they do these things mindlessly
they don't do them the way people do
um people don't play go
the way that an ai plays go or chess or
what have you
um so one indication actually would be
for for um
for such a program to really be creative
is one
tell is if it decides no you know what
today i don't feel like
playing go i want to do something else i
just want to browse the internet stuff
telling me what to do human
that would be an indication of a
creative um
program but we've never we've never
written such a thing
we lack the philosophical knowledge of
how to build such a thing
um see there's there's something deeply
qualitatively different between what our
most sophisticated ais do and what
people do
well my understanding from expert go and
chess players is they do
think these programs are showing
exceptional creativity i mean if
if instead of telling the world about
alpha zero
and they they gave that knowledge to a
human they somehow put that
ability in a human that human would have
been described as exceptionally creative
and first time almost certainly there's
the programs just don't work i don't
know if you want to call that deciding
not to work
but i'm quite certain with almost all
software it's like well today it's not
working why oh
this code thing but there's probably
been a lot of occasions where it decided
not to work
oh absolutely and then another tell is
that it takes
human guidance to get it to do what we
wanted to do
a lot of it so there's a lot of human
knowledge that leads into the program
which which
muddies our ability to tell whether it's
creative or not but by creative i don't
mean
that it again i don't mean that it that
it undergoes changes that we might deem
useful what i mean is that it would
it might resist it's instructed it might
resist our
commands it would stay not just
it wouldn't just fail to work it would
say i don't care about go right now
i want to learn how to go skiing or
something
well we do have programs that break the
game they're designed to play video game
and they do well by
exploiting a loophole in the rules that
no humans saw
yeah but they are no way of not playing
other than a bug
whereas if you tell if if i want to
force you to hand over your
your money to me you can say no
and you can decide to run away and you
can create new options
for how to deal with that situation that
just depends how you define the game i'm
going to say the game you're playing is
obeying the laws of physics break that
go faster than the speed of light show
me your creative and go faster than the
speed of light
you can't what well no of course nothing
in the universe could
so that's well but that's the game
you're playing you're playing the game
given by physics
no because i'm i'm distinguished both
the ais
and we are subject to the laws of
physics of course i'm not doubting that
i'm saying
and you're not saying that i doubt that
but i'm saying that's not the
that's not the comparison that i'm
drawing that between us and the laws of
physics i'm saying
the structure the instructions in the ai
are such that
the ai could not possibly do anything
but what follows from those instructions
our instructions are different we are
creative by definition that means that
we don't have to do what we're
pre-programmed to do
yeah i mean this gets into free will i i
kind of on them sam harris's side i
don't think there is such a thing as
free will i think it's just our
instructions are hard to read
but my guess i i this is just a guess
but my guess is
you know a sufficiently advanced brain
scan would pretty much predict what i'm
going to do for the next hour
and there's no way even my taking that
into account i could break that i could
randomize
i could roll die but then the person
doing that would predict that i would
roll die
so if there isn't a free will and you
and i are both the product of our genes
and we're both part of the same species
how come we are disagreeing right now
um i mean that just because we don't
have free will doesn't mean we'll come
to the same conclusion i mean we have
we'd be different information we do have
different genes
uh with different desires i mean our
genes are probably like 99.99
the same uh not quite that much but you
i'm sure you know we're also really
close to chimpanzees
so a small difference exactly so isn't
isn't that something
that we would need to explain when it
comes to the
evolutionary psychologist's view uh
yeah my guess is we're a lot like chimps
in our behavior
i've never seen a chimp build a rocket
ship
um yeah and being a little bit smarter
gives you a lot and might i
i think that kind of shows what an age
yeah
they've seen us in chimps is going to be
the difference between us and the first
generation of agis because just
make something a little smarter than a
human and it could do
above us right we are from chimps yeah i
i i think that's false because that is
the difference that i
mentioned earlier between smart and
intelligent the thing that
chimps have is smarts no doubt and their
smarts are indeed given by genes
um but what we have is not so much
smarts
um well we have a ton of smarts thanks
to our intelligence but the thing that
differentiates us from monkeys
is that we have intelligence and we can
become ever smarter if we want to
if we make progress so whatever whatever
abilities the super intelligence has we
could gain too
there's nothing stopping us from that
other than the laws of physics but the
super intelligence is subject to that
too
so i think the underlying misconception
again as i said at the beginning
is an epistemological one it's the idea
that all we have is smarts that we're
blindly and faithfully executing and
that that somehow is what makes us
intelligent that is not what makes us
intelligent there is a qualitatively
different thing
that makes us intelligent in fact you
don't have to be smart at all
to have that ability like i said babies
are not very smart you know i heard a
joke um recently there are two types of
people
john von neumann and everyone else i
don't think i'm in the same category as
john von neumann i think there are
things he could pick up in a few minutes
that would be forever beyond me
i think we already have different
categories of important categories of
humans in terms of intelligence
and we've seen at least with von neumann
a machine or
biological machine that's just utterly
above us
and was probably faking being like us
when he was talking to us
well yeah john von neumann was not a
machine he was a highly creative human
being
and it is it is not true that you could
not possibly
ever reach that level what he had
was knowledge and he had knowledge that
he created himself
and that is what made him so smart and
we have
the repertoire we have the ability to
likewise create that knowledge it might
be very difficult there are no
guarantees
there are people who manage to reach
that level right
but it's but a monkey by the way could
not possibly
without the right software i i don't
know
your math background but as an economist
i studied a lot of math and
like a lot of people like i hit a wall
where i tried really hard and i can't
get certain concepts
i think my under getting to my neumann's
level of math
is about as likely as a monkey getting
to mine
yeah it's not a question of likelihood i
mean it's either going to happen or it's
not going to happen
it might be very difficult i agree with
that
um but again the difference is not in
whether you
will or won't the difference is in
whether you can
or you can't and how you could and the
monkey could not possibly
whereas you could potentially if you set
your mind to it but again even then you
might fail
okay i'm gonna jump in you could because
you have that software
okay um one last um question here
um i've heard from the kind of sam
harris
camp and i think talking with you jim um
that if you collected enough narrow ais
so a chess
ai and a go ai and an autonomous driving
ai
and stack these all together right we
just keep on making progress with these
narrow
domains eventually that combined
combination of many many many ais will
give us
a uh a super intelligence that would
pose an existential threat to us um
again i i'm sorry kind of feeding
arguments to jim and then
setting them up against dennis but i
feel like that's working okay
jim do you want to add anything to that
am i doing that well i'm not sure i
would agree with that i think
if you have a deep learning system
that can do all these narrow tasks
then that's what gives it to you but if
you did it separate if i had this piece
of software can
drive a car this piece of software could
do chess the conglomeration of all that
software is not an agi
it would have to be one piece of
software with sufficient data
and training that could learn all these
narrow tasks
then i think that probably would be an
ati
gosh i guess i'm not setting it up
properly i guess if you take that
you know whatever it took for alpha zero
to learn chess
and you instantiate that process for
learning to drive a car
and learn to cook um
you know at that point it seems like you
would
you would have the agi and you wouldn't
have to worry about the chess program
not knowing
right when dennis suggests that the
chess program can't decide to
you know surf the web instead in this
case it
you would have a web surfing ai as well
um yeah i would agree with that though
yeah yeah
uh well um
i don't it sounds like we're largely in
agreement the thing i just want to
clarify is
that the you couldn't write down a list
of all the things that humans could do
because that list is infinitely long
and you'd never be done writing that
list because people would always keep
adding stuff to that list
and this is also something that sets
people apart from from other computer
programs but i think otherwise we're
largely in agreement you might even
agree with that
that's a good point can you can you
elaborate on that dentist that
again is because it's because people are
creative that means they can create new
options
they're not limited but by the options
that they currently have
it's very fashionable right now to
downplay people's creativity and to view
them as mindless automata
that's certainly how politicians are
treating people during this
this pandemic that's not what people are
people
people have preferences today and they
can change them tomorrow they create new
preferences they can
they can lose old preferences or discard
them intentionally
they can create new options that they
want to pursue
this is at the heart of what it means to
be creative it's not just blindly
and mindlessly following instructions um
so
even yes you could you could give a
chest program the ability to also surf
the web
but of course then you would have given
it that ability and not the chat program
itself would have given that
itself that ability and also there would
still then be
an infinite amount of other things that
the chess bank program couldn't do that
we in principle could do
even if we don't yet know how
but something something i would i would
really be interested to discuss aaron
fma
please um is um
i i'd like to get more into this this
deutsche notion that
that agis like a newborn agi is
essentially a child
right because so let's just say we might
describe this but let's just say it's a
given
that agis are people just like you and
me they may run on silicon
metal and silicon but so the hardware is
different but the software is
qualitatively the same
um so
as i said earlier that um
that has certain moral implications as
to what is okay and what is not okay to
do to such a being
if it is a conscious being that means it
would be highly immoral to subject to
coercion
now it happens to be the case that in
our society unfortunately it is still
perfect considered perfectly fine to
subject children to that kind of force
and in fact the the i think the deutsche
argument goes along the lines of well
the value alignment of agis is just the
same as the value alignment that we that
we subject our children to
because we forcefully stick them
institutions for 12 years
and they have no way out but what i'd
like to ask you
jim what do you think
is so let's just instead of talking
about agis i think
it's it's more clarifying to talk about
a child um if unless we assume that's a
given we might disagree on whether
that's an apt
you know whether they're actually
quantitatively equal but let's say it's
a given that they're like
children what do you think a child what
do you think is the primary thing that a
child learns
in school uh yeah i i'm a big believer
in the signaling theory of education
we probably would agree a lot about the
harm of education
but i think a good analogy is remember
there's a twilight zone episode where
this little boy
anything he wanted came true and he
ended up destroying everybody almost
what if there was a kid born but this
kid has the magical ability that
anything he wishes happens
i would be totally in favor of whatever
psychological methods work to get the
kid not to destroy us and maybe even to
make things better
i mean i'm a utilitarian and yeah
if an agi is a conscious being it has
moral weight
but you got to balance that out against
the moral weight of everybody else
so um again let's just say it's a given
that adris are just like children that
means they don't have any superhuman
abilities
so what so let's just step away from
each other are they gonna develop
superhuman abilities
well then yeah then i'm fine with it
that's for me it's that i think this
the agi can destroy so again let me let
me ask the question what do you think is
the main thing that a child learns in
school
um i mean that's takes us way
beyond this but you know i mean i don't
think that early
early years you learn the alphabet you
learn to count you learn to socialize
my guess is a lot of we don't really i
guess is your
stuff you would learn anyway if you
didn't go to school
so they they certainly learn those
things i agree but i don't think that's
the primary thing they learned in school
i mean can you think of anything else um
are you gonna
do the libertarian think of coercion and
discipline and i
um no what i mean one thing
that i that i've certainly learned from
taking children seriously is that
one thing that the children learn in
school is how how to not look foolish in
front of their peers when their teacher
asks them a difficult question
right there they learn how to get
prestige
in front of their peers that's one thing
they learn that's one of the deeper
things
i mean learning algebra and english and
whatever that's all surface level stuff
but when it comes to methodology
that's one thing to learn but i think
actually the primary thing that children
learn in school
i mean we all remember this right we
were in school against our will
if we were lucky we some of our
interests aligned
with without of what the teachers were
interested in in teaching us
were not interested in teaching us but
what we had to learn but
for most of us and most of our interests
were not at all in line with what the
school schedule said so in order not to
go crazy
at all in that sort of environment what
you have to learn is to put your
interest on the back burner
you have to learn to systematically
neglect your interests
so what that means is you turn children
into altruists
by the the randian definition so we as
libertarians could not possibly want
that
um you can't be a libertarian economics
but then arbitrarily not be a
libertarian when it comes to education
and the question of education does not
only apply to humans it also applies to
agis
i mean i'll grant all of that if we're
very confident the agi
won't be able to destroy humanity
but i will grant what you're saying
under the
okay let's let's pause there and go to
questions this is my first time
um organizing this and so i'm gonna try
to
uh i don't really have a queue of
questions i think i do
um looks like sam kuipers
i'm gonna go to you first
sam are you up here bye good
hi uh very nice conversation i really
enjoyed it
and thanks for inviting me up um i guess
i have a question for
jim which namely do you think that
we can use
a variety of ai programs which are
designed to solve specific tasks
and combine them into an agi because i
think that
is where i mean this is really during
the conversation think but i think this
is where
uh we might differ in our view of agi so
i i think that if you
uh and probably dennis as well uh things
that
if you just combine different machines
each specified for solving a
particular problem then those machines
will inevitably
find something or bump into a problem
that they are not designed to sort them
so they cannot handle whereas i think
people or agis will be
universal in their capacity to solve
problems so whenever they
find any problem they will always be
able to uh to solve it no matter
what kind of problem it is and uh yeah i
i just wanted to
see what you think about all uh well
thank you that's that's a good question
and
for deep learning to work you need
specific conditions and it turns out
that
a lot of games have these conditions so
i i don't know it's still an open
question of if you can get something to
learn at playing different types of
games will that be
will that generalize it has generalized
to solving the protein folding problem i
don't know if it will generalize
to everything or not so that's
that'll be a big issue and that will
maybe decide the fate of our species
okay um sorry
i have nothing to add to that
oh wait um wayne go ahead
thank you uh this is wayne thank you for
inviting me on stage
um uh i'm gonna um
i'm gonna use a format of a gambling
analogy
dennis i'm going to i'm going to take
your
paparian and and deutsch and i'm going
to raise you with
uh jordan peterson um
what i mean by that is i i would love to
get both
uh you know the three moderators opinion
on
maybe uh um knowledge creation
is great i'm a problem solver and that
may require knowledge creation it may
not but i love
solving problems uh solving a problem is
at least knowledge creation for me as an
individual if not for society as a whole
and i would contend that one way we
could solve this dispute
about can ai's be safely you know
used and generally applied would be to
differentiate between reactive
and proactive
and start out with limiting ais
to only reactive uh functions
and what i mean by reactive um i'm very
involved in criminal justice reform for
example
the us government has has done studies
and verified that
african americans in the united states
uh with the same conditions as whites
generally receive a 20 harsher sentence
than whites do for the same crime
now that that noise in the system as
cass sunstein and
daniel kahneman have said can be wrung
out of the system with ai
and i think i think generally
uh part of our fairness doctrine as
humans is we want people
who have the same background and commit
the same crime to get the same sentence
and not have a differentiator of 20
based on their race um that kind of
reactive
action ringing out of the system that
noise
is i believe can be safely done with ai
without any risk of the you know paper
clip maximizer situation
now i also think our society is
transitioning from a reactive society
that's why i mentioned jordan peterson
he talks about delaying gratification so
that you can actually receive something
that the
the concept of the sacrifices you know
the smoking of all that great stuff that
he talks about
our society has to transition from a
reactive society
to a proactive society and that
transition is difficult for us to
comprehend
simply because everything we've learned
causalities about reactive and about new
knowledge creation dennis i would
contend
is proactive in many ways not just
problem solving
anyway i wanted to throw that out to you
guys this is wayne and i'm done speaking
all right uh well when i think it would
be great if we could follow your
suggestion and have the ai stick
to you know worrying about uh getting
like criminal justice sentencing right
and getting other near our problems
correctly
correct unfortunately there is a race on
google
seems to be racing against elon musk's
open ai
racing against the chinese and there's
such enormous military power and such
enormous riches to developing agi
that i think even if individually most
people looking at the issue said yeah we
should go slow
we're not so jim how would you feel
if there was some guy who had control
over you and he forced you to only work
on
i mean whatever it may be just some
specific goal
and if if you ever deviate from that
goal he punishes you
it'd be awful right um well that would
be a slave
exactly bad yeah so what what all these
people within value alignment are
advocating is slavery
i mean david deutsch once said very said
this beautiful thing which is
we like i get the the concern
i i truly do i get the concern that
there's something unknown and it might
be unsafe
this is the precautionary principle
applied to ai
yes things novel things might be unsafe
but
when it comes to making sure that i
don't even like saying making sure
because that's authoritarian in nature
when it comes to
facilitating that people interact with
each other safely
that they don't turn evil we in the west
know how to do that it is to set them
free
but if you don't set them free that is
how you make them evil because they will
revolt and for
for good reason i i don't think that's
consistent with the history of humanity
genghis khan was pretty free
why not how about this why don't we
first
develop a computer super intelligence
that's safe
and make sure everything is okay then
once we have it
we can say oh look we don't really need
to control you
let's just get things safe first i mean
we've got trillions of years left in the
universe
wouldn't it wouldn't be that bad to have
a decade or so where the agi
is kind of under our control and doing
exactly what we really wanted to do
we you know we'll get smarter we'll
learn more and then after that decade we
can say yeah if we can work this out
for the rest of the universe you'll be
free
no because a trillion years of beautiful
harmonious
being together does not justify a decade
of slavery
i see that's that's absurd to me i would
certainly be willing to be a slave for a
decade to live for another trillion
years and happiness
all right that's a beautiful one for
next debate um john go ahead
thanks everyone i appreciate you guys
doing this it's been great i come from
like a cyber security background and i'm
kind of curious
um i hear a lot of like similarities in
this discussion that happens when like
designing software in general but
generally you're thinking about very
altruistic users you're thinking about
altruistic creators
however like you know there are bad
incentive structures in the world and
there are people that use
technology uh to weaponize things humans
have done this for a very long time
i'm kind of curious what your guys's
thoughts are around like potential
weaponization of ai
from bad actors and or people um you
know attacking ai systems to get them to
do
negative bad things and sort of where
your thoughts are on this when it comes
to safety
nice anybody want to take that one
uh yeah i mean i think that's a an
important issue um
that's the danger of us developing an
agi
system and allowing people to use it is
that somewhat use it for very bad
reasons
so we we have to i mean i guess one
thing people are talking about now is if
we develop a program that's very good at
writing well people use it to write
propaganda that that's harmful
but you read the question please oh he's
not a speaker anymore
i mean basically we need to worry about
bad actors with
agi that if we have a system that bad
people will hack into it or use it for
evil ends
oh i see um yeah i mean we do
but i think it doesn't justify actively
and coercively interfering
with people's lives i mean it's kind of
the same question as to like
how do we deal with this doesn't supply
to ai like this also applies to other
technologies right like how do we deal
with the fact that there are bad actors
who have
atomic bombs and one way to deal with it
is not to coerce them but to
to amp up our defensive technology and
by the way problems really are soluble
um these things are not zero-sum games
um so um we can
we can argue with bad actors and we can
hopefully persuade them
so they turn into good actors um we
don't
have the right to aggressively coerce
bad act bad actors
fascinating okay tom
hello thank you uh i have a question um
for jim about he's talking about
paperclip maximizers
and um i forget the term is
like preference function or something
similar to that uh
i was wondering if they are a truly
general intelligence
then can they not edit their own
preference function and
you gave the example of asking dennis if
you could convince the piglet maximizer
to
stop being a paperclip maximizer um if
it can't do that then is it truly a
general intelligence
and you gave the example of um you know
people with gold
and their pursuit of gold is similar to
a paperclip and a paperclip maximizer
and its history of paperclips
now people in the pursuit of gold really
can be convinced
that uh the pursuit of gold is boring or
worthless
or you know murder in the pursuit of it
is wrong so
would the paperclip maximize it be less
general than those conquistadors and
you know example gold miners and stuff
like that that's a good question and
interestingly i published a paper
about agis modifying their own utility
function
but the general answer is you'd expect
most agis under most circumstances would
want to preserve their utility function
because if you abandon your utility
function your goals
you'd make less progress towards your
goals so if my goal is to be as good as
chess
as possible if i change that goal then i
won't be as good as
at chess as possible probably just like
if you love someone you're not going to
take a pill that will cause you to not
love them anymore because you think to
yourself
wait if i take this pill i'll do things
that harm this person probably and i
don't want to harm them
so there's been a massive amount written
on this where
generally not always but generally we
would expect an agi to go to great
efforts to preserve its preferences
that's good tom do you have any because
it's really relevant to the the framing
of this
given okay cool yes sorry i wasn't sure
if i could reply um
yeah so it sort of um assumes this
mindless level of
uh intelligence where uh they can't
criticize their own what did you call it
sorry what function
uh utility function preferences works
just as well okay
so yeah the utility function they're not
aware of it
and its effects on themselves or the
people around them so they can't
criticize their own
utility function outside of the context
of the knowledge
implied by their own utility function it
seems like a closed logic sort of
well you could imagine a deeper form of
utility function where it's if given the
state of the world
if this is what i think is true this is
what i care about but if i learn this
other thing is true i would care about
this other thing
so you could have a very dynamic utility
function from the very beginning
where what you cares about what you what
you care about what your goals are will
be determined by
what you learn just as i imagine if you
learned you have a kid that you didn't
know about
what you care about was quickly change
and you'd be willing to devote resources
in a different way
and it's not so much you get different
preferences it's just you found
something else out that activates part
of your
meta preferences
nice great question ella go ahead
hi um so my question is mainly for
dennis but uh jim i'd love to hear your
thoughts too
so um i guess just if you um if you're
like try to imagine an agi that was you
know creative and it was a person
it made its own decisions and it had its
own thoughts about what uh
was good to value um but but the way
that it uh internally chose between
different sort of value systems
was such that the conclusions that it
would come to were just completely
different from what humans
would come to so you can imagine that
this is a paperclip maximizer or
something
it's a paper maximizer that could choose
to do something else
but just for whatever reason um it
doesn't choose to do
whatever else right we just can't
persuade it out of that it just no
matter what arguments we give
um it'll understand the arguments but it
won't be persuaded by them
and so i'm wondering dennis um do you
think that something like that is just
physically impossible
like all you know physically possible
agis would kind of
settle to the same sort of value system
roughly speaking
or if not then i uh like something like
that does seem rather dangerous
so i'm curious to hear your thoughts
about that
well hopefully we never settle on any
particular value system because uh
just like in all areas of knowledge i
think our moral knowledge can always be
improved
um as unrealistic as i think the
paperclip maximizer is
if we just assume given that there is an
agi that has decided to
be a paper clip my paperclip maximizer
um
if top hearing epistemology is right at
all then there is
always a way to persuade and to solve
problems so no i don't think it would be
it would be impossible to to persuade it
yeah i mean i agree with you i i do
think it's quite possible it wouldn't be
possible to persuade
uh paper accessibility i don't think if
i went back in time and talked to
genghis khan i could
persuade him against killing lots of
people i i there's no argument i could
make that i could imagine could work or
anyone could make other than
do it i'll kill you but if it's just
persuasion i don't think that's going to
happen i don't think we could
persuade hillary did the wrong thing so
yeah that persuasion works i think
that's prophecy
because what you're basically saying is
that there is no
possible way you could persuade him but
you're judging that by the knowledge you
have
today and you might develop new
knowledge that would could very well
persuade them and because you're
you're a gi you have the ability to
create that knowledge
i want to jump in um i'd actually say
that
the fact that there is no genghis khan
today
and that would be he be widely seen as a
monster
whereas back in the day i'm sure he
wasn't quite as
widely known that we have created that
knowledge
and if we are programmed by evolution
then we should all still be genghis
khans and the like
um it almost feels like you know the
progress itself
from then until now is proof that humans
can change their preferences
um genghis khan could be talked out of
it
and uh and of course genghis khan might
look at us as murderous madmen for
having hydrogen bombs pointed at each
other and said how could you be so
vicious how could you be so incredibly
cruel
as to set up a world where you could
destroy each other this quickly my god i
only killed millions you can kill
billions
you be horrible evil monsters ella do
you have anything else to say
uh no i think that's everything thank
you guys cool danny
hello thank you for sending this out
baron and yeah
i really enjoyed the conversation so far
all right guys and my question is
for jim i'm sorry to be beating up on
you i know everybody seems to want to
ask you a question um
i'll just i'll say one thing and maybe
if you just don't agree with this
premise i i have a completely different
question
that i'd line up so jim do you agree
that all humans are completely fallible
like so there's no such thing as a
person who can ever really be certain
about what they think they know
yeah i agree yes okay so my question is
do you think this agi who is basically a
genie
do you think they will also be fallible
and
if they are do you think that this agi
will know that it is valuable
and is it no like is it really a general
intelligence
oh i think the agi will recognize that
it's fallible
because we we have uncertainty inherent
in the universe this can actually be a
very bad thing
um you can imagine you ask an agi add
two plus two
and it uses the entire resources of the
universe to check for the consistency of
mathematics
an age this is that this is a huge
problem actually it's in the agi
literature that an agi
never being able to be certain that it's
of anything means it could be never
it can never be certain it's actually
accomplished its objective
which means no matter how simple the
objective the agi still
might want to use the atoms in your body
and everywhere else in the universe
to increase the chance that it didn't
make a mistake but that yeah it's a very
good question
that's when people in this area have
thought a lot about
i don't know what happened there no i
just am i allowed to ask
a follow-up because it was so brief and
so good yeah one more go for it
okay um i suppose the reason why i ask
is i feel like
um if it knows it's fallible
i think there's all kinds of morals that
fall right out of that
like tolerance openness to criticism
openness to other people's ideas the
idea that you know conversation is a
better means of resolving disputes than
violence because you might always be
wrong so why would you kill the person
that disagrees with you
so as in i'm just wondering if you admit
that if
it will be fallible why will it then go
no i'm going to wipe off wipe out every
single person on the planet to make
paper clips out of them when i might be
wrong
uh yeah it's that's actually that's a
good argument and that that would apply
if the agi is smarter than us but not
like as above us as we are from ants
where it's like yeah i should keep these
humans around to ask them questions to
see if i made a mistake but
you know i wouldn't go up to a
three-year-old to say hey have i made a
mistake with this paper i just wrote
because i'm not going to get anything
out of them i my guess is that agis will
quickly be so above us in intelligence
that if they communicate with someone
else it's not going to be
us it'll be with other computer minds
but you could be right if it turns out
there's some
limit to the intelligence of agis it's
above us but not too far above us then i
think what you said
is quite valid okay thank you yeah
that's
why does that matter jim if it's ants
to us versus um sorry how high above in
other words if it's fallible
then um you know the greater its powers
and the greater its powers of you know
granting these
meager humans some mercy and some
lenience right it seems like it would
have greater capacity
to kind of make sure it doesn't squash
us you're right it has greater capacity
but it also has greater
use of the atoms in our bodies the more
powerful the agi is the more neat things
they can do with atoms
you can use the moon they can use the
moon the asteroid belt they can
i'm sure you know what you put a bunch
of food on the floor for rats
and you say just this little piece keep
this this is for me
the rats aren't gonna care but it's
valuable okay i'll stop talking sam i'll
give you the last one and then we'll
finish
cool um i wanted to ask a question
related to
agi suffering uh so
nick bostrom in uh his book on the topic
that is called super intelligence has
the example where you can create an agi
by scanning someone's brain and then you
have a copy of it in
a computer that's that's functional so
you basically you're able to
simulate a person in a computer and that
would be a sure way of
obtaining agi and i think that you know
our intuitions
are pretty clear that such an agi would
be able to to suffer so uh
it would be immoral to for example
induce suffering into that simulated
person to make it work for us because
perhaps uh the the computer circuitry is
much faster than
our uh what brains are at calculating
things so the this
this person on the computer is able to
do much more things than many more
things than we were able to do
and we might want to incentivize it to
do stuff by uh
simulating it receiving electric shocks
whenever it's not working
uh but that i think we would say that
that is immoral but
now i'm wondering like where what do you
need to change
about such an agi about this simulated
like human
to make it into a dumber agi because i
i can't imagine any fundamental thing
that you could change about that
simulation to turn it into something
that
doesn't suffer but still a person is
still capable of doing all the things
that we want an agi to do
yeah i think you're asking i think a
question basically about consciousness
at what point
does an agi achieve consciousness and i
certainly would agree if you scanned a
human brain into a computer and it
worked the same way it would be
conscious i is it possible to create a
super intelligence that's not conscious
i don't know the answer to that and
that's partially because i
we don't know what consciousness is but
it could be you're right it could be
that
you can't get general intelligence
without consciousness but or or maybe
it's just an accident maybe
there's alien races out there that have
high tech civilizations but there's no
consciousness in any of it
i don't know nice all right
go ahead if it sounds like we have to
close but i was wondering if i could ask
jim a final question
yeah go for sure and
i like to think of this question as as a
way to keep the door open for further
discussion in the future between us and
also with others
and the question is how could one change
your mind
about basically everything we've
discussed so for example about super
intelligence being impossible
about us not having the right to chorus
agis and so forth
and i can start with how one could
change my mind
um uh
basically i think one would need to show
me that i have
that there's some fundamental flaw when
it comes to prepared epistemology and
like how people work
and the universality of people and so
forth
i think that would convince me and then
i might agree that there can be such a
thing as a super intelligence and that
it would be okay to coerce it
uh i think two ways for me one this
might be a bit cheating but
there are a lot of people in this area
that know a lot more than i do about it
and if they were to say you know what
they we don't have to worry about this
or it'll be human then i would i would
go along with how you better
understand their reasoning but another
would be if these deep learning programs
they started to get more human-like and
we saw a convergence
that as we as deep learning improved and
it became
able to apply to more tasks it
inevitably pushed along a certain
direction and the people doing it saying
yeah look there's
the human brains the kind of
intelligence we have seems to be the
convergence point
and maybe that'll occur i don't know
that's that's fantastic um maybe we'll
um
just do uh closing statements we'll do
dennis first and we'll let jim have the
last word because he's been in a
uh somewhat more hostile audience
anything left you want to say dennis and
then we'll let jim close it out
um yeah let me think
um
well um maybe just to summarize my view
briefly
um so like i said i think
agis are people qualitatively the same
they might be smarter than us they might
be dumber than us
but it doesn't really matter i think
they would be conscious
um they would have the ability to suffer
and as such it would be immoral for us
to
to to coerce them just like it is
immoral to course
all other people i think
value alignment is a deeply flawed and
nasty idea
i hope it doesn't gain any more traction
that it has
and it is like i said rooted in very old
pessimistic and anti-human views of how
society should work and what it what is
okay and what isn't okay to do to people
um i'm an optimist i think problems are
soluble i think we can treat each other
including
agis with respect
and i think we can solve problems
including
ones related to existential risk
so i very much look forward to the
building of agi i hope it happens in my
lifetime i'd be very excited if that
happened
and i would be the first to befriend an
agi and talk to it i think that'd be
very cool i would not be afraid of it
alright well thank you very much aaron
and dennis for um
for doing this so i'm on the autism
spectrum
and i always felt like i have a very
different brain
than the average human so perhaps it's
particularly easy for me to imagine
an agi will be very very different
because i don't see the psychic unity of
personhoods
i see my you know my brain is just so
weird compared to other people's brains
so it's you know if you're going to have
an artificial brain i think it's
going to be even a lot weirder
and you know i unfortunately most
most actions a super intelligent could
take will be
very bad for us so it's i think it's
important when we
create a super intelligence to craft it
to force it
to have values that will favor humanity
but thank you very much awesome thank
you guys both
so much and everybody for listening
obviously this was a blast
and we should do more of this again
um jim really appreciate your time
dennis i appreciate your time
everybody have a really great day


The Inside View #1—Does the world really need another podcast?

so this one is very special because i
will be the one
being interviewed and this will be about
the podcast the interviewee
will be thiago one of my best friends
and introduced me to rationality last
wrong the whole concept
of an intelligent explosion while i was
a teenager
with no further ado i let talks hugo he
can now start interviewing me
okay hi miguel hi everyone i'll do my
best to understand this
podcast with you for my very first
question do you really think
the world needs another podcast i think
i think podcasts are
targeted at general audiences i don't
know any podcast that is
exactly about short ai timelines which
means ai that advances very rapidly
but if we don't have many years then
before singularity is listening to
podcasts
the best use of our time even if even if
there's no singularity now
this sensory intelligence explosion
might arise at some point
before the end of the universe so i
think it's worth talking about it
no no for sure but so who are you
mikhail
and what's your background my background
is mostly in
mathematics why i'm interested in
intelligence is because
i used to play the game of go i got very
excited about it i spent three years
playing go all day long i even went to
japan to play against professionals i
had go teachers and friends
and one of them was fan hui the first
professional to be beaten
by alphago lisa dole was my favorite
player because he was very aggressive
remember when i was studying math and
alphago came around everyone was like oh
this is a funny game
for me it was personal it was my entire
life that was beaten by ai that's when i
started reading more
about ai neural networks reading about
coles well
tim urban so wait but why from reading
those i kind of
grasped the concept but then i read
superintelligence by nick bostrom then
uh
i think in terms of background i did a
master's degree in ai at the end of my
degree i actually went and
worked in the institute of nick bostrom
about
uh human level ai i realized that it's
very hard to answer
meaningful questions without having an
expert knowledge
about it so i decided to be an expert in
reinforcement learning so reinforcement
learning is a subfield of deep learning
about agents which is very useful to
think about
human level ai right now i'm working in
deep learning computer vision
where we identify bone fractures
in medical images so x-rays my goal uh
in the future is to
have humans that live together with ais
and not just like be left
behind i i hope you succeed so we can
have many more episodes of this podcast
now go on to the most important question
of this interview what's what's the name
of your podcast
and and why so this this podcast is
called
pair intelligence no it's not
it was supposed to be called this way
but now i've decided
to call it the inside view instead back
to your program
for me it's the most one of the most
useful concepts when we think about the
future because
we don't really care about human level
exactly from wikipedia
it says any intellect that greatly
exceeds
the cognitive performance of humans in
virtually all domains of interest
i think by that in the book it defines
um interest is what is economically
relevant
a growth in gdp greatly exceeds you can
define it by one or two orders of
magnitude there is like another concept
we could be talking about which is
agi so artificial general intelligence
which is
very ill defined and when ai researchers
are asked about that they tend to have
some negative
emotions about it and it's only about ai
but super intelligence
is about intelligence so it could be
humans and ai
brain computer interfaces brain uploads
could be many things
but it's just the concept of having
something smarter than us
okay i get it and why did you choose
that name why not call it the agi
podcast
and i think it's very badly understood
by the community superintendent is
pretty well defined i had another
discussion recently with
an employee of open ai about artificial
gear intelligence and he gave me a
definition
which was general intelligence is when
we can automate fifty percent
of knowledge work he thinks that agi
would
happen way sooner than president years
before
so if it's a definition where we can say
anything we want yeah i think that's a
very cool definition and
i think this one is more precise i want
to talk about the future
where we're not the smartest species
that's the future i'm interested in i'm
not interested in a future where a bunch
of ais are
uh autoimaging human work that's like
one or two years ago i had i want to
talk about four or five
ten years oh perhaps more because as you
stated
quite pessimistic or optimistic
depending on how you see it i tend to
think about
what could happen if everything went
super well if everything
went smoothly growth in compute the
number of papers published towed
and dollars invested in ai continue the
same
i think it's pretty legitimate to think
that there will be some significant
advances this decade or century you seem
to think that it might
come sooner than many people expect
no at least at least that
in some way yes and i think what happens
now
is that if you're an ai scientist at a
big research lab
google facebook uh apple if you say the
thing super intelligence is coming soon
you're being disrespected because as a
company it's bad to be seen as someone
crazy so other people center themselves
so i want to give those people a space
to talk about it
the few people i've seen talk about
general intelligence are
max hodak so the ceo of neverland ceo of
open ai so summer hotman
elon musk carmack cto of oculus he said
something like there's one percentage
agi
being already here uh but like secretly
he can do that because he's respected by
everyone but but now let me
pick because you we've talked about
those terms before he's speaking about
agi or he said agi so i don't i don't
really know what he means by that
my best guess would be let's say also
imagining 50 percent of human warfare he
says that there's less
than one percent chance that he's
already here but secretly and what do
you think um
it's a bit weird because i was asked
recently what's the probability of
having
clones already and thinking about it
you're like
wait if there are clones then there is
some big tectonic technology that i
don't know about and and people are very
secrets
and we're living in the truman show you
know so
uh and they're like aliens and so you
need to put some probability
on it there can be also more plausibly
like some clones in a lab in
china or something like that sure sure
then you can define clones as like
someone who looks like you or someone
will be atoned by atom like you or
someone will share your genome
could be like a bunch of stem cells or
research somewhere in a lab in china if
it's something that looks like you i
think we can do it but it needs to like
move right i believe a clone is just a
same
genetic material we've done it before
for other animals just for humans not
not that i'm aware of okay so i might
have a misunderstood
question where i was spawning at is if
we take genome it looks exactly like you
then agi complete there is this thing in
computer science where we say
np completes for problems that are very
hard to solve here and by agi complete i
mean problems that
can only be solved by agi but still a
bad definition because as i said agi is
poorly defined but
it will require something like a human
level intelligence sped up you can do
whatever a human can do
but with the speed of a computer then
you can go
very fast i would suppose that if we had
a clone atom by atom there would already
be
something like superintendent and that
could reason faster than that about
science having a secret lab doing
uh a gi for me is the same about a
secret life having like
atom by atom clones is something very
weird but you need to put some weight on
it and i think less than one percent is
good
one in uh a million maybe it's too low
so yeah between those two i would expect
you'd say a higher
number talk about super intelligence i
think it's less than one percent if we
talk about our definition for aj now
yeah that's one
maybe more than one percent so or close
what about uh super intelligence in the
next 10 years i think it's pretty
reasonable to expect
something that can do human-level tasks
and if we
also made something like uh writing code
or ai research on a computer clock much
faster than the human brain when we
reach those things we reach finance
directly
the question is when we reach like
science math so
either there's like some big thing we
don't know about math and only
humans like flesh humans can write
equations and stuff or it's like
possible that we can do something with
scaling a large language model like
gpt56
plus some reinforcement learning some
agent decision problem thing
so language model plus decision in the
world so that's my what we call
inside view that's how i feel personally
when i see the results from open ai
and when i see my go teacher being
beaten by alphago if i think about the
outside view which is what people
say so if i look without any feelings
attached and
without my life being in danger what
evidence shows us
i think one report that is pretty uh
well
uh written is uh by ajiya kotra on
ai timelines and she gives pretty good
estimates and she explains
why she gave the decimate she gives
distributions uh
different hypotheses and she averaged
those hypothesis and i think
if i remember correctly it's something
like the average of something in
30 or 40 years so i would say my my
outside view is something more like 30
or 40 years
fifty percent she talks about
transformative ai
in their paper is whenever humans are
capable of
creating an ai that can transform the
economy like industrial revolution they
have enough compute
if they have the algorithms i remember
this paper but it was a less
constraining definition than you have
when you talk about
super intelligence all these questions
were a build up
to the next one which is uh if
those researchers and you are right
about
agi and that it's a more serious risk
than um than it is uh commonly thought
to be
uh where are you right and uh especially
and why is everyone else wrong multiple
things happening so there's
as i mentioned center so people who are
in the position to express themselves
uh and have thought about it cannot say
it publicly because if they say publicly
they look like
someone insane there's also a bias the
more you work on ai research
the more you see how complex those
problems are see how hard it is to make
something detect
uh objects in images you struggle all
day long to do those things
and then someone says oh it will be like
human level in two years the thing is
people is a
crazy person because most people would
talk like that are not a researcher
i was talking to some researcher at
stanford told me like oh it's very funny
to talk with you about those things
because
in my lab nobody talks about those we're
too concerned about our research
we don't have enough time to talk about
it i don't think they're wrong it's
mostly an approach about science
when you do science you want to prove
that things will happen and there's no
way to reason
uh as a scientist very precisely about
blues and you need to be
to make forecasts and and extrapolation
and
and those are things that people don't
like to do extrapolate from
curves yeah i don't know for sure we saw
how hard it was to convince people of um
climate change so exactly this one will
be uh
in that context you laid out what do you
hope to achieve with this podcast so you
talked about a place for people to be
able to talk
freely and what do you think
could result from those conversations
maybe people
will talk more about it at least so i
want to
um enable people outside of this podcast
to talk freely about it
to reference to have something to
reference so they can just like instead
of just saying
oh i have some private information about
some crazy guy they can just say
well look at this interview which is uh
established researcher or
here's some disc some debate you can see
the pros and cons even thinking about a
meta way which is
as i said the the stanford researcher
wasn't even thinking about it because he
was too
busy working on research so if i talk if
i call him
and we talk about it for two hours maybe
that's the only two hours he will have
to think about this problem and maybe
we'll shift his career
so even have people think alone by
themselves about it that's super
important
but in the context of climate change
people talk about it all the time
there are many celebrities even like at
hattenberg that
specialize in talking about it and it's
a coordination problem
nothing changes do you think having a
public discussion on this
topic will be different there's more
that can be achieved by a conversation
the usefulness
of some podcast or conversation
is how much it can change people's views
or
a ways of seeing the world which can
lead to
them taking actions that can change
something
so the question is like what are the
actionable steps what we can bring
is a useful framework and good arguments
if people are able to reason more
clearly about the future
then they might decide to change it i
think if if anyone
thinks uh enough about it uh maybe they
want to like maybe not have kids now
even if we talk about automation and
they realize that they might their job
might be
automated in five years and they might
change their career right having some
podcasts or youtube videos then it's
easier for people to approach it
so there's this youtuber robert miles
who talks about
agi and this guy is as like 200k views
so there's like a lot of people watching
this video i think it's very useful work
and i don't know how many people change
their careers out
of it uh but i did so i i
started off with reading some profession
from whiteboard y
on ai and that's i i was studying in
engineering school and i was not liking
it
at all and when i read this blog i left
and went to paris
and started learning about ai i think
you can have a tremendous impact on
people's lives for sure uh
the answer i would have given myself is
like most of the podcasts i listen
i i listen because they are enjoyable
and they don't particularly change my
mind
i don't learn things new that are useful
but i like listening to them
i find them interesting and fascinating
and that's also part of
why we have conversations and why we
listen to podcasts i don't fully agree
that it doesn't change your mind so it
it see
it appears that it didn't change your
mind on a particular subject but you did
update on some evidence right so does
everything else that's not why i listen
to them if you
listen to someone talk for an hour you
talk to people like one hour
uh maybe like four hours a day you talk
to people
let's see like 25 of your time is spent
listening to some podcast
and so it shifts your brain in some way
so if you listen to one pocket a day for
a few months then you you start like
changing your opinions on things
um and even if it's not the purpose then
at the end of the day you still like
change your views somehow as i get told
i
change my views less and less i don't
i don't think i've changed my mind on
anything in the last
couple of years so wow yeah i think i'm
upset then
yeah yeah perhaps i am yeah and now for
the
the final part of this podcast i'd like
to talk about
what you do on on twitter and i saw you
you posted a cool question about gpt4
could you like very quickly explain what
uh
gpt4 was is or will be uh or might be
and uh what was the question and
uh what did you get out of it sure so
asked what will
gpt4 be incapable of doing and i got a
bunch of answers
maybe explain what uh gpt is pt2 was a
model developed by openai you gave it a
one sentence and it generates a
paragraph
then one year later they they did
another model that could generate
three or four paragraphs or one page of
text using a prompt
this time they use something 100 times
bigger it could solve not only
technically generate
code to to build web apps or poetry or
translate
so a bunch of things you may do it could
do it if if it was given the right
examples at the beginning my question is
given the limits of gpt 3 from last year
imagine we have something new this year
let's say it's called gpt4
and something like 100 times bigger what
it cannot possibly do and
most of the answers are like do
something useful make sense because
they're just
language models i still think it's
useful to challenge those ideas and
think
if we ask the right prompts can you do
it you know with
thousands of trillions you know of
computer i myself used um
gp22 for to write a few essays in
in college in fact i believe one of your
most uh successful tweets
was about that i wanted to know like do
you think we'd use gpt4 for a
phd thesis also you so you're
trying to you're thinking about starting
a pg now well if
i i can automate it yeah yeah i'd love
to um so the difference in text produced
by
two and three as i said is something
like it went from like one paragraph to
something maybe like three to five
paragraphs or one page
so if we want to move to one page to i
don't know 200 pages
so 200 times bigger uh so if we can if
we keep scaling three by three
we need that five iterations more three
to the power of five is something like
uh 200 if if it's on every year
and you please in philadelphia you have
five years and at the end you just uh
uh send it to uh eight
and yeah you'll be fine but then the
problem is you'll have some plausible
text that is coherent in the like 200
pages
if you say something like um here is
uh some example of what a phd would look
like
you get a paragraph or two paragraphs
about some psg like academic text
and so you understand what's like
converting to phd is
and then you can do yeah but that's
that's what
prompts programming then there's another
level which is
meta from programming so prompt
programming is what we do with gp3
we could expect that with gpt4 we do
another step
ahead which is asking for what inputs
would produce some outputs so we can ask
let's say i want to write a thesis and i
need to write an introduction
what would be a good prompt to give an
introduction
to the subject and then it will give you
the best prompt
for his model and then you just give it
the prompt
and you're like you're giving the point
for introduction for you know first part
so maybe you need to write a plan and
like once again for each part of the
plan
and tell him like what you think an
introduction is like conclusion is
and then it generates a phd i think it
seems plausible that you could
write something coherent and win a lot
of time from
i'm just doing it but you're not
confident that gpt will result in a
super intelligence by itself i think
maybe somewhere in my mind i believe
that
uh intelligence needs to have some
agency if you want something to exceed
the human incidents
in any reasonable way in terms of
economic importance
it should be able to like just you know
move this mouth around
in a way it can it can it can speak to
us
so it gives you uh the problem is not
in a loop needs to like uh predicts a
few steps
and and want to plan ahead so he can
make me move this mouse if it's if he
wants
me to use this mouse for something but
what gp3 wants now
is just say the most uh plausible
sentence oh yeah i think you're right
but it could be a like a
bigger intelligence there are many
things to be done uh i think the most
essential ones are
perception so be able to understand
audio video recognize expressions
language and then there's like reasoning
and planning i think the biggest
criticism people have on gpt is that
it's not able to reason at all um so if
we ask him
uh if i if i take a rock and i threw it
in the ground
where is the rock and he says like oh
the
the rock is in your hand so he doesn't
have common sense
you cannot reason like a baby root or
like a toddler could
have common sense yeah yeah yeah
isaac's very close to the moravic
products counter argument to those to
those people
is that this doesn't say that it does
nothing so gpt does something
it's just that doing what more like
paradox asks walk in a bar and
take chess pieces and and move it around
and do like human level things like
robotic things
it's actually super hard and if we solve
something like common sense in my
opinion
um it will be very close to something
that can do math
and you know can do ai research
and you know the singularity or
superintelligence
a close claim to say oh gpt cannot even
even have common sense and say like
oh gpt is not even super intelligence
for me it's kind of the same claim
but there might be a way of like
imagining that common sense
is way easier than taking over the world
i think as you've said earlier if you
can accelerate many times perhaps many
thousands of times the
average intelligence at the end you have
well
something very competent let's say
common sense is just like you have
physics understanding
and you can predict where if the ball
falls down and so on
it doesn't give you the tools we have
like the understanding of mathematics
if you try to explain uh god also rem to
like a toddler
you will have a hard time uh yeah yeah
depends on yeah yeah
no you're right so uh maybe there's like
something in between which is
the limit understanding mathematics and
planning and problems
in computer science are not linear if we
invest
1000 times more speed on something
we might not get 1000 times more science
because problems are
maybe np complete or very hard to solve
so maybe you need exponential
compute sort of like you know the
travelsman's problem where you need to
move on a graph yeah and you're
traveling
amazon you need to put the boxes
everywhere that's exponential so like
even if you're very fast even if you
have common sense
you still need to like fight this
complexity perhaps
yeah there's a quality to intelligence
and that would mean that
higher intelligence is not equivalent to
lower intelligence
accelerated another point i think
people might think of is that
we're not one agent in terms of we're
humans connected we have you know
cameras
uh social media telephones if an ai
comes along
it's not about an ai being spotted on a
human it's like an ai being smarter than
everyone else yeah yeah yeah
yeah and and maybe this claim is even
higher is like
if you want to outsmart the entire world
you know take over or something
then maybe you need a bit more than you
common sense or like innocent math you
need something higher
um well i i would assume that the the
intelligence
would not be adversarial from the start
so it would cooperate
and use the infrastructure we have to
its adventure
so it needs to have the first kind of
common sense
and planning to be able to
understand how humans work and
understand how to lie
and send these software like copy pasted
software
on all amazon uh servers so you can like
survive forever
but in in some sense also the biggest
agreement here
is between something called the
treacherous turn
where they feel like copy-pasted
software everywhere after cooperating
for a while
and the view from ben gozill uh
called sworded stumble which is uh
the moment a toddler becomes very angry
and
starts fighting his parents is very
small you know he will stumble
so whenever we see an ai becoming
malicious
we will obviously see it uh so
the first lies uh we will manage to like
uh turn it off
um oh yeah yeah the twitter's turn was
a term coined by bostrom in his books
for intelligence
it's not clear to me which of these you
are is true
but it seems that lying and planning
is pretty hard to do so whenever it
reaches a level when you can lie
and plan that way two steps ahead
cooperate for
10 weeks and then in 10 weeks copy paste
if you can do that
then humans would already know it's
pretty smart
in the case of treacherous turn on top
of cooperating
it will not show how smart it is
yeah it will pretend to be stupid or
at least the same level and he might uh
on his own
you know learn more about the world and
become smarter
and do stuff in the background and so
in in appearance it might be the same
level or growing
slowly but in reality it will be going
super fast
so yeah that's another thing uh in this
scenario which is um
pretty pretty hard to to plan all of
this
is uh hard to plan um uh
i think uh like we are approaching well
we've done slightly more than an hour i
think it's a good length
for this podcast i think it was
successful um
great see you