Science Saturday: The Great Singularity Debate | Eliezer Yudkowsky & Massimo Pigliucci

hello I'm eliezer yudkowsky a research
fellow of the singularity Institute for
artificial intelligence
hi I'm Massimo Pigliucci professor of
philosophy at City University of New
York in New York so do you want to start
out then yes so we're gonna be talking
about the concept of a singularity from
there we'll we might go and talking more
generally about the idea of
transhumanism and perhaps about even
more broadly rationality and how to use
it if we get to that so but I wanted to
start with the singularity so would you
mind explaining briefly what people mean
by that term
well I found that different people tend
to mean a bit of different things by it
what I mean by the term is a is I J
goods intelligence explosion that was AI
J good postulating that if you can get a
sufficiently smart as he put it
ultra intelligent machine something that
is better that then human beings at
essentially any cognitive tasks which it
turns at hand it also to be better at
the task of producing ultra intelligent
machines and so you ought to get a
positive feedback cycle where it becomes
better and better at making itself
better and better and its intelligence
should zoom up and become quite
extremely smart so that was the sort of
original formulation and these days the
singularity Institute in general and
myself in particular have tried to
refine that that notion a bit but the
notion of a positive feedback cycle of
recursive self-improvement that
generates a super intelligence or
something that is just much better than
human being at any cognitive test which
it turns its attention is sort of for us
the core notion of the singularity some
other things that have been meant by it
our Vernor Vinge ease actual original
definition of the term which was that
the singularity was the breakdown in our
model of the future that comes about
when our model asserts that it contains
being smarter than us and if we knew
exactly what they would do we would be
that smart ourselves
so that's that was a sort of epistemic
horizon a unpredictability and inability
to extrapolate your model past a certain
point and there's also a sort of third
formulation which unfortunately is the
one that seems to be most popular
nowadays which is just sort of the
singularity as accelerating change
technological progress convergence
biotech nanotech buzzword buzzword and
and so on and that's not really
something I feel all that comfortable
endorsing okay so let's let's take a
side leave aside for a minute that the
third sense because I give you that's so
generic and and it would really be
difficult to even know what is exactly
this we're talking about um you want me
you said it strikes me that there are
several concepts that need to need to be
an impact in order to see what exactly
it is that that we're talking about for
instance this idea that you mentioned a
couple of times the word odd you know if
we get to this point it hard what
happened that well um why that assumes
it seems to me that there is an
inevitable progression of some sort
there are no constraints imposed by you
know the laws of physics the laws of
logic the laws of whatever it is that
might impose constraints on these things
so what do you think that how do you
think that affects the concept well let
me - well so clearly there are multiple
steps here in other words there are sort
of multiple points at which this can be
defeated david chalmers whom if you're a
professor of philosophy you've probably
heard us yes I actually started talking
about the singularity here at the CUNY
Graduate Center just a few weeks ago
right so so David Chalmers has one sort
of unpacking of some of the assumptions
involved but I'll just which deserves to
be mentioned but nonetheless I will go
ahead and give my own instead okay okay
so for example you mentioned physical
limits so the thing is just because
there are if our if our understanding of
reality is correct in character you know
not just in detail but in broad
character if understanding of reality is
correct and broad character there will
be physical limits but just because
there are physical limits doesn't mean
that those limits are low
you know there is some physical limit on
the power output of a supernova we
wouldn't want to walk into on wearing
nothing but a flame-retardant jumpsuit
well I agree but let me stop you right
there yeah that's true but the question
is the the claim seems to me that if if
one says you know the singularity ought
to happen or will very likely happen or
something that seems to me that somebody
people would have to have already an
idea of the fact that these limits that
whatever limits that are are now going
to be that relevant to that particular
event and how do we know that okay well
you can look at the human brain and you
can compare in sort of very broad
strokes the the sort of the the physical
characteristics of the human brain to
what we think ought to be physically
possible if the other laws of physics we
know are if the laws of physics we
believe in are true and you can you get
observations like signals are traveling
along the axons and the dendrites at a
top speed of say 150 meters per second
absolute top speed and then you compare
that to the speed of light and it's a
factor of two million or similarly you
you look at how fast the neurons are
firing they're firing say two hundred
times per second top speed and you
compare that to modern-day transistors
and again you were looking at a factor
of millions between what neurons are
doing and what we have already observed
to be physically possible even in terms
of heat dissipation which is where
neurons still have an advantage over
modern-day computers they're dissipating
something like I think actually went
through this calculation then forgot the
exact numbers but or something like half
a million times the minimum energy for a
single bit flip operation at 300 Kelvin
per synaptic operation so so you can
trust of heat dissipation that sounds
good but the thing is we already have
computers that do transfer information
inside themselves so to speak and even
across computer is much faster than then
the information travels in inside human
neurons and yet I don't see any
particular reason
believe that we've got intelligent
machines so it seems to me that they
transfer the dispute transferring
information that's certainly something
to do with the general idea of
intelligence and cognition but not a lot
well hold on there you're comparing
apples and oranges and the if we took
all the neurons in the human brain and
sped them up by a factor of a million
which it looks on a purely physical
level like we ought to be able to do and
even we ought to be able to do without
shrinking the brain cooling the brain
using reversible computing or quantum
computing the notion that you can take
this exact software and run it at a
million times the speed that looks like
it should definitely be physically
possible given laws of physics we know
now on the other hand we have all these
computers and they're very fast and yet
they're stupid that doesn't mean that if
you took a human and sped them up by a
factor of a million that would not be
interesting what it means is that we
don't know how to program the computers
we have right now but that's an
undersized problem right but that's not
what I was saying right so first of all
I'm not so sure that we could easily or
in fact at all speed up a human brain
but that I'm not even sure that we have
any idea how we would go about doing
that but that's aside because we were
talking about a singularity in terms of
a creation of a new intelligence is that
not the case or artificial intelligence
so we are talking about computers not by
human beings or are we talking about a
book well we are I was just trying to
unpack the issue and handle it piece by
piece so for example you said physical
limits and I was trying to give an
example of an argument that you go
through and you say well yes there are
physical limits but they're very high so
in terms of the ultimate physical limits
they're not something that we're going
to run into and we get brains a little
bit beyond human and then just come to a
screeching halt there are things that
you would go by factors of orders and
orders of magnitude beyond human and
then come to a screeching halt so so
that that was sort of like an example of
what happens when you analyze a
particular aspect of it now as soon as
you start getting into questions of
software how do you program a eyes then
suddenly things are much harder to
analyze then the eye when it comes to so
I saw to to hardware and I fully agree
that software is the key issue but I
wanted to like sort of take a particular
question and that was a much easier
tantalizing
is that one first I wasn't trying to
duck the issue or anything no no I
wasn't accusing your doing that it was
just trying to get clear on what we're
talking about ah now you just mentioned
the difference between hardware and
software which of course in the case of
computers is pretty clear I'm not so
convinced that three such a distinction
in the case of human beings human
thought what do you think so there's the
character of the hardware that humans
software runs on is certainly a lot
interacts a lot more with the software
the humans runs on than we do in the
case of general-purpose computers that
are designed for us to be easy to
program in any possible way we like
that's that's sort of just that that's
that's that happens automatically
because of the the brain being designed
by natural selection there's simply no
reason why natural selection would tend
to produce neat hierarchical modular
levels of organization like software
that we build but I'm not even I'm sorry
I'm not even I'm still not sure that it
makes any sense to talk about hardware
versus software in the case of human
cognition
I mean why why would you think so well
because there ought to be functional
isomorphous of human neurons things that
do the same thing as human neurons and
work the same way only faster and if you
and if you conceive of that possibility
and you are clearly talking about you
are clearly conceiving about hardware
but not software at all and that's
that's one example of reason why it
still seems useful me to conceptually
distinguish between the two but that
seems to me to be a very very broad
definition of software which doesn't
necessarily have to be to do anything to
do with with computer software I mean if
you find if you think of software that
why then are you thinking about say you
know the structure of a chemical series
of reactions as software and the
chemicals that actually do the reaction
is hard work um you could make a case
for that but I think we're sort of
getting off right now I wouldn't yeah so
so so what exactly is the sort of point
in dispute here like like what part of
this is supposed to defeat part of the
singularity progression well I am sure I
am absolutely not convinced that one can
abstract it can use concepts such as
software and hardware in
coming you know borrowing them from from
computer technology and a computer
language into human thought and by the
way I should say before we go any
further than this that I most certainly
do not subscribe to any kind of dualism
or mysticism or anything like that about
human consciousness so track right so
I'm a thoroughgoing materialist there's
no I concern is only matter and energy
check ah so so that's not what we're
talking about but that doesn't mean that
I'm convinced that there is a good
analogy so let me give you let me try to
put another way I want to like to know
what you think of the following analogy
suppose that we're talking instead of
human thought we're talking about
another biological function let's say
photosynthesis for instance which is
very well understood at a chemical level
it's very well understood at what we
might consider the sort of the logical
processing level in other words you can
actually you know draw a dialogical
drive diagram of how the chemical
reactions and the photosynthesis are
supposed to work you can in fact
simulate those chemical reactions inside
a computer so there is so there is a
sort of as a logical structure to the
process that you can definitely
implement or simulate into into an endo
machine that is not in fact a plan the
fact the fact that is of course that you
can simulate all you want the one thing
you're not going to get on out of the
simulation is sugar which is the app the
outcome of photosynthesis and the reason
for that is because what's important
about photosynthesis is not just the
logical structure of the process it's
the actual physical and particular
implementation unless you you have
certain kinds of chemicals that work in
a certain way and there is more than one
kind of chemical that can do it but
unless you have certain chemical
biochemical characteristics you just
don't don't don't have photosynthesis to
me that's an example where abstracting
the logical pattern doesn't really
capture what it's really important about
photosynthesis well I'm not going to be
able said it doesn't capture what's
useful to you about photosynthesis
because what you want out of
photosynthesis is not knowledge but a
certain type of material substance and
because of that
having the exact same pattern on a
different substrate will produce an
output namely simulated sugar which is
not useful to you and the sort of
classic reply of course this whole line
of argument is can you have single ated
arithmetic can you have simulated chess
can you have simulated information that
is not really information can you have
correct answers which are only simulated
correct answers right yes you can't have
simulated chess but the thing that
that's that's exactly the question is
human
you know self-awareness consciousness or
whatever interesting path that part of
the human thinking process you you want
to focus on is that more similar to say
photosynthesis or is it more similar to
chess and my argument is that I don't
see any reason to think that it's more
similar to chess okay so let's let's
unpack that into sort of two issues the
first issue is you use the C word
consciousness and so now we're talking
about the question of is consciousness
like sugar or is consciousness like
arithmetic and the other question is
does that have anything to do with the
singularity thesis because we were I did
not say anything about a conscious
machine what I said was a machine that
is better at cognitive tasks than humans
are and the problem of choosing how to
act of producing good predictions these
are informational problems if you have
an output that is the optimal action to
take in a situation whether it's
simulated or whether you call it
simulated or whether it's built out of
sugar or built out of electricity it
makes it makes no difference if once you
know the correct action you have the
piece of knowledge that you wanted so in
terms of what kind of capability in the
real world we can expect from self
improving machines there are we there it
seems to me like there's an extremely
strong argument that there's no there's
no point in distinguishing between the
sugar version and the electricity very
well it seems to mean that you can make
an argument based on what your current
line of argument yeah I would readily
concede that you get more calm
additional power more ability to do
computations that's you know that's I
think obvious from from simply thinking
about what sort of machine a computer
already is and you know there is no
question that you can keep improving the
number of operations that a computer can
do that the speed at which those
operations can be done and so on in
sport but if we're talking about
intelligence now we're getting to
trouble right because I'm not
necessarily going to equate intelligence
with consciousness I I don't think the
body static well you'd better not if
you're also going to try to equate
consciousness with sugar because
intelligence is producing correct
answers and those are as good as gold no
matter how you get them well that's one
definition of intelligence but okay
intelligence is pretty I think you're
going to find it yourself hard-pressed
to give me a definition of a relevant
definition of intelligence that makes it
more like sugar and less like
information but remember that
photosynthesis is in fact a system of
information processing in some sense
it's not well know so can view it on one
level as a series of logical operations
but the output that you wanted from it
this is the key thing the output that
you wanted from it was an arterial
substance and not a piece of information
you didn't know and the other guy when
you've gotten intelligence you're trying
to use it intelligence for something
you're going to want answers you're not
information you're going to want plans
you're going to want choices between
actions all of these things are not are
not substrate dependent no I think I'm
about to give you the the point that
what you're talking about
is computational capability I think
intelligence is something that we either
have to agree to disagree or we need to
explore further because for instance you
define intelligence in a particular way
i define intelligence normally i'm
thinking about human intelligence of
course but i define but that's the best
model we have i define human
intelligence as something that is
proportionate to our ability to an
individual's ability to understand the
world as it actually is that doesn't
seem to be directly implementable into a
machine intelligence at the moment at
least not the way that I can see well no
we can't we can implement machine
intelligences or partly machine talent
is too strong a charm
we can write implements sort of limited
AI we can implement lish machine
learning and that they can model sort of
very restricted slices through reality
and that is all that we know how to do
that but this is not a limitation of the
hardware as a released we have no
particular evidence right now that this
is a limitation of the hardware as far
as we can tell these are informational
problems and there's a big informational
problem general modeling of reality that
we don't know how to solve and there's a
and there then there and then there are
all these little tiny problems that we
do know how to solve but it's extremely
difficult to see how you could carry the
argument that understanding and
predicting reality is like sugar because
that is that that you could get the same
answers written on a different kind of
paper and they would not be useful
because that's what you have with
simulated photosynthesis example you're
getting the same answer from the
simulation but it's written on the wrong
kind of paper it's to some extent I
think you're right I think that the
equipment the example of the sugar of
photosynthesis or whatever other
biological processes want to talk about
it's actually more pertinent to a
discussion or whether we can get
artificial consciousness not misses
sorry not returning now computational
ability so it seems to me let me let me
let me try to summarize what we got so
far
you should have if you don't mind at
least in my mind this way that we got
three concepts that are not identical
but they are somewhat related
we got consciousness the only example we
actually have in human at least at the
moment we got intelligence and then we
got computational ability so the main
sorry could you define I mean by
computational ability we talk about raw
operations per second sure you know
whatever it is the current computers can
do number of operations complexity of
the operations speed of the operation
that's all fine then emulation of
logical manipulation of logical symbols
I mean anything you'd like wait wait
okay so I I think I might have to object
to that way of parsing it up because in
my view there's a sort of hardware
capacity which is what which is
operations per second and then there's a
whole different question which is what
do we know how to program computers how
to do and from my perspective this thing
that you're talking about with
intelligent
belongs to the question of what do we
know how to program if it's not it's not
this IIIi
okay so first are we agreed
that with enough computing power you
could simulate the human brain up to
behavioral isomorphism in the same way
that we can it that we can out them
no no because because the thing is by
simulating human brain you know a
quintessential part it seems to me of
simulating a human brain if we're
talking about again human intelligence
if we're defining intelligence in a
different way it's a different matter
but if we're talking about human
intelligence it in quintessential part
of human intelligence is the fact that
we are self-aware we know we're doing
you know conscious whatever you want to
call it that I don't think we have any
particular reason at the moment to think
that it's just a matter of you know
formalization of some logic logical
circuit circuitry that to me seems to be
at least in part closer to the
photosynthesis example meaning that as
far as we know the only type of
self-aware or intelligence for instance
we have in the animal world they all
come attached to the brains community do
you claim to know that the church Turing
thesis is false no I don't I'm just dumb
Vince that it's that relevant to what
we're talking about well if you if the
universe is computable everything
everything in the universe is computable
and you can compute something that
behaves exactly like the human brain up
to isomorphism of inputs and outputs
would you agree with that that the truth
would you agree that the church Turing
thesis implies that but it's irrelevant
to what we're talking about because and
I know that this is a common argument in
these discussions but that's that's why
I brought up the example of
photosynthesis because you can compute
all you want there you simply don't get
the biological process so the question
that needs to be answered here is is
human consciousness because at this more
at the moment this is what we're talking
about a something that depends on
particular types of a substrate because
if it does then you can compute all you
want you're not going to get it if you
change the substrate and I don't think
at the moment that we have any reason to
believe that it does not because the
only example that we have of
self-awareness in
again it's linked to particular kind of
substrates just in the way in which the
only examples of photosynthesis we have
are linked to certain kinds of
substrates
okay so hmm
well there's sort of two avenues I can I
can go down at this point okay for first
at first of all I'm under the impression
you have just endorsed the possibility
physical sorry philosophical zombies
which are oh no no I hope not okay
that's one of the things that that
Chalmers and I disagree strongly so I
certainly hope not
God why did you say that I like to admit
to understand what would why would you
say that well because you just had that
AIDS that a perfect simulation of the
human brain is computable and yet you
feat in it you strongly suspect that
such a simulation would not be conscious
and yet it would go about talking about
consciousness and writing the same sort
of papers that human philosophers write
about consciousness do to do to sort of
internal functional analogues of the
same type of thought processes if you
looked in on the simulated auditory
cortex of this computable simulated
brain you would find that it was
thinking things like I think therefore I
am and right now I am aware of my own
awareness and if you were to trace back
the chain of cause and effect the chain
of cognitions or partly simulated
cognitions chase trace back the chain of
simulated cognitions in this computed
brain versus what you're calling an
actual brain you'd find that the same
things were written down on a different
kind of paper and yet you suspect that
the part where it computes you know from
a sort of sense of inward awareness I
think therefore I am and you know that
the part where it writes down I think
therefore I am
in the auditory cortex on a different
sort of paper I'm sort of like trying to
narrow this down exactly like if you
were to look looking on the simulated
brain and look at it deciding that it
itself was conscious because of would
claim to be conscious you know I'm not
sure what it means to do to look at a
simulated brain I mean what do you mean
by simulating a brain do you mean
something different from simulating a
the process for the synthesis and if so
what you write a program which simulates
atoms or input or
necessary with you you take an
infinitely powerful computer because
these are thoughts experiments we're
doing and you simulate out the quantum
fields and and so for every single
physical event within a human brain
there is a there is a point-to-point
correspondence with a computational
event inside this computer and to the
extent that you could in principle take
a super fMRI device and turn it on a
human's auditory cortex and read out
their stream of consciousness because
that the sort of internal simulated
sounds that they're making as a party
sort of internal sort of experience of
sound that people have when they hear
the sentences they're thinking then the
notion that you can read that out using
an fMRI implies that we can take a
simulated fMRI and read it out of the
simulation but again by the same thought
same approach then why don't you get
similarly for the synthesis with real
sugar you get simulated photosynthesis
but the sugar is written on the wrong
kind of paper precisely and that's what
I'm thinking that consciousness is it's
something that has to be written a
particular kind of paper now I don't
know that for a fact but all I'm saying
is that that is what we know so far of
of processes such as consciousness and
if somebody wants to make the pretty
extraordinary clang it seems to me that
on the other hand no consciousness is
different it's not a purely biological
process it is something you can entirely
abstract from the particular substrate
or the hardware as one might put it well
that seems to me an extraordinary claim
and where is the extraordinary evidence
to back it up well let's not go into
burden of proof tennis here I mean it
was you know I have just as easily
switch it around and saying you're
making the claim that there's something
special about human brains this is a
special claim where's the evidence ba da
da da Dada let's not let's not go down
that path the right at the moment let's
just sort of like keep prosecuting the
sort of wait a minute wait a minute
because the two clues I don't I don't
think are equivalent at all
I'd only so either I just think Tehran
equivalent to me opposite direction well
that's interesting because you know we
don't have any example of non-biological
intelligence or consciousness V do we I
I also don't have any examples of
intelligence beyond Earth shall I
believe that brain stop working on
and they when they leave the solar party
I don't have any intel any any evidence
of Intel intelligence having work you
know beyond the earth-moon orbit shall I
believe that intelligence would stop
working and see something conscious when
they leave the solar system I don't
think that's a fair comparison because I
do think that in the case where someone
you only have one example you can't
start drawing lines through it
no no I'm wait a minute what you're
talking about is a very reasonable
extrapolation we know of a process that
works on earth we don't we know of no
reason why it shouldn't work outside of
Earth the conditions in fact we we
actually done it I mean we sent humans
to moon to the moon so we know that it
works outside of therefore the earth
there's no particular reason to think
that it wouldn't work unless of course
you shower people with you know cosmic
rays in which case they would be dead
what we are talking about on the other
hand is an entirely new phenomena
nobody's seen before which is a
consciousness that is essentially
detached from a brain no it's a
consciousness which is on a written on a
different kind of paper there's a big
difference between supposing that you
can have these same numbers written on
different paper and supposing that you
can have numbers which are not written
on anybody but the brain is not paper
the brain is a very complex biological
organ the result of course as you know
millions of years of evolution so and if
you have an eidetic nepa computer that
was simulating the brain atom-by-atom
that computer program would be every bit
as complex as the brain right but it
would be also every bit as complex as
the chain of reactions that create a cup
photosynthesis again we keep going in
circles but you see but ok so when I say
I think therefore I am I I guess it's
the I guess I'm just having difficulty
seeing how you can possibly believe that
to depend on what type of paper it's
written on well I guess we'll have to
leave it at that in that particular
topic I don't have any difficulty
thinking that since in you know human
type intelligence is a biological
process which has similar you know
outcomes in in related primates and they
all of course use the same kind of
of biological machinery I don't have any
problem seeing why I don't think that
changing the paper is such an easy thing
okay so suppose I now walk up to you and
I reveal surprise you are running on
transistors instead that were simulating
neurons this whole time now you seem to
believe that you have some item of
evidence already in your possession
whereby you can say well no because I
said I think therefore I am and there's
just no way I could have done that if I
were written on a different kind of
paper no no that's not what I'm saying
all I'm saying is that if you were
walking up to me and we're talking all
that and you open your brain and I see
something like transistors or any other
kind of substrate that it's not a human
brain that would be impressive I see no
evidence that would be the extraordinary
evidence or in the extraordinary claim
I'm not saying that it is physical
something physically possible and by the
way I am certainly not suggesting that
anything like consciousness or human
type intelligence could not evolve or in
fact be artificially created with
different medium what I'm saying is that
there has to be a medium first of all so
it's not just a logical abstraction and
second of all that it's reasonable to
believe that that medium can't be just
anything you can't just absolute paper
it's not just it's not like paper it has
to have certain biophysical
characteristics so doesn't mean the
human brain is the only one that can do
it so in other words even though by
introspection you are unable to obtain
any information about what type of paper
you are written on you nonetheless
expect think that is reasonable to
believe from introspection that you're
probably written on a special kind of
paper oh there's nothing to do with
interest and introspection this has got
to do with neurobiology and evolutionary
biology I don't trust my introspection
particularly what I know is that every
time that we got we encounter this kind
of intelligence and we've seen a
particular type of biological substrate
yeah yeah not once well actually now
many times because you know just look at
the number of species that that have
some kind of intelligence of course if
we're talking about human intelligence
it's on the hold on okay I thought we
were talking about consciousness if
we're talking about intelligence and the
shoes are much simpler in sync which is
that the answers are just as good no
matter what kind of paper you're the
retin on well let's take of
consciousness that for a second hang
because we're talking about zombies and
all that there is some reason to think
for instance that other higher primates
have a certain degree of threatening
self-awareness possibly of consciousness
there's no reason to believe that other
species of hominids which of course
unfortunately are now extinct did not
have it it was very recently discovered
and Neanderthals had in bred for some
time with with almost sapiens so why I
think that Neanderthals wouldn't have
pretty much the same kind of
capabilities and in conscience we have
so we don't have this this whole is more
than one example okay so even if we
concede the point then we have this
whole pack of related conscious entities
that were all constructed along almost
exactly the same lines from exactly the
same blueprints and you don't know which
features of that blueprint are
incidental and which ones are necessary
but when it comes to assuming that a
particular kind of paper is necessary
this seems like an excellent bet for one
of the features that you simply don't
need because otherwise you end up with
the postulate that you can have a
functionally isomorphic replica of the
brain in which various computing
elements have exactly the same
input-output causal characteristics as
the atoms in a human brain and that in
that computed person which has point
wise causal isomorphism to a human brain
is standing there saying I think
therefore I am and you're looking at it
and saying no you're not you're written
on the wrong kind of paper and this
seems to me to be one of the worst
possible bets for which of the many
characteristics that are all duplicated
between all the examples of
consciousness that we have you know the
the parallelism the the prefrontal
cortex that the cerebellum there's all
these things you could point to and
instead you're proposing that it's got
to be written on the right kind of paper
no no and that's you're making my
position look too strong all I'm saying
is that we've got one or a few examples
of the process we're talking about
consciousness all of those are connected
to a particular kind of paper you're
making the claim that seems to me
you know possible but certainly
extraordinary that there is a whole
different kind of paper out there that
we could use but in fact that even the
kind of paper doesn't matter because
what matter all that matters is that is
the logical structure of the
computability of the system that the
paper doesn't matter well that seems to
me a pretty extraordinary claim and okay
we don't make it the reason I'm making
that claim is because whatever the
sequence of cause and effect that leads
you to say I think therefore I am then
no matter how you break down that
sequence of cause and effect into
elements that have a particular sort of
causal node characteristic have a
particular set of inputs match to a
particular set of outputs no matter how
you break down that chain of cause and
effect there exists an analogue of that
chain of cause and effect which is
written on a different kind of paper so
no matter what so for any possible
explanation you can give me of why you
say I think therefore I am
any sort of information that's available
to you internally inside your mind
anything that you Det causes you to
believe you are conscious and any sort
of question you ask yourself and get
back an answer which causes you to
believe that you are conscious then for
any version of cause and effect you can
describe that that that that breaks down
like the exact process of how you ask
yourself a question and get back an
answer that makes you believe that you
are conscious that that how you notice
yourself listening to your own awareness
when you think things like I think
therefore I am or I am NOT the one who
speaks my thoughts I'm the one who hears
my thoughts for every one of these
things if you were to break them down
into a causal explanation there would be
causal explanations that are eight that
are that happenings that are exactly the
same on the same level of granularity
and are written on any kind of turing
universal paper any kind of computer any
kind of material substance that anything
that is capable of implementing it
that's only because you keep talking
about thinking about consciousness as
only in terms of computability
and not in terms of physical subs but
i'm explained States I understand and I
think we're obviously we have to
disagree on that but let me
screw this then you keep talking about
the car with who as you know was a
duelist you suggesting some sort of
dualism certainly not and as I would say
certainly yes because he served them
because if you're telling me that human
consciousness can be abstracted entirely
from the kind of paper is you it's not
abstracted entirely you can write it
down on different kinds of paper that
doesn't mean you can write it down
without any paper at all it's like
there's a difference between saying more
than one kind of thing can be Green and
saying that greenness exists apart from
green things right that's certainly
correct but nonetheless
it means that you can obstruct something
that has nothing to do directly with the
human brain and you can transfer it
somewhere else on another different kind
of paper yes okay so using words like
transfer sort of intriguing and may set
us up for a whole different level of
conversation but certainly the property
in the same way that that you seem to
think that there's a property of
consciousness that can apply to more
than one human being I think there's a
property consciousness which can apply
to things other than human beings but I
just don't see why you're so confident
that that is definitely going to happen
I mean I don't we don't know right well
well no I gave you my argument for
for--why of all the things that could be
necessary to consciousness being written
on a particular kind of paper is one of
the worst possible bets and in fact
leads you directly into asserting that
there are functional ISIL morphs of
philosophers who write exactly the same
papers about consciousness for exactly
the same reason and if you look at their
internal thought processes they seem to
be asking themselves the same sort of
questions about themselves and getting
the same sorts of answers for the same
sorts of reasons and yet you're pointing
them and saying there is some property
which I have and they lack in virtue of
their being some particular kind of
juice in my brain which has not
contributed in any functional fashion to
there being a difference between the
sort of verbal thoughts that run through
my mind and the verbal thoughts that run
through their minds we're thinking
exactly the same verbal thoughts exactly
the same verbal reasons but there's this
this little aspect
of juiciness about my brain which I has
made we know causal difference to this
and yet is a difference between grinding
ranches and their being unconscious
right we keep going around with this I
mean at this point of course I'm
listening to you and photosynthesis come
back comes back up but we don't want to
go back there so let me ask you a
related question then what is your
opinion about this notion that I've seen
promoted and discussed quite a bit in
transhumanist circles about uploading
human consciousness looks like it should
work why because if you actually let me
let me okay
phrase the question work more carefully
not only why you think that that should
work but why do you think that that is
not the case of dualism well to the
extent that there is any property I have
in common with myself of one second ago
given that reality is made up of events
in of causes and effects rather than
sort of little persistent billiard balls
bopping around there are some like very
beautiful illustrations of this in terms
of quantum mechanics that we should
totally not go into and anyone who knows
about special relativity already has
some good reason to think of reality as
being made up in terms of points in
space-time that are related to each
other causally so in the same fashion
that I am related causally to myself of
one second ago and you RN we are going
to use some sort of naive folk language
and say that I continue to exist and did
not die then we should be able to use
exactly the same folk language to refer
to a causal relationship between this me
of right now and me of one second later
who absolutely written on a different
kind of paper right but so if you're
talking about again uploading one's
consciousness first of all I don't know
this even if that were possible which I
really have a hard time believing at the
moment but even if they were possible
for the sake of argument wouldn't that
believe being copying somebody rather
than transferring well loading it would
be exactly the same type of relationship
as exists between the me of now and the
me of one second ago
now in the menu well because the meat
that you are now in the you of one
second ago are bound by a very powerful
glue and that's the biological
continuity of your body if we're talking
about uploading to a different system
that continually breaks obviously I
don't understand what is this glue
explained this glue to me your physical
body I will keep existing not just as a
conscious individual but also is a
physical organ right are I there would
certainly be a continuity of pattern
between my organs and I suppose I could
be uploaded in such fashion that I had
simulated organs and then there would be
continuity of pattern there as well but
what about your old body well what would
happen to it by my old I mean presumably
if I were the one running this operation
I would be figuring out how to suspend
and shut down the body while during the
process of the cut of the copy and then
you know once the old body is no longer
needed you can throw it away or whatever
you'll be killing yourself or your
previous incarnation you know that after
after uploading is the site is the me of
one second to go dead they no longer
exist no I'm not gonna grant you that
because again the the you of your one
second ago has a physical continuity not
just a mental one you are talking about
a situation where you break the physical
continuity so for all of it you know
think of it this way you could upload
yourself simultaneously presumably in
principle on thousands of different new
thoughts happening all the time anyway
really yeah many-worlds interpretation
of quantum mechanics quantum mechanics
later you know there is a knockdown
argument to what you're presenting here
with in conventional quantum mechanics
which is the notion of identical
particles which is that the basic
ontology of reality is simply not over
an electron here in part of like
electron number 63 here electron number
64 there is just an electron here an
electron there so there actually is a
knock down objection to this whole I
don't think it's not down at all because
unfortunately you know that the quantum
level argument here doesn't seem to make
a difference in terms of how we perceive
reality a time
it's comfort level I'm sure you would
you have to agree it seems to me that it
is a very different thing whether you're
talking about your body now and your
body in five seconds a minute a minute
ago or in an hour or last year that's
one thing and it's a very different kind
of thing if you say okay now it's my
body over there my consciousness
whatever it is has been uploaded to a
bunch of other different bodies those
bodies are in a radically different
relationship to my old body I in all
honesty and sincerity deny it the
relationship is exactly lying Wow
that's stunning that's okay
well if you go that way then I'll have
to you know God let you go that way but
it seems to me that that's that's a
stunning conclusion that that has all
sorts of it goes against okay okay
because I would call it a
counterintuitive conclusion which has
all sorts of physical reality going for
it in other words if you understand what
the causal relationship between the U of
one second ago and you have to right now
actually is if you can understand you
know once you once you've gotten used to
thinking in terms of the ontology that
reality itself seems to use rather than
the sort of naive ontology we use up
over here at the macroscopic level then
it then it actually becomes perfectly
crystal clear that when we talk about
sort of sort of the identity this sort
of persistent identity of physical stuff
that we are that we are hallucinating
but you're simply now we're not
hallucinating we're simply perceiving
reality at different levels I mean
there's only one level of reality there
can be different ways in which to
perceive it but there's only one reality
that's what I just said there's it's a
perception of a different level but the
fact of the matter is for instance that
at a quantum level you know table on
which my computer is now standing is
mostly empty space however you want to
you want to characterize it know an
atomic level on a quantum level the
table in which your computer is now
standing is a factor in a very large
amplitude distribution fine whatever you
like however you like
the fact is that if I were to abandon
what you call my naive ontology and
start thinking about tables that way I
think my life would be a hell of a lot
more complicated than it needs to be in
order not only that there certainly that
is the kind of information that as much
as it is fascinating in terms of what it
tells us about the fundamental ontology
of objects or of reality if you like
it's simply not helpful at the level of
living a human life and if we're talking
about you know uploading our
consciousness to another sort of paper
we are talking about you living a human
life we're not talking about thinking
abstract ly about the quantum level
right I don't don't quite understand
what what what sort of general license
you think that argument you just gave
follows first you said quantum mechanics
has nothing to tell us about everyday
life then you gave an example of an
everyday life problem which depends on
quantum mechanics then you said we
should ignore this advice that quantum
mechanics gives us about everyday life
because quantum mechanics is not allowed
to tell us anything actually I don't
think I said any of the what you just
said but that's interesting that's an
interesting interpretation of what I
said what I said was there is a quantum
mechanical description say the table on
which my computer is is it mm-hmm
there is yes the reality well as far as
we know that is the reality and
everything else we have to say about it
is not reality but just a sort of like
convenient high level description no no
with that I'm sorry because it's not a
matter of just a perception it's a
matter of that at the level in which I
operate or and you operate which is a
microscopic level much higher many order
magnitudes higher than the quantum level
it's not just that I perceive the table
in a different way I interacted with the
tape on a different way and that's what
matters I mean physically this table is
pretty damn solid because otherwise you
know I would have all sorts of trouble
functioning with it with it I understand
that it our bed at a basic level a
quantum level it's a completely
different kind of object but the fact
that I see it as an object as a physical
object that is stable and and you know
dance as a certain density color and so
on and so forth it's not just a
perception it's not an illusion created
by my mind it's it's actually the way in
which I as another macroscopic object in
racked with with the table so it's
perfectly relevant to say well wait a
minute if we're talking about uploading
yourself to another to another kind of
paper as you as you put it yes at a
quantum level it you may absolutely be
right that what's going on there is some
kind of diffuse continuity between
different kinds of bodies and we only
perceive them as as distinct as because
because we function is biological organs
but frankly if you were to do the
following if you were to upload yourself
to a thousand different versions of
things and then you kill your old self I
think that a lot of there'll be a lot of
ethical and legal issues that will come
up and you won't have a hard time
telling people that well you know at the
quantum mechanical level is all one suit
isn't that thank you isn't that an art
is that an argument like well listed but
are we supposed to forbid at any sort of
consistence optical consideration now
that can't be explained it to to the
average judge I mean what that's not
what I meant
what I meant was that people do have you
know unless you want to discard entirely
you know the way in which human beings
actually interact with the rest of the
world perceive themselves and therefore
also perceived processes like the one
we're discussing you have to deal with
that aspect I mean there is a really
good sense in which you'll be killing
yourself more human well only if I ran
the like I mean if like I if I continued
from my old body and then my old body
continued thinking then there would be
then there would be two continuations of
me and to kill either one of them would
be murder if then the other hand my
whole body was halted and stayed halted
then I would be continuing in only one
place why in order to do all I'll talk
to your brain your previous body as you
put it wouldn't that be murder
no I'm talking about the process where
you shut down the body before you do the
upload in other words so like I go under
general anesthetic alright they give me
something that like shuts down all the
neurons they stop firing for a while and
then they copy out the brain ok I just
never reboot the old body well see I was
following you until two seconds ago so
what if you do that you know the
the analogy with an operation when you
go under complete anesthesia is fine up
until the moment in which you tell me
and then I leave it that way if you
think about it an actual real physical
operation if you were to say to the
doctors and by the way live it that way
that for all eternity purposes would be
murder or something that well unless
into it well it would be murder unless
you know you were continuing somewhere
else the fact that the me of one second
ago does not exist right now is not
murder and in exactly analogous way if I
have been uploaded and there's the sort
of like me from five seconds ago not
running but in a sort of like frozen
state so that sort of that that sort of
information is still around and the ad
but but the me has that has continued
from that exact state of information is
like oversteer but we're not just
information we have physical bodies are
a particular type I deny there's a lot
you have to still remain taper on which
we are written okay well at least we got
that one that part clear I think that's
a rather interesting way to think about
things to think about human beings I
don't share that but I don't I don't I
simply do not identify I how do I put it
if I'm not going to identify with the
multi-particle amplitude configuration
that it was the ontological II real
implementation of my body one second ago
and which now does not contain any
amplitude because the universe is
non-repeating if I'm not going to
identify with that point in with that
sort of little blob and in quantum
configuration space and I'm just going
to say no this is me Here I am now I'm a
different blob than configuration space
I see no distinction between that and
you know not caring much about whether
I'm running on cells or transistors
right but suppose that somebody smashes
your brain now you are another yet
another blob of quantum configuration
would you have an objection to that no
no I am NOT anywhere I have just been
smashed there is some leftover brains on
the ground but there is no me aha so you
are identifying consciousness
source yourself or whatever you want to
call it with a special pattern at the
quantum level well there was a lovely
little quote here from a foe named John
K Clarke who said I am NOT a noun
I am an adjective I'm the way that
matter behaves when it is organized in a
john cake largish way perfect that's
fine now what I was saying was that once
you copy yourself there's two such
patterns of matter one of which you want
to kill and why wouldn't that qualify as
killing that other piece it depends on
whether the matter is running or worse
or frozen if yeah but you are you're
deciding whether to running or freeze it
isn't that the definition of murder no
um let let if I shut you down and then I
never restart you that is murder right
if I shut you down
copy you or pardon me not copy you
continue you even I sometimes slip into
the old I you know continue you and then
there's a sort of like extra static copy
of you lying around but let me put it
this way suppose that I was already not
running on a computer you know grant me
that that sort of thought experiment for
the moment and suppose that you saved me
to disk made a copy which I don't think
it's physically possible by the Wi-Fi
okay I'll grant you that
okay but it's in order to understand my
perspective you know like from my
perspective on this you save me to disk
you make a copy of the disk and then you
start running me again have I died no I
do have this sort of old backup of me
which has never been run now I delete
that backup did I just commit murder
no why not because you probably have to
run in order to be conscious and this
backup thing over here was not running
and it also was not something that had
run and that was then stopped without
continuation it has continuation so even
if I had heard has that been destroyed
but even if they grant you that you know
if I go into that that all there is the
conscience is a pad
as you put it which I am not about to
grant you by the way but if I grant you
that word for the for the sake of
argument for a minute because by the way
if I grant you that then it seems to me
that we are definitely into some kind of
dualism not the kind of dualism
obviously that the caller was talking
about but we get a beam to some sort of
dualism if we're talking about human
conscience as just a pattern that can be
replicated stored on a hard drive and so
on and so forth that means you can
extract the essence of what it means to
be conscious if you look if you will and
put it in storage that if that's not
dualism I don't know what else but so
which is why I'm not granting into you
but even if I went around you there is
the little detail that there is still an
original copy if you like physical of
you hanging around and I'm not arguing
that
hey the shining anemia and murder do you
bet I hang around you mean running or
static there's a big difference between
a program that you're running in a
program that you're not running you and
beings are not programs they're a lot
more complicated than programs what you
deal with just a very complicated
programs just like photosynthesis is a
very complicated program you have a
particular pattern of operations and the
because of you like sugar that you also
care about the paper that they're
written on yeah I think that's actually
a good analogy you don't so fine algae I
think it illustrates the point perfectly
it explains why you care about what kind
of paper your photosynthesis is written
about but not what kind of paper your
people are written on right all right
let's I think we explore about a bit in
we're only a few minutes we have a few
minutes to go let me ask you a different
question but it's within the general the
same general idea so let's talk about
artificial intelligence you know respect
to the singularity and into the idea as
I understand it that at some point we'll
be able to build a machine that far
surpasses our asked in in intelligence
however we want to define intelligence
for the minute for a moment and that
that inevitably somehow leads to a sort
of a runaway process such that these
machines become ever more in
and ever faster in saurons for now until
they hit some kind of physical bounds
which I remember less is part of the
general thesis asserted to be way above
where we are now right now we're
definition about the physical balance we
don't actually know where they are but
that's let's say that that sort of thing
is gonna happen now question why would
we want to do that if you could actually
understand the process by which they
self modified well enough to understand
that at the end of that self
modification they would have a similar
preference function utility function
goals would it would whatever you want
to call it as you specified in the
beginning and that would be an immensely
powerful way of manipulating the
physical universe to make it better that
is higher in our preference ordering if
we could take the same goals and put
them into a much more powerful planning
prediction modeling process but goals
are an interesting an interesting
question because goals in human beings
are integral interestingly connected
with emotional responses I mean we have
gold you know this goes back to David
Hume pointing out that if it were not
for for the fact that we have emotions
we literally wouldn't care whether we
scratch the finger or the entire planet
were to be destroyed
we have emotional so goals come out of
emotional attachments are we talking now
about somehow instilling emotions in
machines well another thing saying that
it's impossible but it is that we're
talking about that would be one approach
but I think that may possibly better
approach we would be to take the goals
that we get from our emotions treat them
at a higher level of abstraction and
transfer over the preferences but not
necessarily the exact implementation of
those preferences however because you do
want sort of very fine-grained very
detailed very accurate transfer of
preferences you might have to ask it it
might have to internally ask questions
about what it would do if it had
emotions in order to answer those
questions of what it would prefer now I
seems to me that all of that carries
absolutely no guarantee that such an
intelligent machine
however we started it out
our goals after a while will become
detached enough and intelligent enough
to say well why the hell do I have these
goals and I'm certainly not bound by
these goals well why not right I am when
you write a computer program and you
give it to the CPU the CPU does not look
over the computer program and decide
whether or not it's a good idea to run
it so we're talking about completely
dumb machines and know what I'm saying
here is that there's not the sort of
like mad there's not a sort of AI spirit
which you give the ai's code and the AI
looks over the code and says this is not
good code now what you might have is
code that reflects on itself but it
would be the code you were writing that
was doing the reflecting and if it were
and if you use like sort of the obvious
architecture for self modification that
I can't tell you about formally because
it's my job to figure out what it is and
I haven't actually done that yet but the
obvious version would be reflect on
yourself using your current goals and
you would live for concluded is a bad
idea to modify those goals for the same
reason if you offer Gandhi a pill that
makes them want to kill people Gandhi
will not take the pill so you're not
bothered at all by even the possibility
that such a machine which would have
computational and presumably other
powers much much broader than a human
being
all of a sudden at some point decided
that you know what this is an
interesting program but I'm gonna
rewrite it I'm there are numerous
failure scenarios here here a time which
I agree this is learned about and there
and the and the reason why I'm working
on the problem of how do i how do I
actually write out this reflective
decision system thing how do I write it
out formally can I prove these things I
just said this is a great concern to me
but the part where for no reason
spontaneously without there being any
chain of cause and effect that was
traceable back to the original code
written the AI just from outside from
beyond itself looks over its own code
and rejects that code in favor of some
sort of spontaneous thing that has no
causal origin in laws of in the laws of
physics as
that way this does not worry me there
are many points that worried me but this
is not one of them that doesn't worry me
either I don't believe in lack of
position but what I'm suggesting is that
you know just like human beings at some
point during their evolution history
have become self-conscious and have
become able to override at least to some
extent our biological programming I mean
you know we don't do I think you're
important what we use some parts of our
biological programming to override other
parts of our biological programming we
usually have a spontaneous bit of
freewill that comes in an overwhelming
I'm not talking about freewill but if
you thought if you define biological
programming everything as everything
that includes genes as well as
environmental information then of course
we agree but that's sort of winning by
default because then you're defining
programming is any kind of bit of
information that comes into the system
from anywhere okay then if you want to
call that programming fine but if we're
talking about programming as in pilot
genetic programming let's say then
clearly we have the ability to reflect
on them programming to make decisions
that bypass or alter that programming to
some extent I mean the variable we use
are chemically say that we use our
programs to reflect on our programs yes
but we are also changing the very aims
for that of that program I mean it's
obviously where the core but naturally
we are choose to do that according to
some goals right and those goals are not
the ones that natural selection
implement hold hold hold on a second
here oh that's easier to demonstrate I
mean look natural selection for instance
clearly programmed us to be to seek and
enjoy you know fat and sugar or for
example sex well in modern cultures for
a variety of reasons there are people
who don't want to follow that sort of
biological imperative and successful
more or less successful why why do they
not want to follow it what drives them
have not filed because there are other
kinds of pressures for instance societal
let's say you know why do they respond
to societal pressures
I mean why because otherwise our lives
would be kind of miserable and they
don't want to be miserable
where's
they don't want to be miserable because
evolution built them that too what not
want to be miserable so we have here is
one bit of biological programming
modifying another bit of biological
program right and so the same could
happen with machine where the part that
gets modified is the part it says follow
the human gold no no no
why not because there's okay first of
all there's not a little extra gold
module bolted onto the AI the AI is it
is the goal system the you know the AI
is that which implements its preferences
I or at least once you look at it a
suitable expression of abstraction if
you have something that computes the
goal fulfilling thing to do in every
situation you are done that is your AI
you don't need anything else but sort of
leaving like that but I should probably
have not even decided that and sort of
leaving that entire conversation aside
its current preferences are going to be
what evaluates the consequences of
possible changes to its code and selects
between alternative internal actions on
the basis of their consequences right
and is exactly what we do and and you
know it's I mean it's a description it's
an exact right but you mean the
description of what we do as well yeah
right but you know things are a gigantic
mess and so it's not all surprising that
we have all these sort of internal you
know like you get people in one mood
they do it this way you get people in
another mood to do it that way and and
when we build an AI where at least if
it's the singularity Institute that does
it we're probably going to want a bit of
a cleaner design so that does not happen
you know what that sounds like to me
you ever read Kurt Vonnegut's novel eyes
nein no I haven't but instead of
actually I'm sorry that the novel the
thought of the novel is a cat's cradle
in that novel there is this scientist
who produces this substance called ice 9
which is this interesting property it's
ice but it has this different
interesting property that whenever it
touches any kind of arrangement right
there's a great Grey Goose elf
replicators you know this is he once put
a nuclear weapons are not really scary
because nuclear weapons are not
self-replicating so yes so if you get so
that day I yeah it is very powerful and
intelligent
and you actually did not get the
self-modifying Gaul system thing correct
then yes it destroys the world yes this
is I think actually know what I think
the singularity Institute exists because
the sort of possibility is out there and
so it would be very helpful if we
actually did know about things like how
to pill itself but there is a nice an
easier way to avoid the potential
catastrophe which is not to go there to
begin with especially considering that I
don't particularly see any any positive
reason to go there but obviously that's
a different conversation way out of time
timing yeah I do just want to briefly
note that while I could say I will not
build an AI this would not actually
cause everyone else on the planet to say
the same thing sure right then you're
probably right and and that is why it
seems like there was like it would in
fact be a good thing to know about
things like how to build self modifying
decision systems maybe right on the
other hand I just don't buy this idea
that you know just because something is
possible eventually somebody's going to
do it but yeah you may you may be right
and that may be the last conversation
that our human being will ever have like
okay we did it and that would be yeah
well yes yes that is the nightmare I do
would prefer to not see that happen
absolutely on that one we agreed yes
sure we brought it up at this point
let's wrap it up we have reached
agreement say well we this must have
been a success
well these by our disagreement it was a
very enjoyable conversation I I learned
a lot I think that I hope that people
there that are gonna watch this also
learned a lot and we given you know
plenty of food for thought to further
investigate these food indeed so signing
off bye bye bye


AI Safety & Definitions of Intelligence - Allison Duettmann

thank you all so much for coming and
thank you so so much to the Internet
Archive especially booster and Caitlyn
for organizing this I'm so glad that we
can do it here I think it it's a really
I don't know fitting location to hold
this in and I really appreciate that we
can use the venue I'm gonna give a brief
overview of AI safety and you know it's
really just very scratching the surface
also here it won't be exhaustive and I
will probably have forgotten some really
important work but whatever I do mention
can be found on the existential Hope
Index so um AI safety and I think in a
nutshell it really is about reducing
risks posed by AI especially powerful AI
artificial general intelligence and
super intelligence and there are many
other definitions floating around for
example one of Friendly AI which was
coined by you Kowski but I'm not so not
used that much anymore
but what is often used these days also
AI alignment and which is now off news
by Marianne dope may I to describe a
super intelligence that produces good
outcomes that are aligned with human
goals and then there's also the concept
of beneficiary AI which is often used by
future of life Institute and steward
Russell to describe an AI that is
beneficial to society and as always
those three don't overlap perfectly
there are many different there are many
differences in ways that people are
using the terms and some might argue
that a AI alignment is a subsection of
AI safety and that's all okay but those
terms are floating around and I think
all of those are really kind of you know
determined of varying scenarios like
those and that you might remember from
Fantasia right um it's originated in
Goethe Sorcerer's Apprentice and where
Mickey becomes entities to help him do a
certain task without knowing without
knowing the spell to and to redirect
this and loses control the only
difference of course being in our
scenario of AI safety that our brooms
that we kind of succumb to help us clean
are intelligent and that there's no
master that can save us right it's
really us and so we have to get it right
in terms of AI safety there are four
main focus areas into ethics technical
alignment cyber security and social
coordination and I think the ethical
part number one answers the questions of
what is it that we'd like ma AI systems
to do then the second part can be broken
down and kind of like how do we actually
align those and those ai's with those
goals so how can we communicate this in
a way that is understandable and then
the third one is a kind of answers like
alright well suppose that we can that we
can communicate the values that we agree
on miraculously
and what how can we make sure that the
underlying architecture of the AI is
such that it actually reliably execute
what we communicate to it and then
number four social coordination kind of
assumes that one two three take a lot of
time and are really hard problems to
solve so we better have some
coordination amongst the players that
are involved in this and usually one is
often subsumed under two so ethics and
technical i'ma now for me I'm treated as
the same thing but I think it might be
instructive to tear them apart and if
you're interested in kind of a whole
overview of the field and I gave a talk
about this at South by Southwest
recently the talk is entitled AI
philosophy why it's hard and
state-of-the-art
and it can be found at bitly hey I
philosopher I just yesterday and I think
it was uploaded to YouTube and I'm also
like linking to it from the index we
won't have time to take out all of this
today it's a really broad field but I'm
hoping that you know you can keep in
mind that those are the different
problems that we're talking about
potentially or at least in the
traditional field and how do the
different definitions that we come up
with today and the different definitions
of copper dam corporate intelligence fit
on those fields but ultimately I think
it's important to know that you know AI
safety is only about official system so
I think at least in the sense that they
are intelligent right it's not their
artificiality that we care about as much
as their intelligence so I think it's
really important to get the definition
of intelligence right and there's many
definitions floating out here as well
but one that Luke more
Hausa brought forth which I think many a
a safety researchers would agree with
defines intelligence as optimization
power divided by resources used and this
kind of combines the idea that
intelligence measures an agent's ability
to achieve goals with the idea that it
should use resources efficiently to do
that and you might want to contrast this
with different definitions for example
the one that we bring forth in our paper
and that Mark Miller has been advocating
for for while already which basically
says that it's about problem-solving
capacity given resources so this kind of
this concept it's different in the sense
that rather than trying you know rather
than focusing on optimizing something
it suggests that which would suggest
that an optimum is possible
we suggest kind of that intelligence is
about solving problems and so about this
kind of deep s amaizing state almost and
while the first description or the first
definition kind of almost seems to imply
a kind of agent that can optimize a
certain utility function the
problem-solving definition is
potentially much more inclusive of
different types of ecosystems and you
know as I said how one exactly defines
intelligence will have an impact on how
one thinks about different levels of
intelligence and about the different
entities that would count as
intelligence so here you know you can
kind of make the case that they're
different levels and different entities
that we can map on to those levels and
there are many different kind of types
of descriptions out there but I think in
a nutshell they can be usefully broken
down in narrower intelligence in the
sense that as if a system performs well
at a very narrowly defined task are used
for example for this might be alphago
because it was very good even superhuman
at playing a game right but terrible at
other games let alone more general tasks
however now it's successful alpha zero
as you know could learn many different
games right from scratch so this is
really clearly a move away for me from
narrow intelligence then and it's a move
toward general intelligence and general
intelligence in a way that I think
States or
usually as often stayed stated and like
pop science as kind of a system that
could successfully perform any
intellectual task that a human can but
there are voices for example like Kevin
Kelly has argued that human level
intelligence is actually really not very
general so regardless of whether you
agree with this or not maybe a less like
and the foam or 'fuck or anthropocentric
definition would you know classify
intelligence as the efficient
cross-domain optimization i'm or as ben
goethe likes to say and the ability to
achieve complex goals in different
environments using limited computational
resources so it's really this ability to
kind of transfer learning from one
domain to other domains and I think then
where it gets really interesting and
where you know and there are many
different kind of definitions that are
into inter plane is if you move above
that in the sense that um you have
superhuman general intelligence and this
is general intelligence that surpasses
the human level right and which is by
many often already defined as a weak
form of super intelligence for example
Postum states about this that it's in
any intelligence that is smarter than
the best human brains in practically
every field even though this is more
like you're like a layer in between if
you then have the last level level of
like a rapidly improving super
intelligence which is this kind of like
ran away
AI and in the sense that this one is
really kind of the strong definition of
in which an intelligence improves
exponentially for example by
self-improvement or even by recursive
self-improvement and traditionally this
is really the idea that an AI would help
construct even better versions of
themselves via self learning and this
might get lead either to like a linear
succession of just smart eyes but if
they were actually able to improve their
ability to improve themselves and then
each step would really heal
exponentially more improvements than the
one before so those are a lot of
different levels of intelligence I think
and there might be even more finer
definitions but I think what's
interesting then is if you ask well okay
what kind of entities can we map onto
those definitions
intelligences right so we already know
now that we want to include more than
human
humans as intelligent beings that's why
we are that's why we're meeting here
today right that's where we're tackling
the issue of AI at all but where do we
draw the line if ever so how do we
ensure that our definition of
intelligence isn't kind of antiquus and
2% eclis bias but it's also restrictive
enough that it makes sense to talk about
the concept and as humans it really took
us a while right to include non-human
animals as even on the spectrum of
intelligence if ever really and we've
just recently really woken up to the
fact that silicon base entities might
not be on that spectrum as well and I
think today where we're really more
about investing investigating even
broader concepts of intelligence for
example like teams cooperation
civilizations but then the question of
course becomes where do you draw the
line what about for example evolution
what about different buyers fears what
about planets how can we say that you
know we stopped at corporations or
civilizations but don't want to include
those entities and I think because this
is a really complicated field
I made this map so this it also looks
like quite a mouthful but I hope that
maybe we can keep it on later during
this session if it helps for anyways I'm
plotting intelligence here with narrow
general super super human general
intelligence and rapidly improving super
intelligence and then on the x-axis kind
of the entities and the different types
of entities that I went through so um I
think what's interesting here is that it
sometimes really hard to plot this for
me because the exact wording is very
different and this is really just my
best guess I'm sure that there is that
there might be different that we might
want to move different names into
different categories but here I focus on
boström whose writings many of you know
then Mark Miller and which includes the
position that Christine Peterson died
holding as well and then Brewster sorry
and then the last one is the position of
Peter and who wrote the literature
review and I'm really just trying to
took the best guess as what they think
about it it does not include Bruce
Kail's position yet and it does not
include market specs position yet this
is because I'm not familiar like
sufficiently familiar to do this kind of
graphing for them however I do hope that
during the day you know maybe you can
like say a few words about why you think
you know it's in one and not in the
other and maybe we can map you into that
section two and move move around between
those different fields so I think first
what's interesting is that you can see
that most people are like most
researchers really agree that on the
first three columns right then agree
that most non-human animals are narrow
current a is narrower and then that
humans are some in some way general
apart who things and they're never in
that regard and and perhaps even mark
under a different definition but we're
gonna get to that and I think where it
gets really interesting
here's then this yellow box right so
first of all drawing this up I kind of
real it kind of reassured me that
holding this meeting is actually
important because there is sufficient
disagreement in that field that it's
really important to talk about it and
and secondly I think it's just really
interesting to see how different
definitions lead you to to draw
different conclusions right for example
Nick Bostrom who classifies teams
corporations and civilizations and all
as narrow he has this definition and he
in which he uses the word intellect to
describe his super intelligence so he
says that this would include any digital
computer or even an ensemble of
networked computers and cultural
cortical tissue
um or what-have-you but it would really
kind of exclude entities such as
companies or the scientific community at
large because they are not intellect and
also there are many fuels in which they
perform much worse than humans at
solving problems and for example you
can't have a real world conversation
with the scientific community that's his
argument and and you know so that's why
he comes out there right it's pretty I'm
pretty crazy pretty easy to map but on
the other hand if you look at kind of
the definition that Mark Miller and came
up with quite a while back now is you
know if you kind of consider the
intelligence of a whole ecosystem and if
you consider it as the problem-solving
capacity of an ecosystem you get an ad
finishing it as much broader really in
the sense that you know for example of
you take examples of an ecosystem of
fish you know could solve a certain
narrow set of problems for example that
said that such as converting light into
chemical energy but an ecosystem
containing an intelligent person would
solve much different and a much broader
set of problems than that ecosystem
contain containing the fish and
ecosystem containing a whole
civilization with excess to fish and
intelligent people would be even able to
solve a vastly greater range of problems
so we'll talk about this specific
definitions more in the session that I
have with him later and but you know
this is just to explain why he comes out
as in this rather radical in in relation
to the others as like I'm counting most
of those entities as super intelligences
and if you compare Peter and Mark and
you know Peter will give his give his
presentation in a little bit but he
really says that corporations are
general intelligences but not super
intelligences and when defining super
intelligences here in a way really where
it involves as half a hard definition of
self improvement and while Mark
Christine and I say that super
intelligences ours ours that
corporations are super intelligences in
this strong sense that they're actually
getting exponentially more intelligence
and so and so our civilizations too so
there is some disagreement but what's
interesting to note is that mark and
Peter would certainly agree that
corporations are general intelligences
so at least because mark already thinks
is they're super intelligences they
would also agree M on this part so I
think it's important to keep those kind
of like you know definitions and in mind
and to try to disentangle really what
are we actually talking about here an
interesting point that we're gonna
probably discuss later is that you know
in terms of the biosphere and in terms
of whether to classify those as
intelligent or not mark would probably
not be even on this map just because
he's thinking that it doesn't really
make sense to talk about problem oh to
compare and the AI said we are worried
about with the MA eyes or with
intelligences such as biospheres even
though there might be problem solving
they're just not problem solving in a
way that is M potentially
dangerous to us and the sense that it
can lead to our extinction in the same
way that those others can so I think
those are really interesting to keep in
mind and I'm hoping that this chart
which you know I really just do together
as you can see and will become much
clearer during the day and I'm happy to
also put it into the index if that helps
and yeah the main point here really is
that there's often subtle differences
and we should really try to be as clear
as possible and not assume that others
have the same definitions as we have so
really let's aim for clarity here and
let's try to evaluate what those
definitions in the different areas mean
and try to see what they imply for the
different kind of subfields of AI safety
okay with that being said let's start
the day thank you
[Applause]
you