Science Saturday: Dreaming of an Artificial Intelligence | Eliezer Yudkowsky & Jaron Lanier

hello I'm Eliezer yes research fellow at
the singularity Institute for artificial
intelligence a reductionist a naturalist
a rationalist and generally drawing on
the daniel dennett side of the force and
with me today is I'm Jaron Lanier let's
see my titles are nutty these days I'm
the interdisciplinary scholar in
residence for the engineering school at
UC Berkeley and I'm the scholar at-large
for the Microsoft Corporation and I have
various other odd affiliations and I
don't think it's possible to be a
rationalist and also a follower of
Daniel Dennett although I'm very fond of
down personally I don't consider him a
rationalist I consider him to be a
religious extremist with just a
different religion so few years back he
wrote a rather famous article called 1/2
a manifesto in which he spoke out
against cybernetic totalism which from
your perspective had a number of points
part of it was excusing the flaws of
machines that instead of machines
becoming smarter rather we become dumber
some parts of but not precisely we lose
the empirical loop that lets us tell if
we're becoming dumber and that's a very
crucial point but please go on
ok some parts of that I did agree with
for example the critique of the idea
that natural selection of replicator
dynamics the neo-darwinian synthesis
would also be able to explain human
cognition which from my perspective is
just not what it's for it's a successful
theory of biological systems not as
successfully not of how biological
systems are designed not necessarily a
successful theory of the brain and I
would and by the way this notion that a
Darwinian explanation should explain
cognition in the brain originated so far
as I recall with Linus Pauling which is
an unlikely character but it was really
Pauling who started to come up with this
notion he would say that the secret to
his success as a chemist was that he had
many many ideas
they pointed out as if they were in an
ecology it's really of all things the
sweet liberal linus pauling who came up
to that one it's I think it's rather a
magical type of explanation really
because even after you claim I mean if
you claim that a living organism has
been designed by evolution you can
derive all sorts of specific predictions
from that if you claim that thought is
running on evolution what do you know
after that that you didn't know before
use you came up with that hypothesis as
far as I can tell nothing right well I
mean as we know it's difficult sometimes
to resolve what lessons we should learn
from the theory of evolution - and
that's not a critique of evolution it's
just to say that it's a theory that is
that requires more care and work in its
use than some other sorts of theories I
mean you can come up with competing
narratives or for evolution that are
equally good at explaining different
things so it's it's it's tricky it's
it's not one-to-one and onto as we say
it's it's a it's a it's a different sort
of theory than previous scientific
theories you can't just plug it into a
computer and run it to get the
predictions back out you certainly can
and also Moore's law which I'm aware
that a number of my colleagues and
artificial intelligence that it's not
very great Storen I think that has
rather that that just as it runs
exponentially so - it has
logarithmically diminishing returns at
best and I would say that five IQ points
is easily far more valuable than one
order of magnitude and computing power
on all sorts of interesting problems and
and you know just to state the obvious
any improvement in our ability to write
software has come very slowly indeed and
right now we don't know how to write
multi-core software by many years of
trying to sort that out and so if you
know unfortunately the world is still
helped and hindered by insight more
any sort of magnitude that you can apply
to some system measurement you know to
some technology and we're lacking on
that though I will note that I visited
the computer Museum the Computer History
Museum and was struck walking down the
aisles of time just how much of the
improvement in our technology had been
an improvement in what it takes to
program a computer I mean the the morsel
on one sense wasn't even as impressive
as walking the road from rewiring things
by hand to Python mm-hmm
well sure I and I that I would accept
certainly so but more importantly onto
the things that we actually do disagree
about and I understand that you are
generally not a fan of the notion of
artificial intelligence or I should say
rather strong artificial intelligence
hmm well it becomes a little subtle
because in terms of my work and the
contributions I think a lot of folks in
the AI field think of me as one of their
numbers so for instance I've worked on
creating computer models of parts of the
brain I helped Joe Rosen build the first
prosthetic peripheral nerve the nerve
chip many years ago I worked on this
stuff called virtual reality which Ray
Kurzweil believes will all live in after
the singularity or at least he seems to
sometimes believe that and I've done
many other things that are considered
part of the program by a lot of the
folks in the AI world the difference is
a subtle one it's not so much a
difference in belief of what
technologies are possible or even which
ones should be developed and certainly
not what scientific idea should be
explored because I want to understand
the brain profoundly and I'm very
devoted to that quest instead it's an
aesthetic a spiritual a political a sort
of a subtle difference it's a difference
in how we conceive of ourselves in the
world not so much the particular as we
do and I'll tell you later reminds me of
more than anything else I grew up in red
state America and rural southern New
Mexico and a lot of the people I knew
believed in
rapture you know they believe that
Christ was coming in that the good
people would just vanish and then after
they vanished their cars would crash and
all this and my belief is that the core
values I shared with those people are
our morals our sense of life had much
more in common than that then they're
not but this this notion that a rapture
would happen in your lifetime like that
it's like this thing you're planning on
really does make you nutty and that's
where I have to part company with a lot
of my friends in the old days and I
think it really did them some harm and
made the nutty like believing it might
happen if some arbitrary time in the
future is one thing but to believe it
happening and that's kind of how I feel
with the singularity singularity folks
it's you know actually keeps you even we
believe this is happening right now you
start becoming kind of nutty and you
start designing software with the wrong
goals in mind you start losing touch
with your own objectivity well heaven
forbid that we should suffer the
negative social side effect of that
software design usually you know the
people who raised this concern are
concerned with other things and fighting
that software but I'll admit that it's
an increasingly important problem one of
the things that I want to sort of did
wanted to raise when I was reading 1/2 a
manifesto I wasn't quite sure I could
untangle your predictive moral and
aesthetic claims I wasn't quite sure I
could figure out what you were saying
this will happen or this cannot happen
versus this should happen this is a good
act this is a bad act and this is a book
I would prefer not to read I mean
there's all sorts of subjects that don't
happen and we wouldn't want them to
happen but they make great subjects for
books and conversely there might be
things that will happen and even that
should happen but that you don't quite
like the way that people are writing
about them and I wasn't quite sure I
could untangle that well I have to
confess I haven't read 1/2 a manifesto
in some time myself
I remember quite what I said and I'd be
appalled if I still agree completely
with what I would have said a few years
ago I hope I'm a dynamic yes I was going
to a particular way since then which is
a basic human right
I would even even a responsibility but
what I will say is that I hope that I am
NOT taken to be well certainly not a
Luddite and but also I hope I'm not
taking to be some sort of traditionalist
who hopes to see an eternal
solidification of the biological design
of the human that we happen to have as a
result of where evolution has gotten us
so far that's not my intent at all I
consider myself a radical and in fact I
think a lot of the particular things
I've worked on and a lot of the ideas
I've expressed are quite a bit more
radical than what the standard fare in
the singularity movement if I may be
ever say so but but the the thing the
thing that I think is very important is
whether you are believe in people or
machines for so if you believe in people
more where that where that leads you is
first of all you accept just let's say a
bit of metaphysical ambiguity it's just
possible that this consciousness thing
is not just an illusion that there's
something kind of special going on here
and that possibility that touch of
metaphysical possibility I think becomes
the foundation point for humanism or the
idea that humans are not just bogs in
the machine well if I can sort of yeah
jekt hear it by machine either you don't
mean lawful physical system or you mean
to imply that humans are not lawful
physical systems so what exactly do you
mean by machine such that humans are not
machines well um first of all as an as
I'm sure you'll agree lawful physical
systems can not be measured to represent
to totality right i gil agree with that
statement i trust and i think it much i
mean i i have theirs well actually that
that does sort of like toss me off of it
because it might mean that you're about
to say something about quantum mechanics
that i might not necessarily want to
agree with Edwin Jaynes once said that
ignorance I'm trying to remember his
exact phrasing if we are ignorant of a
phenomenon that is a fact about our
state of mind not a fact about the
phenomenon itself so you know that we
can't have total knowledge of a system
that's just a fact about us it doesn't
necessarily affect the behavior of the
system itself in any way well I'll see
it does though because the only way to
know the system is to know it so
epistemology can't just be thrown away
and and that's where Daniel Dennett is
really breaking theist because he
believes that you can throw away
epistemology he believes you can read
epistemology as this sort of wrapper
that's not really the thing you care
about and you can't do physics that way
and I would argue you can't do computer
science that way so epistemology has to
be part of the picture or else you just
end up with your fantasy about what you
say hard reality is but no one else it's
just your belief system so you see the
thing is throwing out epistemology and
pretending that you can know hard
reality makes you into a religious left
case that's the thing I wish my friends
and I could see like you like you have
to be balanced you have to find a
moderate position that incorporates the
reason a touch of metaphysical doubt is
actually important to science is that it
brings epistemology back into science
it's as soon as you believe you can know
things perfectly that you start to screw
up you believe in yourself basically I'm
a scientific narcissist Am I am I
allowed to have extreme probability
distributions and if not what is the
probability level at which I break down
you are allowed to have extreme
probability distributions or
controllable experiments and you are not
allowed to have probability
distributions for unknowns so you know I
mean I think this is this like in in
normal parts of science this as well
establish the reason it gets confusing
in computer science is because we can
create virtual reality we can create
these worlds out of our own imagination
that seem real and therefore we believe
in our ability to know the real world
too much because we can create these
things that are so similar to it this is
this is both the power the power of
computers is exactly the thing that
confuses us and actually a great example
of this in my mind is the current
financial troubles where the power of
computers to express a complicated
finance I think is essential to survival
now because we need to coordinate you
know actions to degree we never did
before and yet exactly that powers what
can confuse us to a degree that never
would have been possible with more
computers the most of these stories that
I've heard about the finance troubles
involve sort of expert machine learning
programmers committing errors that
statisticians themselves would not
commit in other words they are putting
simplifications into their model that
may be very convenient and yes lo and
behold they work for a few years but
they're not true a statistician
professional statistician would know
they're not true and those were
ultimately the ones that fit them
conditional independence where there is
actually no conditional independence
would really be the main thing that
would bite you in these financial kinks
yeah and so but but one lesson to learn
from that is that smart people and use
computers to confuse themselves in ways
that would be pretty difficult to
achieve otherwise and this is why
epistemological
modesty is so incredibly important it's
so incredibly important to doubt your
models do not believe that you have
access to some sort of fundamental truth
when you're staring at a computer screen
so we make it I do that the I do get the
impression that you would eject on some
level I'm just not sure if it's
predictive moral or a aesthetic to the
statement that humans are machines and
of course that generally turns into a
question of what do you mean by machines
so do you mean to say that human beings
aren't as predictable and shouldn't be
forced to be as predictable and it would
be boring to read about them if they
were that predict
Boal as say a steam shovel or something
mmm you know that's completely the wrong
dimension there are many many reasons to
say that people and machines are
different the most important one is
pragmatic which is if you believe people
machines are the same then you'll trust
human-like software too much and you'll
screw up and behave like an idiot so
this is this is the problem
that's a parent in the Turing test that
you can't tell whether the machines
getting smarter or whether you're
lowering your standards to make it seem
smarter well hold on here because
there's the saying of the world's
stupidest man may say the sun is shining
but that doesn't make it dark out I
think Robert Pirsig invented it I'm not
sure exactly and the the there's any
number of perfectly true beliefs which
if poorly summarized poorly understood
misused in any number of possible ways
can lead to a strain all sorts of
gruesome fashion and of course the
classic example here is quantum
mechanics it's true it also drives
people and saying when they hear a
poorly summarized mmm-hmm yeah I know
the classic example is perfectly true
and there's all sorts of easy ways to
misunderstand it yeah but you see if an
AI doesn't say anything like there's no
benefit to it like you can absolutely
every project and I like to remind you I
do this stuff that the AI people think
needs to be I mean like I work on these
models of the brain and I've worked on
machine vision and I've done all these
things and the ideology they I doesn't
actually help you accomplish anything
it's a it's a religious idea like if
there's no pragmatic reason for it as a
belief system and there is a pragmatic
reason to oppose it as a belief system I
mean there's specific propositions right
you can't just bundle all the
propositions together and and slay them
with with one mighty blow that consists
of one thing that you can do wrong with
this entire bundle of propositions the
only logical positivists would believe
you can separate it into propositions
and that stuff certainly
so you know I mean like once you the
promise once you go down the line of
reasoning you're going you've already
you've already lost the main point here
okay but what can you give me like an
explicit example something AI people say
that is false morally wrong or ugly well
the point is it those aren't the
important issue the important issues is
that they say things that make people
stupid so for instance you shouldn't say
that the command line in that search
engine like Google knows what you want
what you should say is it's a piece of
crap and that you do your best to
manipulate it the second statement is
correct and can help people learn to use
it the first statement only make people
stupider and stupider so in other words
attributing these qualities to machines
as if they were people actually does
detract from the moral responsibility of
the people who use them providing that
the attribution is wrong in other words
if you say that but you know discovers
no matter there's no like meter off to
the side of humaneness or anything I
mean like you know like believing in
ourselves is irrational and yet it's
this thing we do that's the core of
humanism it's the core of democracy we
believe in ourselves you know we don't
have to we can all just die I mean it's
it's like this this is like this this
this strange it's like um it's it's like
Pascal's bargain you know it's like you
believe in yourself because what's the
alternative not believing in yourself
and the questions whether you should be
that to machines too so all I'm arguing
is you go to machines everything that
you believe because it's important what
can you give me an example of a
statement that is in fact false but
which you believe because you're better
off that way no no I can give you
questions that are undecidable though
that I'm better off with so these these
questions are undecidable I mean I think
they are real I mean whether there's
something special about people is is
undecidable I think we are not
epistemological a privileged enough to
have any definitive answer about that
don't we make a Pascal's bargain we make
a decision to believe in ourselves which
is irrational and yet that's I mean to
me that's part of the core of a
spiritual approach to
I'm not afraid to use words like that so
in fact you don't so in fact there is no
like known answer as to whether or not
people are special but you choose to
believe that they are yeah yeah I think
that you have an epistemological problem
I could sir mainly do you not know
whether people are special or do you
actually believe that they are special
which of these this reminds me so much
um when I was a kid I would argue with
these sort of people on the spectrum of
Marxism to logical positivism who would
try to organize the world so you're
assuming that your question is even
meaningful but the truth is that the
foundations of your question or far
Shaker than what I've said I mean
there's all this stuff about whether
statement can be true or no nor anything
like this is like ridiculous I mean like
real language doesn't even consist of
true propositions a statement of this
kind is is inherently on the side of the
like and but the reason for that is that
there's this thing called meaning that
we don't understand that attaches itself
to language and I mean what you're
assuming is that we know more about what
meaning is that we know more about what
information isn't may actually no we're
still kind of swimming in a sea of
mystery about some of these things well
obviously what I'm trying not to do is
let you is let you get away with saying
we should believe X for some reason
other than X is probably true that that
is one of my hot buttons and I'm sure I
don't have to go into all the reasons
why this would be a would die this would
be a hot button it seemed to me there
that you were saying that on the one
hand we didn't have enough information
to know whether humans were special and
on the other hand there were emotional
benefits to bleed and get therefore we
ought to believe well see the problem is
that they're very very very few and very
rarefied statements that we can know are
true and they're mathematical statements
and I love math I love mathematical
proofs and yet if those were the only
statements I was willing to act on I'd
be really quite and I would be such a
nerd I would never any God I never would
have been able to have the daughter cuz
I would be able to communicate with
another person and I wouldn't have a
wife I have to accept my ground banana
what and gravity right now I'm holding a
pen I'm about to drop it I bet it'll
fall I win well that's a you see that's
not a provable truth that's empirical
and there we get into this innate this
thing about the nature
which is very different from
mathematical truth and you know I mean
we don't know the limits of what gravity
works or it doesn't I mean we're just
like learning about gravity we don't
have a good theory of physics yet we
don't know we can't resolve we don't
know I want it I want it I'm not see the
uncertainty I'm are doing four shouldn't
be applied to any particular belief
system whether it's the rapture or the
singularity or anything like that I
think it should be applied to yourself
though I mean like this is this is the
thing like you can't believe in nothing
given a life events you know we find
ourselves plopped in this reality filled
with uncertainty and to believe in
nothing is to become a mute idiot where
you can't even take the first move or
say a thing so what you should do is you
should be aware of your irrational
beliefs you should embrace them but you
should also you know they should be
modest they should be moderately should
be balanced and I think believing in
yourself is a good start and if you find
that there isn't I mean this is like
this crazy argument I used to have the
Dan Bennett which was very entertaining
years ago where right so I you know what
a zombie is oh yeah I've written a bit
about okay King Dan Dennett side of
things as one might expect okay so I'm
just for anybody who doesn't know a
zombie is like a person who has no
internal experience but you can't tell
that they think if every appearance of
being regular and in some versions they
are atom by atom identical to people and
nonetheless lack consciousness I when I
do generally prefer that you specify
which of these are speaking about I
couldn't care less because the whole
thing is just fun I mean it's all under
specified you're pretending that there's
all this precision but it's specious
precision I mean you're precision means
nothing but let me go back to this city
requires both that you be ignorant and
that you'll be willing to relinquish
you're ignorant you seem to be attaching
a lot of importance to ignorance here
and I wonder if maybe that's sort of
possible importance to acknowledging
uncertainty instead of pretending that
you don't have any but let me let me
just unsound ease so I published article
many years ago where I said the only
measurable and curricle evidence of
whether person is AA is
he or not is if that person happens to
be a professional philosopher and then
if they take the sort of Dennett
position that there is no internal
experience and all that then you know
there are zombie so the thing is it's
entirely possible that you and I are
different I like to have this magical
internal experience of consciousness
that you don't have I mean that's that's
a logical possibility that could explain
our difference I you know to be
conscious when you are I think you
actually are a real person I bet you are
I think you're kidding me no no hold on
you just said it's possible it's
logically possible that you have a
magical internal experience and to me
the concept of magic seems a bit
incoherent because almost by definition
as it were it's that which if you know
how it works or if it's possible to know
how it works have stops being magic you
don't you don't get it I mean there's a
certain point where okay I mean I'm
conscious myself but confusion exists in
the mind not in reality if I'm confused
about a phenomenon even my own
consciousness cannot enact about
consciousness masters this is not a
phenomenon there's no empirical method
to study it I mean it's a it's a it's an
epistemological it's something rather
but it's not a phenomenon it's not
evolution for rocks or something it's
it's it's not something that's out there
you're saying on the one hand that
you're conscious but consciousness is
not real or did I just misunderstand
what you said over there no I'm saying
that here's a way to think about it
imagine you have a mathematical
structure that consists of this huge
unbounded continuous yield of scalars
and then there's one extra point that's
isolated and I think that the knowledge
available to us has
something like this it's uh there's this
whole world of empirical things that we
can study and then there's this one
thing which is that there is there's
also the epistemological channel of
experience you know which is different
which is very hard to categorize very
hard to talk about and not and not
subjective you empirical study it might
you know and if I can ask yeah I'd have
interrupt and ask a question here
suppose that you know sort of super
Dennett or even if you like super linear
shows up tomorrow and explains
consciousness to you you now know how it
works have you lost something or gained
something it's it it's an absurd thing
to say I mean that's like saying it's
just it's just a nutty it's the whole
construction is completely absurd and
misses the play I mean wait are you
ignorant about consciousness or
something else going on here I don't
quite understand you not know how
consciousness works are you making some
different type of claim I don't think
consciousness works
I don't think consciousness is a machine
but but actually to be precise I don't
know what it is and this is where we get
back to the point of the custom
illogical modesty I do know how a lot of
things work I've built models of parts
of the brain I've worked on machine
vision algorithms that can recognize
faces better than I can and I'm really
interested in the olfactory bulb and it
worked on there so I'm really really
interested in the brain I just think
that's a different question from
consciousness ok so if you don't know
how consciousness works where do you get
the confidence that it cannot be
explained to you because this would seem
to be a kind of positive knowledge about
the nature of consciousness and is sort
of very deep and confusing positive
knowledge at that for decades for
decades and it's a peculiar thing I mean
it's really like talking to a religious
person it's so much like talking to a
fundamentalist Christian or Muslim where
you have this I'm the one who's saying I
don't
and you're the one who's saying you do
know and I'm asking you to acknowledge
that if you're ignorant then you can't
go around making positive definite
sure statements this can't be explained
to me the possibility is absurd that to
me sounds like the religious statement I
mean saying I don't know but I'm willing
to find out a sign saying I don't know
and you're not allowed to exploit that
wasn't my statement you know I mean what
I'm suggesting is that it's the
possibility of experience itself is
something separate from phenomena that
can be experienced I mean or another way
to put this consciousness is precisely
the only thing that is introduced if it
were an illusion and that that does put
it in the class by itself and you know I
and so therefore you can't
it becomes absurd to approach it with
the same strategies for explanations
that we can use for phenomena which we
can study empirically now I think that
actually makes me a better scientist and
I think it makes sort of hard AI people
sort of more superstitious and weird
scientists and why happen humanism is a
better it's a better framework for being
a sort of a clear-headed scientist than
the sort of party on religion well well
of course I consider myself a humanist
and I expect that Daniel Dennett would
say that he considers himself the
humanist so I and we taught is
irreducibility an issue here do you
think that there are things that you're
not allowed to reduce to simpler
components well that's a different
question entirely and um I you know it's
as a technical question that interests
me a very very great deal and so but I
don't think it has any bearing on this
question but consciousness but I think
it's a better topic to talk about
because that's something we could we
could find a common ground to have an
interesting conversation about so for
instance it seems to me that the speed
with which evolution has proceeded gives
evidence that evolution is able to take
place at some what we might call higher
levels or some reductionist levels that
we haven't yet identified and that
fascinates me and I'm very curious about
what they could be so I think at the
very least more reduction as possible
then we know how to achieve in some
complex phenomena well because we can
also do English cases where centers
where ultimate reduction isn't possible
beyond what we've done very interesting
well we have to distinguish between the
question of is this in fact composed of
simpler elements and do we know how do
we understand how and can we computed in
a reasonable amount of time if you say
that evolution has elements we haven't
yet reduced that's a bit different from
saying that evolution has irreducible
elements if you say that evolution is
cheaper to think about if you consider
it on the high level rather than trying
to figure it out atom by atom well
that's true how we think about evolution
think about evolution thinks of itself
what I'm saying is that I think there's
evidence in how evolution has proceeded
that it's active at some levels of that
it has in that it has somehow embodied
in it representations that are
reductionistic in ways that we don't
understand and and that and that's very
interesting to me so I'm not making any
comment about us and our she's talking
about evolution but so I don't think you
can talk about reductionism at all
without bringing in epistemology in the
notion of an observer because reality
itself there I mean reality does not
explicitly contain the higher levels
reality itself just computes it using
the corpse I once met a fellow who
thought that if you used general
relativity to compute your artillery
trajectories he was in the military that
this wouldn't just give you an answer
that was uselessly detailed and slow he
thought it would give you the wrong
answer he thought you had to use
Newtonian calculations for artillery
trajectories because general relativity
would give you the wrong answer I was
trying to explain to him that this is
not quite what laws at different levels
means and in the same way a 747 it's
much too expensive to calculate out the
747 quark by quark but reality is not
calculating out the 747 using a
simplified model reality is calculating
the 747 court by quorum sure I would
agree with you everything you just said
makes perfect sense to me and that
actually provides enough I mean if we
want to talk about the consciousness
debates that provides another route into
it so so um let's reconsider
consciousness and we can ask the
question if conscious one way to think
about whether there's some special thing
called consciousness is what would be
different if it just suddenly vanished
if you removed it from the system what
would change and well the brains would
have to change is the first thing I
would say you cannot remove
consciousness without changing the brain
well that's one answer one answer is
that the brains change and people would
behave a little differently you know
another answer is nothing at all would
change because it wasn't anything to
begin with or or maybe there's the sort
of wheeler style answer that reality
would suddenly disappear because it was
part of a or something full-fledged you
know like physical zombie-ism where
you've got people who are atom by atom
identical yeah but don't have subjective
experiences and this to me has always
been a problem for the obvious reason
that these people are going around
talking about subjective experiences for
exactly the same physical causes that we
do and yet they're not conscious I don't
know where your stance is on well but
let me let me ask you question so let's
suppose let's just suppose for a moment
that consider the answer that some kind
of deep or big problem consciousness as
it's called or experience of experience
whatever terminology kind of finally
gets you there for the thing I'm talking
about let's suppose it goes away and so
all your one possible answer to what you
have left is all of the particles
continue in their same trajectories
everything about the physical system
remains the same however there's no
reductionism there there's no gross
objects there's there's neither a word
there's still you know all the particles
that compose a brain are still there but
there's no brain there's no word there's
nothing the word refers to the whole
layer of meanings just it hasn't been
all there hold on articles ok so I'm
sorry but I have to insert it I have to
start asking questions at this point
yeah let's say you've got a 747 yeah in
my mind I can try to imagine there have
been quarks
they would only be able to imagine one
quark and it won't even be a real
mathematical course yeah
or I can imagine you know using my cryo
cortex I can build this whole model of
the southern 47 shape hmm you seem to be
talking about subtracting the 747 but
leaving the quarks and this sounds to me
more like you know like a mistake you're
making in your model of the world and
something you could actually do to the
world itself
I mean maybe from your model we don't
understand quarks well enough yet to
make a model just to be blunt about it
cuz you don't have the unified theory
but let's see we did let's say we did I
mean yeah you said something for that I
thought is absolutely correct which is
that you can't talk about the idea of
reductionism without including an
observer and then that leads the obvious
question well what's an observer you
know and in quantum mechanics you do you
do have an observer you can't describe
quantum mechanics as you know purely in
its own terms it's always something
that's observed by somebody outside of
its universe and what's this and the the
notion that this is a that this is a
conscious observer is to say the least
not uniformly accepted I mean there are
people who think that all you're talking
about there is entanglement and I don't
think we should get into that
particularly the last thing I want to do
is enter the world of mystifying physics
but the only thing I'm pointing out yeah
you know you don't need to contain a
mystifying physics it's just this is
just a very simple a little bit of logic
that if you really really unravel the
idea of the observer if you really have
top of these things you'll ultimately
come to this seam where things don't
quite hang together and that's where you
are you know it's you you actually are
an observer and we don't know quite what
that means that's the epistemological
channel that's the thing you can't quite
throw away listen whether you believe in
yourself or not is really not my
business the only thing I really do feel
that I should be able to ask if you guys
is to not design software based on the
fantasy that computers are turning into
people or people are turning into
computers or any of this okay I don't
think that computers are naturally
mystically as a result of no human
action upon them changing or going to
change into people I
do you think that if you understand how
a human mind is put together or even if
you understand some of the basic
principles that would be involved in it
on a on a level that's simpler than the
the full happiness of the human mind
that you can put together your own mind
using equal or better principles and you
know do a better job of it yeah I don't
think it's something that happens as a
result of Moore's law I think you
actually have to understand what you're
doing and I've really worked on brain
stuff you know I really worked on
simulating parts of it I've worked on
some I've done I worked on frameless
might be impossible and while I work on
the brain you know I mean I think warp
drive is possible you know I mean I
think I think it's when you really work
with these things and you start to have
a feel for them and you start to respect
them I think and I and I think you once
again there's this the sense of humility
becomes so important I think I think
this note I mean there's this problem
that a lot of computer science people
have that they sort of imagine that if
you can sort of get a little toehold on
some abstract plane that might contain
something then it's as if you've become
the master of that whole plane and I
once again I just think it's a way of
fooling yourself I think it's a form of
immodesty that actually decreases the
quality of your work as a scientist but
I just think it's a it's a poor way to
think do you think it's possible for
someone to understand intelligence and
then build an intelligence
well intelligence is a different
question to my mind and here I would
throw her phone I would say human this
is very important I would say as a
hypothetical question understanding
intelligence to me is similar to
understanding warp drive in that they
might both might be possible someday I
believe that um a belief that either our
imminent or that we have even the
beginning of a foothold on either is
probably just confusing in the in our
own lifetimes so so in a pragmatic sense
I would say no but in a hypothetical
temed sense
yes okay so in that case I well first of
all let's also ask about that's what I'm
telling you is better - intelligence is
a phenomenon which is different from the
epistemological Channel in my it's also
bit different from warp drive because we
know that at least human intelligence is
possible whereas we have no such a surge
with respect to warp drive I think the
evidence is fleeting but how the
election turns out then ask me again ok
you know for humanists you don't seem to
like humans much so I mean not that I'm
necessarily disagreeing with you here
I'm just making the answer you know I
like humans and I you know I
so intelligence let's let's talk about
that for a second um there's a history
of people trying to measure and
understand intelligence so there's this
variable called G that's a correlation
of various things it's the basis of the
IQ test and I want to point out
something about the IQ test which is it
has the result has 3 digits
ok now that is a classic case of
specious accuracy I mean like if you're
gonna review a movie let's say you might
give it 2 stars you might like I think
the highest number of stars I've ever
seen in a system for rating movies is 10
and surely a person's intelligence is as
subtle as any like movie so like 3
digits and accuracy is everyone realizes
that that IQ points should be measured
and that IQ should be measured in
standard deviations and that the hundred
that in that the 100 is zero is just a
convenience there are right but the
point is that it's possible that people
are given people are given a card that
tells them tells one their intelligence
is 120 and another one is two other
intelligence is 121 and the fact that
that's even possible is an obvious
misuse of math and it's also but there's
also it's a it's an obvious failure of
appropriate intellectual modesty on the
part of scientists or honesty about how
much is real
understood it's driven by a bureaucratic
need on some level rather than
confidence in in a measurement and that
in a sense so in other words pretending
we understand something like
intelligence before we really do I think
is damaging and the evidence that we've
done so is specious accuracy and saying
you know I'm saying that here so I'm not
saying that I'm not saying intelligence
doesn't exist nor am i saying that it's
absolutely impossible to measure and I'm
not saying we know nothing about it
all I'm saying is that we pretend to
know more about it than we do and and I
think that that is an obvious thing it's
an interesting point in an interesting
illustration on the other end I would
also point out though that that IQ is
not what scientists are trying to infuse
into an AI B or at least not with this
not the same ones are trying to get a
grasp on because that's sort of like
saying while I'm going to build a flying
machine and I'm going to administer a
fly Q test that was standardized on
pigeons and that's going to be my
measure of how well the Machine flies
well look as I say our dispute is not
about technological activities per se so
for instance I I was the chief scientist
of an outfit called I'm addict that
eventually turned into the Google
machine vision group and our machine
vision algorithm for tracking faces and
facial expressions won the Miss
competition for such things all the
times it was held I believe yeah I
believe so so so um that would probably
be categorized as an AI activity by many
AI people trying to figure out how a
program can recognize something in the
visual frame and and so forth and indeed
it was inspired by neuroscience it
started off actually not with a bunch of
students of a neuroscientist that have
been in Chris Kristoff and Amar's Berg
at USC who is the guy he originally
noticed that synchronous firing in
neurons was important so it's a
neuroscience inspired project that made
the best of class algorithm for
something that brains can do I love that
stuff you know so in terms of the
engineering agenda and actual
achievements I'm
one of you guys you know it's more the
sort of religious overtones of it where
we differ okay but well first of all I
do think that you and I have certain
empirical disputes about our probability
distribution over future or humanity and
say the year 2030 but before that
there's a station that's that's that
some people are promoting which which I
agree with Ben Gertz alone the
artificial general intelligence Research
Institute in particular between
artificial general intelligence and
narrow AI or domain-specific AI the
notion being that if you look at most of
the organisms on this planet they tend
to be specially specialized to their
domains and don't really cross out of
them a B builds hives a beaver builds
dams you don't find a beaver building
hives you don't find a bee building dams
now human can look at the bee and the
beaver and figure out okay so that's
what they're doing okay and I'm going to
go build a hive now I'm going to go
build a dam we learn to do things by
examining the domains and figuring out
how to engineer within them we aren't
just created we aren't just limited to
what our ancestors were evolutionarily
optimized for we work across domains we
we probably aren't really literally
general intelligence is because there's
some things were better at than others
we are better at throwing spears than
writing code for example because our
ancestors through more spears than rode
computer programs but nonetheless we
exhibit significantly more generally
applicable intelligence than chimpanzees
say and there's I do think it's
worthwhile to make the distinction
between AI where you program the
computer to do one thing and it just
does that one thing and it's not going
to go and learn about the the cross
domain links in the world it's not going
to examine new domains it's not going to
figure them out it's not going to do
them on their own versus the the lost
Holy Grail that very few still now
follow in artificial intelligence of the
AI is going to learn how to do things
not just that we didn't program it to do
in a particular domain but it's going to
actually learn whole new domains and
operate in them in the same way that
human beings managed to leave the
savanna and had
to the oceans the air in space I first
of all want to be emphasized how much AI
people have been sort of my family since
I was a kid like the crucial mentor to
me when I was a kid was Marvin Minsky
who I still adore no I disagree with
these sort of metaphysical things and I
spend a lot of time these days with Eric
orbits is the current president of the
AI Society and blah blah blah you know
and I'm like part of the community you
know but so I hope I don't sound too
harsh but everybody's used to me by now
I suppose so the question I'd ask you is
why not just pursue these things as
engineering projects or science projects
on their merits and leave out the
ideology I mean what purpose is served
by having any definition of AI it's just
like it's just this useless thing it can
confuse you but it can't enlighten you
just like like just say we're interested
in how vision works we're interested in
how to represent how you build the tool
we're interested in making robots that
can navigate an environment interesting
what about technology and whack over the
world I mean okay I mean I'm generally a
fine with the idea with the rationalist
ideal of being able to discard the usual
words that youth to describe things
narrowing things down as much as
possible the particular empirical
predictions or the things they're trying
to accomplish but if you're trying to
build an AI that improves itself the
choirs capabilities creates for self new
technological capabilities exceeding our
current er current ones including
molecular nanotechnology and then goes
on to apply large-scale modifications to
the world in accordance with a moral or
meta moral system you know even if you
strip out all the words you're still
talking about something read where if
you could actually go out and do it just
looking at the results this would be the
result of immense significance and I
don't know if you could call that
leaving out the ideology but if you just
imagine that we're talking about a
physical system that's supposed to have
this effect I mean you can described in
whatever terms you like and it still
seems like something was rather
important I still think I think the work
is always clearer when you leave out the
ideology I think that
how vision works is clear making
artificial vision is clear making
artificial intelligence is not it
becomes ideology because you're bringing
in you're bringing in metaphysics you're
bringing and there's also this whole
agenda about achieving immortality with
the singularity and all this stuff which
we haven't even talked about it it's
tough and I just think it's like a big
it's like this big confusion like just
like drop it all just be a good
scientist and a good engineer you don't
need all that stuff but what if I
actually would prefer to all else being
equal unleash this AI armed with net
what learn a no technology to wipe out
AIDS and aging what if all else being
equal I'd prefer I'll tell you one thing
that bars me from implementing my
preference I just think you waste half
your cycles on ideology and you'd be
better if you weren't doing it that's
that's my personal but where's the
ideological aspect of this is it the
part where I want not to age and die at
the age of 100 or whatever no I think
the article part it's the part where you
become you have to believe with some
sort of certainty that you're a machine
according to some principles that you
approximately already understand in
order that you can make that machine
immortal and what you might be doing is
just maybe you're reducing yourself more
to have that belief or whatever but the
point is it's just this extra belief it
has nothing to do with your real work I
mean if it really is what motivates you
and you do some good work as a result
believe it ok fine but but I'm just
saying it's a you know all I'm saying is
that it's this it's this stuff that is
separate and then it like off to the
side and unneeded it's not part of the
real science or the real engineering and
I just think we'd all be happier if we
if we weren't so concerned with these
ultimate believes and you know I mean I
the truth is like I have I have the
ideas I've expressed and obviously you
disagree with them but like we shouldn't
even like ah I just I I just think it's
so much better like when when I I love
just giving a talk about say I'm how
parts of the visual cortex work the
parts that we can understand a little
bit about it it's just like so clear
everything makes sense the science makes
sense the engineering makes sense and
the unsolved questions can be stated and
understood as soon as you start talking
about this agenda
artificial this and that and and
immortality and blah blah blah all of
these things you enter into the southern
realm which is which which for one thing
you can leak when you go the other
responsibilities verify so that it's as
nice and neat does the discussions that
we have of the visual cortex right begin
and then you could go ahead and build
via the super machine you know I'll tell
you something um just to be let me
alright I'm gonna just get a little bit
tougher here most of the flaws in
software I'm gonna argue a lot like the
unreliability and computers can be
traced to somebody's a I believe 25
years ago that a lot of a lot of the
problems we have with technology are
because somebody was pursuing an
ideology instead of just good
engineering sense a lot of the cases
where we say we've designed an
architecture say so that it can have an
arbitrary number of users in our you
know was part of this this old ideology
of trying to build a multi sub brain
machine you know like they're all these
all these weird little AI things that
are traced through so they're all these
design criteria that come from the
ideology instead of from the problem at
hand and it's contributed to you know
probably trillions of dollars in an
efficiency on a global basis by having
an appropriately designed machines the
SMTP system the the mailed email
transport system had been designed a
little better back at the beginning so
that we wouldn't be having as much of a
trouble with spam now I mean sure you
can over design but there's also
problems where you under design things I
mean look at the the current financial
crisis I think we'd all be a lot better
off if you know some of those traders
had been a bit more driven by such
ideological considerations as Talib's
talking about black swans that they
can't yet see but no they were hardcore
they were pragmatic they stuck with what
they could see working right now and
then economic conditions changed the
whole thing blew up so I you seem to be
arguing against technological foresight
in general and I'm not entirely sure I
could back you on that one I think
absolutely everything about my career
you know proofs proves that wrong it's
obviously not what I'm doing
so but you know what I have to ask you a
question we've been going at this for an
hour it appears to me how long is it
supposed to go on okay we've been doing
it for 53 minutes and they'd like it to
be one hour oh okay all right then we
shall we shall proceed seven minutes so
you know look I'm there's just so many
cases where oh gosh okay so since
Microsoft is supporting my research I'll
give them a hard time two great examples
for Microsoft when is the paperclip
which is which has been which was a
terrible imposition on computer users
for generation and has finally gone away
was a direct expression of the AI
culture that instead of a computer being
comprehensible it was supposed to
comprehend you and of course that
doesn't work so this whole thing was
built in a hope in Russia people are
trained on ideology you're not
criticizing in the activity of having
dreams you're criticizing the part where
you overestimate your own abilities and
you think you know Manabe wasn't in
trying to invent Clippy the problem
wasn't failing and I'm including him
into the application anyway you have
been fine if they tried and just said
okay we failed we're going to leave this
and see the problem is that you AI guys
you you form a community of belief where
you reinforce each other and you get you
get all extreme in a rally just like any
group of people and then you you fail to
see when Clippy isn't working that's the
problem that's the problem the problem
is not wanting to make Clippy the
problem is when you can't is when you're
so attached that you think you can't see
that you failed and you know I'd say
I've seen people doing that classic
story that the yang communities does
that again and again and I mean so I
mean I own sees that it's an ideology so
they're bears where there's a pragmatic
problem with it and the question is what
advantage does it bring so if you feel
it brings in
if it helps motivate or or organize your
work great I'm not going to tell you
what to believe all I'm saying is that I
think that everything that would be
placed under the banner of AI and much
of my work certainly has been by other
people and just as well be motivated and
understood in a simpler way and you just
leave out the ideology and everything
gets better and you get a guy fflippy
problem I don't think you actually do
the same work if you're not coming in
wanting to to build the the artificial
general intelligence that self-improve
I'm doing it and I'm like the crazy
hippie mystic guy of course you can do
it I mean it actually just makes things
clearer just like this this ai ai it's
basically a modern version of a
religious belief system and there's no
purpose to it keep saying that and I'm
and you could go into that but give me
an answer life out of it because that
you'll be you'll be on the inside track
when the singularity happens it's got
all the trappings of religions that's
okay well you know that's sort of an
easy thing to say when we've only got
four minutes left because that's a whole
large topic there for one thing there's
at least three different major things
that are now meant by the word
singularity and I only talked about one
of them which is the self improving its
self improving AI part of it not the
Moore's Law part where you can predict
exactly when it's going to happen down
to the Year by drawing the little graph
and and not be like sort of gigantic
poorly specified breakdown part or the
and not even really necessarily and you
know sort of like that the minor
sideline and the fundamental change in
society brought about by the presence of
things smarter than we are I have like
something up called three major schools
of singularity thought I mean there's
some people who think this smarter this
thing smarter than us is already around
like George Dyson thinks that Google's
already alive you know yeah you know to
call something a religion I it seems to
me that you've got to sort of be making
a claim that they believe certain things
that are not true and moreover this
isn't just a case of there's lots of
people who have religious beliefs about
quantum mechanics
but that doesn't make quantum mechanics
or religion and and and the notion that
you're going to take the notion of you
know while humans are intelligent we
know we can be done let's build an
intelligent machine and just sort of
label that a religion because various
people have followed that and gone wrong
in various ways III think that that to
me that shows a sort of lack of you know
sort of scientific historical
perspective on how confused people were
about biology say in the days of
vitalism and and how many things they
did wrong or the early alchemists who
were poisoning people left and right in
pursuit of what would eventually turn
into chemical knowledge but had a whole
lot of mysticism mixed in with it
everybody who opposes this sort of hard
cyber certainty ideology is accused of
being the guy who put Galileo in
shackles or or something rather euros
but I mean once again I'm not that guy
I'm at the cutting edge of the research
that you say is the important research I
think undeniably I mean in terms of
achievement I'm doing that stuff I'm not
just talking I'm walking the walk so I'm
definitely not the guy shackling per
Galileo or you know sitting down I just
I just think I can do it without having
to believe in this program and I think
it makes me better you know I think
morally but you look but you object when
I want to know things and even when I
want to do things that look like they
ought to be physically possible
not exactly objecting I'm just mildly
making fun of you and it's in a
good-natured way but I just I just I
mean the thing is you have to understand
is that for 30 years I'm the one who's
been being told oh you really you know
if you don't buy this stuff you're some
sort of irrational list and I I just
don't see it that way I think that I
think that this sort of attempt to be
irrational to the point of totality just
makes you more irrational because it
pushes you beyond what you're capable of
you like give me an example of a
specific way I can't be rational even if
I want to be actually that's a pretty
easy challenge I need
I gotta go I'm working on some
algorithms that you would think of as a
lie even though I don't and I'm gonna I
want to get back to him so well actually
I don't know I actually tend to be a lot
striated and I do distinguish between AI
and AGI so if you say something isn't a
I then I'm very unlikely to consider at
AGI artificial general intelligence so I
don't think I need to worry about that
these are these are these are these like
you know these disputes between these
these different like tiny distinctions
between the different sexes are
impossible for an outsider to understand
let me assure you okay okay have a nice
day
hope you're an Indian person sometime
soon okay