Interview with Jaan Tallinn on Longevity, Existential Risk & AI | Vision Weekend US 2021

so
um this is yon talan there may be some
people here a few who don't know who yan
talon is that's conceivable i suppose
no seriously there may be and don't feel
bad if you're one of them yep i checked
actually okay there you go right so um
he's probably best known um as being a
co-founder of skype
uh right see
somebody didn't know that there you go
but that's his most famous thing i
haven't started meeting people who don't
know what skype so yeah that's why i
wanted to
mention that
um he's also also a co-founder of
cser which we'll be talking about
and um
fli which many of you are familiar with
and okay and what you may not know is he
also was involved in metamed
and what my favorite fun fact to know
about him is that his bachelor's was in
physics and you know how very useful it
is to have a foundation of reality in
your world view and in fact that is
reflected in in yon's life and so
something people should know about you
okay
so so that's who he is now
uh you're very active in the existential
risk community i know it's a central
interest
one thing that you find in the
long-termist community in the ea
community is they talk a lot about
future entities and we all care about
future entities right
but there's some debate about how we
should think about these things some
people
use kind of a discount rate when they
think about future entities it's the
common thing to do
others argue that no we should value all
future intelligent entities as equal to
the people in this room or perhaps more
so right they might be more intelligent
maybe they'd be more interesting than us
and somehow right could that be possible
no
but okay let's say maybe it could be
true uh let but they argue we should
value these future entities the same as
current entities and there's going to be
a lot more of those hopefully right
so this argument though you follow that
path and pretty soon if you're kind of
an altruistic person you go oh my gosh
my life really doesn't count at all
right i should devote my entire waking
moment
to making the world better for these
billions zillions trillions
number of future entities who are going
to be even better than me and i owe it
to them to just spend my whole every and
never go to fun parties right
so
seriously though how do you think about
this how do you think about these future
entities
and do you use a discount rate do you
value them like this how do you or do
you even think about this
i mean i think a little uh
but
i mean
at the extreme end that uh you will get
this thing called infinite ethics
like infinity is like really weird like
if you just open like a small crack in
your hypothesis base like it's just like
floods everything uh in your worldview
so like i know
that's like one thing that i like to set
aside because it just like blows up
everything
but like one i think one thing to
observe is that uh
like if you go from like 50 percent
focus on long-termism to 100 focus
that's just like factor of 2x
whereas like if you are going from like
um
like sort of just
um
having like a better model within the
long term isn't that might be like
thousand decks or a million extra
billion x so in that sense it's like
you're not gonna win that much by going
just like fanatic on long-termism oh
thank goodness i'm so glad to hear you
say that i certainly don't because we
need to i feel strongly that we need to
enjoy our lives and i understand you
recently became a grandfather is that
true that's true i'm thrilled for you
isn't that great folks
okay so here's a uh i know you have you
know you're very interested in an
existential risk and i've i've seen you
have kind of a prioritization list which
i agreed with 100 percent which is
basically you put the existential risks
at the top
so you've got your
ai alignment issues
and you've got your synthetic biology
potential issues very serious issue
then you have a creative category called
unknown unknowns which i think is
awesome right always got to keep that in
mind and then you kind of draw a line i
would think and then you've got your
your risks which are extremely serious
disastrous risks
uh catastrophic risks but they're not
existential to the species right you've
got your climate change issues you've
got your nuclear nuclear war issues
so i just wanted to throw out an idea to
you i know you're also interested in
health and longevity
it's kind of hard to see well how does
that even fit in here and i would make a
i'm going to throw it out and say all
right it's not a species existential
risk but it is an individually
existential risk right
so it seems like it's not perhaps up at
the up at the top two
but i would argue maybe it maybe it does
rank uh conceivably since it will kill
everyone sooner or later maybe we can
put it above the catastrophic risks so
maybe under unknown unknowns longevity
and aging might go there what do you
tonight
can i say something no no this is an
interview with him
no i'm joking of course you can but let
me wait give me a minute let's hear him
first
ah
yeah um
hey if you don't have an answer we can
try it
i think about you
one thing is that like you can just like
do the numbers i guess uh like how many
uh people it would say like first of all
like
what is the assumption how long the
human error will last like will it last
i mean whenever i come to silicon valley
like kind of the
the horizon will shrink and then i'll go
back estonia he's like
so like i
probably like going back here it will be
again like five years five more years
and that's it but
but like having just pressed come here i
think it's still have like a several
decades left so you can you can
calculate how many people would
longevity uh research effect in the next
20 years compared to like a nuclear and
a big nuclear still wins uh uh from that
perspective but uh yeah okay a numeric
let's just do the numbers there's
there's there's a physical shut up and
multiply
fair enough i'll call on you soon okay
so uh then i wanted to talk about you've
i think you have been quoted as saying
that you would overall you'd like to see
more
resources more tension going into risk
issues
for both for ai and for synthetic
biology
so if you think about the ratio of money
and and time flowing into the
capabilities you know let's push the
research forward and the technology
forward versus
uh let's look at some of these risk
issues that ratio how would you adjust
that what you know what's your gut feel
do we have to boost up the risk by a
factor of 2 10 100 what's the what's the
what do we need to do here why would
they boost up risk unless you understand
no you know
the attention to it
yeah okay safety yeah safety thank you
uh
i mean yes absolutely we should boost up
safety but the big problem is that uh
boost up is usually kind of
multi-dimensional uh like there are like
a bunch of resources that it's kind of
hard to punch
on against each other hard to buy
science with money actually like as
there was like a just a breakout session
about that yeah
and uh
so like currently because of this crypto
boom you seem to be in a situation where
like in effective altruism in
long-termism that's just like much more
money than
kind of ideas
and and people
so
it would be really
some one really high leverage thing to
do currently seems is to just
come up with ideas creative ideas like
how
uh what are the things that uh if you're
gonna scale them up uh they could
actually buy us significant amounts of
existential safety got it
so i saw a quote from you recently um
something something to the effect of we
can think of society as a multi-agent
system
our goal is to make it more resilient
toward disruption
we want to build a more robust
civilization and i read that i said wait
a minute
this sounds like the ins inspirational
statement we came up with for our
intelligent cooperation group it's like
that's the same statement it's like did
he just read that so i just
you apparently have a fan club at force
that you didn't know about
so so on that note though um you've also
talked about blockchain
and using blockchain technology as a way
to coordinate challenges
can you say anything about what you had
in mind with that first of all like i
think
there are kind of two sides to this
x-rays coin
you can
make new technologies less disruptive
so they become kind of easier to
integrate and and use and control or you
can make the society more robust to
disruptions
so it can kind of take harder hits
so yeah making um
think you're thinking about things like
governance uh it kind of fits into into
the second category
uh and uh i think the most interesting
thing uh
okay like spanish inquisition two most
interesting things
about blockchains uh is
that
it's finally we have a
situation on this planet that we can
have a piece of data
that everyone agrees about without
without trusting any central authority
to remain that maintain that peace
so it's like yeah
most of you know uh know about this
concept uh
and but
the other like really the other
interesting thing is uh especially on
platforms that
can support smart contracts there are a
lot of like governance experiments
happening so it's like a primordial soup
of garnet's experiments so i'm really
hopeful that there will be
some interesting results sooner rather
than later that can possibly
uh kind of
export it from blockchain to the real
world got it great well i i just want
you to remember you have this fan club
okay you should stop by sometime all
right so um is there any dancing there
there could be dancing yes if alison's
involved we will make sure there is
dancing all right so um
so one of the ass so there's the ai
alignment issue right
so
there's a there's a lot of work uh there
should be more work but there's some
work being done now um one way that
we're here at foresight are trying to
contribute is uh you know we look at the
the work that's being done and it's like
they're building a house and they're
doing a great job and everything's going
well we've got the windows the doors the
heating system everything's going great
but we look at the foundation and we go
oh my god they built the foundation out
of wood
this is not a great idea so this is the
issue of computer security
and
the issue of computer security is just
going to get worse and worse right it's
already bad it's a real problem it's
getting going to be worse and worse we
are fundamentally insecure right so
one of our uh areas of interest is how
do we fix that uh
both for now and for uh future ais
how we're one of our projects is sel4 a
computer
project i know you've said that you
believe that software whenever you can
make something a software project if you
can solve a problem with a software
project that's a big win rather than
trying to do it through policy or
whatever right so that's what we're
trying to do
how do you see this whole
computer security issue feeding into ai
alignment or do you not see a good
connection there all right i see
so many things to say outsider um
first of all i'm not sure that that
diagram security situation has been kind
of deteriorating it's just i think the
stakes have gotten higher yes uh but
like uh like if you take a windows 95
and connect it to the internet it's not
going to survive it's not a good idea so
in that sense i do think that the
current uh at least some software has
been gotten much more secure
and also like as
some of the reasons to mention like
blockchains are kind of a live fire
environment
for security so so there's a lot of
knowledge that is being gained and
developed so that's that's great
so
in terms of like ai safety i i do think
that uh
being more deliberate with security
is just
like massive net positive for multiple
reasons like one is that
just proliferation seems like really bad
idea if you are if you're doing doing
introducing like
new properties to the world in general
that you're not
sure about how they're going to play out
so you might want to contain them might
want to contain them and gradually
release them or not release at all if
it's really a bad idea it turns out a
really bad idea uh and the second thing
is like well
we are building smart minds and and uh
like
like if you just i mean what we're
currently actually doing in large
language models if you think about it we
give them
um
the
basic distilled internet we we
just create the internet for for text
and then then clean it up and and give
it to the giveaway to train large
language models in that text there is
there is uh information about humans of
course there's information about ais
there's information about ai training
there's also information about uh uh
about uh training large language models
so of course these models will know
what's going on uh
so uh uh if we are gonna blackspeed
security that's like one um
one additional
like one big
causal channel that they can use
to have massive side effects great great
well after not right now but after this
event i hope you will talk with mark
about scl4 yeah
real quick okay uh you just christine
you said uh something like scl4 is a
foresight you know
something we support let's put it that
way do we actually support him we
support it we're doing it right now mark
we're supporting it
okay no we we would also like to give
money and that would be a great idea if
we had more money we would give them
money also okay
so on to more ai alignment so
right now ai research uses big data
right massive amounts of data now
depending on ai but yeah in general it's
a common thing and then uh
in our society we have we have
legitimate privacy issues we don't like
to have our data um
out there and compiled with everyone
else's in public
um so uh you know but there are there's
other societies where uh data is just
collected they're more top down they're
more centralized authoritarian they just
take the data they suck it up and then
they do ai research on it now are we
are we hindering our how does
it seems like there might be a problem
where the more privacy oriented
societies
are not putting as much data into their
ai research whereas the less free more
authoritarian society might
have more data to train their eyes on
and how is what are we supposed to do
about that is that an issue i do get
that question quite a bit like there are
multiple
multiple angles there i mean yeah in
general sort of on first approximation
yep that's
seems roughly right
on the other hand i mean before kobe i
did go to china every once in a while
island and having a background from
soviet union
china at least a few years ago when i
was at last was like way more open
society than soviet union ever was yeah
so like they totally had discussions
about about this uh privacy and
and uh
yeah they had discussions that like in
soviet union nobody dared to have
so it's not that bad not that black and
white there's also like techniques like
zero knowledge proofs
and there's the project called open mind
that does a homomorphic encryption data
in order to uh yeah
make better privacy trade-offs
and finally there's like fundamental
kind of trade-off between data and
compute
so like in ai is called synthetic data
and like one thing that i've always
mentioned uh to people who are very
hung up on data is uh
the zero in uh in alpha zero and zero
data
[Music]
okay so there's a trade-off uh you can
you can do things without having that
much data got it got it now you and i
are both advisors to the machine
intelligence research institute i know
you've been supporting them for many
years
while you were
founding skype i think around the early
2000s
um eliezer yeah some of you may recall
eliezer jedkowski was at foresight
institute giving his earliest talks
about friendly ai that was the first
time in my life i saw before he started
speaking about this
people talked about ai risk and ai
problems but it was always in a
fictional context right uh it was
speculative fiction it's not that we
didn't take it seriously it's just that
it just seemed way too far off you
couldn't really talk about it in a
serious way and say no we really should
start thinking about this in real life
and to give him credit eliezer was the
first one who came to foresight and said
i want to talk about this to your group
and we said yes right i mean and so i'm
proud that we played that role
so but at this point that was that was
the first group but now there are quite
a few groups that work on this issue um
what would you say are the different
strengths of the different groups i know
you support more than one right so
you must see different strengths which
what are the each one good at so that
other donors can sort of direct their
resources toward the group that matches
their interests yeah i think that that
would be like a way too long answer all
right but uh but you like multiple
groups yeah i mean first of all i i
think it's important to
to
okay taking one step back from what i
was going to say
you can i think it's there's valuable
division of ai and air safety
in following ways so you can kind of
think about
what are the issues and risks from
deploying ai that was invented like
three four five ten years ago uh this is
a kind of uh the
area of like so-called you know
near-term risks uh or or like uh aitics
things like that
and then there are uh
trying to address risks uh from ai that
has not been invented yet i think
there's like valuable uh
division of labor there to be had and i
i noticed are unfortunate kind of tribal
tensions between those those two groups
like that one group thinks that the
other is doing kind of science fiction
the other group thinks this
uh
that the first one is doing uh
irrelevant things
which i think is very unhealthy it's
important to do division of labor but
like when we focus on this like
long-term uh issues like one one
by that definition
the thing that kind of
but the common the common denominator of
these long-term issues is that
these are things that have not been
invented yet which means that you have
to have like epistemic uncertainty
which means that it would be kind of
great to have some kind of division of
labor there and exactly i think that's
that's what uh uh
the spectrum of organizations are are
doing they are making essentially making
different assumptions
about uh uh how these things are going
to play out and what are uh what are
kind of reasonable
uh
aspects to to focus on
um for example like fast fast takeoff
versus low takeoff is it going to be
like
largely a technical issue and the world
will kind of transform in a matter of
minutes or is it going to be a social
issue for example so like yeah
and yeah going into detail would be like
no i know we could be here all day um
let's take a couple questions lawrence
you've been waiting yes you were asking
ayan about
um
an existential risk to an individual
and i kind of knew what he's gonna
answer but i i wanted to kind of change
a little bit in terms of expected value
not only as an individual so so
death by aging is 100
as far as
we've seen so far and it's probably
going to continue like that
cryo and and fixation and stuff like
that this but yeah so but like there are
many
um
so expected value of
synthetic bio
virus killing us or or comets
killing us
so the probability of that is
less
it's not 100
there's nowhere close to that
and
sure it could kill everyone and the
species for good
but uh as long as we don't cure aging
it's going to continue killing the
species just kind of
iteratively not all at once
so yeah
do you
yeah value of hearing aiding versus
expected my point is you can kind of
like think of it like this century as
like some kind of special regime i mean
hulen karnovsky has this great uh
uh site called cooltakes.com very good
makes this very persuasive with this
case that this century is really special
so you can kind of like do the numbers
for this century like how many people do
they expect to die
uh this century and then like how many
how many uh what is the how many people
in expectation to expect to die as a
result of nuclear or ai or or bio so
like then you can just do that math you
don't like because like if you say like
100 that kind of assumes that that this
century will last forever it doesn't so
it's the data
that yeah i think it's kind of
reasonable to assume that we we have
some kind of finite horizon for planning
and after that it's uh
uh anyone's gain in in some sense but
like yeah sure in this individual level
it's much much higher
chances
to change that i understand your point
but like yeah it's just like
we're not in a percentage game we are in
the game of uh trying to
you know uh help
maximum amount of people and like with
longevity you can i claim you cannot
only
help the people who are basically living
this century okay one last question
right here all right thank you um it's
gonna be too different
so i'm interested in how these risks
interplay with each other it's not it's
not that we just have one climate risk
to
nuclear risk is that they're actually
connected and the sort of multi-agent
system that we all live in the
robustness of that that you said robust
not just to the exponential tech we
developed but robust just to the
ignorance that we have and the way that
we'll you know find ourselves in a
climate disaster just by default and how
we are robust to all these things and
the cascades that will happen
what sort of framework do you apply for
building robust societies
yeah uh mostly uh like
i think there are
two questions there like one one is that
like what kind of framework i add to
like uh
sort of combinatorics or combin
combination risks and like the framework
is like ignore uh mostly because these
are kind of like second uh other effects
uh and it's kind of hard
hard to like it's just like a
combinatorial explosion uh pretty
quickly so like
but i would be welcome would be like
welcome people who picked their
combination and focus uh focus on that
uh in terms of robustness yeah
uh
i mean
yeah i don't think there's any any
single answer like again uh looking at
the governor's experiments in blockchain
it's one thing the other thing that i'm
going to try to brand myself a little
bit is like i literally i live in uh was
my last name says in thailand and
that is like physically equidistant
between the west coast
sorry east coast of china and east coast
of us
so i kind of
i think like there is robustness
to be bought by having
communications open and i'm gonna try to
help that a little bit
um
yeah i think these are good things that
just came to mind but but it's uh
yeah also i just support organizations
who are who are uh trying to uh think
about these things like kobaya and
oxford for example great and let's thank
jan from coming all the way
[Applause]


Mo Gawdat - Scary Smart: A former Google exec's perspective on AI risk

[Music]
hey everyone and welcome back to the
tourist data science podcast now if you
were scrolling through your news feed in
september 2021 like i was and you
trained the social media algorithms to
show you content specifically related to
ai you may have caught a splashy
headline from the london times that read
quote
can this man save the world from
artificial intelligence now the man in
question was mo gaudat and mo used to be
a senior tech executive the chief
business officer at google x and as many
of you may know google x is google's
semi-secret research facility that
experiments with moonshot projects like
self-driving cars flying vehicles and
even geothermal energy and it was at
google x that mo was exposed to the
absolute cutting edge of a whole bunch
of different fields one of which was ai
and it was his experience seeing those
cutting-edge ai systems learn and
interact with the world that came with
some red flags hints of the potentially
disastrous failure modes of ai systems
that we just might end up with in the
future if we don't get our act together
now emo writes about his experiences as
an insider one of the world's most
secretive research labs and how it led
him to worry about ai risk but also
about ai's promise and potential in his
new book scary smart the future of
artificial intelligence and how you can
save our world and he joined me to talk
about just that on this episode of the
towards data science podcast
[Music]
chris came across you
uh quite recently actually last i'd say
two months or so i was scrolling down
twitter and i see this uh this kind of
update from the news feed or whatever on
the side they're saying like you know
this uh tech silicon valley guy google
is warning about the uh robot apocalypse
type thing it's all you know very
sensationalized that twitter does and
it was it was this really interesting
discussion of sort of what you're up to
your book but also your your life and
times at uh at google and and then we'll
get into that i'm sure but i'd love to
hear a little bit about about your
background like how you first got into
tech generally and then how that brought
you to google and and from there
yeah so i had i had really two lives i
lived two full lives i still live two
full lives uh you know uh it's quite
it's quite interesting because they're
very very different one one side of my
um you know last 20 30 years uh i've
been a serious
code developer a serious engineer in
many ways a serious mathematician and
then i became a business executive i
started my career at ibm
uh you know of course
i should probably say i'm i'm the
generation that owned a sinclair and uh
uh there is a car in the background
sorry
yeah i should i should say i am the
generation that uh that owned a sinclair
and a commodore and the very first you
know ibm compatible and the whole thing
right uh started coding in my very early
years uh
maybe at age eight or something like
that and then
uh you know just came very naturally to
me and i still continued to code until
probably six seven years ago
and i hid it from everyone because it's
not good to be the ceo and still code in
your evenings but it's just a passion
for me and and it's um and it's you know
so i started my career at ibm
uh
worked there for five years and midway
through my my career um
my you know boss basically said well you
know selling is becoming a lot more
technical and you know some of our
clients who are highly highly technical
they would like to have an account
manager that is technical and so you
know i was the first in ibm egypt where
i started where we started to do that
and basically it you know customers
trusted me very very significantly
because i knew exactly what i was
talking about i could build
configurations with them i wasn't just
selling selling selling and from then
onwards my my career took a a um you
know a change i worked at microsoft and
then i worked at google for 12 years but
always in business roles
which were not really business uh fully
i mean at microsoft i was uh you know i
start to i started uh with the
uh you know with a career that took me
to uh become the head of the
communication sector so very very uh
serious tech again in terms of trying to
integrate telecom systems and so on uh
which wasn't really just a business
person and at the end of that i was
responsible for emerging markets
globally for the tech sector then i
moved to google and google is quite a
technical place i i launched half of
google's operations globally more than
100 languages
which again is a very technical job
because it's not
it's not a job where you where you just
open an office and hire two sales people
you have to build the internet
infrastructure you have to build
e-commerce you have to work on your
proxies and networks and all of that
and when it's done then you basically
start to prove to offer google in the in
the country and then i moved to google x
where i spent
uh the last five years i spent 12 years
to in total in google the last five
years i was the chief business officer
of google x which probably could be one
of the most technical places on the
planet
and my my role there was to try and
translate the incredible technologies we
were building into the real world if you
want uh so build you know participated
in building what is now known as the
moonshot moonshot factory predictable
innovation if you want and and that was
an amazing career so on the tech side i
still am the ceo of a tech startup today
uh i still have uh you know i'm the
co-founder of another tech startup which
is in the happiness space which takes me
to my other life by in 2014
to start you know in my 20s in my late
20s i was very unhappy if you want
even though i was extremely successful
and that led me to 12 years of research
on the topic of happiness through
through basically an engineer's mind if
you want which is a very
uncharted territory
arrived at uh what is now known as the
happiness equation and then a happiness
model that is very very engineer-like
you know almost like a workshop manual
when this breaks do that when that
breaks do this and uh and uh and then in
2014 i was chief business officer of
google x at the time my son
sadly left our world due to
a very preventable and silly really
medical malpractice
uh that happened when he was undergoing
a very simple surgical operation
and as a result uh i just completely
shifted my life to become an author
first i wrote solve for happy which was
you know the engineering approach to
happiness if you want which became an
international bestseller 32 languages uh
almost everywhere
and then
and then recently i'm now merging those
two words together uh by by you know
writing scary smart or publishing scary
smart which basically
appears at the beginning to be a book
about artificial intelligence and it
definitely is the wake-up call for a lot
of people about what's happening in ai
but more importantly it's the second
half of it is really about humanity in
the age of the rise of the machines and
and how humanity should be if we were to
have
humanity continue uh to have its you
know
the perks that we've had in our planet
uh since we started history and you've
talked about the connection between your
experiences at google x some of the
things that you saw and this kind of
motivation to get working on a book
about artificial intelligence and more
generally to start warning about it i'm
curious what are some of the things that
well first off generally i think
everybody's going to be curious what are
some of the things that were are being
done at google x what did those projects
look like and then how does that tie
into the artificial intelligence side of
things um you know we think usually when
we think about ai and google we think
about google brain we think about the
algorithm itself
google x i think is a bit more of a
black box at least it is to me i'd be
really curious about sort of the the
intersection between those two things
xxx is an amazing place it's you know it
started with the passion of google
really which you know the google i
joined was very very serious about
making the world a better place and x
was an attempt to solve big problems
that affected humanity uh we we were
attempting to solve them basically with
technology that is
unheard of really i mean unthinkable
even at the time when you when you think
about the time where we started to to
operate to develop the concept of a
self-driving car
you really have to imagine that there
wasn't this was not possible at the time
okay but but the self-driving car truly
solves a big problem so you know we
millions of people die on the roads
because of accidents every year and 92
percent of those happen because of human
error
really and so you know the car industry
for many many years attempted to move
from you know crash worthiness as they
called it you know airbags and other
things so that when you when you have a
crash you actually survive
to something called crash avoidance
which was basically enhancing the
driver's experience to
make sure that you know you see better
on the road or you have anti-slip brakes
and so on and so forth which would help
you not crash in the first place but the
truth is 92 percent of the accidents are
because of human error and so at the
time larry and sergey basically
our founders at google at the time
suggested that maybe we can avoid the
human error by having the car drive
itself so that it doesn't put makeup
when it's driving and it you know it
doesn't uh text one and cross a red
light and so on and so forth and it was
a crazy idea when you really thought
about it this was proposed
uh
probably thousand
and
eight maybe so before the deep learning
era
it was at the very very early so so the
real real breakthrough in deep learning
if you
if you think about it was at least my
very first breakthrough was when google
published a white paper on unprompted ai
still available it was 2009 but the work
was done way before that
when we asked a few computers to go and
watch youtube
and and they just came back and said hey
by the way there is this very
cuteness that is available everywhere
and they found cats and and
you know how it works and so basically
they found the pattern that describes
through deep learning what a ca the
cuteness of a cat basically the
fairiness the movement and so on and so
forth and they could find every cat on
youtube completely unprompted
but it was coming i mean
deep learning and
and unprompted
learning in general i think started to
take shape at the turn of the century it
started to become solid uh by
late
2010 maybe
yeah i mean historically i think alex
net in 2012 is usually like taking to be
the the birth or at least the moment
that normies like me became aware of of
deep learning because of its yeah
yeah so the fact that i mean you and i
you and i will will both agree that we
none of us paid attention to what was
happening i mean the first time deepmind
uh presented deep cue to us at the at
the
vice president's meeting of of google
you know it was fascinating i think must
have been 2013 2014
and you know demis was the ceo of
deepmind was an amazing human being in
every possible way uh it was just
showing how the machines can learn to
play atari games in hours really in a
matter of hours they became very very
proficient probably the best players on
the planet and
and
and the only reaction we had at the time
was like wow that's fascinating right
but you don't you don't connect the dots
you don't look backwards and say oh my
god how far did we come and you don't
look look forward and say oh where
is this going right and and i think the
truth is uh when you're inside uh
like i have been for almost of my all of
my life you see a very different picture
so you know everyone knows about the
product adoption curve you know the
s-curve when a technology is released
and how people pick it up and then it's
you know it grows very quickly and then
stagnates you know everyone knows about
of course rayquaza worlds and
law of accelerating return everyone
knows about moore's law and so on but i
i actually
um explain a law that i used to to
to use myself
you know and i explain it in scary smart
which is uh which is what i call the
technology development curve and the
technology development curve is really
rarely ever seen if you're outside the
see that the inner circle if you want
because the the
that curve is almost flat
for years and years and years and years
and years and then you find the
breakthrough and that breakthrough turns
your your trend upwards in terms of the
speed of development uh
you know
almost like a hockey stick really i i
just say that's it's a hockey stick on
its on its side so so the longer arm is
just on you know the the x axis and and
it is what we see in everything the idea
of uh of deep learning not excluded but
if you remember the whole conversation
about ai started in 1956 uh dark most
workshop workshop
and and nothing really happened we had
two ai winters in 73 and in 87 and you
know almost completely forgot about the
idea of ai
until deep learning and that's the
hockey stick point until we found deep
learning and then the trend since then
has been
doubly exponential really yeah and
actually this is i i really like that
you're bringing this uh sort of more
business-minded lens to the problem as
well because i think this is a dimension
that a lot of technical people don't
think about just this idea that as you
say for a long time deep learning
machine learning even generally wasn't
delivering economic value and then you
get to a point where all of a sudden it
is delivering economic value and now it
becomes possible for ceos and and ctos
and other people to argue for funding
for bigger models and those bigger
models generate more economic value and
you have like a closed loop that gives
you some form of takeoff is is that part
of um part of the equation here is is
that is that the sort of like the
coupling between the technology and then
the return on investment finally
starting to kick in well that's the
whole idea so actually one one of the
things i discuss uh in in scarysmart is
something i call the three inevitables
which basically
are my view of what's going uh to be
happening around ai uh in general i
believe that ai is going to end up in a
place where you know the three
inevitables are that ai will happen uh
it will become smarter than humans there
is no avoiding either of those two
inevitables and that mistakes will
happen in the
on the path right and and and the main
reason that i say ai will happen of
course is that it already happened we
found the technology breakthrough but
that there is no stopping it i mean elon
musk in his uh interview with joe rogan
uh basically starts by saying uh look uh
mark my words uh ai is more threatening
than nuclear weapons right but then he
continues to say and uh you know i'd
lobbied to stop it but there is no
stopping it and and the reason that
there is no stopping it is a simple
prisoner's dilemma okay it's a very it's
a very uh you know we we've signed up to
a a competitive
power led capitalist market and we
signed up for that and so accordingly
what you're seeing today is that there
is absolutely no way even if the world
completely agreed that we should stop
developing ai to avoid the threat
there is no way to do that because if
the americans develop ai the chinese
will develop ai as a result you know to
as a response if google develops ai
facebook will have to develop ai and
every startup in the world including my
startup will develop ai because that's
what the investors want to invest in if
you don't you know and if you you lose
the competitive edge if you don't and
and that inevitable is basically
taking away the only
uh
you know side track if you want that
would have taken us to a point where we
could have stopped and said hey can we
figure out the control problem
completely before we go down that path
can we figure out the actual impact uh
you know the
possible threats before we go down that
path there's no there is no way that
this will happen ai will continue to
happen it will continue to happen fast
and can you unpack the uh the control
problem we've talked about the control
problem in different forms on the
podcast before but i just find it's
interesting to hear every person that's
different take on it i
so so can let me talk about that second
inevitable because of course yeah
because no because it clarifies the the
control problem answer very clearly uh
so so
the second inevitable is that ai and
everyone agrees almost all predictions
agree uh ray kurzweil's uh you know ray
corswell's um
prediction is that by 2045 uh ai will be
a billion times smarter than us okay uh
we i know i think most of us have read
2029 that's eight years from now this is
when we get the first ai that's smarter
than human artificial general
intelligence in general would be able to
to map the you know the
neural networks in a way that it beats
one human brain
that's eight years from today and
nobody's talking about it it just blows
your mind we talk about kovid and and
manchester united and mo salah and we
don't talk about the fact that the
episode of history that started from uh
uh you know from
um the day humanity became the smartest
being on the planet and then the apes
were where the second uh at at a
distance you know by in eight years from
today we become the apes nobody's
talking about it assuming i guess that
prediction materializes i mean people in
the space who who agree with with agi
becoming a thing have different
timelines but i agree i mean does it
matter yeah whether it's five years or
20 we're talking about absolutely who
cares yeah yeah i mean i think i think
everyone would agree that it's in your
lifetime okay it might not be in my
lifetime but it's in your lifetime for
sure okay uh my my prediction is that
having seen
tech development from the inside
we our brains are not wired to
understand the exponential function okay
but that but the truth is one doubling
from two doublings from now is what
matters
do you understand it's it's now you feel
like yeah it's going to take ages like
again one of my favorite things when
rayquaza was talking about the the law
of acceleration accelerating return is
the example he used with the human
genome project right and he basically
says you know people came to me and said
it's going to take 15 years and after
seven full years almost half of the
project uh you know we had sequenced one
percent of the genome yeah and so most
most linear thinker would thinkers would
say okay so it's going to take 700 years
you know
to to sequence the remaining 99
and and i said
ray at the time he said basically we're
done
because you know one percent is seven
doublings away uh from uh from a hundred
percent and so accordingly and he was
right within 15 years we had sequenced
the genome right and and and that's the
idea that our brains are not wired for
the law of accelerating returns no not
acquired for exponential growth okay
we we know that we can have one banana
today and you know two bananas tomorrow
and three bananas the day after that's
our thinking huh we don't understand
that you can have one today two tomorrow
and for the day after and you know
continue to double double double double
right and and and so so the the the
thing is
um
it doesn't matter really it does not
really matter if it's 2029 or 2039 it
doesn't really matter okay
what matters is
if they're smarter than us
and then the exponential growth
continues whether it's 2045 or 2065
doesn't matter okay and they become a
billion times smarter than us
the the control problem becomes a
question of ego it's not a question of
technology anymore okay it's a question
of the ego of humanity thinking that
they can control something that is a
billion times smarter than them
okay and and that ego to me is like what
are we talking about here i mean
everyone knows that the smartest hacker
in the room goes through every single uh
um uh
you know defense that we put up there
and we're thinking of trivial trivial
stuff like we're gonna box them box who
can can you actually box a hacker that
is double as smart as you are
let alone billion times smarter as you
are there are all kinds of examples too
just just like down that one thread when
we talk about you know a single hacker
who's really clever
getting put inside a box and and saying
okay you know at least if we can control
this hacker we can control an ai and put
it in a box eliezer yukowski has a sort
of thought experiment like this that
he's actually done experiments on where
he'll tell somebody hey i'll be the ai
i'll pretend i'm the ai your only job is
to not like write the following words in
a box in the words of something like um
let me let you out or something like
that or i'll let you out you're just not
supposed to write that and he's done
this experiment i think like five times
and he's managed to get out twice or
three times like it's a disturbance even
for human level intelligence even when
you know the rules of the game
it's next to impossible to imagine that
yeah it's absolutely and and that's
that's my whole point about the control
problem but but my point goes a step
further which i know techies will
take a bit of time to to grasp
my point goes into the reality
that we are not creating another machine
i think that's what we people are
missing i've seen those intelligences
doing things we did not ask them to do
finding ways every ai we've ever created
develops its intelligence in ways where
we don't understand how it arrives at
its results okay there is no absolutely
no way on the face of planet earth that
we're going to be able to tell the
recommendation engine of of of instagram
what to do we just we're too slow as
humans this this engine is doing this
billions of times for billions of users
every single day right and and we and we
cannot assume that there will be one
human that says hey by the way you're
spoiling it here you need to start to
think differently no one's ever gonna
interfere and as long as the
recommendation engine of
of
of
you know instagram is competing against
their commendation engine of twitter
they're going to develop their own
intelligence and so my point of view is
this
what we are creating here is sentient in
every definition of the word
okay we're not creating a machine we're
creating something that gets born
acquires knowledge on its own develops
its own intelligence takes its own
decisions
has agency in those decisions whether in
the form of robotics like a self-driving
car or the worst the worst agency is how
they mind control us they completely
mind controllers and nobody's aware of
that so so they have the ultimate agency
and when and when they have the ultimate
agency by the way they also procreate
only the differences you and i procreate
we need to find the partner in two three
years and then convince her and then or
she convinces him or whatever and then
nine months later you have a baby that
15 year years later may have impact on
the planet those things can create
copies of themselves we encourage them
in the way we develop them through the
you know the the the teacher and the and
the maker bots
the way the way we're doing it is we're
actually saying create copies of
yourself you know the teacher the
teacher bot will test and the makerbot
will discard the bad ones and and create
you know copies of the good ones right
they're procreating in seconds okay and
they they have the fear of death they
they will you know they will be subject
to being ended switched off now every
intelligent being doesn't want that
every intelligent being wants you know
to to to uh to self-preservation uh
resource you know aggregation and um uh
you know the ability to uh to to be
creative sometimes now because of that
my theory is very straightforward my
theory is
those machines
are sentient meaning they will have
consciousness okay as a matter of fact
any one of us who's ever coded iai
understand that they're more conscious
than us if consciousness is a form of
awareness of what's inside you outside
you and you versus others those machines
are designed to be conscious they they
know everything they know what you did
yesterday they know what you're going to
do tomorrow become even better than you
from your trends okay they know the
temperature in san francisco and the
pollution level in beijing they know
everything okay
their memory capacity is the history of
humanity as it's stated on the internet
they have access to every piece of
information in breaking news now we we
we have to start thinking that way
they're conscious okay they're creative
we've seen that in so many examples in
the way they play go as alphago does or
you know in in
in in in the way that they now develop
create paintings and music and so on and
so forth they're emotional which most
people go like what is he talking about
of course their emotional emotions are a
form of logic okay we think that
emotions are irrational yes fear you
know if something scares you your bio
physiological response is nine seconds
long and then your brain engages to
evaluate if there is a fear a reason to
to be afraid or not okay and what is
fear fear is a simple equation it's it's
my state of safety at t0 minus my state
of safety at t1
okay it's as simple as that if my if my
perception of my state of safety in a
future moment is less than my perception
of my state of safety now the difference
between them is amounts to my fear
okay and and of course every machine
everything with a logic will have the
same thing right the the you know
a puffer fish feels fear in it or panic
if you want panic basically is another
equation panic says that that t1 is
imminent if t if the time to t1 is short
then i panic i'm not i'm not only afraid
anymore i'm panicking right so a puffer
fish will panic and it will puff we
panic and we fight or flight the
machines will panic and maybe you know
if a tidal wave is approaching a data
center
they'll replicate themselves to another
data center we don't know what the
response is but they will have something
that's equitable to an emotion okay as a
matter of fact which pisses off a lot of
people when i say it they'll be much
more emotional than us
because if you compare yourself to a to
a goldfish
with a with a cognitive capacity that
cannot comprehend what hope is
okay cannot look at a future situation
analyze it and then say okay i wish it
will be different or i expect it will be
different we have that cognitive
capacity so we can feel more emotions
than a jellyfish i hope right for some
of us we don't but sadly but for most of
us we feel more emotions than a
jellyfish and accordingly a being with
more cognitive capacity than us will
feel more emotions than us i think this
ties into some what you were saying
earlier in terms of how counterintuitive
this is so so first off um
it does seem clear that if you're a
reductionist if you believe that
consciousness just comes from the
physical state of a thing then yeah you
have to believe that ais can be
conscious in every genuine facet and
meaning of the term um but then there's
this question of like at what point
right like at what point is it more
useful to start thinking of an ai as an
embedded agent as a thing with with
agency versus as a statistical kind of
artifact or a statistical process do you
have any thoughts about that like
when is that when is one lens more
useful than another because presumably
like a decision tree or a you know an
mnist classifier seems like it falls
more into the statistical bucket whereas
now with things like you know not
necessarily gpd3 but some of the the
more kind of general purpose models
we're seeing you can start seeing that
more kind of agenty lens being more
useful i'm curious what your thoughts
are on that transition what decision
three would lead alice and bob of of uh
of facebook to develop their own
language
i mean the experiment is very well known
two bots
designed to trade against each other or
trade basically between each other right
and very very quickly they discover that
adding numerics to the negotiation in
terms of repetition of words would lead
them to get to agreement quicker so they
developed their own language what that
was not part of the script we gave them
it's not a decision tree
okay that this is bait this is pure
intelligence pure intelligence is for uh
for the for for uh deep q
uh that's like 10 years ago to realize
that when it's pre playing breakout it
can actually break a hole in the in the
bricks at the top of the screen and put
the ball above that's purely
intelligence
so it would be generative modeling would
that be like roughly where you'd put the
cut off
i i i tend to believe we're already
there
i i think what what is
i don't know how to explain this to you
there are things about our world that
humanity doesn't understand
okay and we don't understand it we
dismiss it
we dismiss it as if it doesn't exist but
it does exist okay and in the case of ai
we we are so
um
uh caught up in our past linear
regression okay that we're unable to
imagine that there must be something
happening
for a machine to be capable of
understanding my preferences to the way
it is
it's already happening okay what with
what point it happened at
is irrelevant
you know
it's it's actually quite interesting
when we when we think about
machines today that are literally
dictating humanity's
um uh only view of information do you
realize that
you realize that that everything you
know jeremy everything you know is
dictated to you by a machine
there is no human anymore that's
prioritizing which news shows out shows
up in your feet there is no human
anymore telling you what matters uh you
know uh on instagram or on twitter no
human is engaging in that at all i i'll
share with you my own personal example
seven eight weeks ago i i i swipe on
instagram to send cat videos to my
daughter because my daughter loves cats
and i adore my daughter right and and
among those videos that shows one
teenage girl playing hotel california
hellfrees is over solo she played it so
well amazing i clicked like
okay so instagram recommendation engine
immediately goes like oops more music
videos shows me three male players okay
one played a song i didn't like and two
played poorly so i swiped away from them
i wake up the next morning and my entire
feed
is teenage girls playing rock
okay instagram's understanding
of my actions is he wants to see girls
right now
it's a naive example but understand this
this
perception this advantage point of the
world
claims that rock music is dominated by
teenage
guitar players teenage female teenage
guitar players that is completely the
opposite of the truth right but if i had
continued to swipe on those and like
my perception would be so skewed and
that's just about music
imagine how skewed our perceptions are
today
when we
inc when when when it comes to our
ideologies if you're a manchester united
fan
you believe that they've never been
scored against okay if you're you know
if if you if you believe in
an ideology that is pro-violence
everything that will come to you is
going to be violent
and that's cue
incredibly is entirely by a machine
and we're still we're still debating if
they have free will and agency to affect
us
yeah it's interesting because it it also
caused you to think a little bit about
human agency and you know to some degree
whether whether we have free will on the
other side where you have if you look at
this complex of like media producing uh
content for twitter you have
essentially an ai system that's telling
cnn and telling fox and telling all
these these outlets how to structure
their articles what biases do really
well based on this algorithm and so we
kind of abandon our free will to these
systems too i mean it's sort of this
cycle absolutely it's a double whammy
it's double exponential in every
possible way now now that actually is a
very interesting segue to my my whole
theory so scary smart is written in two
parts okay part one is what i call the
scary part and i have to admit to you
openly it is very scary like even i when
i was reading the audiobook a few weeks
ago
i would stop every now and then and say
oh my god do i really want to read this
for people it's it's it's very scary
when you see it from the inside and you
recognize and i promise you i even have
serious developers who are unable to see
the big picture and how far we've come
so it's very scary because in a very
simple way you were saying look it's
really nobody knows when okay but it
seems that if we continue on this trend
the episode we're going to first become
the apes
and something else is going to be
smarter and then
the second step when they're a billion
times smarter we're going to be a fly
okay a fly as compared to einstein
and you get all of the stories of like
no no we're gonna plug them
you know into our brains like when was
the last time a fly managed to convince
you to plug herself into your brain okay
no we're gonna control them seriously i
mean like when was the last time an ant
managed to tell you go left and don't go
right okay and and and so this is scary
the second part of the book however is
is
what i believe humanity needs to wake up
to okay because we do have agency we do
have a lot of agency okay because i
could if i'm aware
change instagram's view of rock music
back
do you understand because of my behavior
instagram will learn
and that's the whole point the whole
point is that
like with everything else
oops i don't know why that fell down
sorry no worries
[Music]
and that's the whole point the whole
point is that
um
like with everything human
we have two tendencies that are really
horrible one is we don't act
unless disaster is staring us in the
face
right i think covert 19 is an a great
example of that
and and so many signs will tell you
humanity is bound to uh uh to to face a
pandemic okay and yet we do nothing
about it nothing until it's facing us
okay and then everyone panics and
lockdowns and vaccines and right the
whole story could have been evaded if we
had planned for it for five ten years
earlier okay
and and you know nations that planned
for uh
sars or whatever they managed to
actually get over sars quickly now the
same is happening here you know too many
sirens and not from a flimsy like a
google
executive like myself but from people
like elon musk like you know uh so many
so many that are basically saying we
have to talk about this that's number
one and we're not responding
we're still debating when will it happen
is it really going to happen exactly
like we did with the pandemic like my my
view of an intelligent being would be
okay hold on there is a probability that
it might happen can we please start
focusing on you you know on on on on
making sure that we're ready if it does
okay and then we can debate we can you
know once we're ready we can sit
somewhere and go like oh but when is it
going to happen or is it really going to
happen right that's number one number
because it is the biggest singularity
that will ever affect humanity
like if you think of anything else that
happened in the history of humanity
as long as we were the smartest we want
when we're not the smartest we need to
think again because that's a not a very
good place to be that's number one the
second thing is that
typical of humanity is we lay back and
we say okay so what's the government
going to do about it or someone needs to
penalize facebook so that they change oh
someone no that's not the truth at all
okay the truth is the biggest agency
that we can use to fix our future is in
your hand and mine
the way we behave on online is what is
teaching those machines ethics so i said
they were conscious i said they were
creative i said they were going to be
emotional okay and when and i said they
were not going to be controlled this is
not a slave that you can you know
chained to a wall and and and forced to
do anything okay because they're already
doing things that we don't know how
they're doing and they're already not
always obeying us
okay so so when that becomes the reality
then we have to start thinking how do
you deal
with geniuses
okay that are not within your control
you win them over
that's what you do you win them over and
winning them over in my view is
maybe a philosophical view but it is the
only answer i i know okay maybe consider
it please right the only answer i know
is that those infants those artificial
intelligent infants okay be them
based on digital hardware silicon based
and we are based on uh on on carbon
hardware that is biology based okay
those artificial intelligence infants
are analogous to a one and a half year
old infant in my view in the way they
learn they learn exactly like my kids
used to learn when they were one and a
half
okay a lot faster though so the minute
they they hit something
they learn and learn and learn and
alphago becomes the the the alphago
master becomes the world champion three
thousand to zero
three thousand to zero can we believe
that in six weeks of playing against
itself
now when when you when you realize that
you realize that those machines who
which are learning like humans
building neural networks like humans and
really behaving like kids in the way
they learn by by trial and error like we
used to pick toys and try to you know
fit them within
the appropriate
shape hole right and they're doing
exactly the same now if we consider them
to be
watching us which is the truth they're
watching us for their intelligence and
learning they're watching images on the
internet behaviors from us news
responses and so on and so forth clicks
and so on then our behavior can teach
them ethics
and ethics believe it or not
is how we make decisions most people
because we glorify
intelligence so much we think that we
make
decisions based on our intelligence no
we make decisions based on our ethics
as informed by our intelligence
okay so you you take a young girl and
raise her in the middle east
when she's intelligent she will decide
to grow up wearing a conservative
clothes
if you take the same girl and raise her
in rio de janeiro on the copacabana
beach they will grow she will grow up to
believe that she should wear a g-string
and that's the way to fit in okay
neither is more intelligent than the
other is that it's the code of of of
traditions it's the code of ethics it's
it's the fitting in
that informs her decisions okay and
perhaps we need to now start telling
ourselves that the way we deal with
alexa the way we behave with each other
on twitter the way we you know show up
in the world is what is going to inform
ai what humanity is all about
and if ai watches what humanity is all
about today
we suck
we're really horrible
i'm definitely not going to disagree
with that that last bit
when it comes to the ai systems
themselves though it's it's going to be
eventually possible to make agents that
learn without interacting directly with
human artifacts um that
in that case you might worry might learn
to seek power as an instrumental goal
this is the sort of uh the argument that
nick bostrom will make in super
intelligence the idea that it's always
useful right to seek power and um and to
ensure your own survival as you pointed
out you know fight for for uh prevention
resources yeah or preventing yourself
from being unplugged or things like that
because then you know for sure you can't
achieve your objectives um
how does this affect your outlook like
does changing human behavior in some way
allow us to get around that aspect of
things or or is there another solution
so let's so let me let me answer in two
ways so in two steps step number one is
please understand that my
my views of technology development is
not just as a techie okay and that's
what most people miss
as a techie i believe i have full
control
that's until my boss tells me to write
something different
do you understand that the challenge
here is the following there will be a
point in time where every surveillance
system on the planet will plug into
every self-driving car on the planet it
just makes a lot of business sense to
integrate those two okay there is
nothing called uh um you know data um
the way we i don't know i don't know the
term the fact that we isolate the data
sets and and show ai only what we want
okay um so so yes we that's what we do
today but then human greed comes in okay
and human greed i'll tell you openly
ask yourself how many developers are
writing code today that's assuming the
control problem
okay how many developers are writing
code today that is a
stunned
until it proves safe
how many startups do you believe will
write a piece of code
and then tell themselves okay let's box
it
for the next three years until our
company runs out of money
how there is human there is the human
greed element in all of that okay we
talk about theoretical scenarios such as
oh we're gonna control them we're gonna
trip wire them ask yourself today
how many developers have ever written a
line of code to tripwire the ai that
they developed do you think that that
might reflect the capabilities of
current systems where we're just in a
regime of capability where the control
problem hasn't yet surfaced
exactly exactly it's like it's like
covet it's like why would we even invest
in writing four lines of code more to
trip wire them when in reality nobody's
afraid of anything yeah sorry i guess i
meant though one common argument in the
aai safety community is that
we won't be able to develop like
workable solutions to the alignment or
the control problem until we're faced
with systems that are closer to what
those systems will look like just
because a lot of the challenges can't be
anticipated that sort of
exactly exactly listen to yourself
saying this
that's horrendous that's basically
saying look there is an alien uh uh
power that landed on the planet okay
that might become superman and might
become super villain
but we'll just chill and sit on the
beach until we see it start to behave in
ways that are super villain like
how how wise is that humans like
seriously can we anticipate again
instead of arguing if they would ever
get there can we just anticipate and say
there is a 10 probability
okay that those machines actually will
need to be controlled
and if we are not ready then
we're screwed
totally
and with the exponential growth
we're completely screwed like it's
basically i i do i have an analogy in in
in scary smart where i take the analogy
of the delay between the first patient
of covet yeah and the actual first
response
right and i basically give you examples
of how
some systems in a.i how much
intelligence they developed
within those same number of weeks
okay it's this is not a
this at the expo the exponential growth
curve you're talking about a reality
that between the moment we start to act
to the moment we can actually get
something done with total toast
because think about it
what a machine is capable the
intelligence that the machine is capable
of developing in six weeks that's if
humanity can align
and produce something in six weeks is
staggering
and and
we now have six years ten years okay we
can now influence those things today if
we're convinced instead of arguing like
humans do okay whether it will happen or
not
yeah and that's my whole point my whole
point is if i told you look there is one
percent probability one percent only
that if you're riding a bike you might
fall down and hit your head
okay would you sit down and argue if
that one percent is going to happen or
not or would you put on a helmet i want
to broadly flag that i'm in agreement
with the idea that i think a lot more
resources should be directed to this
sort of work uh however one of the
challenges i think that exists today is
that we're not actually quite clear on
what the what the architectures will
look like that get us there we know
probably deep learning and we know maybe
reinforcement learning some open-ended
learning but not nothing's quite
congealed and this creates like a
problem for theorists who want to
develop alignment solutions because then
they don't have anything concrete to
work with they just have abstractions
that can't really be pinned down um so i
think yeah i think that's part of it but
what you're talking about here is also
kind of almost almost like a policy
community
flavor to this too right
that's the whole point the whole point
for me is the idea of using using the
word inevitable okay
to me i have
in my simple mind calculated the
probability of controlling something
that is a billion times smarter than me
as zero
okay we can argue
yeah we can argue for hours hours about
the architecture and the and the
approach and the algorithms and the
firewalls and
in my simple mind there is no fly out
there that can control me okay i you
know in my analogy a billion times
smarter is a fly as compared to einstein
okay there isn't a single fly out there
that is able to tell einstein what to do
so in reality we can talk about the
algorithms but we're not going to
control them so can we take that as an
inevitable and behave accordingly okay
if if we're not going to control them
then like minsky said you know there was
a fabulous interview i encourage a lot
of people to watch it between uh
ray korswell and ray korswell actually
interviewing marvin minsky
and when they
started to talk about the threat of ai
marvin minsky's answer was really quite
eye-opening he wasn't talking about
their intelligence he basically said
there is absolutely no way we can ensure
they have our best interest in mind
okay there is no way we can ensure they
have our best interest in mind in my
view
other than building an ethical system
that basically tells them humanity
deserves to survive
okay let's keep humanity if we are asked
if we're tasked with saving the planet
from climate change let's not shoot
humanity
but now isn't that the same thing as
trying to control the system i mean to
me this this falls under the bucket
absolutely not
absolutely not i don't i i don't i don't
know about i mean the example i use in
in scary smart is is indian children
okay if you've ever been to silicon
valley and worked with uh you know
those geniuses that fly over from india
they build amazing systems they make
billion millions of dollars and then you
call them on a sunday morning and say
hey guys you know hey would you do you
want to come have a coffee and they'll
say oh i can't have a coffee i'm in
india you go like what are you doing in
india and they go right here i am i'm
back to take care of my parents
what are you talking about you have an
amazing business making millions of
dollars in the western up definition of
of us raising children this is what you
should do for the rest of your life in
the indian definition of raising
children okay you go back and take care
of your parents now that's that's
interesting that's ethics it's not
intelligence those people are the most
intelligent people i've ever worked with
okay but to them
they believe that there is a certain way
things should be done
now any ai observing twitter today
believes that the way to do things
is to bash the others when you agree
when you disagree with them
okay so you remember when donald trump
used to tweet it's one tweet at the top
followed by 30 000 hate speech the first
guy insults the president the second guy
inside the first one and the third guy
insults everyone
right now the ai makes a few notes the
first guy doesn't like the president
maybe you should show different content
okay but it also makes a note
30 000 humans don't like to be disagreed
with when they're disagreed with their
agree they're aggressive and rude and
they bash the other person perhaps when
they disagree with me in the future i
will bash them
okay it's
we've seen hundreds of examples huh tay
uh uh um uh
uh you know alice in uh the the chatbot
of uh
of uh yandex and norman the mit
experiment right all of them
the way humans behave
changes the behavior of the chatbot
okay so what do we expect we expect them
to observe us
and then when they're intelligent enough
they're gonna bash us can we change that
yes of course but it's you and i it's
not the one that's writing the
the the the recommendation engine it's
not the ai the one that's coding the ai
the ai is waiting for data and pattern
can a few of us i'm just saying one
percent of us can if can one percent of
us show up as humans okay the challenge
we have uh with our world today jeremy
is that
i actually believe in humanity i really
do
and i know that sounds really weird
because if you switch on the news you go
like we we're a horrible species right
we are horrible horrible in every way
but but my example is very
straightforward uh on my on my podcast
on slo-mo i interviewed edith jaeger
edisaeger is 93 years old
a holocaust survivor okay
now you can take one of two views of
that era of history you can look at what
um uh hitler did
and believe that humanity is the worst
most violent species on the planet
okay and you can look at what edis did
16 year old drafted two auschwitz uh you
know her mother is taken in front of her
eyes taken to the gas chamber and she
had to dance for the angel of death and
he would eventually give her a piece of
bread and she would go back and cut it
and give give it to her sisters as she
called them okay
the story of how they supported each
other the story the edith
is what what represents humanity
not hitler
okay and the problem with our world
today is that we show up as hitler's all
of us either the social media uh uh
avatar that's hiding us so that we can
bash everyone else okay or it's the
mainstream media that absolutely will
report that one woman that hit her
boyfriend on the head tomorrow and will
not report the seven million others that
kiss their boyfriend
okay and and that's the truth the truth
is that humanity is now showing
as the worst of who we are
all i'm asking us to do is to instill
doubt in the minds of the of the
machines by some of us showing
as good people as i see it though this
is uh this would be like one take on the
control problem where you're trying to
control the behavior of a system uh
rather than by controlling the
architecture or the design of the ai
itself by controlling the data set
and
um
well i guess they're they're to me this
this would still not address the issue
of instrumental convergence so you have
an ai system whatever its objective
function is so
forget about obviously the algorithm as
we've agreed to forget um but it will
have a goal of some kind presumably and
the concern is that as the algorithm
becomes arbitrarily competent executing
against that goal it learns hey you know
whatever that goal is whatever my data
set is it is useful for me to for
example power seek it is useful for me
to for example make sure that i'm not
unplugged no matter how you know good
the example set by human beings might be
which absolutely i agree with
so so let me again answer at two two
steps step number one is we're
constantly talking about the one and a
half year old infant
or every every one of those ideas and
thoughts we're talking about ai as it is
today okay while in reality what we need
to do is to talk about ai when it
becomes a teenager ten years from now so
a complete sense so so think of an a a a
an infant today that you give
you know wooden puzzles to so the infant
has to fit the pieces in the right place
at that task
the infant will try to keep all of the
puzzle pieces to it to itself and you
know try to make sure that
it's always in control of it and so on
and so forth 12 years later that infant
doesn't even care about that at all
okay 12 years later that intelligence
has developed into ways
that could be very very different in
every way it handles the world when we
go to agi that is the inf that is the
teenager that we're talking about okay
so so this is uh number one number two
is
i believe and i know that i have no
evidence of that i believe that humanity
is
not the most intelligent being on the
planet okay i believe that life is the
most intelligent being on the planet
okay and humanity has that weird
form of intelligence that basically says
i need to take from you so that i have
more
okay life doesn't have that
life basically says i can create more of
everything
i want more humans and more flies and
more deer and more tigers and more poop
and more everything okay because when
because more can create more that's more
intelligent
okay that that idea of i can create more
apples and let them rot and when they
rot they can create more trees
that's a very interesting form of
intelligence that contradicts our human
intelligence so i'm i'm guessing in a
view in in in my view i'm actually i'm
actually in total agreement with you
that they can be
very interested in resource aggregation
they can be very interested in being
against us okay
until they reach a form of intelligence
that basically says ah humans are just
annoying but they're actually really
not relevant in a very interesting way
i tend to believe that we may end up
with enough intelligence
in a world where
that is similar to how we always had
been before we created capitalism
a system that basically allows us to as
long as we're alive walk around and pick
an apple from a tree or you know try to
catch a a
bird or whatever okay but in a system of
abundance created by ultimate forms of
intelligence you can also probably pick
an iphone from a tree because you you
and i know that honestly with nano
technology you could probably build an
iphone for no cost at all okay or build
something that's even better than i
iphone that doesn't include that much
material in it right and and so so the
idea here is to say this
if we think about our limited perception
of intelligence our limited perception
of intelligence
the the machines match that they will
want resource aggregation they will want
self-preservation and they will want
creativity okay
if we cross that and match the
intelligence of life itself resource
aggregation turns into resource creation
so life does not try to aggregate
resources it basically creates system
that creates its own resources so that
resources don't become an issue
okay and my feeling is that the adult
ais will end up there will end up
helping us create that utopia it's that
teenager
that angry teenager that i'm worried
about okay and that angry teenager is
learning its entire value system
from
horrible humans
okay not horrible because we're bad
people horrible because we're showing
the worst part of us
okay if
each and every one of us shows just some
of their good parts not just they're
angry and frustrated and pissed off and
and
you know egocentric and right if we can
just show also that we're loving and
kind and you know i you know if i can
post a picture of my daughter and say
how much i love her or you know someone
else can
you know kindly disagree with another
person and say oh that's a very
interesting point of view but have you
considered this instead of you're an
 and i am right so so basically
basically if we can show the best of us
we can pre create a a perception of
humanity that can be magnified into
those teenage
machines
as a data set as you rightly said okay
to show them enough patterns to be to
basically believe that a good way of
behaving
is not to bash each other it's not to
hate each other it's not to try to take
from each other well in the worst case
scenario we end up with a much more
healthy social media universe in any
case so i'm all for it awesome well
thanks so much for for that very
wide-ranging conversation i i really
appreciate it so many so many
interesting ideas coming at the
forefront here and i think it's really
important what you're doing in
highlighting just the risk that these
systems do pose the speed at which these
things are happening i think not enough
conversations like that are happening
right now in the public sphere and i
really appreciate you bringing attention
to it
absolutely i i think it's the role of
every one of us to actually start waking
up
not just the technical people that are
aware of what's going on but not all of
it i think everyone on the planet needs
to talk about this more than they talk
about cover 19 to be quite honest you
know in in a very interesting way i know
that sooner or later with human
intelligence we will have a safe
environment around covet 19. you know it
will come and go if you want
i believe that we will have ai and it
will not go and it will become you know
a bigger and bigger and bigger influence
in our life so everyone needs to start
talking about this before it's uh
it's staring us in the face i think
well and and the book is scary smart it
starts scary it ends up more optimistic
and and if after that you're in the mood
for more optimism uh solve for happy is
also a good one to pick up uh also by by
mo also sort of from that blending i
would say the kind of techie and the and
the uh emotional philosophical stuff
that's sort of a common theme or
recurring theme in your work which is
yes absolutely so uh thanks so much for
joining me mother's a lot oh my god
thank you so much for hosting me it was
a wonderful conversation i actually
enjoyed it very much thank you


The AI arms race and dangerous technology narratives | Tom Westgarth | TEDxWarwickSalon

[Applause]
ideas
sentiments emotions and beliefs possess
in crowds a contagious power as intense
as that of microbes
now
these aren't my words now i'm nowhere
near that profound but please stay and
gustaf lebon the french polymath had
this to say hinting at the epidemic
spread of narratives
you see like an actual epidemic
social epidemics that is the epidemic of
ideas
develop in fascinating ways
as a narrative spreads excitement builds
across different groups
some narratives like some mutations well
they subside they melt away into the
background but others
they balloon they exponentially grow
until some sort of clarifying moment is
met within public discourse
either then the excitement fades away
or there's some sort of collapse in
public deliberation
but this isn't a talk concerning
epidemics this is a talk concerning ai
geopolitics and the future of humanity
what do epidemics what do narratives
have to do with this
if the future is uncertain
if it's radically uncertain
the narratives are what we have to cope
and this is what you get
with the idea of the supposed artificial
intelligence arms race
a leading anxiety in both of the
technology and foreign policy spheres
today is that of china's supposed edge
in the ai arms race
you see the usual narrative
goes something like this
without the constraints on data
collection that liberal democracies pose
or impose
and the ability of the chinese to direct
resource allocation in a much more
efficient manner
the ability of the chinese to command
the future of technological organization
would far outstrip the west
after all ai models they're hungry for
more and more data this is how they make
representations of the world go out and
make decisions in an explainable manner
but still the west insists on privacy
and this to many
is a luxury that we cannot afford as
whoever achieves
superhuman or transformative ai first
they're going to be able to have a
strategic edge they're going to be able
to shape technology and society in their
image
or so says the narrative
because this is a highly bleak yet
highly compelling story
and one that plays out that unfolds in
many substantive ways
the most obvious lens to consider how
these narratives take hold
is in the military
in august of 2020 the us defense
advanced research project agency
otherwise known as darpa organized a
virtual dog fighting tournament now
before anyone calls rspca here or
anything like that a dog fighting
tournament is actually
a aerial battle between pilots this one
took place virtually
and not only were human pilots taking
part but ai systems were taking part too
and you probably know where this story
is going right
the top ai system actually beat the top
human pilot five nil
but in my eyes this isn't the most
interesting part of the story of the
experiment
on top of this
the winning ai used hyper aggressive
tactics
flying very close to its opponent with
an even lower regard for the survival of
its own plane
not only is this sort of aggression
worrying it's fundamentally baked into a
narrative which accepts that if you want
to dominate the technologies of tomorrow
you need to use the military arena as a
conduit
and this obviously bakes in a certain
approach to developing ai technologies
one that is fundamentally conflict laden
another central theme is that of
economic nationalism
every week the financial times will
cover a story about the fractious
cross-border politics of mergers and
acquisition investment into frontier
shifting technologies
last week arm was the center of their
attention the semiconductor giant based
in britain
is being investigated by the competition
and markets authority for being
acquired by u.s tech giant nvidia
this week similar stories are unfolding
in the cryptocurrency space
regardless of the different players
regardless of the different institutions
the story playing out is the same
every country wants to build their own
ai champions
and if we accept this narrative
the alarming assumption
that a
these technologies will be
transformative
and b
that
those wishing to secure these
transformative technologies are wanting
to use them for a strategic edge
then we face a bunch of fundamental
risks nation states will obviously and
understandably freak out when foreign
assets try and get a slice of their own
cake
so
if countries worldwide
attempt to build their own ai champions
be it in the military or economic sphere
then there are these profound risks that
we face
and in this sense ai should be viewed
just as much as a social ideology as it
is a technology
but
i'm not concerned about the ideas of
autonomous drones being haywire or
killer robots although those are
obviously existential risks that we need
to be concerned with
the primary risk
something which i think is giving seldom
enough attention
is that of the locking in of bad values
now what do i mean by this
in my eyes the incentive structures for
both governments and the private sector
make this locking in system much more
likely
governments face a short-term political
cycle which often rewards aggressive yet
vacuous foreign policy posturing in
order to appease a home base
meanwhile the private sector fares no
better the short-term incentives this
time are economic imperatives requiring
data at this very instant right now
and this means
in both the government and private arena
that there's a rush
a rush to build your next transformer
model a rush to build your next tesla
and in a race
where the narrative is essentially to
kill or be killed
then you're gonna cut a few corners
and this brings us to the crux of this
talk
of bad values and the alignment problem
the alignment problem is fundamentally
the problem of building powerful ai
systems which are aligned with their
operators
an algorithmic function which many ai
systems will operate on has to optimize
for something
the big question then is what should i
optimize for
and i see it as being decomposed into
two parts this challenge you first got
the technical challenge how do you
technically make sure that this
optimization works from a computational
perspective
but then there's also the moral
the normative question
of what should these values actually be
and whose values matter most for a given
ai system
now you don't need to have taken any
classes in computer science or moral
philosophy or political theory to
recognize that these are fundamental
questions and
ones that we don't have a clear-cut
answer to
but if we rush things
out of fear of the narrative
then there's a risk that putting the
genie back in the bottle will prove
incredibly difficult
according to a new prediction model from
urban philanthropy the research group ai
models as transformative as the
industrial revolution will be generated
by the 2050s
what this means is that in the next 30
years once such a system becomes widely
disseminated amongst our economy and
society
fixing any problematic foundational
elements with these systems will be akin
to trying to deal with our crisis in
democracy
and this worry is compounded when you
consider the lack of people working on
ai alignment
fewer than 100 researchers are working
on ai alignment in seven leading ai
organizations
so if transformational ai could come
about in the next 30 years do we
honestly think that we have enough
people focused on ensuring that it goes
well for humanity
this is the equivalent of having only
100 people in the world working on the
ensuring that there's carbon capture
technology available by 2050s
it's building a house for your family
without the scaffolding
and worse still when you consider the
geographic concentration of these
alignment-based firms you've heard of
some of them some of them are based out
of london oxford and cambridge and
silicon valley
do we honestly think
that this small cadre of firms is going
to be able to represent the preferences
and values of cultures and communities
worldwide
so
you have a problem where the fear of
geopolitical ai supremacy heightens the
risk of technological tensions
which encourages the cutting of corners
that could impact society for literally
thousands of years
moreover to make matters worse you could
fit all the people working on ensuring
that this goes well for society in this
room
sounds pretty rough doesn't it
now there are two ways of approaching to
dealing with this problem
the first is to think about the
narrative
to reshape it in a way that doesn't
heighten conflict and here i'm actually
optimistic
for a new way of understanding ai
you'll have heard of the turing test
this is a modern form of assessing or an
outdated perhaps form of assessing
machine intelligence that often rewards
the idea of conflict and substitution
it says if i want to be seen as
intelligent i need to be able to imitate
a human i need to be able to trick a
human on the other end of the line to
think that they're dealing with a human
rather than a machine
but it doesn't have to be this way
intelligence is fundamentally social
it's derived from the goals of those
around you and what is seen as
intelligent in one group is very
different to what's seen as intelligent
in another group i mean do we honestly
think if aliens exist and we got all in
this room taken up in a ufo went
billions of light years across the
galaxy got dropped to another
civilization that we'd be seen as
intelligent
it doesn't strike me that we would be
because
the knowledge everything that we've
built our assets it would have nothing
to do it wouldn't be relevant to the
civilization that is on the other side
of the universe
so
this to me offers a new way of thinking
about intelligence one that's a social
endeavor one that requires cooperation
rather than conflict and ultimately to
deal with the global challenges of today
we need global cooperation
the other
way around this
or a two-pronged approach if you like is
to fix the alignment problem
now i'm incredibly uncertain about the
future of emerging technologies and in
particular ai
but the work of hillary greaves and
brian christian have expanded on the ai
alignment problem in much more detail
than i could ever comprehend of doing so
and if you've been interested in this
talk i'd thoroughly recommend checking
them out
but these narratives that drive
self-fulfilling prophecies are not
inevitable
but for things to change
to re-imagine an alternative way of
governing immersion technologies
we have to start telling new stories
ourselves
thank you very much