Nicolas Miailhe - AI risk is a global problem

hey everyone jeremy here welcome back to
the tours data science podcast and today
we are talking about
the global side of the ai safety picture
with
one of the co-founders of the future
society nicola miyai now nikola himself
is a globally recognized expert on ai
and ai policy he's advised a whole bunch
of really important international bodies
including the global partnership on ai
which we'll talk about a fair bit in
this conversation nikon's approach and
the approach of the future society more
generally are actually quite interesting
and unique and they're focused on
finding consensus
not only between groups of people who
worry about short-term ai risks versus
long-term ai risks
but also between countries around the
world that see ai
quite differently as you'll see during
our conversation i think that's just a
really important area to be working on
and it leads to a lot of interesting
downstream discussion so
i hope you enjoyed this conversation and
i'll get out of the way and let it start
all right well nico thanks so much for
joining me for the podcast
thanks a lot jeremy and good morning to
all the joggers that might be listening
to us as they are taking their run in
the morning or in the evening around the
world and congrats for all the posca
series and all the great we're doing
and thanks for having me yeah thank you
actually to nico's comment we were just
talking about how the vast majority of
people
who listen to this podcast really do
listen to it rather than watch it so
it's kind of nice because if i'm having
a bad hair day you know i can get away
with a lot here but nico you are doing
some really interesting work
at the future society you're one of the
the co-founders of the future society
one of the things i find really
interesting especially for highly
capable people who have
fairly technical and sophisticated
backgrounds
you know when you choose to dedicate
your time to an initiative like the
future society something that's
very forward-looking it's not like
you're raking in piles of cash or
anything like that you're doing this for
a reason
i'd love to understand what what brought
you to this sort of forward-looking set
of concerns
why the specific concerns about the
future that you have and what are those
concerns
well you're right to point to this and
fremita's public service
i have very much impression with my team
and have the honor and privilege of of
being the president as well as the
founder of the society and i co-founded
it with other people
and we see this as a form of public
service
we see ourselves as an actor of the
emerging
global civil society sector now
why we do that and and why we have
decided to call ourselves
the future society which is a big lofty
name that serves also as a mission
that serves also as a claim because it's
about
nothing less but this nothing less
but trying to invent the society of the
future that we want
nothing less but trying to reconcile in
practice
align in practice the ecological
transition
and the digital transition spur headed
with ai
and align them in a way that does not
sacrifice
but actually upholds the best for more
values in terms of freedom of values in
terms of democracy and self-government
and of values in terms of solidarity and
equality
doing that in practice is not as easy
as it sounds because well
ai is a can be framed as a general
purpose technology and
let's state that it has uh profound
uh consequences that implication in
terms of national security
and international security or lack
thereof and
a redistribution it is not
following out it's not falling out from
the sky it is really emerging from the
ground and the ground is that of the
digital revolution
and as such it is reinventing the way in
which we work
leave socialize and in many ways it is
disrupting
some of our most fundamental social
contracts
pushing forward notions of consumers and
our identities as consumers
and questioning our identities as
citizens centralizing power
at the top and at the center
so why do we do that and why we focus on
ai governance but there is also a sense
of
pragmatism versus timing there are
big policy decisions being taken right
now
so for us to act as an independent
evidence-based
small very much underfunded but still
active
global think and do tank we believe is
needed now you know in other words there
is a policy window
around ai but around other things
underpinned by
the digital revolution uh neural
technologies
um cryptocurrencies so our very small
global ai think and do tank we think is
needed
and maybe a few differentiators or what
would how we try and do things a bit
different
well the first one is that uh what you
know opinion the how
and the what of whatever and however we
do and decide things are
inextricably connected you cannot think
of a policy question and not thinking as
to how you are going to
to impact it the how and the what and
that means that we also have started to
develop our own machine learning based
tool to try and
impact policy decisions one example is
our project called ai against modern
slavery
where we work to develop a machine
learning based tool using
existing models to help expedite the
review and compliance of modern slavery
statement that corporations have to file
every year
and the goal is okay to enhance the
accountability of governments and
businesses but also to try and
usher a new generation of ai ready
ai capable ai powered public policies
and legislations
um but i i'd say that or mandate is
definitely beyond ai
and for more observations and that's why
we're called the free society because we
also want to be looking at
gene editing neural technologies there
is right now momentum
on the eye and it's important to impact
this uh
this moment of to to have an impact on
this momentum
well i find it really interesting that
you you highlight ai against modern
slavery on the one hand and then there's
also a bunch of other ai policies
that address the use of ai by companies
and corporations and countries
it sort of highlights what you were
saying ai being a dual use technology on
the one hand
you know it's great it can be applied to
a whole bunch of social good it can
unlock just generally
great use cases but in the same way that
gpt3 for example might be able to do
great text summarization or language
translation
it might also help people carry out
better phishing attacks and things like
that so
there's always like kind of two sides to
this coin it's very important another
key tenets of our approach is that the
opportunities
and the challenges of the ai are
inextricably
and irremediably connected and we stand
and our mission is to help advance the
responsible adoption of ai and other
emerging tech for the benefit of
humanity because we think
and that's that's an important in a way
principle of action we think that
we do need ai and we are going to need
ai
to help solve the global challenges that
we confront including climate change for
example
and help usher a new era of human
flourishing
and human evolution but we are so we are
cautiously techno-optimistic
but worried about the level of
collective wisdom
that or specie and its manifestation
through governance institution exhibit
right now to be able to capture the
benefits
while uh minimizing the downsides and
dealing with the risks
that's a question of balance or rather
imbalances
actually that's that's its own
fascinating topic and math weirdly it
maps on to some some more technical
conversations
that we've had on the podcast with
people who focus on like technical ai
alignment issues
where there's this big open debate right
now about whether
for example we want to build ai
capabilities and then kind of like
correct safety issues as they arise or
do we need safety to reach really far
ahead
and then like try to anticipate where
the technology is going
increasingly it seems like people are
saying look we just can't do anything
with this safety wise until we build the
capabilities
these things have to come in ten and
they have to be in lock steps sort of
like what you're suggesting here
the technology has to has to assist with
the safety and the fairness
well i partly disagree in the sense that
i i believe in the need for what we call
disentangle research there is value
uh in a lot of value in what people like
miri are doing for example
hey everyone it's jeremy i just wanted
to jump in here and provide a little bit
more context on the organization niko
just mentioned
since i'm guessing it'll be new to at
least some of you miri is the machine
intelligence research
institute and they're really the first
research team ever dedicated to
addressing potential catastrophic risks
from ai
miri was founded back in 2000 and their
founder eliezer yudkowski
was one of the early pioneers of ai
related existential risk research
he also happens to write some of the
best harry potter fan fiction you'll
ever find i'm dead serious about this
google it and you'll thank me later
there is value uh in a lot of value in
what people like miri are doing for
example
trying to develop ai safety protocols
and pathways
way ahead of agi i think there is a lot
of value in it and i want to recognize
what they do because it's never easy to
know
and to do research and something that
does not yet exist you know it drives
everyone crazy
it's it's it embeds a schizophrenic
dynamics so they have to be recognized
because there is a
it's a lonely work but it has to be done
why
because part of the ai safety pathways
will emerge from
dealing well with ai ethics the world of
tomorrow will emerge from the world of
today
but not exclusively we have been
strategically surprised in the past
the case of the manaten project is a
case in point so there is
value in baking at the core of what all
community
the ai development and adoption
community is
ideas and pathways try and project a do
the effort
of projecting to the unknown i think
it's very valuable
and part of what we need to do is
reconcile these approaches
and we are part of the committee that
tries to do that
so it's it's that's why i say only
partly disagree with what you're saying
i want to recognize the need for
disentangled research besides
other words where you try and align
the ultra short if not immediate short
mid
long and ultra long term that's uh i
completely agree i mean i think
so i suppose what i might have meant to
put it maybe more correctly
there is a portfolio approach on ai
safety there's a portfolio approach on
ai policy where as you say a variety of
different time windows need to be
factored in
we have to make sure that we're
incrementing correctly but also that
we're projecting correctly
organizations like muri are really
important for the latter and then this
kind of like
gradual escalation of safeties and
capabilities research is another
component of that almost risk portfolio
that we need to be developing
by the way one of the the most
remarkable things i think
at least to me that you just said there
was it is so rare for me to hear an ai
policy person
or somebody who who focuses on ai policy
talk about miri talk about
um an organization that most people like
it usually have to be in this technical
niche to know what miri is
i'd love to understand like was miri
part of your
um and elias or yukowski's work all that
stuff was that part of your journey to
discovering the ai safety space
or is that something that you've
discovered along the way like i'm really
curious about how you came to this uh
this area of course it was part of my
journey
i was born and raised in paris in france
but i spent 10 years in india
so i've been exposed and i worked in
industry so i've been exposed
to the huge potential of algorithmic
correlation to help advance
questions of development and access to
basic services at scale india is a
billion people
when india decides to launch a dar
project to establish a biometric based
meaning algorithmic correlation based
a central identity database to fuel its
social programs you know
this will have a global impact on
development
and on ai and so at that point i
realized that there was a growing gap
between
the urgency of development and the need
to plan the
long run in terms of the potential
second third
and fourth order consequences of the ai
and algorithmic correlation revolution
and and because of that
willingness to reconcile in a way the uh
the universe
the ultra long term the the cosmos with
the immediate
uh i have from the get-go decided that
it would be one of our mission to try
and reconcile this ultra long-term
challenges with the ultra short term
because if we don't do that we'll have
neither of the two
because we need to also respect the
urgency of development
and the fact that a lot of organization
a lot of people
still today have a need for access to
basic services of health care
transportation
banking and and so much others and
this tourist for uh algorithmic
correlation needs to be quenched but
quenched in a way that is
both ethical to reconcile this end of
the day with the end of the month
fairness privacy and also paving the way
for a reconciliation of the
if you are law end of the world this
question of catastrophic
if not existential so this question of
reconciling
this ultra long to ultra short term in a
way that reconciles the global north
uh yotkowski and all the great people
working at miri operate out of
california
they don't operate out of india there
are a lot of great indian researchers
and super
capable and into genius indian
researchers in the silicon valley
but the fact of the matter is that a lot
of this a safety perspective still
emerges
from california from oxbridge
from paris not so much from the global
south
and so this effort of reconciliation of
this end of the day
and of the month and of the world in my
view is especially important
to base concerns of a safety in
the quest for ai development as much as
in the quest for ai ethics and vice
versa
it's not easy because these communities
come at the problem with very different
perspectives
and very different epistemological
perspectives
and world views so it's hard and at
times you know
one risk being considered as a lightning
rod because you try and bring people
together
but one of our mission is to try and do
just that around
scientific processes the idea that we
researched and put
forward for an ipcc for ai which then
became the global partnership on the eye
it was not our idea but i think i can
say that we have
a bit of a significant influence in its
pathway from the research to the
deployment
initially aimed at doing just that we
need to come together
as a community around a robust knowledge
creation
process it is called science science is
a process to create knowledge
rub its knowledge and replicate that
knowledge and the effect of the
knowledge we have on the world
we have to come together to create a
solid base of matter of fact
built from science that informs
our collective understanding of what do
we mean by ai
there is still no consensus on that as a
complex socio-technical system
we defer a lot as to what we mean by ai
number one number two what are its
dynamics
on the short mid and long term the
question of
the agi remains highly contentious
there's deep steel in oil community
disagreement over the possibility and
the time
frame of the possible emergent of a
human capable
ai and the consequences so so definition
dynamics and consequences when you and i
jeremy
go out today virtually and we look up in
the sky
will we agree that the sky is blue today
we will
but we won't agree today as to or we may
not agree today as to what ai is and its
dynamics and its consequences so
this the idea of an ipcc was there to
create the foundations to lay down the
foundations all the time
to come together and create a solid base
of matter of fact
globally to inform policy debate we
don't need to agree on all the social
contracts
but can we agree on the fact that the
the sky is blue
and that the sun appears to either as
yellow today
so i think to to listeners who maybe are
following
agi a a little bit less closely if
you're if this is
closer to an introductory conversation
for you i think one thing that
might run by you or get past you as it
did for me back in the day
is just like how important this idea is
of the disagreement between people who
worry about short-term ai risk
and people worry about long-term ai risk
it's very common to hear people say
uh we need to be focused exclusively on
one and not on the other
there's like an active there are papers
that have been published in the ai
safety community talking about
how this seems to be like an impasse
trying to find different ways to kind of
reconcile these things
it's really rare to have somebody like
an organization focused on reconciling
those two perspectives
i think that's actually one of the
coolest things about what you're doing
here not only i mean you're doing it
along the time axis you're trying to say
okay long term is in short term it's
let's bring it together on this
but your answer is i was going to say
physics status time and space are the
same thing so why
don't we start implementing that
reconciliation in policy it's not time
it's space time so it's north south east
west
short immediate mid and long and ultra
long
yes it's hard to do yes we're going to
break our brain on this
yes i agree but it's either we do that
or we expose ourselves to potentially
catastrophic risks on the long term
and uh that's very important and another
of so why governance
why governance why focus a lot on policy
and the state
as a manifestation of a human identity
and a process to influence
and govern human lives yeah i think a
lot of people wonder about that because
companies right now dominate ai research
and
well do they do they when you look at
the figures
of course they do now but come on
don't you believe that a lot of what
was made possible for google
amazon open ai i would say
is the result of massive massive
government investment i would
argue even that it's only taxes
which only governments can mobilize in
other words the fact of having
your grandkids who are not even born
start
paying today for long-term patient
high-risk capital that enables this kind
of a
very deep techno-scientific revolution
not necessarily the innovation
the private sector is really good at
taking techno-scientific
revolution into innovations yes these
lines are blurred and indeed i don't
disagree with the fact that
now these multinationals these
oligopolistic multinationals are
investing at par with many governments i
don't disagree with that
but i want to recognize the fact that
defense policy
innovation policy care policy through
their institutions such as darpa nsf nih
have are and will continue to play a
huge role and therefore
citizens should and having a way a say
in the foundations of these
techno-scientific revolutions
and let's take an example of uh of today
open ai
what has happened to open air over the
past two years well
massive capital infusion from
private corporations whom we know have
also
big and receive big you know investment
from governments
it it it has been a cause of worry for
some in our community that
you know microsoft has invested one
billion in open ai and at the same time
microsoft has received a significant
investment from the
department of defense through the jedi
contract and i want to oversimplify it's
more complex than that but
i just want to say that even in that
case to grow its model
in in an output a model with more impact
you know the investment of government is
and remains
very important and and where did all
these researchers that went to open air
come from initially
and i would venture to say that a lot of
policy support for education
and a lot of policy support for best
universities was at play
that's why we tend to oppose and
oversimplify the role of
private versus the world of the public
it is my individual stance not
necessarily the stance of the future
society that
governments through research and support
to research funded by taxes and look at
what happened yesterday in the u.s the
buying and recovery plan
over almost two trillion us dollars tell
me which company
is capable of laying that on the table
not all of that will go to research we
know
but this ability to mobilize long-term
patient high-risk capital you know
over time in large numbers is still in
many ways the realm
of the government justified by many good
and not so good ideas including national
security we support
national security but when national
security is used as a tool
for expansion it can also be a problem
you know
president has nowhere upon leaving
office
you know in his farewell to the nation
as a general himself
warned the nation over what he framed as
the military-industrial complex
so we should not over-emphasize and go
into conspiracy theory i don't believe
in that
but but even if the president in his
farewell address mentioned that i didn't
we need to pay attention to all the
great benefits we went to the moon
with that and we will go to mars with
government support where would spacex be
without nasa support
let's be honest yeah it it they're an
interesting series of questions for sure
and i think disentangling the public and
private thing is is highly non-trivial
i think it's also the case viewed
through another lens that
there are areas of ai safety research
that private companies are not going to
be incentivized to
work on like i'm thinking here
especially of like the alignment problem
right
right now the work that miri is doing i
think is a great example no one at miri
is making the argument that by doing the
research they're doing
they're going to accelerate economic
development at any point in the next
like five years or something like that
they're not building new tools not
helping new companies launch
what they're really fundamentally doing
is trying to avert a catastrophe this
has the profile the risk profile of
something like buying insurance
and the people who are going to be
funding this are not
companies that are looking to optimize
shareholder value
they're going to be you know people who
can transcend this sort of like
a tragedy of the commons i mean
ultimately that's what this is there's a
risk
that everybody bears in part in maybe
it's a tragedy of the commons
over space-time over community that is
not fully aware of itself which is
called the
humanity so to speak but if we claim
the global if we claim the species
then we have to be careful as to who
claims it that's why we at tfs
the future society puts enormous
emphasis on inclusion
and civic engagement because the moment
we claim that we are doing something for
humanity
we have to be careful as to who are we
to make that claim and how
in a way representative we are and
representative
repositiveness is not only a question of
legitimacy it's a question of blind
spots
i grew up in paris i did not grow up in
kenya
i don't have the same understanding of
this reality over there
but but i say kenya because we know that
africa is going through a demographic
boom
and a lot of the the insurance policies
were developing today
are for them in a way that's why this
this question of reconciling the north
and the south east and west is very
important
but we have to do that in my view in a
way that does not paralyze
action that's why i very much support
the work of miri
i very much support the work of a safety
because it has to be done
i'm just saying that and they're aware
of it that it has to be done in a more
representative way
um even and i would say maybe they won't
agree with that but
even if we are looking at disentangled
research which is ultra long term
because the way in which we look at
ultra long term is not the same
if we come from a philosophical
tradition of uh let's
i don't know ashkenazi european secular
american or um buddhist
or african with or without any given
epistemological philosophical and even
spiritual tradition these we need to be
humble
these traditions have an impact on us
whether we like
it or not so we need to consider them
you know and try and build diversity and
inclusion
um so so i i'd love to to uh push
not push on that but like prod at that
because uh that that to me seems like
one of the biggest challenges we face
today
is you have this massively impactful
technology and
i mean it seems clear that with the way
gpt 3 has shown us you know ai can just
grow by leaps and bounds every every
month every couple of months
like we're going to be surprised as you
say by new technological advances power
starts to get centralized in the hands
of the people have you know so on and so
forth
but if we can't agree like it seems like
everybody will have a different lens
through which they're seeing ai
some countries can't afford to worry
about safety because like they're just
trying to lift themselves up right now
like
developing economies cannot put
resources towards safety to them ai
um and maybe you'll disagree actually
okay it's a reconciliation problem it's
so important to build capacity
in the global south for the responsible
adoption of ai and governance of ai
to have them part of the conversation on
global aitik's local aeratics to
have them part of the conversation on
their adoption so that they can share
the benefits they can have control
it will never be perfect control but we
are at a stage today
where you know it can be labeled or
framed as cyber
colonialism and when i say cyber
colonialism i'm not talking
u.s and kenya i'm talking france and the
u.s
france today is a digital colony of the
u.s
so if it's that level of power
imbalances between
two g7 countries what will it be
between europe and africa between the
u.s and africa and that's therefore
imperative that we help and
not help empower them in building a
capacity for responsible adoption of va
to avert that to reconcile that because
otherwise
we accelerate the race dynamics because
if you tell someone
either you wake up or you are going to
be potentially
enslaved to the vision and the identity
of someone
you know someone with pride and
willingness to defend her or his
identity will react
and that reaction will translate into a
potential race
some would say that the capacity in
africa are so small that there is not
much risk and we should focus only on
the
u.s and china and i would say well no we
need to triangulate
where is the big market of tomorrow
where are those needs that need to be
satisfied
so where is in a way the potential
battleground
between europe china and uh and the us
well potentially africa to oversimplify
that's why
harnessing ai for the sort of our
mission at tfs we work a lot on that we
are right now
finishing a project with the government
of rwanda
to help rwanda develop its national ai
strategy and implementation plan
to try and and create a pathway to
accelerate african empirement through
the responsible adoption of yeah it's
very important
it's very important for ai safety i'm
arguing
that's that's really interesting because
i actually i think i see things a little
bit differently on that score i'd be
really curious about your thoughts on
this so
um in the 90s there was this policy with
respect to china in particular
where you know economic liberalism will
lead to political liberalism
we thought that by you know sending
setting up trade agreements that were
favorable
generous um and kind to china that
ultimately we would have a global
partner
and unfortunately the relationship there
has become adversarial
including nai where now a lot of people
in the ai safety and ai policy community
are
explicitly concerned about the u.s china
rivalry as one of the main drivers of
international instability and ai
dis safety so i guess the idea of
multi-polarity that you might have
you know not just a u.s china uh or
competition but
potentially other players who are
equally as capable when it comes to ai
um might be argued to have a
destabilizing effect at least my
intuition takes me in that direction
so while i'm i'm certainly very um
i i very much like the argument that
look everybody kind of needs a seat at
the table
it isn't right to just have a set of
ideas that are developed in shanghai or
silicon valley dominate the world
we need yeah hold on so i come from
before working in industry i worked for
the ministry of defense
okay and i was entrusted with a small
element of the french military nuclear
policy
which was to try and envisage uh nuclear
deterrence post 9 11. that was a work of
analyst and study
and so and i very much see so this
this notion of orphans defense strategic
balance
stability comes a lot with very good
reasons from the nuclear paradigm
and we should not forget the nuclear
strategic paradigm because it's
an existential policy paradigm but i
want to draw at least
a few different differences between this
padding and what we're facing
the nuclear paradigm was a fundamental
party which was
which could be represented in a
simplistic way through the following
market
there are risks associated but there are
rewards in the form of access to cheap
energy
you have to surrender your ability to
develop military nuclear power
and we will give you access to civilian
nuclear power
which decreases the cost of access to
mass energy and
energy has a big impact on our lives but
the the way to in which it was
intermediated
made that bargain quite binary and quite
distant to most of people
except in the case and we're celebrating
the anniversary of the fukushima
nuclear catastrophe in except in certain
cases which i don't want to turn a blind
eye to let's let's
let's face the reality too but the
bargain
that ai as a general purpose technology
offers on us in my view goes way beyond
that as i said in introduction
ai reinvents the way in which we leave
work
and socialize in a way that potentially
rewire our brains
even so that's why this this this
question of
well we'll decide for you u.s and china
and you can sit there don't worry well
the positive externality of stability
will be peace for you
but people see that their very
identities are affected by this bargain
and they want to have a sit at the table
because about
living working in socializing in a way
which is very direct and not distance
nuclear add that influence of course
energy life death
but now it's much richer ai is all
pervasive
really all pervasive there's something
more fundamentally
i mean it's fundamentally threatening in
a way to
somebody having ai dominance over you
rather than having
nuclear dominance i mean nuclear
dominance is kind of like you know the
country can
can threaten you they can tell you to do
whatever i guess there's yeah something
more insidious about ai
where you use twitter yeah it's it's
much
it's much deeper it's not i will destroy
or let you live
is i will really reinvent these social
contracts it's it's very deep
influence your thoughts influence who
you think you are in terms of
redistribution mechanisms of your
country influence the consuming patterns
difference production patterns
it's it's it's deep it's not binary
and and i wouldn't say that nuclear
shallow it's not shallow mind you but of
course
but it's it's in my view not as deep in
terms of the influence on people's life
and again at a time where we have
recognized the value of the individual
we want individual consumers in the
global south and we recognize their
identities not only as communities but
people with needs with ideas with
volition and and so the individual we
leave in a way
at the time where the global
civilization had reached a point where
the individual is the center of the
attention
so it's a and therefore this depth is is
is to be
dealt with in my view so the that's
interesting is almost like an argument
for
ai proliferation which is which is very
like the reverse of what people usually
talk about in the context of nuclear
proliferation um what do you make of
arguments that and i think nick bostrom
might have an argument along these lines
where um over time you know as
technologies improve
ai in particular improves becomes
cheaper becomes more flexible
the destructive footprint of malicious
actors starts to grow and arguably grows
at the rate of this technologies which
is exponential
um yeah how does the proliferation
gambit play into this in your mind
it's you're right it's a problem the
offense will always be
and always have the advantage and
therefore the problem with the
software intermediate society and then
you can enrich software systems with
more and more and more complex
algorithmic
processing and correlations is that yeah
there is a
strategic advantage given to uh
malicious actors
and indeed we need to deal with that and
that's one of the reasons why we need to
be careful in how we adopt
we we systematically underestimate the
cyber risk
systematically and that's one good thing
that the
the military community is there to
remind us that guys as we as we delegate
more and more
critical decisions to its systems and
powered by ai
and as design systems are more and more
black boxes we want to be a bit careful
because a black box is a strategic cyber
risk
uh so yes i think he's got an argument
and we need to be careful and that's why
this proliferation is potentially risky
and that's why
but again if people want food on their
table and ai is going to deliver amazing
productivity gains in
crop yield production the risk is high
that the adoption will go faster than
the cyber risk is
and that we will expose our collective
to a strategic risk because the internet
is global indeed so yes i don't deny
this these questions i don't deny these
problems and that's why
we believe that ai governance is a
systematically complex
system to govern that requires
unprecedented effort in collective
intelligence
to deal with those things and perform in
practice this process of reconciliation
not in the way it is perfect it will
remain dirty it will remain dirty people
who look for perfect governor no it will
remain dirty
approximate uh partial but we need to
move along that line humanity is like
that anyways
yeah i actually i i'm a big fan of that
i think that
framings of ai safety that revolve
around like perfect safety
or framing of ai governance that
revolves around perfect governance like
these are intrinsically doomed to fail
and we're gonna end up feeling
you know and and looking stupid if we
end up saying if we try to claim that
like this is achievable
such an underserved area i mean it's one
of these it's
i think by far the most important policy
governance challenge we're going to have
to address in the coming years
um can you speak a little bit you've
alluded to the partnership the global
partnership on ai
i'd love to hear from you so what what
is the global partnership on ai
what role do you play in it and how do
you see it fitting into the the
framework that you provided
okay so the partnership on the ai uh was
launched last year
within the g7 as a result of a
leadership effort from the president of
france macron and the prime minister of
canada justin trudeau
to bring together a series of
like-minded countries and when we say
like-minded countries around these
liberal
and democratic values particularly human
rights to try and invent
better understanding and practical
solutions to adopt ai responsibly for
the benefit of humanity so we feel
perfectly aligned with that and that's
why we are very much part of the
partnership
i myself i appointed to one committee
which is a data governance working group
i also chair the newly launched
committee on climate change and
biodiversity
and the first society as a nonprofit has
been retained
as a knowledge partner and help the
responsibility working group
and the pandemic response subgroup which
is a very important
focus of the the partnership devised its
strategy for
uh 2021 and identify areas of uh
action and assessing potential
initiative and potential priorities
and for us it's a very important tool
it's a tool that has evolved from the
way in which the the
it was envisaged from 2016 to 2018
we at the pitch society did some
research borrowing from
a theory of regime complex in arms
control
in climate change in international
finance and trade and in
internet governance and we with others
we're not the only one but we came up
with this idea
that we would need an ipcc for ai an
inter-governmental panel for ai adapted
for the
inter-government panel on climate change
why and why now and why this as the
first
building block of this governance regime
for the reason why
which i mentioned before we are in dire
need
of bringing around science the global
community of
experts to help build a solid base of
matter of fact
as to what is ai what are its dynamic
over space time
and what are its consequences over space
time and and
this ipcc for ai was envisaged as a way
to do that a bit like what was done to
understand climate change its dynamics
and its consequences
and along the way um the ideas evolved
in the
international uh negotiation first
bilateral between france and kerala and
then
multilateral within g7 to realign the
mandate around more of an action tank
um i still believe that there is a need
for this
evidence base collating evidence and i'm
trying also to get
and see how the partnership could not
abandon this very important mission
but i want to state that this important
mission of base of matter of effect
needs to engage everyone including china
and gpa being an initiative that
brings together like-minded countries
around human rights for example
it cannot include china at this stage
and so
we need other forums probably in the u.n
to do this work of an ipcc foreign
because if we don't do that work we will
not
be in a position to bring experts on a
common diagnostic
and it's very important for ai safety
but it's also
very important to align on the balance
between ai
ethics and ai and development if you
want
so that's uh i think that's just a great
summary and it it dovetails very nicely
into the next question i was going to
ask which has to do with china
you know it's often we do often frame
the dynamic between the us and china
as competitive people use the word the
term arms race a lot
um it's unclear like it's a very complex
arms race
obviously but as you say like in the
context of an ipcc for ai in the context
of the global partnership
on ai that puts china at the margins
there understandably so
um but they are a massive ai power and
they're poised to become
more so what um so what do you think
concretely like what could we do
to bring them in the fold at the same
time recognizing that
historically with things like bioweapons
we know china has basically disregarded
a lot of
international law on on that um the idea
that they'd be reliable partners and
this is certainly a big question mark to
say the least
so like it seems like there's an
enforcement piece
and understandably china won't trust us
either i mean it's a two-way street
so i i don't know if these
considerations are things that um
that you've thought about yet it's
obviously early days we not only
think about that we work on that just to
give you an example of one thing that we
did
we organized in silicon valley in june
of 2018
the first and so far only u.s china ai
tech summit
bringing together business and tech
leaders not government
we envisaged this as a 2.0 truck
diplomacy let's bring together business
and tech leaders
around two things can we try and agree
around the latest trends in
business models and technology
development but in a very applied way
and looking at business models can we
recognize that china
is a very innovative country it used to
be
that silicon valley you know with a
monopoly in disruptive business models
these days were over already then
so how can we because these business
models are heavily underestimated and
under value in how they
trigger impact on the world we spend too
much time on the technology
not enough on business models it's part
of the multi-stakeholder thing we need
we need industry around the table
because we need to understand these
business models that bring
this machine learning based innovations
into
impact on our lives so that's
engaging and finding the right form of
engagement between china
and the us between china and europe
between europe and u.s
is at the center of what we do and and
yes i have a few ideas
the first one is that you're right and
you alluded to that we should not show
any form of
candor there is no need necessary to
engage china on everything and china
will not want that
because there is a competition we need
to acknowledge the competition
but at the same time we need to find the
not lowest the highest single common
denominator of engagement
and we have hopes and we should have
hope for that and the hopes are derived
from what happened in climate change
what made the paris accord possible in
2015
was the fact that before this accord
there was
an agreement between china in the us a
bilateral agreement between china and
the us
on climate change i don't think that
that will happen on the ai
between for a while but what i'm trying
to say is that why that agreement
happened
in my view and it's a multi-factor uh
decision of course but
one important fact is that china and the
global south beyond china
had been part of the ipcc for years so
this common base of matter of fact that
could trigger the fact
guys there's a problem we have a common
urgency to deal with
we need to react together this knowledge
were created together
and therefore the diagnostic was shirt
gave the possibility
which was exploited at an opportune
moment for this deal to happen
and so that's why i take the example of
the sky we need at least to share a
minimum set of diagnostic over
ai definition dynamics and consequences
with china to identify
according to diplomatic and
international negotiation processes
areas when we can collaborate around
international security
around ai safety i think that if we do
our job well over the next
15 years we can identify ways in which
to engage
and i think that diplomacy works not too
bad and then
governments in the us in europe and
china if they are well informed by
science can identify
areas of cooperation and what we call
confidence building
measures will it be perfect no is it
guaranteed
no but it's either we start on that
pathway or we expose ourselves as humans
to indeed a potential risk because the
race dynamics is on
and if we celebrate and
raise towards hgi without thinking
carefully
about second and third order
consequences it's it is my opinion that
we expose ourselves strategic risks
indeed yeah i think the importance of of
embracing imperfect ai governance as a
reality
is is so important and plays into this
on so many levels um
and and maybe just one more point on
that is and one fear
in in diplomatic circles which i think
is legitimate and we have to be careful
about is the fear of what we call
normalization
some say that okay in particular in the
u.s that they don't want to engage
china on too many topics because it
would normalize many others and they
want to draw a line
and hold the line i respect that my
point is
can we create a common base of matter of
fact at least because we share that
we share the air we share the sky we
share space even
we talk about that and yeah to be able
to see what we agree and what we
disagree on
and find a way to both hold the line
where our polities want to hold the line
and engage across that line where we
think it is needed the soviet union
and the united states of america engaged
throughout the cold war
throughout the cold war so we need to
find these ways draw the boundary
but engineer the pathways to traffic
across the boundary
yeah i i think this is something that's
under to some degree under
emphasized in in general conversations
about this for understandable reasons
i think like the average person has um
a completely understandable and largely
legitimate knee-jerk reaction when they
think about
engaging with china with you know weaker
concentration camps the way falun gong
you know organ harvesting political like
all this stuff it's all legitimate
there is absolutely an analogy you can
draw between that and neville
chamberlain appeasing nazi germany
like there are death camps and that is a
fact
but but at the same time
when you start talking about the
categories of risk that we're discussing
here
as you say i mean like whether it's
climate change whether it's ai
ai existential risk this is really like
it's an all or nothing play
um and the level of pragmatism that's
required here the kinds of moral
calculations are
so complex i mean i i don't envy anyone
who has to
take any stand on this it is so
complicated and difficult to
to address one more point on that which
which is of worry to me
and it it speaks to the question of
alignment it's a
it's a strange way to look at the point
of alignments it's a very different
i will take a small tangential for a few
seconds but i think it's important
i worry a lot in the case of china over
the paradox if not the contradiction
between the
ideological software marxism socialism
and its translation on the ground
capitalism
so the reconciliation between an
ideology that remains marxian if not
marxist
and its manifestation in reality which
is at times very
wild capitalism needs to be reconciled
and in many ways this distance that
stretch
has been in a way reconciled through
technological means and techno
surveillance means
an orwellian system where technology is
used to reconcile
or deal with this contradiction in a way
that controls the population
and it is clear that the externalities
of that
paradox resolution are extremely
negative in terms of
risks in terms of instabilities and in a
way
we would need it to be better aligned
and and that's why it's it's important
to also
find ways to have china moving towards
the pathway of resolution meaning
moving towards much less control because
that's control outputs shorten problems
export surveillance technologies but
creates blind spots
when you lie to yourself the the more
blind spots the more strategic risks
and and that's that's that's that's a
big problem i would say that's that's a
strategic misalignment which is
dangerous between the the ideology
and the reality on the ground that's its
own fascinating topic and
i i mean i got that okay i'm so tempted
to go down that rabbit hole because it's
such a great observation i mean i think
i think a lot of people
tend to think of china either as like
effectively a
purely socialistic marxist state or or
an effectively capitalist authoritarian
right and chinese want privacy we tend
to oversimplify
chinese beliefs in chinese views you
know one billion people
we tend to say chinese don't need a no
one privacy i think that's totally false
that's cross that's disrespectful to
them
i think that they have they have
preferences in terms of ethics
they have great people who work on air
ethics
one problem is that we envision i think
with the degree of reality the control
of the state
over which the how these ideas are
translated in reality
that that is problematic but it's not
because there are these problems that we
should
oversimplify projections of preferences
of the chinese people they are way more
complex
they are humans of course but it's all
they want privacy they want ethics
they want redistribution much like us
yeah
and then that distinction between the
chinese communist party and the people
between the
sort of socialist marxist authoritarian
framework and then the capitalist
anything goes like those those two
things seem to be so important in terms
of understanding what's going on
and they they do tie into what i think
is like it's a really
important question that i think a lot of
people including myself don't know
enough about
which is how are chinese companies
thinking about ai safety you alluded to
the conference you set up that seems
like such an important step
um you know we talked about open ai deep
mind active ai safety efforts active ai
alignment efforts
what about the chinese counterparts what
about baidu 10 cent
what about these companies how are they
thinking about this well is facebook
active in a safety uh
i i guess facebook ai research does have
yeah fairness initiative
no fairness a safety if you were to
bring jan lucuna set up because he's
french yeah on the podcast and ask him
about
human level intelligence and the value
alignment problem
that's a good thing because he's a very
nice guy he would not discount it but he
would treat it and say guys let's not
worry
about overpopulation on mars as we have
heard that's andrew yeah andrew ing's
fitness line yeah so let's so even uh in
in the so-called west uh in
multinationals we still have to
do some work on that and that's why and
and and i addressed this question from a
scientific program in the sense that
i address on i master in my point
uh janukun as a scientist that's why we
need to first reconcile
and build a bit of scientific consensus
on these questions that's why this is
where i would really start
now to address your question i think
that much like
in the west there is uh i don't know i'm
not a china expert
i don't claim that i know uh baidu
alibaba tencent xiaomi and all the
others
but i think that much like in the west
there is under strategic under
investment
in these questions of ai governance
particularly in these questions of a
safety but also
in these questions of air ethics look if
there is no pressure from the government
that's why
politics matter if there is no pressure
from the government
there will be any big incentive to work
on aitik
except for consumer satisfaction which
is a common ground that we have
and that we identified back in 2018 in
in the silicon valley or u.s china
summit
there are common grounds on consumer
satisfaction around robustness
around part safety of the software to
deliver the functionality
but if there is no pressure on what
is not incentivized by the market
long-term ai safety
ethics i think yes we have a problem so
though i'm not a china expert i would
say that
yes china surfers and the industry
suffers from the same problem that we
suffer in the west but
probably with the with the higher degree
uh of issues because of the lack of
governments
probably sincere involvement to push
these issues
uh on the balance sheets well
and as you raise uh yen lacun and andrew
ing
that really does highlight your earlier
point that the technology is sorry the
conversation begins with the science and
the technology
we will not make progress until we can
get people to agree on what the risks
are and if we
if we have a faction that says look
there is no there is no
risk like the probability of an
existential catastrophe from ai is zero
um then that really kind of hampers the
conversation
ai safety is all that we should care
about yeah and we should try and slow
down the pace of ai you'll have people
in here say
sorry yeah i need water i need
electricity so
it goes both ways and it goes it stands
at the center but goes way beyond
science that's why i think we
what we need to build is a solid
science-based science-driven
but civic epistemology or series of
civic epistemologies
and that's very hard of course some
would say it's impossible
i would say this is what it takes to
build human flourishing even
safe human flourishing over the long run
well into your earlier point about
imperfections i mean like it may not be
possible to do perfectly but
surely some compromises are possible um
you know my biases certainly were
showing there when i when i was talking
about like the one side of the coin
people who just refused to acknowledge
the ai risk
yeah i i tend to think that there is one
um but you're right i mean there is this
balance between you know how big is the
risk and how do we meet in the middle
with countries like india that
yeah and and i don't deny the need for
some people to
isolate themselves from that
conversation and do disentangled
research
in their own corner at peace and even i
would venture to say
hide it and hide the result from
potentially
strategically risky interests including
for example
defense so that's why the fact that uh
philanthropy funds to give an example
miri to conduct
research in his own little nook i think
it's very much needed
i'm not none of my views are totalizing
yeah none of it i'm just saying that
we cannot do only that otherwise we run
another risk of misalignment and lack of
coordination but that is very much
needed we need to
place different types of bets but as we
place best look at
what are the trends what are the
exceptions that's what i'm saying
yeah i know i i'm a big fan of that i
think it's a very meta portfolio risk
approach
and something that you know we all have
this risk of like becoming very
polarized and seeing
a certain problem very clearly and then
to the exclusion of all others
this seems like a really important
approach to it
what do you think the uh the role of
governments might be in terms of funding
maybe not necessarily the kind of ai
safety research that's being done at
miri but
like do you see an appetite there are
governments open to
funding like pure ai safety research or
are they still more focused on economic
development
i think that there is enough money in
the government to fund more ai safety
research
i think it's in the interest of
governments who declare themselves in
favor of human flourishing to invest
more money there and at the same time i
know that a lot of these investments are
dual
and therefore are bound to be seen with
suspicion that's why i see the role of
philanthropy there
to deal with the duality problem right
and that's number one and number two
i think that uh government need to do so
give an example
i started the first society initially at
the harvard kennedy school of government
as a student because i wanted to create
the free society and to really learn new
skills new network
and operate from a position of in a way
um
you know influence because harvard is
quite influential in the world of
government and the kennedy school is a
renowned institution and
the imprint of john f kennedy and his
vision over technology as a new frontier
had a stunning impact on me and
potentially was a good message to carry
as part of that we we had at the kd
school the then director of darpa
we know that darp has had an immense
effect on techno-scientific pathways
of ai and beyond internet gps
self-driving cars so many things and so
one
one topic of engagement we had across
the table was
how can you in your funding scheme
fun better prediction over the second
and third order consequences of your
research
right these externalities which go way
beyond your vision of national security
should in a way apply back to you
because you know you have a global
impact
so this frameworks of responsible
innovation and looking at
how do we deal with sec how do you in
your tenders
push and internalize part of the third
and second or
second and third other consequences have
to be done i think that
trying to influence public research
around that can be done too
i think it's uh in everyone's interest
on most people's interests
that's another password for which it can
be done trying to invite
including through regulation those who
fund and do
excellent research to be more reflexive
over this research
in the shorter ethics in the immediate
term development and in the ultra
long-term
safety yes there are points where they
will be in a way
inadequacies and orthogonalities over
short and long term
but in a lot of areas it's it's
potentially compatible and has to be
explored
very interesting yeah i mean again that
portfolio approach
showing up as a theme i just it seems
like such an interesting and
very uh differentiated strategy too for
the future society just the way that
you're doing it
i think it's the geographic aspect of it
most people do talk about portfolios
with different like
you know investing in in safety along
different axes with respect to time like
short term ai safety long term and so on
it's that that geographic piece that
really does seem necessary i mean
there is no way the ai equation works
out without some kind of global
coordination around this stuff
it's just really cool to see the work
you're doing there nico thanks so much
for sharing your thoughts it's a really
stimulating conversation
is there is there a website you'd like
to point people to i think people really
should check out the future society well
come and see us come and join us if you
have money that you don't use
donate it thefuturesociety.org
www.thefuturesociety.org
and thanks a lot jeremy for your
invitation and please
keep on the great work you're doing my
absolute pleasure i'll provide links to
all that as well in the blog post
that'll come with the podcast so folks
if you're listening you can check that
out as well thank you so much nico