Synergies vs. Tradeoffs Between Near-term and Long-term AI Safety Efforts

[Music]
this will be a session on the trade-offs
between traders and synergies between
the short-term and long-term safety
problems so there's a lot of different
ways to talk about the trade-offs and
synergies between the long term and
short term here so before I define what
we'll be discussing in this session
concretely will have a short word about
work to really not be discussing in this
session which is this very common
framing of this problem being roughly of
the form well X is just a distraction
the real problem is Y and then X is
going to be something like algorithmic
fairness and Y is going to be something
like long term a alignments and
Catherine I never know her thoughts on
this yeah so we all sat down to have a
quick like preliminary chat yesterday
about what we thought was interesting to
debate in front of this group and what
we agree on and what we don't agree on
so we all agreed that we're not gonna
address the question of whether like
governance or policy solutions synergize
our trade-off because we all broadly
agree that if you're looking at
governance or like institutional
decision making about what's good to
deploy in what order that there are
synergies there and there's gonna be a
continuous path so we're gonna focus on
technical research when we say
short-term we're gonna talk about
systems that exist today we're not gonna
try and debate what's gonna happen in
five or 20 years in which of those is
short or long and then we say systems
exist today that could be like
reinforcement learning systems that are
running in stimulation they're not
necessarily stuff that's deployed in a
product when we say problem we
identified sort of a bifurcation between
a problem is something that's causing
harm and a problem that's something
that's not acting the way we expect or
were confused about in some sense and
we're gonna focus on problems as stuff
that's not going the way we expect or
confusions rather than concrete harms to
the individual or group people right now
and we're also not gonna tell anyone to
stop researching anything we think
research happens best when you're really
fascinated in the topic that you're
working on and feel really engaged in it
and
so we're not gonna like go around finger
wagging at anyone for recently they all
doing great yeah and we also identified
that there's a difference in framing
between well there are problems now and
those synergize with problems later and
what we can implement these solutions
now and we can implement those same
solutions later and we think both okay
so I guess like what where we found the
most disagreements I guess is to what
extent is near-term and I said to
research contributing to the solutions
to the long-term safety problems
would you agree there was yeah okay
and of course there's a question of
course like what are we then here mean
by short term and long term problems and
these concepts on the self creep fuzzy
roughly the they mean people when they
talk about and I mean about two
different clusters first cluster being
these like as Katherine has mentioned
earlier these technical problems with
the current or new future a systems
involving narrow a is these systems
they're harmful right now and problems
and solutions for this tend to be more
model specific examples of these
problems are as a properly algorithmic
fairness differential privacy but also
stuff like over sale examples or a bus
of the distributional shift on the other
hand we have safety problems with the
far future very powerful AI systems
maybe even a GIS they are we expected to
be harmful in the future and we talked
about solutions to these problems that
anymore model agnostic these problems
include a value alignment credibility
naturalized induction Larry's problems
we had explained earlier really well by
Rohan of course they're problems across
the scale and maybe this is not the best
way to split up the space of a safety
problems so and we are going to discuss
this today as well so I guess we're
going to start with the opening
statements first remiel okay so one way
that I'd like to try and put forward a
way of thinking about this particular
question what it turns on is when we
talk about thought Turner stuff we
typically can
learned about technology which is
available now and the question that we
find ourselves asking the questions are
kind of of the format are the problems
that we're facing now Houston traffic is
there something about the way that we're
currently pursuing AI technologies now
that we're gonna abandon these problems
later when we have better technologies
then we have different paradigms or are
these things going to persist so the
answer to that question is going to be
some determinants of what the relevance
of the current safety questions that we
are discussing ends up what relevance it
has for the longer term questions so
like in terms of this energy aside I'd
like to break down the answer this
question the question of our current
systems idiosyncratic and of the
problems that they're facing
idiosyncratic by answering it hopefully
every possible way
because I have absolutely no idea of how
the answer this question and I think if
any of us claim to have the answer to
this question we're probably deluding
ourselves so ok so is my current systems
I mean things like people learning for
instance or machine learning in general
so let's look at the possible answers to
this so one possible answer is yes
everything about our current systems is
going to be thrown out and we're gonna
have better paradigms later in 10 years
or 20 years or yes and the aga the
long-term Adi systems that we that we're
going to build eventually are gonna have
no relationship to our current
technology so if this is the case then
the answer for synergies in our position
for this would be one of be confusion is
that look we don't have any other way of
talking about these long term abstract
ideas unless we try and resolve them in
terms of current models and let's do
that as an exercise to make sure we get
our terms down to make sure we know how
to think about them and to effectively
practice for the case where we have to
deal with these in another paradigm in
another realization in which case we
don't that have to be the first time we
deal with on the other extreme side
there's the case of no that basically
everything about the currents this
is going to be the basis for AGI so this
is kind of like the AGI is deep learning
scenario and in that case then there is
almost a direct link between the
short-term problems that we face in the
AGI level problems in which case we
better think about them now in terms of
the current system so then this is point
in the middle of maybe this some overlap
but there's going to be some other
paradigmatic shifts and in which case
there is a lot of value in us trying to
figure out what actually are those going
to be those short-term problems which
are going to transfer over and to try
and expend the time now to figure out
what the nature of those problems are
and what potential solutions those may
be and finally there's the kind of
answer to this of will who cares about
what the answer that question is but
there's actually a whole lot of new
problems which surfaced as we start
developing the current AI technologies
new fundamental problems that are
arising that we realize are going to
hold even in the case of whatever future
technology whether or whether it is not
based on the same technology and these
are sets of fundamental challenges that
arise that that I mean I'll just say
very shortly are things like and I'll
unpack these we can unpack these later
of things like the specification gap
that what we asked for is not
necessarily what we get and what we want
is not necessarily what we asked for or
it could be this problem of
distributional shift that we build the
system for one it like aiming to resolve
problems in one space but then we have
to apply it in a different space or
another possibility is another potential
fundamental challenges just having
learning in the mix means that we we
understand one nature of the one part of
the problem but we don't understand the
rest of the problem and we rely on
learning to basically fill in the gaps
of our specifications and that means
that there is some kind of epistemic gap
between how we understand the problems
were trying to solve with our AI or AGI
systems and what we end up building and
this leads the lack of transparency and
then also the problems about robustness
which may manifest at the moment in
terms of ever sterile examples that good
man
in any possible way in the future so
basically whether these systems are
idiosyncratic or not the current systems
idiosyncratic or not then there are
relationships between short-term
problems okay thank you mate I'm gonna
go next yeah I disagree with this frame
on my model if even in even in the
extreme you noted where even if we
imagine that the deep learning path
leaves fairly directly to general
intelligence my model says that the
reasons why why those systems might be
dangerous is not so much that they may
be susceptible to adversarial examples
but rather this for a problem where what
you select for may well not be what the
system is internally targeting in the
same way that the human brain was
selected very heavily by natural
selection for its ability to promote
genetic fitness to get its penis into
the next generation and yet humans a
general reasoner do not you know when we
wake up in the morning we're not like
gosh how do I get my genes in the next
generation today our our thoughts and I
mean some people might I just not every
day
my thoughts are also based on all sorts
of things like how do I have fun how do
i how do I learn things I want to learn
how do I you know bring about fairness
how do I like solve this one math
problem that I have in my mind and this
is a case where the brain was optimized
heavily for one thing and the internal
targets of the resulting optimizer were
quite different so so you can imagine an
analogy in reinforcement learning where
you really you you train a policy by
reinforcement learning for some target X
and then the resulting system if it's
powerful enough to be a general reasoner
may well not be optimizing for X this
seems to me to be the core challenge and
alignment and a challenge that I think
is not very well expressed in short-term
systems because they sort of aren't
clever enough they're sort of they sort
of aren't doing enough reasoning for
those they're sort of really have
internal optimization targets and for
this become a part but preventing those
from coming apart and sort of
understanding how to have the internal
optimization targets match some target
of our choosing this seems to me to be
the core problem and it seems to me that
trying to better understand adversarial
examples or a bust missed distributional
shift I don't attack this for challenge
well I think they have some use I don't
see them as sort of striking the heart
of the problem okay thanks um Katherine
yes so one thing I just want a caveat is
that I personally work on adversarial
examples so my uh my examples tend to
draw from that but I don't think that's
all of what we're trying to talk about
when we say like short term problems
where we mean stuff were confused about
in current systems or like neo for
example works on interpretability so
with that caveat maybe one thing I that
I want to highlight is that you're
saying that systems future systems won't
be dangerous because they're susceptible
to adversarial examples will be
dangerous because
this mismatch between what we've
attempted to select for with our
training process and what it's actually
doing and well I fundamentally agree
that nothing we have today is doing
enough reasoning to come up with these
like complex long-chain optimizers for
other targets I do think the reason the
adversarial examples problem is
interesting is because it has a much
milder version of a problem of that same
flavor where we've attempted to get a
thing that recognizes dogs versus cats
in the way we've done that is we've
grown dog and cat pictures on it and
we've gotten something that is just
fundamentally not doing what we want
like it hasn't actually learned the
problem and in this case I think like
the strongest piece I want to try and
put forward is that these problems today
can inform problems we'll have tomorrow
I'm not sure that any fixes to
adversarial examples today will also be
fixes to the sorts of long-stay
optimizers or inner optimizers you might
be worried about but I do want to
highlight that and for those who throw
at my lightning talk I mean sorry to
interrupt maybe we get to define what we
mean by interrupts misers it was
mentioned a couple of times but I'm not
sure if our audience knows um let's see
better what's that
I hate that nerve we can forget I used
it it's roughly the thing with Nate was
describing not not according to me I
think you're gonna drop that for now
okay um but I was saying yeah but
examples of problems now I think are
useful to like inform what might be
problems later
good so we've got a couple of slides ok
so this is a slightly different issue
than the relevance of say adversarial
examples to long term safety but I just
want to address some issues that have
already come up in the conference like
in Tim Wong's lightning taught yesterday
and other discussions I've had so so I
want to say that in general there are
big differences between the
narrow AI safety issues and the
long-term issues the Adria issues when I
emphasize it doesn't mean the short-term
issues are unimportant and thinking
about this to be I really think that air
researchers have a crucial role in
thinking about the short-term issues and
that they would really benefit from
sustained technical work like the
short-term issues and not just like
policy issues or like issues for like
activists or policy people to work on I
think they were like rich technical
problems there so I just want to sketch
how I like one of the big differences
that I see between long-term issues and
short-term issues so in terms of AGI
that is going to be super human in
performance humans can't reliably
evaluate its actions on a sort of action
by action basis it's on a very near-term
basis and we can't just wait around for
the results of the AG eyes actions so
you can't just say well we'll let this
agent try and work on cancer research
and try a bunch of experiments on humans
and we'll just wait five years to see
whether they have any long-term side
effects because we know that the agent
is smart enough that that kind of action
could have very bad consequences so I
think this is a pretty fundamental
difference when it comes to comparing
like what we need for AGI research with
the kind of paradigm for training
algorithms that is really effective
today and so if we think of some of the
short-term safety issues where like
these are issues where maybe people are
actually being harmed today by
deployment of AI or they will be in the
next couple of years so there might be
like physical harms or in Justices so
autonomous vehicles so in terms of the
safety issue of autonomous vehicles
humans can recognize the errors that the
cars are making for the most part like
we don't need superhuman drivers from a
superhuman like autonomous vehicles like
if they were just the level of a really
good human that would be amazing but at
the moment like human humans can
recognize the errors that the issue is
not like how do you train the system or
what's the training data but more the
systems have very weak
models of the world right they don't
understand human drivers and how human
drivers have a theory of mind and do
reasoning they have like don't have an F
world knowledge to drive in a very wide
range of conditions but the training
method there can be the standard method
that we used to like try named erté
humans trying to get feedback from
humans have humans hand-tuned reward
functions things that won't work
fundamentally when we're dealing with
AGI and then if we look at fare
algorithms that are making real-world
decisions they're often that errors
these systems make are very easy for
humans to recognize and they come about
again because of like weaknesses of the
system it doesn't understand law it
doesn't understand morality it doesn't
understand causality and which features
might be gay mobile so I think that this
is like this is a really important
difference and I think there yeah there
are a lot of continuities so on
adversary examples friend like I think
the even though there are other issues
they're making deep learning systems
into safe AGI is that we face though I
like I agree with Nate's
characterization there are other
important issues there it seems to me
that ultimately we want AGI systems to
be extremely robust like robustness
becomes more important over time an
adversarial examples seem to me like a
very good way of studying that where we
have the amazing you know advantage in
that kind of research that we have
actual empirical studies that we can do
and empirical studies have just
historically been incredibly important
in area I think if you ask people like
10 years ago what would the adversarial
example problem thief and you're on
there's I don't think they would have
just would be able to come up with all
the things we've learned from doing
empirical tests
lots of people thought are these these
problems will go away easily it seems
like that the actual picture has been
very different so I think if there are
areas where empirical study can be done
I think that's pretty advantageous in
terms of making research progress so
yeah there seems to be some of this
agreement in regards to how
representative are the near-term
problems of the long-term problems
so Katherine you said you believed for
example the recent examples are milder
version of the problems NATO was
discussing can you maybe explain why it
takes that well yes so the specific
thing that I was trying to point out is
that if you zoom out and say we had a
training procedure which involved it's
throwing examples at it or saying words
at it or hand waving at it that was our
training procedure can we expect this to
do exactly what we want we already know
that with current systems the answer is
no because we get things that you can
change a single pixel on an image a cat
and it thinks it's a dog like literally
one thinks it looks like very obviously
not doing what we want and then Nate is
is is looking at sort of a problems that
would arrive with different systems not
necessarily just like an image
classifier like gosh what kinds of
training processes or ways of giving an
objective to the system would like cause
it to bend and do what we want and I
think that like being aware that these
problems already existed that we don't
have a good way of like taking them out
right now is like one is a helpful thing
to notice I guess I'm not necessarily
trying to claim but like fixes to the
change of pixel and the cat becomes a
dog problem are also gonna be fixes to
the kind of stuff I'm I also just wanted
to say that I for the conduct of this
debate I'm pretty convinced that trying
to talk about stuff we're confused about
whether these stuff that's causing harm
today is going to help us narrow on
stuff that's like more helpful to talk
about right now because you're trying to
like fix the fact that self-driving cars
are crashing
there are probably policy solutions
today that are much better suited to
that or you know you're trying to fix
like unfair algorithmic bail decisions
like yeah those fixes our policy fixes
fixes right now we refer to this like
the confusion research right yeah yeah
we expect it needs to have expected to
disagree I mean do you think that there
are any properties of neuter systems
that will generalize to long term
systems let me just respond sort of
directly to your take I think the
analogy does not seem
direct me like it seems like if we try
to if we try to draw a line around the
problem that I'm trying to point to
which I readily admit I'm probably have
not successfully pointed to you with my
flailing words but it seems to me that
we try to draw a line that includes both
the problem that you're pointing to in
this tensed and or the problem that I
was trying to point to in the end and
the sense in which image pacifiers are
not doing what we wanted them to do when
we when they are susceptible adversarial
examples and that we're just no longer
talking like it's just we sort of zoomed
out to a level generality where I no
longer I no longer feel this captures
sort of my confusion if you will about
what seems to me to be the core problems
so like for example let me see if I can
articulate it like this model I was I
was saying earlier where it seems to me
like the key difficulty is positing some
general reasoner to be optimizing more
or less for an objective of your using
my model is that you sort of can't
really start talking about this all that
much until you can take what Ramana told
me yesterday that then it would all of
the intentional stance in the sense that
we have a whole bunch of of nice
theorems which say that if you have you
know some some piece of matter that's
more or less acting like you can predict
the world and is more or less acting
like it and allocate scarce resources
successfully dissolve a variety of
problems then regardless of what's going
on in there it's it's a good predictive
mechanism to talk about it as if it has
beliefs and has desires is making no
claim about whether it has a belief X
file or a like desires Python function
or whatever but we can sort of take this
fast or like ah you know it did that
thing because
it had deleted it previous relevant
knowledge because it didn't think it was
going to see this problem again I'm
making no claims about the internals
there's certain types of systems that
will behave in such a way that this
language is descriptive and predictive
and my the sort of problems that I'm
trying to point at of like how do you
get like there there's this way you
could take the intentional view and say
we can ascribe goals and these will be
predictive and the problem I'm trying to
point out is the one of life when you
have a system that has this property
we're ascribing goals and beliefs is in
fact predictive how do you get it to be
the case that the goals you would
abstractly prescribe if you were trying
to get the most predictive described
goals match the match the target against
which huge brain day the target against
which piece collected it and my my basic
model is that an image classifier is is
not close to being the sort of thing
where this intentional stance applies
like I don't seem to have enough of the
machinery to me that it makes sense to
talk about it as having believed having
goals I just don't see a way to draw
that line I suppose so I would agree
with that on the case of image
classification systems that it is a it
feels like from a human perspective
that's a bit of a stretch but I think
you only need to go to the world of MVPs
and pom DPS and reinforcement learning
in order to make that kind of intangible
stance very natural as though just
routine you routinely when you see any
RL researcher talk about their work they
immediately adopt the intentional stance
when they show their agent doing stuff
they talk about the kinds of goals that
this agent has in within this complex
temporal decision-making some audience
questions at this point
so I wanted to throw something out
I guess all of you especially concerning
the near term safety issues with AI
systems um I'm a little bit surprised
especially I think having an engineering
background when I hear about the short
term AI Spacely problems then I mostly
hear about you know image miss
classification
right now we have I think a lot of
pretty complex automation systems
operating in the real world that are in
fact safety-critical they're in control
of physical systems in the real world
where if the automation goes wrong
something really bad happens they can
have serious personal injuries or loss
of life or it is the big material
damages and it's often systems that have
little to do with you know machine
learning classification they have a lot
of different moving parts that are
interacting with each other often
distributed automation you have anything
from I guess the self-driving cars that
we're trying to build but also existing
systems liking you the power grid or
nuclear power plants that are heavily
automated and it seems that and I'm a
little bit worried that we might be
getting a little bit of field myopia or
tunnel vision and we're looking at what
we're doing right now in machine
learning systems but we're not looking
at the much broader field of automation
and what they are doing about safety
with their you know currently existing
and very real very present you know
safety problem so I wanted to do you
kind of throw it out humans and if you
have any thoughts them out what they are
doing about safety should inform how we
think about the safety of our you know
increasingly intelligent AI systems
going forward yeah so I want to strongly
agree with you that I think that the in
as much as there are problems with
safety critical systems that are
currently deployed that like an
engineering product society integrated
systems view is like entirely
appropriate for and that calling the
stuff that we're trying to scope this
debate to myopic from that stance is
like a valid criticism in some sense
like in our
the preliminary discussion he
deliberately decided to focus what we're
talking about on like D confusion like
what are we confused about today and not
what's causing harm today and I think
those are quite different problems which
need quite different solutions so like
one thing I've been paying attention to
thank you know Peter Eckersley bringing
this up is that California recently
passed a law that mandates the use of
algorithmic systems in making bail
decisions this is like some sort of
linear regression making like life and
liberty decisions for real people I
don't study that but I think it's
critical and I think that like groups
like the partnership I and I have a role
for like addressing this and
coordinating responses to this and
that's very important I think the policy
and government standpoint these sorts of
decisions how we make them as a society
from a government standpoint is like
absolutely on the like set of steps that
we're gonna take to making similar you
know life and liberty decisions in the
future but that they are different from
what a researcher who's trying to
understand a problem is gonna be looking
at in their toy problems in the lab and
I think adversarial examples are a toy
problem in the lab and they have value
in that sense but if your claim is that
studying adversarial examples is gonna
make self-driving cars any safer I think
that's a reasoning error but hey I kind
of want Neal and hey or whine know me
[Laughter]
greatly person Nate nail was overriding
Nate in my brain anyway but usually to
talk about like their closest examples
in current RL systems that we might see
to like what the longtime concerns us so
I think I remember in the Kappa flag
paper you have like a step for reward
predictor module so you've got you've
got the like at the end of the game you
win or lose but that's not very helpful
like making decisions within the game so
you have somebody that has things like
who has the ball as it my team who has
the flag as it my team or the end enemy
team and like you might then get the RL
agent like learns to do things that get
you the flag which isn't exactly the
same as the end goal when you get into
whether that's like the same thing yep
okay so so I I think this is a good
point to go back to a point that Nate
made which I feel like is much of the
crux of the disagreement here which is
about whether the particular challenges
of alignment or how I would describe it
as also being specification whether
there it's well expressed or not in the
current deep learning and reinforcement
learning paradigm and whether like in
some ways is this a difference in
quantity or a difference in kind so what
I mean by that is we're basically in
this situation where we say look we want
we want X we don't really know how to
say we want X we kind of implicitly want
X we say we want Y we write down that we
want Z and we end up with the system WWI
then we end up with the system which in
order to get Z ends up getting W because
our expectations of what Z actually
means in mathematical language we're
still kind of fudging our way around
this and we don't know what the
consequences of those particular
specifications are so whether that
difference between W moves the original
thing asks the thing the difference
between W and X I mean the cycle
fantastic whether that actually Furious
or not and what the consequences of
those that missed bests that ultimate
misspecification are if it's already
present and it's already deeply
problematic in deep learning and
reinforcement learning like the example
adversarial examples is one phenomenon
of this case reward learning and inner
loops of reward learning using
algorithms that try to take very complex
optimization algorithms there are
complex optimization problems that are
like very fast reward problems and try
and enrich them and create auxilary
targets is one way that you end up with
behaviors that you're not really
expecting at the end of the day how bad
this problem ultimately is whether it
involves this I know we didn't define it
but in the loops of optimization or
instrumental goals or anything really
potentially like run away in the middle
that might really really really not be
what we want I don't I
we probably don't have systems that are
that dramatically bad that we end up
destroying ourselves when what we wanted
was a coffee that we're not in that
situation yet but we do have models of
those and I think that there is good
work to be done to explore what the
consequences are what the signatures of
these things are and what potential
mitigation strategies
return David yeah two quick questions
the first one is but the people who
think that working on short-term like
things were confused about is important
how much better is it have people
working on that who are thinking about
applying those D confusions who long
term issues compared with people who are
working on those problems just for the
sake of those problems and then secondly
when we talk about long-term issues
being very important and also short term
issues being in very very important
right I think under like both of these
things are through right we don't want
to discard them but at the same time
it's difficult
how do we separate out say something
that's orders that like if you believe
that something is orders of magnitude
more important than another thing but
both of them are very important how do
you like they that like prioritization
in a way that doesn't sort of minimized
it doesn't sound like you're minimizing
they short-term issues yeah I didn't so
the first one was is it better that
folks doing short-term D confusion are
also thinking about long term and the
second was how to talk about this before
minimizing concerns
I can take the first one I mean I think
the I think it probably does make a
difference for people to be concerned
with long-term issues so if we take
adversarial examples there's a huge
number of papers on this topic some of
them aiming at fairly practical things
and there are practical problems with
them for several examples some of them
aiming at like proving theorems doing
verification thinking about this in a
very abstract context so like what can
you say about any classifier rather than
just that particular neural nets we have
today so I think there will be
advantages in people going at it this
with the mindset of you know how can we
build robust machine learning systems
where we don't know exactly what machine
learning system is we need to make
robust and so we want to always aim for
a level of generality so but I think
even people just focusing on the
questions today without an interest in
the long term may still do really
valuable work yes I wanted to try and
disentangle a couple of a couple of the
claims and see what people thought so I
heard two core claims from Nate one is
that this problem which I would like to
call you know how precise is the central
problem that needs to be solved and
another one is that we are not in a
position to study it today so I'm
curious if well if I've characterized
that correctly and then if other people
disagree which part of those do claims
they disagree with and then I guess I
was also curious although I don't know
that we'll have time to hear Nate
justify the claim that were not in a
position to study this today this was a
final question I I think I'm not super
interested in justifying the claim that
word not in position to study it today
because I consider myself to be studying
it today
oh I'm sure it could be studied way to
other than what I do although I do agree
it is central and be studied then Nick I
think I can feel it not clear that it's
essential I don't feel I have a great
handle on it like I think that
evolutionary analogy is pretty
misleading like we're trying to build AI
we're intelligent evolution is this dumb
process so there's a lot that we can do
that evolution was not able to do so I
think like you know it's unclear exactly
whether this will be a problem if you're
working with deep learning type systems
I think we need to eat if you need to
bring the rest of the discussion to
lunch so I'd like to thank our debaters
for a really interesting discussion
[Applause]
[Music]


Artificial General Intelligence: Racing and cooperating | Seán Ó hÉigeartaigh

[Music]
thank you very much for those of you who
don't know me I'm Shaun Hegarty I'm the
executive director of the Center for the
Study of existential risk which works on
global risks and long-term challenges
and I'm a program director with a center
for the future of intelligence which
works on the opportunities and
challenges of artificial intelligence a
couple of years ago myself and my
colleague Stephen cave wrote a paper in
which we outlined some concerns about
what we saw as an emerging narrative
framing artificial intelligence
development as a competitive race or
strategic global advantage this is most
famously identified with some quotes
that we've already encountered at this
conference from Vladimir Putin in 2017
and this point from the State Council of
China's new generation and artificial
intelligence development and framing
these these goals in some more
competitive language now as we've
already heard and Alan and Brian pointed
these things that nicely it's unclear
how much we should read into this but
where putin's quote came from a
discussion with schoolchildren and the
chinese phone well chinese leaders have
repeatedly emphasized how important and
how essential of global cooperation is
going to be around artificial
intelligence nonetheless one of our
concerns is that certain types of
narratives can easily sort of build up
and sometimes even reinforce themselves
so somebody writes about artificial
intelligence as a race other people pick
up on it and run for this people are
influenced by this some of them perhaps
decision-makers who then take actions
which makes sense if we're in more of a
kind of a race dynamic and that again
lends credence to the race framing and
the whole thing kind of spirals and
we've seen a lot more and public
articles and policy reports that frame
intelligence in these terms I've been
asked to give a framing talk for this
issue of races and artificial
intelligence I'm going to do that by
making a number of argument since the
jessyca n--'s
one is that competition isn't
necessarily bad a lot of competition is
good secondly contemporary artificial
intelligence i'm not sure it makes sense
to frame it as a race full stop and said
what i think is happening is a lot of
different types of competitions on
different levels and in different ways
at the same time thirdly there is not a
race so to speak towards artificial
general intelligence happening right now
the goal is too far often too poorly
defined for something that however
fourthly at some point in the future a
critical mass of people might see this
as an imminent thing and such a race may
emerge basically i think it would be why
there are some serious concerns about
artificial general intelligence being
developed as part of a competitive race
for security and other reasons
sixthly there may be opportunities
certain key points of which communities
like these can really influence the kind
of bringing them of how we develop
artificial general intelligence
particularly at key points where for
example a critical mass of people
researchers decision makers and which
one seeing this is a long-term
speculative issue to something more
imminent and we should be thinking about
what we can do to have a positive
influence there one way is by having
well develop proposals and
well-developed and plans in place that
have buy-in from a number of key actors
who might have been thinking further
ahead on these things and the other
thing we might think about is the
culture and the environment in which an
artificial intelligence is developing up
until that point both in the research
community and in the broader governance
policy and other stakeholder ecosystem
around that and by that I mean practices
principles and precedents types of
collaboration other things that do or
don't alliteration so right now
I don't think that it makes sense to
treat AI development as some sort of
global race what we instead have are a
lot of different types of competitions
there are research groups working as
hard as they can to achieve certain
fundamental and breakthroughs and
applications earlier there are companies
looking to apply artificial intelligence
the various industry and business
applications whether it's in medicine
healthcare finance manufacturing
what-have-you
we have companies and nations trying to
be in a good position to bring
artificial intelligence new emerging
markets we have national strategies
being developed by a whole range of em
nations where policymakers are really
looking to make sure that there are
nations and their societies are best
supported and by artificial intelligence
and not left behind and so that M the
economies do as well as possible in the
future with artificial intelligence this
is all quite natural and healthy
competition can drive innovation
progress and beneficial societal
applications now obviously there are
things that we need to bear in mind
though you know in the drive towards
automation we should think about making
sure that people whose jobs are replaced
don't get left behind we shouldn't
deploy AI systems cry maturely or
carelessly in situations that might
exacerbate biases in historical data we
don't want to make sure that privacy
isn't undermined as we push to gather
more data to make better applications
but competition isn't necessarily
inherently about things looking at a
more geopolitical level a lot has been
recently written about concerns about
for example power becoming concentrated
both in leading corporations developing
AI and in the nations in which those
corporations are based are embedded
scholars like - Lee and others have
written persuasively
about the concern that we might end up
with nations that have a huge advantage
that are effectively offering
services to developing nations in such a
way that would be very difficult for
those emerging nations to pull
themselves up and be equal partners at
the table companies might end up with a
very difficult to overcome advantage in
things like research talent in access to
computing facilities and access to data
and leading groups might be able to play
at those role in influencing national
and international governance and
regulation but and there's been a lot
written about concerns about a military
arms race and by this I don't just mean
lethal autonomous weapons and this also
takes into account application of AI to
decision support logistics information
gathering intelligence fibre security
supplies chains the whole gamut of
things that makes up offense/defense
security and given that artificial
intelligence is such an inherently dual
use technology in which fundamental
progress can be turned into applications
and applications can be repurposed for
other purposes so easily these issues
are going to remain quite intertwined
with an industry a military which is has
at its basis a competitive angle and a
security angle that might be somewhat
antithetical to the cooperation that we
might want now looking a little bit
further ahead to this idea of an AGI
race and our special general
intelligence right now there isn't a
race war exist and most policy documents
talk about artificial general
intelligence as a speculative future
prospect that is decades or perhaps
hundreds of years away it and outside of
places like this an awful lot of mao
researchers feel the same way about it
it's not clear how much it makes sense
even think about this as a race when the
end goal is so poorly defined and and
difficult to say anything meaningful
about however that may not always be the
way things are if we are some decades
away from artificial
intelligence at some point it's going to
go from speculative to seeming more
imminent and at that point we need to
consider a few scenarios one is that
this server drive towards artificial
general intelligence might result in a
different sort of dynamic than all of
these different sorts of competitions
that we've been seeing before this
general intelligence has in the
definition generality a system that
defined as Anthony did that would be
able to basically do everything that
human intelligence is able to do as well
as us are better would imprints we'll be
able to occupy a huge number of niches
perhaps all of them which might mean
that there isn't very much room for lots
of different winners if this technology
allows us to make faster progress on
scientific goals economic goals
financial goals military goals it might
give the group or nation that achieves
the technology first an opportunity to
shape the future of the world in a way
unlike any technology we've seen before
and if as has been hypothesized this
technology was able to help us with the
process of developing artificial
intelligence itself it may mean that the
group who have achieved this technology
can continue progressing faster than
those who don't have it now I don't
think we can say that any of these
hypotheses are correct or incorrect
right now we've heard a lot of good
arguments why it might not be the case
we might have diminishing returns as
yahshua point to that earlier there
might not be a single point of which
something is AGI and whatever was before
that wasn't it just may not be away a
sense if they think about these things
it may be that we have a world in which
some tasks are best amenable to AGI
systems but narrow AI has an advantage
in other things but it seems at least
plausible to consider a world in which
developing something that we can call a
GI gives whoever achieves this burst
this huge global advantage and to think
about what this might mean for the world
and I'd point out that at the point of
which this may become imminent seeming
to people beliefs will matter
so the
people who are pushing forward towards
this what they think a GIM will be
capable will affect their actions and
what they think that their competitors
believe about AGI and what their
competitors will do will also influence
their actions now I think there are
really serious reasons to be concerned
about artificial general intelligence
and the drive towards it emerging in the
context of a race narrative one key
concern is this idea of safety ideally
we want and this technology that is has
such transformative potential to be
developed in such a way that am we move
slowly and carefully and paid due
attention to safety issues and that the
scientists who are working on the
forefront of it are sufficiently
convinced that they've solved the safety
issues before it reaches certain levels
of deployment it may be much more
difficult to do that if AGI is being
developed in a competitive setting where
other groups are working towards the
same goals perhaps not taking the same
sort of AM safety precautions and so
there feels like an a lot of pressure to
and keep pushing ahead quickly similarly
it may be a lot more difficult to answer
a lot of the key questions about how you
would deploy this technology once it's
developed in a way that's fair and
beneficial for the entire world how to
involve the voices of all the global
stakeholders who would be influenced by
this technology in a fair and equitable
way if it's happening as part of this
competitive race it may be more
difficult to share expertise between
leading scientists working on safety
issues between companies are between
nations to establish the kinds of
cooperations and collaborations we need
this sort of competitive dynamic might
even spill over into tensions of
different sorts geopolitical tensions
and even conflict and above all it comes
down to this question of well what if
somebody wins the race how can we make
sure that if this race is won by a
particular group the benefit is one that
results in benefit to all of us and that
we all feel is fair and equitable
in an ideal world we might want to see
artificial intelligence developed over
the last couple days as some sort of
globally cooperative venture some sort
of perhaps global Stern for AI where we
have scientists from across groups and
across nations working in together and
the benefits of the project accrue to
humanity as a whole rather than to any
group RNA and nation where scientists
work on this and don't bring it past
their stages until key questions are
being answered about safety issues and
issues of m control and where global
stakeholders are convinced that the
benefits are going to be distributed
worldwide now it's not clear how
feasible or practical this is but it's a
good goal so something we might ask is
in the world we're in right now where we
don't know if artificial general
intelligence is 20 years away or 200
years away or 2,000 years away and we
have all these kind of competitive
dynamics playing out what kind of things
should we be doing now so one thing is
rhetoric rather than amplifying this
rhetoric of AI as a competitive global
race and putting forward other types of
narrative artificial intelligence as a
shared global goal for a global good a
shared global challenge another is
establishing the kinds of cooperations
and collaboration that we need there's
already a lot of collaboration happening
within the research community it's
essential that as competition heats up
in different aspects of artificial
intelligence we maintain that spirit of
cooperation and particularly on issues
like research relevant to safety as we
develop more powerful capabilities
within artificial intelligence there may
be roles for different types of openness
of research publication and models and
different types of sharing models which
is something that we in a number of
other groups explored in the malicious
use of artificial
and report that was released last year
double engagement this conference is an
aspiring example of research leaders and
thought leaders coming together from
leading hubs in Europe the United States
China Japan and this is exactly the kind
of thing that I feel that we need to be
doing now it takes time to figure out
how to work together you know
particularly across cultures to
understand what our shared values are to
understand what we disagree on and why
and to figure out how to work together
if the stakes around artificial
intelligence are only going to become
higher now is the time to start working
together in this way and similarly
organizations like the partnership on AI
and play a powerful role in encouraging
globally engaged in global even
countable governance of artificial
intelligence
lastly precedence Yun and others pointed
out scientific communities can play a
powerful role in influencing how a
technology develops and the steps that
our community takes to put in place
certain principles and certain
precedents and may influence the way
that the technology develops in a
substantial way I particularly like this
principle from opening eyes charter that
they released last year or they say if a
value aligned safety-conscious project
comes close to building AGI before we do
we commit the stop competing with and I
said start assisting this project if
more groups were to adopt similar
principles and similar practices of
collaboration then I think we'd be in a
very good position when it comes to the
emergence of this drive towards
artificial general intelligence ahead of
this drive emerging I think there are a
number of topics that we can meet and
thinking on working on and questions
that we can be asking ourselves one is
what might be plausible and feasible
models for a joint
and general intelligence development and
project how could we incentivize
research leaders and experts who
currently reside within national efforts
or companies to take part in this as
well how would we make sure that this
project would be able to really achieve
alongside those what might be ways in
which this wouldn't work simulating
models for a global artificial general
intelligence benefits agreement who are
the stakeholders who need to be
represented what are the key issues how
can we make sure that benefits accrue to
everyone how can we make sure that if
leading stakeholders commit to this they
can be held and binding towards it one
question is what will be good enough as
I think mal I want to feel there's a
point there it's great to think about
ideal worlds but we also need to
consider the worlds that we're most
likely to end up in and the ideal
scenario might not be the option on the
table so we need to think about what
might not be ideal but what might be a
good enough solution that at least rules
out the worst-case scenarios I might be
concerned about lastly there's this
question about how we even talk about
and write about this issue there's an
irony to me it's talking about you know
races being bad while being up here
talking about races and there's it
points to a fundamental tension in that
we as researchers have a responsibility
to consider all the plausible scenarios
in order to be honest in order to
develop good ideas in order to engage
key stakeholders on those ideas and in
order to open up our ideas to the right
sorts of scrutiny but at the same time
it seems essential that by our actions
we don't serve emphasized or exacerbated
be exactly the site and types of
dynamics we might be most concerned
about all of these seem like really
useful things for us to be thinking that
ahead of time and to have done some
really solid thinking of about so that
we're in a position to influence the
global community when this comes into
view or a kind of a critical mass of
global stakeholders this is a very
powerful community and I'm now finished
it's research leaders as I said from you
know across the globe
people who represent the research
leadership and a number of companies a
number of academic groups and some of
the key thought leaders all of you
people thinking deeply and carefully
about these issues not just for the
future of your own group the future your
company the future of your nation but
for the future of humanity itself and
that's an inspiring thing it's I mean
it's been really a privilege discussing
all these issues with all of you over
the last couple of days to me it seems
like exactly the kind of leadership that
we need in order to make the best
possible progress and being the best
possible position when we get to this
drive towards artificial general
intelligence so thank you all for taking
the time to do this and to the
organizers for organizing this and I'm
really excited to know what will come of
it
I'm personally feeling more optimistic
than I have in years intelligence with a
GI but rather slow progress where a lot
of different actors are improving on
different aspects of intelligence and
might be less easy to convinced of your
a nice project of you know having this
AI Stern kind of thing because I think
we still want to have this coordinated
development even if we're not afraid of
her essential risk this because this
very high constitution of power could be
highly disruptive for society in many
ways can you think of other arguments to
convince a players especially a military
in different countries would be part of
something this is a very good question
it may very well be that we don't have
this single point that's very obvious to
point to that says now you know the
alarm bells going off or we need to work
together and it may be that what we have
is more of a kind of a frog boiling
water scenario where we've got this kind
of gradual development it may be harder
to make the claim in that case but I
feel like there's so an awful lot of
technology develops in this way where
it's
gradual and you can have the technology
developing gradually where it's still
the case the certain threshold or
thresholds are passed in the impact this
technology has in the real world where
something goes from not really measuring
that much or seeming fairly sort of
manageable to suddenly being very clear
that it requires a new type of
governance and I think you know we've
seen smaller versions of this in terms
of you know social media manipulating
elections and a range of other things I
mean maybe we would at some point have a
major cyber attack that makes us take
you know issues around cyber offense and
defense a lot more seriously even though
there hasn't been some sort of you know
qualitative shift or rather a continuous
development so it may be that we need to
think about in that case is well even if
what we're seeing is something more and
continuous we can see that the effect of
this technology on the world that we
live in means that there's something
that's very different now and that will
be very different in the coming years
then what the governance structures and
the collaboration structures that we've
set up for this are set up to account
for so I think there are ways to
approach this where even if the
development is more incremental we can
recognize that something that wasn't a
problem in the past is going to be a
problem now and needs a different
approach
[Applause]
[Music]


Opportunities for Cooperation on AGI at the Governance Level

[Music]
okay thank you very much for joining us
the structure of this panel is going to
be fairly straightforward we're going to
have statements from each of the
speakers within five minutes or less and
if I've moderated you before you know
that I keep times then we're going to
start with three questions just to
really get into the practical aspects of
what we can and what we should do
and if there is time hopefully we'll
have questions from the audience so
Cyrus would you like to begin
okay you know we're statement to stop we
gather and put one together and maybe
explain of the work that I'm doing in
degree with the UEA I'm minister and
possibly what's between global
governance of AI one table that we
started convening last year we have a
look at least we have approximately 20%
up to this audience are being there so
we have a strong focus on on on the a
safety so these front table is being
gathered at the world government summit
which is happening every year in Dubai
and as its name says you know it's about
convening and discussing stories around
technology and others that will have an
impact on society and how will
government link with that so since we
have a minister of the ISO 4 year now
and it's also bidding this this summit
he has this fantastic convening power to
bring a really truly multi-stakeholder
approach with people from the civil
society from the scientific community
but also from governments international
organization and others to gather and
discuss this so that's one of the
question that you you're gonna ask too
but I'm getting it up right
it's basically what we have in the UAE
is truly a unique geo strategic location
we extremely friendly we with the u.s.
we extremely friendly with China so
obviously this is these are the tools
with AI superpowers that we so this on
its own isn't
but on top of this we're friendly with
Russia we're friendly with India we're
friendly with Europe so so we have this
unique neutral platform where we can
have a true global conversation
discussing about governance of the iron
impact of the iron society part of this
discussion and that's what we're doing
right now we you know with many of you
are present in this room is bring the
dialogue on on the on hei and the impact
slightly and starting thinking about the
eye safety issues okay thank you I'm
gonna suggest that we use the Cold War
as a role model not as like a general
rule but just for about five minutes I
mean it should stop being a role model
for us but there's several things that
we can learn from the competition
between the United States and the Soviet
Union which was possible much more
hostile than any relationship right now
between AI powers and yet was still able
to support risk producing cooperation
over decades the first form of
cooperation that was useful was informal
engagement of the sort organized by the
Pugwash conferences and very much in the
spirit of this conference was an effort
to bring together international
technologists scientists technocrats in
order to develop lines of communication
that later ended up being quite
important for back panel discussions
during crises so doing more of this sort
of work and possibly similar conferences
that seek to focus on the policy
communities in their respective
countries and are known as for crack 1.5
discussions which you invite government
advisers as participants but they don't
play any formal role plus if it's run by
government there won't be any free food
so that's another advantage of not
having governments run the conferences a
second form of cooperation that was
important during the Cold War were low
stakes technical collaborations
one example is apollo-soyuz in which you
had technologists from the US and the
Soviet Union collaborating on a
constructive project that
didn't involve anyone trying to kill
each other and that was good both as a
confidence-building measure and and also
to help build this connective tissue
between the two communities a third
example of cooperation where high stakes
technical collaborations
one of which was the development of
locks that secure nuclear weapons
against unauthorized use and one example
where the AI community on that might be
an international collaboration to pursue
AI safety research or possibly AI
security research may be one in whose
goals have short term deliverables
something which is concrete and can show
the benefits that sort of cooperation
another example of cooperation would be
agreements to submit oneself to certain
kinds of monitoring for
confidence-building one example during
the Cold War were open skies agreements
in which countries would submit
themselves to aerial reconnaissance so
that we wouldn't become too anxious
about what the other side was building
or arming itself with in the context of
AI as miles and others have talked about
computing does look to be one bottleneck
and semiconductor manufacturing is
fairly centralized that's something that
actually is amenable some level of
monitoring so it might be interesting to
explore what a international agreement
to monitor at least that sort of general
volume of production for semiconductors
would look like the last form of
cooperation I think the most difficult
are arms limitations treaties and we
have many examples of of three DS which
sought to either ban or limit categories
of certain destructive technologies in
the context of AI one that I think could
be an opportunity in the near term is
for the next 40 G arms limitation treaty
to include some ban on the use of
autonomy with a nuclear commanding
control so with that I'll end and say we
should no longer use the Cold War as a
role model
great Thank You Charlotte yeah so I'm
gonna do a mixture between what we just
heard I'm gonna kind of recap what I
look at I'm also going to bring out
through questions that I think are
interesting so my main focus is AI
policy in the European Union why the
European Union well because the European
Union was initially founded as the
European Coal and Steel Community so
Germany and France couldn't go to war
anymore to have shared resources that
would not make it
ethically unthinkable but also
practically unthinkable that these
countries Padres and Europe would be
dipped into World War 3 so I'm not
suggesting that we should all end up in
a global single market and we need to
have shared judicial or legislative
branches or executive branches but I
think it might be worthwhile thinking
about what could be a thing that if all
actors collaborate on it would make it
impossible for one of them to break away
and actually start racing again by
themselves so that's point one point two
is what is the European Union doing
right now well it's actually surprising
still somewhat functional after this
very inception point it is currently
looking at corporation for artificial
intelligence so exactly the type of
topic we are discussing here they have
signed on by day I mean all the 28
member states plus Norway have signed an
agreement to cooperate on artificial
intelligence that is for financial means
that is for ethical and socio economical
reasons so in terms of the labor market
displacement in terms of how do we
educate the next generation how do we
upskill the workers that we already have
and there's also in terms of what common
vision and ball do we want to have which
has been more specified recently in a
plan for coordinated action on AI by the
European Commission and I assume I'll
get the chance later on to speak a bit
more in depth about this but the
Europeans measure
is basically to have ethical AI and
trustworthy AI and I will explain later
on why I think that is a bit more than
just a nice cover up term for not doing
anything else and how this can actually
eventually translate into a mechanism
for governance that doesn't necessarily
require all actors to agree on the same
thing okay okay I think before I go on
to talk about my advice and
recommendations I want to actually I
have a question on the question that's
being raised to us and then we're
talking about the race but I was
thinking what race if we're talking
about you know the AGI race I mean the
whole research project are so stupid
what do you mean no one agrees on the
actual path to AGI so for me it's at the
best this race is a bit Silla it's a
demon lighted cave we don't know what
the finishing line is we don't know what
the shortest distance is and then so in
a way it's fortunately fine what the
race is all about if you don't if you
can't define that and then how can you
talk about you know how to coordinate
between the different you know
stakeholders and stuff so so that's my
question on the questions that opposed
to us but I think you know perhaps you
know this community can help but some of
the idea mistakes about about this race
I don't I'm also helping in some of
their speakers there earlier so before I
talk about the recommendations I want to
get and you know take another I don't
say one step back I want to take two
steps back I think you know I wanted to
talk about my understanding of the comp
the conflicts in clash between US and
China I think essentially what we're
seeing right now it's not just merely
trade disputes and it's really sort of a
civilization or culture clash so let me
use on a metaphor so they will allow us
to understand better what US and China
is all about it's let's say you if
imagine it's an oil painting you have a
multiple layers of paint so at the top
layer paint it's essentially US and
China are speaking the same
we're talking about you know economic
terms and national interest competition
efficiency materialism consumer
consumerism okay
capitalism that's throughout the same
language that in the past four decades
that both US and China have been talking
about so we understand each other but
that's the only the top layer if you go
to the bottom layer the grounding color
of the oil painting to civilizations are
very different so if you look at the UH
the grounding color let's say if you
look at the the US I'm you know I
apologize that I'm sort of resorting to
the oversimplification yeah it's much of
a monotheistic culture individual
individualistic and then a lot of the
emphasis on zero-sum competition and
then then actually when it comes to more
for intellectual let's say purity and
the pursuit there's the you know focus
on logical purity okay so if you look at
the China side the Chinese culture first
of all we're not religious but we're but
we are spiritual okay there's a
difference in here and there's a whole
commitment to this way of Dao and then
some of the the Chinese colleagues
talked about the the unity of of the the
haven't Yin and then and then in humans
a lot of those kind of a spiritual
concept and then we don't we don't have
a modern monotheistic culture and then
also the de Chinese concept focus a lot
more on inclusiveness and a hominid okay
and then and then also the concept that
we have towards human beings it's a
relational be okay that kind of concept
it's not an individualistic and then we
are we pay a lot of attention to the
context driven context driven content
you know contextual driven decision
making and thinking process and then of
course related to that family based
decision making so in the u.s. thinking
the best way is my way the Chinese
thinking we're actually looking for the
most appropriate way there's no such
thing as the best thing
there's only most appropriate depending
on the context and circumstances okay so
so that's why I want to highlight the
different way
of approaching things and thinking that
does inform all the geopolitical tension
as well as the the approach to our AI
developments and the effects engine so I
just want to raise you know the example
let's say the concept of privacy and
then you know privacy in the Western
contact might be very different
a differently understood in the China
contact if you look at the broad
spectrum this that is individual this
that is a social good or now associate
social stability or you know the and
then on this broad spectrum China's
approach to privacy spa problem probably
much more towards the social good side
and then much less on the
individualistic side so so that's why
it's I personally find it very hard to
describe general principles AI
principles without understanding how
they're gonna Inc applied in the
contacts so I just wanted to make use
that as an example so but in terms of
the recommendations some advice I have
we should probably just do that within
the questions just so that but really
quickly I just want to say this is a
great forum for that kind of a
discussion but I do hope we should act
just bring out more
value-based civilizational kind of or
cultural kind of dialogue because I want
to go to the bottom because that's the
thing that means forms everything that
we were discussing about and then
secondly is about the global project
actually Shaun was talking about as well
rather than receiving a big very sort of
berm
you know abstract concept I was thinking
about it was one thing which is less
sensitive is care robots or elderly care
that's a global issue and then so forth
but in a way we should actually
cooperate in the most specific context
is then we can resolve all the privacy
issues and then safety issue and
security issue with us with respect to
one specific product and then we will
see how different that auto you know
understanding will be brought to play so
anyway that's it just a quick note I did
a lot of work on care robots and I could
not agree with us
but we'll get to that later okay
first questions from the panelists from
your point of view what is the most
pressing problem and the most promising
opportunity with regards to
international collaboration whoever
wants to start the dred so you asked
both three I asked her no I asked for a
one pressing problem and one promising
opportunity okay let's keep it simple
the most pressing problem is the most
obvious problem why would people want to
collaborate what are the incentives to
cooperate if you think that you can do
it by yourself so like I think again
very obvious but that seems to be that
is the pressing problem now how can you
change that I think that's what we're
all here to discuss and figure out one
immediate death one immediate step I
think that harks back at what I said
before about the European Union focusing
on ethics is so to embed what I say in
the broader pictures the European Union
is currently working on guidelines for
the ethical development of AI so in
these guidelines that are open to input
which someone mentioned on Friday you
have a statement that the European group
on artificial intelligence the expert
group wishes that ethical AI should be
trustworthy human centric robust and
secure which are all quite exciting
statements I assume for everyone here in
this audience now the question is how
does that translate into something
larger so the idea behind this is that
these guidelines will probably be used
by the European Union in order to find
partners in an international context
that agree on these types of values so
everything the European Union does is
based on the Charter fundamental rights
so AI as well is going to be based it
needs to adhere to the Charter
fundamental rights basic
so which other countries on the globe
agree with these types of values and you
could go further beyond taking that as a
rather straightforward and easily
achievable basis of cooperation agreeing
on ethical principles because you don't
want each cut each country to come up
with their own ethical principles that
are kind of the same but kind of
slightly different and then you have
duplication efforts within countries and
by different groups and across the globe
you could say well maybe we can come
together on this and this being the
ethics guidelines or an ethical charge
or ethical principles but also trickle
down further in what I imagine this is
just an assumption so take that with a
pinch of salt into certification
standardization processes so once we
agree on how do we want AI to be
developed and deployed there are already
organizations like the item four in the
AIA so looking at AI standardization and
certification particular the ILO has a
working group on trustworthy AI so I
assume that in all of this diversity
there will be someone who looks at
different countries or groups statements
and says well actually all of this makes
sense let's consider writing standards
or safety certificates around this and
to me that then in and of itself so
standardization certification which has
also popped up in a lot of different
policy papers across the globe that
x-country wishes to develop AI seals of
quality for example that then in itself
is a kind of indirect way of governing
the broader AI development and
deployment to me at the very least it's
not a complete solution it can go
towards preventing certain small
shortcomings that we already see right
now in an indirect manner okay in the
interest of time and the interest of
coffee if no one wants to take a stab at
the first question we can move on to the
second which is if you could change one
misconception in emerging AI AGI
governance structures what would that be
and I think you alluded to that so we
would
go ahead actually better with the cold
war analogy you really scared me and
then then also in early panel talk about
this fixed I again I think we are using
way too much so the legacy framework and
the legacy concepts to think about the
governance and principles and governance
structure for for AGI or for the future
I think we really need to change our
minds
mindset and we really need to focus on
you know all inclusiveness that's not to
forget that we're talking about HDI
we're talking about a potential exists
initial risk for the humans but also
potentially you know a good relationship
between humans and the machines in
future so so in other words this the
issues of that scale it does not you
know we can't use the past experience I
mean we can use very little past
experience so we really need to think
about something new and then so I would
I would you know think that we would
much prefer this kind of more of a moral
persuasion and then more you know that's
why value based the discussion and then
come up with something more inclusive
and then really want to pitch at the
level for the global narratives that it
is a global issue it is initially faced
by the entire humanity it's not just us
wings and the China loses and vice versa
so that's like the one huge
misconception which we should have we
should really correct this otherwise I
just feel this kind of a confrontational
stuff and by the way guys they learn how
we humans treat each other and how we
treat our machines so we better this is
the time to reflect on ourselves so
let's let's be really you know don't
confrontation don't be confrontational
don't use thick with each other and then
don't do that towards AGI is what
they're gonna learn right so interesting
so you commented about learning from the
Cold War and I wonder how would you
translate that into one misconception
that you think you could change or you
should change in the race dynamic
yeah so I think first I mean so the Cold
War is sort of about as bad and hostile
a competition as there is so the that we
were able to succeed
in building forms of risk-reducing
cooperation during that period suggest
that we should be at least as ambitious
as our counterparts or during the Cold
War I think one misconception is that
right now people at least in the US
policy community really are not cracking
AGI as a topic they're just they're not
enamored with this topic so you in
trying to find ways of engaging with US
policy community at least and maybe
others could talk about other countries
policy communities
I wouldn't frame this in terms of AGI I
mean I think I could have unintended
consequences anyway but I would cream it
in in terms of other things that are
more concrete and accessible
you know like nearer term AI security
issues that may overlap with some of the
longer term AGI safety issues that I
think is a framing that's much more
likely to capture the interest and
support of policymakers in ways that are
constructive and not distracted just one
thing also that Yan was mentioning a
years back AGI is more and more accepted
in in the global discourse now you find
it more in that you know also in public
policy although it's not I haven't seen
it really in any of the AI strategy
documents they've in rule written by
various government so one thing to do is
might be also to educate parents and
policymakers about that we're talking
about guidelines on policy makers so
that's something we could include there
also so it's also educating them on the
notion of value alignments and the why
it's important there so that's something
I would that bring AGI out into the
world
whispering Charlotte yeah I want to kind
of get off of your points um this might
be slightly controversial I'm not sure
I'm gonna say it anyway um I think when
we discuss governance of AI and
collaboration and cooperation we need to
be very certain and quite honest with
our
selves do we mean cooperation between
the nations we think right now are
racing and winning the race
so China in America basically or do we
mean global corporation or do we mean
cooperation between all the large
nations so Canada Australia that
European Union China Japan etc etc etc
because the mechanisms are going to be
drastically different that we need to
employ in order to achieve cooperation
between two countries five countries the
entire globe so I just invite people to
like actually think my kind of
disentangle what we mean when we discuss
cooperation for artificial intelligence
I mean I mean the reason I use your
spanner is pretty easy that's what the
key robust project I'm thinking about
I'm sorry that would be that's a global
institute not just China has it
Japan has it and then you know Western
country all have it right so the aging
population so so I thought that would
there will be in you know kind of a less
sensitive project for but but I'll be
great Global part to be honest so that's
my we may be education we have a very
short time to take audience questions so
quick no statements actual questions
[Laughter]
fourteen standards working groups we've
had them for three years they're open to
anyone to join it's focused on things
like facial recognition multiple things
on personal data and standards are a
form of soft governance they're a way to
build consensus before you get the legal
place so anyone's welcome to join those
I wouldn't ask about China a lot of
feedback we've beginning on our paper we
did three draft the second draft a lot
of people said wow this is Western and
what was great is Shannon valor if no
one's read the book technology and the
virtues I think it's one of the most
important books to read in the space
today at least in terms of short term AI
and what she talks about is that
utilitarianism because you said this and
in your pudding comment utilitarianism
is really about engineers
let's build the elevator so it doesn't
crash and that's kind of important
then deontological stuff is what you're
saying and here a lot of things the
philosophers in the room are not one but
I can hear them on my voice dang we say
things like let's do the right thing
let's do the good thing that means
nothing in a room of philosophers
because you have to say what does right
mean for who virtue ethics as Shannon
describes brilliantly in her book is
this sort of way of staying we actually
have to find those things not care robot
but those things were when we agree on
them together in unison then the good is
actually a unison thing also in the east
virtue ethics seems to be something
through Confucianism this is the
statement does all that make sense
by the way I can't speak on behalf
country speak on behalf of myself so I
would say a lot of it does make sense
but I don't know how that's gonna be
applied in fact I've been thinking hard
about the ethical framework for
regulating up or regulating humans and
machines you know future references I
was thinking about something like a
compassion ethics and so the compassion
ethics is you know in a way that there's
so much we can draw from our from our
tradition and the Chinese and
Confucianism and then you know and then
and also its peak about people or agents
in the power relationship when you say
that compassion you know you have
parents towards children you have you
know people in a power position towards
their weaker ones and they saw those
rich resources in Chinese philosophy
about how that relation should be
regulated what's the people the people
in power position the kind of
obligations you have towards the weaker
ones and then of course you know so I
think that would be very useful
framework at this point I mean we could
use that framework at this point humans
still ruin in the world appears to be
anyway and then you know we use that
framework you know to see it regularly
how we treat the animals and the weaker
beings hopefully in the future
AG eyes will learn from from this and
then if they become more cognitively or
all this level they're more powerful now
they're gonna learn they're gonna be
compassionate towards us so we all have
our space space in
the universe so I was just thinking
about an actual so had discussion with
that with the e about that as well but
it's still evolving so have a nice come
up with you okay thank you very much for
joining us and I hope you enjoy coffee
[Applause]
[Music]