Near-term AI security risks, and what to do about them | Shahar Avin | EA Global: London 2017

our next speaker is Shahar Avene Shahar
is a researcher who examines challenges
and opportunities in the implementation
of risk mitigation strategies
particularly in areas involving high
uncertainty and heterogeneous or
conflicting interests and incentives
mixing anthropological methods and
agent-based modeling Shahar works with
other researchers at the center for the
study of existential risk and others in
the x-rays community to identify and
design opportunities for impact he
completed his doctoral thesis which was
called breaking the grant cycle on the
rational allocation of public resources
to scientific research project at the
Department for history and philosophy of
science at the University of Cambridge
under the supervision of Professor Tim
luhan's and dr. Steven John Shara has
also worked as a software engineer both
in a large corporation and at an early
stage startup in Israel and in Cambridge
to speak about AI security risks please
welcome Shahar Aviv
[Applause]
so feel quite lucky you're getting to
see a preview of the report that if we
stick to a schedule will come out in
less than a month on preventing and
mitigating the misuse of AI the scope of
this report is
people doing bad things with technology
that is kind of odd on these stages or
can easily be imagined so within the
next 10 years all right so not the more
long term issues of AI wasted we are
familiar with
most of the work has been done by miles
bondage at FHI but he is outside of the
UK so you have me to talk about this and
all of these people are co-authors on
the report this is me form lots of
places this is slightly unusual for n
research output and we can talk a little
bit about that but why are you here
you're here to
learn about AI misuse risks
and kind of how big is it is it a
neglected problem should you do
something about it how scary is the
world and I'm gonna tell you that it's
somewhat scary
but there are plenty of things we could
do about it I also want you to give you
a little bit of information about how to
play advertise those types of risks
relative to other risks because you
Hales and I've been told that's kind of
useful for you and finally I wanted to
showcase a little bit how do we do one
type of research project its Caesar and
FHI how do you do access research there
are many types of research this
particular one which is falls under
expose elicitation well you have a big
question you bring in a bunch of expert
to talk about that question and you try
to write a comprehensive report about it
hopefully to make some change in the
world we can go into the details of how
that project went and we can discuss is
this useful is it something that you
might want to do one day for another
area
so let's position this a little bit the
AI risk landscape can be broken down
not comprehensively into accident risk
that is the system not doing the thing
we wanted it to do and people suffering
or dying because of that and misuse the
technology is kind of working as
intended but people with bad motives use
it to do various bad things in the world
and cause harm or suffering we will
specify this as safety and this is a
security notice that this is not
comprehensive there is also for example
systemic risks well the system is doing
roughly what's intended and no one is
particularly aiming to cause anyone else
any harm but the systemic interaction of
lots of systems being deployed puts us
in a somewhat dangerous space it's a
little bit accidenti and little bit miss
you see but it's out of the scope for
most of the deposit have been done in
this space
so for example technological
unemployment would not naturally fall
into neither safety nor security but is
risk
also we can break things into short term
and long term there's a nice quote that
says long term is far away until it's
not but I'm gonna use it as short term
being we can kind of imagine what the
system is gonna look like by looking at
the R&D landscape now we can kind of
look at the papers look at what people
are deploying
make some sensible assumptions about
where we're gonna be five years out or
ten years out we could be wildly off
because we could get black swans coming
in and changing radically well we are in
ten years
but maybe some scenarios we can start
painting now we will actually bail out
in five or ten years that drops
massively for something like 20 years or
40 years so long terms would be beyond
our ability to predict fairly accurately
but the technology landscape looks like
but still is really important because we
can make some general statements about
how this domain is developing so it used
to be the case that accident long-term
risk was massively neglected especially
in respectable a research that is seems
to no longer be the case because of work
of people in this field which is
wonderful and I think kind of the
definitive publication that people look
at and say this is really good we can
now talk about it sensibly was Postum
superintelligence so 2014 we kind of
took a stab at that quadrant and made it
better
but then people said well but that's
kind of very philosophical and very far
away we can't connect it to their
systems that we have now we don't know
how to do technical machine learning on
it so Dario Emma died and Ola and a
bunch of other people came together and
we said well look here are concrete
problems in a safety and that's kind of
a very good paper but a bit longer so if
you want to consume it in a more video
engaging way what miles has a series of
videos about this on YouTube that I
sounded comment and then we were like
okay so we start to have some of this
and we can kind of start building a
talent pipeline in today's and people
can go in walk for did mines I'll go
look at a guy on technical area safety
problems and
misuse risks started to become a bit
more neglected so if I tell you let's
have a big report on what are and what
can you do about shorter misuse risks so
hopefully this will say 2017 if we are
very bad it'll say 2018 but it's
definitely coming and I don't want to
talk about this puzzle install because
it makes me very sad but at some point
someone should write that as well
so this is like the very short version
for years what's the scale of the
problem
it mostly looks like much worse versions
of things that you've already familiar
with so a cybersecurity attack taking
out critical infrastructure or making
lots of people whose lives miserable
because they can no longer connect to
the computer or get help or move around
is a version of the cyber scuba dress
that we have only familiar with the more
connected systems they're all the more
digital security becomes important
machine learning can be used to make the
domain scarier
so it's global and it could lead to
catastrophe but it takes an existing VCR
and makes it more likely and a bit scary
oh and the same for applications in
physical security and the same for
applications in political security
tractability
we think that there are several
important things that can be done today
particularly when we look at the entire
domain together which is what I'm doing
the report it seems like some areas are
more important than others and more
urgent than others and we'll go into
what they owe an uncrowded nurse there
is lots and lots and lots of work on
specific areas where machine learning
might be used might be misused and what
should we do about it cyber security is
one of these domains the governance of
this lot of most weapons is another one
of these and fakeness is definitely one
of those
but there seems to be things higher up
the chain that are more general
comprehensive or that are harder to do
because the illegal coordination problem
or their technical research that people
don't necessarily know is relevant to
this problem that those are much less
crowded and we would like people to do
more walking soon if possible so without
these lists right let's kind of make
this more tractable again most of this
should be familiar to you if you've been
reading the news it's just it's bigger
than what you see on the news so
automated spearfishing okay is anyone in
the audience audience who doesn't know
what automated spearfishing is where's
your hat
okay so let's talk about we'll start
with phishing phishing is when someone
pretends to be someone else often on the
internet often so they can get access to
something that they shouldn't have
access to like your passwords a website
that you own or something else it'll be
part of identity theft or cadential
theft and then once they have kind of
sent you a link to a website that looks
a lot like a website that you tossed but
it's in fact not that website that you
trust and you put in your username and
password on that malicious website then
they now have access to your accounts
and can send emails on your behalf or
can move money from your bank account or
do other things but in lives I'm trying
to say hack into an organization this
may be the first step to get the account
of the administrator which then lets
them connect about yourself those and
download various points
spill phishing is
when instead of trying to hit lots and
lots of people's credentials but maybe
saying you have won a million whatnots
click this link to get all of your
whatnots
and that leads to a malicious website
and most people know not to click that
link nowadays but it's very easy to send
that ten to ten billion people instead
you find one person who you think it
would be really useful to get their
credentials and you spend some time
researching them and they're like oh
that person really likes model trains
we'll send them an advert for a website
that sells discounted model trains and
that looks kind of really good and comes
at the right time and comes from an
address that looks more reliable they
are much more likely to click that link
and then you get a credential something
you can do evil things with that and of
course that is quite labor-intensive so
we don't see that hitting almost every
one of us today but if you are quite
good at creating models of people from
the online behavior then you could
automate this process
this is really machine learning
discounting so in general in
cybersecurity we'd say that there is a
trade-off between
spray-and-pray methods that are very
easy to scale up to lots and lots and
lots of people but have very low
likelihood of succeeding and kind of
take other extreme and is advanced
persistent threat well you have a team
of people working on one target for many
months
kind of mapping out the network maybe
mapping out the use of behavior and
designing vectors to attack that target
and
there are genuine risks that cyber
security experts agree on that machine
learning can break this division so that
you could have these highly targeted
attacks on a billion people at once yeah
and that
example
now this is so cute so I was looking for
a graphic for drones with guns and this
came up and so I okay who made this what
is this this is a submission for an
innovation competition by a student in
India it's like well he was mostly
fixing about how do you make the kind of
balancing of the gun on the drone to
make it work and also described in his
proposal is that the drone has face
recognition capability not shown on the
diagram so we know we have face
recognition capability my facebook is
but ago did it like to make that happen
from a drone that's moving sounds how
but in fact there our commercial drones
that would kind of lock onto your face
and move around with you they are for
reporters who are covering fields in
using the field like a demo or such like
and we have it raw leeks and we have
guns in some countries more easy to
which than others
not super hard to imagine that you can
put those things together the machine
learning part is of course the face
recognition and also some amount of
control
we know that there are drones we know
that they are
somewhat bad but also somewhat good if
you want to do if you want to find
guerrilla and you think that they will
never get hold of them but of course we
now have evidence of both Isis and
Hezbollah using drones in combat by
repurposing commercial technology and it
just seems that you can massively scale
up your capacity if you use machine
learning so instead of one person one
drone which is kind of demoed we used to
you can have one person a thousand
drones so one person a million drones
right moment you ml is able to translate
high-level commands into all of these
now go and target all of these people it
gets really scary
it's caveat
so also apparently kind of democratic
societies and also not democratic
societies are somewhat susceptible to
what people believe and that can be
affected by the channels in which they
consume information about what happens
in the world around them and if you can
inject falsities into that world or just
highly emotive things then that could go
badly for all sorts of ways know that
we'll have a happen in our world just
hypothetically
but this is limited by the time it takes
to craft the messages and by targeting
so if you can find the people that you
need to target automatically and if you
could craft messages that look authentic
say videos and audio that look and sound
exactly like a real person maybe a
president of a country maybe a CEO of a
company and it's extremely hard to tell
this is a forgery then the world
suddenly becomes much much harder to
coordinate the good people to do the
good things
technical terms so
what does what do all of this look like
if you go one level up what if you ask
machine learning can be applied to all
of these domains by bad people for
fairly low cost all right in fact we'll
really driving hard on making these
things low-cost and accessible by
everyone be using kind of online courses
and
platforms that allow you to deploy
machine learning modules and kind of
rushing to publish everything that you
can because and that's good right we
want more people to do machine learning
and use the technology to do good things
in the world but it also means that the
space of possible attackers or miss
users is going
so let's break this down kind of and
this would apply across the domain so we
have we've analyzed the digital domain
and the physical domain and the
political domain but across these
domains we are expecting to see novel
attacks and specifically of two types
one type is
if you can now go superhuman in an era
where you have an ever way I system that
is just able to do things better than
any human previously could do for
example make a hyper-realistic video
it's something that's very hard for
humans to do but you could train a
machine to do then this is now new in
the past no one could do this attack and
now you can do this attack another thing
that's very important and maybe even
more scary is if there is now a machine
learning system operating in the world
and you can attack that machine learning
system that's new that did not exist in
the world before machine
so data poisoning and adversarial
examples I'm going to be things that are
going to become very scary I don't know
if you've seen this recently that
someone had 3d printed a turtle that to
a machine learning algorithm looked like
a gun from every angle that you looked
at it and similarly some stickers that
you can put on a stop sign that would
make a machine and classifier mark it as
a 45 miles per hour speed limit on
more attacks
this is mostly through
kind of scaling by automation if there
was something that previously would take
me a lot of time to do but now I can get
a machine learning model to do for me or
I can get some of the pipeline for
machine learning model to do for me then
the cost from going from one attack to a
billion attacks has gone down massively
so again advanced persistent thoughts
which is something that we use to see
only against very large corporations and
companies could now happen against
individual users or SM is more tackles
just means more people who are able to
do bad things
no targeted attacks we talk about
customization we can build a profile of
the target and kind of automatically get
suggestions for who to attack first how
to defend against attacks if an attack
can kind of autonomously respond to
whatever the first defensive measure you
have in place then it becomes much much
harder to make defensive measures we've
kind of seen it with antivirus software
and we'll likely to start seeing it in
other domains as well and
already a big problem in Seibel but
might move also too well to some
something political less so in physical
at the moment how to attribute attack
right something has happened something
bad happen and you want to deter people
from doing it again you kind of need to
find the person who attacked in the
first place and punished them if it's if
you're not possible to go from the
attack to whoever carried it out because
there was an independent agent out there
in the world who is performing it and
you cannot go from the code whoever
deployed it
then you cannot punish if you cannot
punish you cannot detail
we just means more people willing to
carry out attacks I mean there's a bunch
of other stuff that makes it harder to
punish even if you cannot be used like
this thing the fact these things are
global and the fact that these things
happen at this speed of quick
telecommunication
okay are you scared yet
let's talk about what to do because well
it does a whole bunch of things you can
do right now
so if this is the pipeline there's some
things you can do with machine learning
some bad person is using those
capabilities to do something bad to you
or to someone then what you can do is
first figure out what this pipeline
looks like because currently kind of I
can put point at some examples but I
don't have the full list of things that
bad people can do if they can make money
off it then they are much more motivated
to spend time figuring out what they can
do then I do I also work on other things
and also if I knew all of them might
maybe a probably shouldn't tell you
because then you could take this
information and go also and it's
recorded
so there is a difficulty but you kind of
want to be in a position where the good
guys know what it is that the bad guys
can do probably before they can do it by
forecasting the technology and running
some simulations
then you can try and prevent the tackles
from gaining access the capabilities as
I mentioned machine learning is now very
much a field well all the capabilities
for everyone all the time and maybe it's
time to start rethinking that particular
meme
and finally we can use technology and
institutions to defend victims because
we know that they're going to be subject
to more attacks
so here are some top interventions miles
reminded me to tell you that this is
very tentative because the report is not
final yet and and we are still churning
through what is most capable and most
tractable and so on but it seems like
something along the lines of
publication risk assessment and security
sharing so a little bit more like what
happens in cybersecurity right you
realize that you can hack into someone
else's system you don't immediately go
and publish it you kind of contact the
vendor or give them some time to patch
once they have patched you publish if
they don't patch maybe then you publish
to kind of put pressure on them unless
you think it's gonna be really really
bad in which case you still don't
publish
so we think that there are some ml
capabilities or some deployed
capabilities that should just not be
made public it's probably only a very
small percentage of all capabilities out
there in the world and it gets more
difficult because you can have some
capability that's very generic in only a
very small Delta for me to something
that can be misused nonetheless we could
do a whole lot better than this is not a
problem
next up is
formal verification and hardware
security
these are tools that make software less
hackable people have been helping on
about them form at least the sixties but
it's been it used to be really really
really how to implement this in any
meaningful commercial sense this no
longer seems to be the case and we just
want all people who are developing
software to at least know that this is
an option and if people are in charge of
critical infrastructure really they
should go to implementing those and this
would also apply for advanced machine
learning systems
leverage existing centralization this
one is tricky I will mention the fact
that it is tricky a
general thing that we might see is that
the ball becomes more dangerous out
there in the wild the people who are
able to protect you
their companies and maybe a few
governments and so everyone moves to you
those offices and you get even more
centralization of data which means that
as long as they are good everyone is
somewhat protected if they ever stop
being good or if someone inside them is
no longer good then their capacity to do
something bad has gone up massively so
this is with a suggestion with a pinch
of salt but if you have a service and
you have once on the last place where
you get it say face recognition rather
than everyone being able to deploy it in
the home then you could apply terms and
conditions which means do you have at
least some legal recourse when someone
is misusing it you can rate limit how
much they are using bit soft well you
might ask them to identify who they are
before they use it there is more that
you can do by preventing access
continuous red team effort everywhere
the world is fairly broken there isn't
that we don't know it is that it takes a
lot of effort to break it
some companies are providing these kinds
of services some governments are earning
these kinds of services by trying to
actively hack and I'm giving a lot of
examples in a digital security domain
but you could easily translate these
into the physical and political domains
as well try and break systems see if you
succeed if you succeed tell the people
who are running the system how to make
them better this doesn't take that much
effort it just needs change of mindset
and we need lots of good people to go
and try and track things and it kind of
needs the institutional protection so
that if you break something you get good
reputation and some money rather than
you go to jail we solved it for
cybersecurity we need to solve it for
other domains
and we need even more people working on
how do you use AI to defend people in
cyber and information domains right so
they'll do seem to be technical fixes
both for cybersecurity especially with
satellizer and for fake news for use of
a better term we just need to have them
developed and deployed probably
yesterday
how does this link to otherwise domains
so of course there are digital elements
to biosecurity and nuclear security and
systemic risk
the clear links to political security
and bad governance alright all of these
can scale what seems like somewhat scary
into really really scary
it is time to establish norms in the air
research community around maybe not make
everything public all the time maybe
think a little bit more about how your
technology is gonna play out in the real
world this seems like it will also be
useful for accident risk and long-term
risks
capacity building in policy kind of
these technology is moving very fast
it's very hard for people whose day job
is filled with emails and shaking babies
and kissing or something to catch up
with everything but it's important all
right so we need to find ways of
employing small technical policy or in
people in government to welcome these
and it will also help with some of the
long term risks we think and it's really
important for the ex's community to
start getting a reputation for only
crying wolf if and only if there is a
book so it's really important that we
don't go take to any one of these
particular points and say this is
terrible this is gonna end humanity it
won't but rather this is a very
considered view of what this looks like
these are the races of the benefits this
is what we can do taken not particularly
a lobbyists position on this so that
they will keep coming back to us when we
say actually this thing will kill it
okay
little bit some I was told I'm allowed
one joke so this all started from a
workshop that we held in Oxford that was
called the bad actors workshop bad
actors being a term of phrase meaning
people with bad intentions
this is Schwarzenegger in Hercules in
New York he is clearly in that movie a
bad actor okay
so how did this happen basically we said
this is one eclectic quadrant of the AI
risk domain we need to have lots of
expertise because there is expertise on
this compared to very long term things
we just need good generalist with good
plains this is happening now and we need
the cybersecurity people and the least
autonomous weapon policy people and the
people who are studying the use of
drones by terrorist organizations and
the criminologists so we got them all in
a room for two days
we had technical machine learning people
as well and we asked
what scares you the most and we got them
into small groups to talk about what
scares them the most and they got very
scared and we got very scared that was
great and then we send them out to
dinner and have some drinks and talk to
each other and get to know each other
and then we bought them in following day
and said okay what can we do and then
they came up with a very long list of
things that we should do and then we
prioritize them and talked about them a
little bit and then we decided we're
gonna have a report which is what I said
is gonna happen soon which you're
getting a preview of this is roughly
what its gonna look like you should kind
of get it when it comes out and read it
and talk about it and make youtube
videos about it
and hopefully this will lead to some
real change right so hopefully we can
get some meaningful take-up of these
results in the machine learning
community particularly on the nodes of
sharing and responsibilities about how
they might be misused within the
cybersecurity community that already
getting to know a little bit that cool
things are happening in AI then they
should get more involved but we think we
can massively scale this up
national policy of getting into this as
well the Americans in particular whether
you're happy about this or not is up to
you but capacity-building needs to
happen and there are important things to
do now international policy is slow but
a lot of these things will require
international governance and kind of
talking to the media and the public so
either we don't manage to fix everything
and everything is super scary in which
case we don't need individuals do
anything because it won't help oh we are
super super amazing and we fix
everything big security into the
technology so that individual users
don't need to do anything enter safe but
there was a big chunk in the middle you
just need to take sensible actions
during the day-to-day life and for that
you need to know what the risks are and
what good behavior looks like and we're
doing a fairly bad job of it in
cybersecurity and online information and
it seems like the stakes just got a lot
higher so we also need to address that
and like who owns making all of this
happen seems like it's gonna be partly
us and partly you questions
[Applause]
all right you can submit your questions
you know the drill Londyn be a global
org slash polls Thank You great talk a
bunch of questions already have come in
maybe starting just kind of from the
premise that
bad actors or bad organizations are
gonna have the resources to do all this
stuff the first question we got is just
how how confident are you that that bad
people will have the computing power the
machine learning phd's the the resources
in general to do the bad stuff that
you're talking about
so they're kind of DEFCON presentations
on the applications of machine learning
to
so cyber security at least is one laptop
and these are people who have no
background in machine learning they just
picked up some kind of tutorial online
and implemented something in terms of
load and it walks
you can
because there is so much trying to get
people to speed up on machine learning
because it's a growth industry and you
need lots of people who know machine
learning there is not a lot that you can
do with not a lot of resources and not a
lot of training
this kind of hits more on the proof of
concept target individuals target SMEs
which is scary but maybe not the most
scary
do I think that kind of the most cutting
edge capability stuff is going to be
directly translated into misuse probably
not
though there is one argument that says
look at the economy try to apply machine
learning somewhere within that economy
to make you more money that you can put
back into more machine learning research
it's much easier to apply it to taking
value that has already been created by
others than it is to creating more value
I mean advertising is also a little bit
in this area which is what we see a lot
of machine learning happening and
cybercrime also seems to be one of those
domains but this is kind of laughs
sketch argument that could easily be on
so maybe just a very practical question
for those who are not experts in the
domain themselves what are the kind of
handful viewed a handful of the core
measures that people should take to
protect themselves
what a wonderful question um
so
to protect yours well patch your
computer right I mean to the most recent
version of whatever it is that your OS
and your Bazo is the basics right and if
the NHS did it then Wanaka would not
have been such a big thing
[Music]
don't open links that you shouldn't open
this is gonna be much harder
so kind of title check for identity
rather than content if this looks like
the kind of thing that I would click on
is no longer a good test for whether
this is genuine or not but is this
supposing that I trust to send me this
thing is better and this might take more
kind of searching for what the person is
but
yet could make it more safe
but some of the worrying things almost
is tarek right so you are not likely to
be a person targeted by a face
recognition drone because you don't live
in one of those countries for the people
who do live in those countries maybe I
have some suggestions but it's pointless
talking about them now I
think for the people in this room the
more important thing is to get involved
in making the system safer and I've kind
of given you top 5 but I can show you
the list of all possible interventions
if that's interesting and you could just
pick the ones seem to speak most if
for the group that you're kind of
representing here with this report how
mainstream is that group visa vie just
kind of the center of
IT security in today's world do you
mention DEFCON for example so the
question asks you know when to talk like
this be welcome at DEFCON would it be in
the mainstream or would it be sort of
far a fringe in the kind of biggest
conferences around these issues so this
was leading one to us we got
really serious suspected cybersecurity
people to the workshop not all of them
are named on the report but
yeah that kinda list is going to be I
mean we dine it on Chatham House so I
cannot attribute any position to an
individual but
Sanderson was the then lowly was the
Eden show hot was though so I mean we
had people who go to DEFCON kind of
frequently and get everyone come to them
and say hey you got that guy that's cool
which was important so I think none of
this is going to be particularly
surprising to the people that have come
just they won't accept it as a talk
because it doesn't have us breaking into
things and making lots of mayhem but
this is fairly mainstream thinking and
Afghan I think they just it'd be made
more well that there is so much to be
done in air and AI you suck so much of a
hot area anyway that's I think it's
gonna be very low barriers to getting
much more attention to this book
you mentioned that one possible path for
the future is centralization into a few
kind of core clouds or you know certain
companies that can keep us safe how
would you evaluate the the leading tech
companies today in terms of how they're
doing in preparing for a I risk Google
Facebook Amazon Microsoft Apple Apple
just rolled out face recognition on
probably what will be a billion iPhones
over time the question is which type of
AI asked if you looking at
short term issues so how good are they
on cybersecurity
surprisingly good better than academia
definitely better than government
Google I think of pushing the envelope
on what is good security practice and
are looking forward to new threats that
are coming online I could probably say
similar good things about Apple and
Microsoft
as well well varies I definitely can say
good things about Google
but I would like to see those results
showed more widely not just to getting
an account with those companies but
getting the expertise sure this will be
sharing this so that smaller vendors can
also buy them but but it is also very
resource-intensive and there are
economies of scale you can just you know
you can hire a team of a hundred people
to do it to the response if you're
dealing with a billion accounts you
might not be able to do it if you run
Inga 500
another question about kind of the the
plausible paths of the future
often AI progress is kind of analogize
to an arms race where everybody's just
kind of competing to have the best and
you know a small edge could sort of lead
to a winner-take-all scenario do you
think we're sort of destined for that
version or do you see hope that we can
sort of solve that and get into a more
kind of sane
you know more moderate progress sort of
future yeah
so that is what can be done right on
this idea of beneficial yeah yeah I feel
good
you have no advantage on not sharing
safety practices with others everyone
benefits if everyone shares safety and
invests a lot in safety and these are
ideas that are being put out there quite
a lot and several companies are
receptive to them
farther down the line it's how to do
related question do you think that we
will need and and ultimately have some
sort of global surveillance paradigm to
try to watch after everybody and and
identify bad actors I
guess it depends a lot with how your
capabilities all these to be right so if
you think that everyone should be able
to deploy extremely powerful technology
then that probably means that those this
folder I mean if I'm allowed to to have
the capability to do something really
really really bad and should probably be
some mechanism to make sure that I don't
do something really really bad with the
technology if I'm not given as much
power then there is much less need to
surveil and this is a choice of gonna
make a society and probably the choices
that we're gonna make is society's
for somebody who's interested in getting
into this area professionally do you
think that government is a good place
for them to go or would you think about
academia large tech companies or other
dependent to do so we think there's a
lot of technical research it needs to be
done on securing systems on analyzing
the economics of various threats
academia and industry are probably the
places to do that I mean definitely in
terms of the top teams and the top
talent and being able to run experiments
on the other hand there are definitely
institutional and legal fixes that we
will need to have in place and the
government is the best one to do those
kind of a change of pace you originally
divided risks into near and long-term do
you think those are highly correlated in
terms of the way that those risks
ultimately are realized or not or are
they sort of more independent where the
the near-term outcomes don't tell us
much about the long-term outcomes and
then depending on your answer there what
do you do you think people should be
focused more on near or long-term I
think there's a lot of heterogeneity and
a lot of disagreement about how things
fall within this heterogeneity so I
think
if you think the long term systems are
going to be
blame children of the systems that we
have today or gonna have a lot of
components that are made up of the
systems that we have today then securing
the systems we have today is gonna
translate directly into the long term
systems you could also say that without
training in the knowledge that is
required to secure present systems
you're gonna be the disadvantage
compared to people if you have the
knowledge about two thousand eight
systems then you just get a really good
training about thinking in certain ways
then walking on present-day systems is
gonna translate kind of epistemic lis
into thinking about long-term systems if
you think that there is kind of a deep
difference in what it takes to secure
short on systems and long term systems
which some people hold and I think is
there are arguments to lead you to do
that then you probably want to kind of
just work on the long term in that
particular framing I think there is
there are enough problems and not enough
people that just find the one that you
kind of feel most passionate and capable
to do and there wouldn't be a place
little that's all the questions we have
through the app right now you're gonna
be doing office hours at the next break
correct tell us where we can follow you
online to keep track of your work and
learn more just go on the caesar website
cser dot AC dot uk' and also follow the
FHI website and the FLI website
you will stay up to date through those
channels for helping us understand AI
risk in the near term round of applause
for Shahar Avene
thank you very much great talk


The technological landscape of affective AGI (lightning talk) | Daniel Eth | EA Global: London 2017

hi so really briefly what is artificial
general intelligence also known as AGI
it is it's defined differently by
different people but it can generally be
thought of as artificial intelligence
that smarter than most people in most or
all areas of intelligence and the
development of AGI would be expected to
be hugely impactful to the world this
can be seen by analogy to when humans
came onto the world stage we affected
the world more than probably any other
species ever due to our high
intelligence and then even higher
intelligence in AGI could be expected to
potentially affect the world even
further so notably many experts expect
that AGI will be developed at some point
over the next few decades and in
addition to that there are several
different paths to AGI and they don't
all necessarily lead to the exact same
outcome so if there are different paths
that we can choose to take and the
different paths will have different very
large impacts that means that the
choices we make and how we pursue AGI
could be very impactful choices in my
work here I investigated three major
paths to AGI and compared them for
safety considerations first I looked at
de novo AGI and that's AGI that's
developed from scratch so if you imagine
a group of computer programmers who have
a series of insights that leads to AGI
the second is neuromorphic AGI the idea
there is AGI that's Bill on based on
principles that we discover from the
human brain from neuroscientific work
and the third is whole brain emulation
the idea there is to emulate a specific
human brain so if you took someone's
brain you scanned it you translated the
scan into a model you ran the model on a
computer and presumably also put the put
the
gave it a virtual body that it could
control and a virtual world to interact
if done correctly then the emulation
would be assumed to act similar to how
the human whose brain was scanned would
act in a similar situation
so there are
previous work in particular by Nick
Bostrom has argued that neuromorphic AGI
is the least safe of the three and
whether de novo AGI or a whole brain
emulation would be safer is an open
question and there are several arguments
on both sides of this in my work I came
to the conclusion that whole brain
emulation would be safer to achieve
first but there is a caveat here this
caveat has been pointed out by several
other people and that's that pursuing
the whole brain emulation might actually
lead to getting there amorphic AGI first
so if we pursue what arguably is the
safest we might end up with the least
safe I think this makes sense but I also
think that as it stands today the vast
majority of work that leads to one of
these three types of AGI is not directly
pursuing one path but instead it's it's
focusing on underlying technologies
and so we have to consider each of these
technologies separately for whether what
the effect of trying to advance them
would be so from that I developed a
technological landscape of the major
technologies affecting these three types
of AGI and also how they affected each
other and the underlying technical
trends I looked at were AI research
computational hardware nanotechnology
research nanoscale neural probes and
neuroscience and
one of the main insights I had was that
I thought that nano scale neural probes
would increase the chances of getting
whole brain emulation first as opposed
to neuromorphic AGI first and
the main reason for this is that there a
lot of the information processing that
happens in the brain actually happens on
the sub cellular level so within the
neuronal bodies within the dendrites
within the axons and so on and so if we
want to have a model that's high
fidelity enough to allow for whole brain
emulation will likely need to be able to
probe this level the nano scale and very
large-scale distributed manner and in
vivo and
nano scale neural probes are the only
foreseeable technology that could allow
for that on the other hand neuromorphic
AG I might not require that level
fidelity so the conclusion I came to is
that pursuing nano scale neural probes
could be good from an AI safety
perspective because it could increase
the chances of getting whole brain
emulation first while also possibly
decreasing the chances of getting
neuromorphic AG I first thank you
[Applause]


The Open Philanthropy Project's work on AI risk | Helen Toner | EA Global: London 2017

morning everybody
is that working yeah okay um
nice to see some more of you I think 10
a.m. maybe was a bit ambitious for some
so this morning it's my great pleasure
to introduce Helen toner Helen toner is
a senior research analyst at the open
front group philanthropy project her
work focuses on policy governance and
strategy issues related to progress and
artificial intelligence this is across
domains including geopolitics national
security public policy and AI safety
before joining apron philanthropy hadn't
studied chemical engineering and Arabic
at the University of Melbourne while
studying she led a Melbourne worked at a
finance startup Veszprem capital and
interned at the Boston Consulting Group
so please join me in welcoming Helen
hi my name is Helen I work on policy and
strategy at the open philanthropy
project as you probably know as well as
work on criminal justice reform farm
animal welfare funding scientific
research and some other things potential
risks from advanced artificial
intelligence is a major focus of ours
which is why I'm going to talk about it
today so I'm going to talk through how
we're thinking about this space some of
the things we've done so far some of the
things we think we've learned and also
close with some thoughts on how you can
get involved in this area if that's
something that you're interested in I'm
going to assume that you're basically
familiar with the open philanthropy
project with the basic case for AI as an
effective altruism cause area but I'd be
really happy to talk over any of the
more basic aspects of this at office
hours if you'd like so I'll be doing
office hours at 1:00 p.m. in the senior
common room so feel free to come and
find me there
just before I get started on these
things I wanted to briefly touch on how
I decided that artificial intelligence
is what I wanted to focus on in my work
I think it can sometimes seem like
people in the EA community have always
thought what they thought
but it seems valuable to me to tell
stories of people changing their mind
because I think that does actually
happen quite a lot and I think it's a
really important part of the effective
altruism community
so my story is that I was at university
I was actually planning on a career in
aid and development and
then I came across effective altruism I
made friends with the two organizers of
the EA Melbourne group and they pretty
soon started telling me about
existential risk artificial intelligence
I thought that they were firstly
philosophically confused and secondly
just over enthusiastic science-fiction
nerds and I told them so which they took
very well to their credit
uh-huh but I didn't want to just dismiss
them out of hand I wanted to understand
why they were wrong so I read more moral
philosophy than I had in my life which
wasn't hard because it had been zero up
till that point and I spent months
debating with them about history ethics
epistemology all kinds of things I don't
want to rehash that whole discussion
because it took several months and I
think there are better places for you to
go and read about the basic case for
thinking about existential risk or
thinking of artificial intelligence as
an ear cause area but I thought it might
be interesting to highlight a couple of
ways
of looking at it that were fairly
important to me and then I I don't see
obviously represented in other places so
the first one of these is that looking
back over history it really seems like a
small number of major transition points
so for example the Agricultural
Revolution or the Industrial Revolution
were you know had radically larger
effects on how people live and on how
civilization functions then on many of
the smaller changes put together and so
you know one implication of this is that
if it looks like something on the
horizon might you know have a reasonable
chance of being one of these major
transitions and it looks like there
might be reasonable work that you can do
to make that transition go better for
Humanity then that could be a really
valuable thing to work on even without
taking the existential risk angle into
it and the second thing I was just
looking at how likely I thought you know
this whole space artificial intelligence
machine learning how likely that is to
be a big deal compared to you know
relative to other things so relative to
how much how much attention and
resources it was getting if you compare
it to
you know other ways of doing good for
the world or if you compare it to the
amount of resources going into
straight-up AI research
you know the amount of time and thought
going into how to make sure that AI is a
good thing for Humanity is comparatively
tiny and so I think that means that you
can and I can think that it's a good
idea for more attention to be being paid
to this issue without thinking that it's
the only topic that matters all right
enough about me on to open philanthropy
so several of my colleagues at open
field also do some work on AI so Daniel
Julie is our program officer for AI and
he leads the technical side of things
Luke Mel has a nick Beckstead hold on
Karnofsky and i also all spend a
significant chunk of our time on AI
related work and the overall approach
that we're taking to thinking about this
space is based around the idea that it
looks pretty plausible that in the next
10 or 20 years something that we call
transformative AI will be developed
which is a term we use to mean an AI
system or AI systems that will have an
effect on the world of roughly
comparable magnitude to the Industrial
Revolution there's obviously a somewhat
vague term and they're a bunch of
different things that you could imagine
transformative AI looking like maybe it
means that we figure out a way to do
scientific Rd in an automated way or
maybe it means we have you know the
advent of automated corporations or
maybe you know AI systems get really
good at learning across many different
domains and we see something like a GI
but but so we frame our you know our
work on this around the possibility that
something of this you know level of
importance might be developed in the
next 10 or 20 years and we think it's a
long way from inevitable that that's the
case
but we think it looks likely enough at
this point so for example we said in a
blog post last year looks pretty safe to
us to say there's a greater than 10%
chance that something like this is
developed in the next 20 years so by
2036
20 years from last year and we still see
that as a reasonable estimate and so we
think that that's enough of a
probability that someone should be
thinking seriously about this and we
don't see many people doing that so I
guess that's where we come in so within
that framing of
transformative AI we tend to get down
into two main areas the first is on the
technical side so building AI systems
that work as we intend them to basically
so this means that they're aiming for
the objective that we want them to aim
for they're reliable in different
contexts that control a ball or
something goes wrong and properties like
that and we see a lot of disagreement
among AI and machine learning
researchers about exactly how hard this
is likely to be and how much extra
research is likely to be needed to make
this the case over and above the amount
of research that will be needed to build
systems that work at all
so we you know we we think it's valuable
to be continuing those debates and
continuing to try and get clarity on it
on how difficult this will be and how
much extra research will be needed while
also funding a variety of approaches to
hopefully I mean you know increase their
probability that that the systems that
are built work really well and then the
second area is non technical so even if
the systems do work exactly how you want
them to what happens then if you you
know if you do have AI systems machine
learning systems that could create
massive changes in the world
you know who should be able to use those
systems what should they be able to do
with them who should they need to
consult with first those kind of
questions and then in both of these
areas the common thread for us is that
the main thing we're aiming to do at
this point is to help build fields to
help set things up to improve the
chances that the transition to a world
that contains these kind of systems in
it you know goes well and so by building
fields I mean that we're aiming to
support the development of the kind of
ecosystems of people thinking about
these topics and building on each
other's work that you see in existing
fields like particle physics or
sociology or whatever
and so we think that this kind of field
building work is something that
philanthropists have a decent track
record record of and so that's that's
part of why we think it's a good idea
for it to be what we are focusing on and
you do run into a kind of chicken and
egg problem when you're doing this kind
of field building you know does it make
sense to start by funding senior people
who can take more junior people under
their wing or should you start by
funding the junior people who then go
and find that they're you know
supervisors or how does that work and
I'll get back to that chicken and egg
problem a bit later in the talk okay so
and what we've done so far
on the deployment problem which is the
you know policy strategy side of things
I would say we're still at a pretty
early stage of figuring out how we want
to approach this space we have made a
few initial grants including funding the
future of humanity Institute at Oxford
and we've had some conversations with
people in government academia I think
tanks about how they're thinking about
the space and and how they want to
approach it I would say a big thing that
we're learning is just that there's a
huge number of potential questions and
potential approaches to use in this
space and it obviously doesn't come for
granted that people were talking to
share enough of our assumptions or
values to land on the same questions
that we think are most important for
example the the transformative AI
framing and it's also becoming really
clear to us that there's no obvious sort
of home field for this for this work and
where it should slide in you know is
that Technology Policy is that
international relations is it law is it
cybersecurity and so that also makes it
hard to know where to target out
conversations
the grant that we've made in this space
that I'm most excited about is to
Professor Alan Defoe who's an
international politics professor at Yale
and at the future of humanity Institute
at Oxford and we're lucky enough to have
him giving a talk right after this so if
you're interested in this side of that
of the AI space my main recommendation
is to stick around and hear what he has
to say on it on the technical safety
side the thing where most aiming to
support right now is the development of
a subfield of machine learning that is
focused on safety so two areas that we
think are especially promising for this
firstly reward engineering so basically
figuring out how to convey to the
Machine what exactly you're hoping for
in a way that conveys the the nuance of
what you want and doesn't let the
Machine doesn't end up with the Machine
cheating or misunderstanding you in some
important way and then the second area
is reliability so building systems that
will work in different contexts that
weren't malfunction in important
situations or if they're put in a
context it's very different to what they
were trained on that type of thing I
think I'll work on starting to build
this field is going pretty well
our approach to solving the chicken and
egg problem here has basically just been
to fund everything at once that we can
find that we think is great so we ran a
closed requests for proposals where we
worked closely with professors at some
top machine learning research groups -
and AI research groups to you know
develop plans for them to do work on on
AI safety in their groups and and
develop those plans so that it was work
on AI safety that we were excited about
and that's resulted in grants to groups
like Stuart Russell's Center at Berkeley
yoshua bengio is group in Montreal Percy
Lee airing at Stanford and uncle Dragan
and Sergey Levine at Berkeley
and we also just on Friday closed
applications for the first year of our
fellowship which we're running for
machine learning PhD students where we
will find top students who we think have
good ideas to do work on safety and
through the fellowship we'll also
introduce them to each other introduce
them to the AI safety community and
start to build a network that way we've
also supported workshops at major
machine learning conferences as well as
running a workshop of our own which also
helps to create these links between
researchers and help them critique and
build on each other's work which we
think is really important and lastly
we've also made a few grants in this
space that don't fit quite into these
categories so most notably the two would
be a grant to the machine intelligence
Research Institute in Berkeley which is
doing AI safety work but within quite a
different paradigm to the type of
machine learning safety that I'm talking
about young and we also have a
partnership with open AI where we're
working closely with them to shape their
work on on safety and governance in
terms of things we've learned I think
the biggest thing on the technical
safety side is which is not so much a
new thing as something that has been
really confirmed for us in how important
it is is that it's really important and
helpful to have agendas of problems that
people can work on so an obvious example
of this is the paper concrete problems
in AI safety which came out in 2016 and
was kind of a first real attempt to
translate people's concerns about the
longer longer term safety of AI into
concrete and attackable machine learning
research problems so this one paper
seems to have been really helpful in
giving machine learning researchers the
feeling that there are specific problems
that they can work and publish about in
this space rather than seeing it as kind
of an amorphous blob of of you know
worries that people have that it's very
difficult to tap into okay so what does
this mean for you if you're interested
in technical safety I think the most
obvious path is to learn as much about
machine learning as you can which
probably step one of that is to get a
PhD
young Laika who works on the the AI
safety team at deep mind David talked on
this at EA global San Francisco and I
think Hawaiian and Vika also gave a talk
about this yesterday so one thing you
could do is look up the recordings of
those talks if you're interested in
learning more about this path I think a
harder but even more valuable thing you
can do on the technical side which is
related to what I was saying about the
importance of having agendas that people
have concrete problems people can work
on
so this this harder thing would be to
read as much as you can about current
thinking on how to approach AI safety so
this is including agendas from Mary from
Paul Cristiano and ammonia towers that
this concrete safety paper
and to you know read those through think
about the problem yourself really go go
deep and come up with your own ideas
about what you think the critical
problems are going to be what people
should work on when and what concretely
can be done
if you're interested in the the policy
and strategy side of things the
deployment problem
the first thing I'd suggest is looking
at Carrick Flynn's post on the ei forum
the upshot of the post is that he thinks
the main type of research that's read it
in this that's needed in this space
right now is what he calls
disentanglement research which is again
kind of similar to what I was saying
about the importance of agendas so
disentanglement research so-called is
you know taking this big messy vague set
of problems you know what might happen
is more and more capable systems are
developed how to very in fact as respond
to that taking that big mess and trying
to kind of slice it into more concrete
and more manageable sub areas I think
this type of research is really
difficult and there's relatively small
number of people who can do it so I
think it's really worth giving a try out
if you think you can do it
fantastic if you think it's not so much
your thing I think another really
valuable thing to be doing right now on
this side is to be building up expertise
and experience in areas related to AI
policy so that you can be well
established in in one of those areas at
a later time when when this the you know
the other side of things is a little bit
further along and and we have more of a
sense of what kind of policies might be
might be valuable to push from from
various actors which could include
governments AI companies and others
and I also think that if you're
interested in the policy and strategy
side of things I do think it's really
valuable to learn as much as you can
about the current cutting edge of
machine learning research so that you
can you know be as informed as possible
about how the technology works okay and
then lastly if you hear all this talk
about AI and you think you know maybe I
should work on that but I'm just really
really excited about this other thing
then I think great go do that instead I
think as much as I personally think that
AI is a hugely important topic
I even more strongly think that the best
version of the EA community is going to
involve you know a wide range of people
with deep expertise on a wide range of
topics it's not going to involve
everyone going off at the same thing or
everyone trying to stay abroad so that
they can keep their options open so if
some other topic has grabbed you
fantastic I think do that that's all I
have to say I think we have a bit of
time for questions I put up a few links
that might be helpful if you want to
read more so feel free to photograph
that slide if you like and
yeah come feel free to come find me at
office hours which is 1:00 p.m. I think
in the senior corner thank you
yes you can still submit questions at
London dot e a global dot org forward
slash polls
thanks very much Helen and so you spoke
about the different areas to work in AI
and specifically if you're sort of maybe
a younger point your career when you can
do a PhD what about somebody who's
perhaps sort of further along their
career path yeah I think it I think it
depends a little bit where you are in
your career and and what you have
expertise in I do think there's I do
think AI will be relevant for a wide
range of different different fields so
if you think there's potential overlap
between the field they are already in an
AI that could be a good angle to try and
try and pivot towards so a great example
is Alan de veau who's speaking next is
you know his background is in
international politics which you know in
fact AI is going to have a lot of
relevance to that so he's you know
relatively later in his career relative
to you know university student he's
started to learn a lot about these
issues and sort of switched his focus to
that or I think if you're in a totally
unrelated area and you really care about
AI you could consider a switch
my last comment about if you really care
about something else as well or I guess
donate right if you haven't got a
realistic chance of maybe
sort of making an impact then maybe
donate yeah absolutely yeah okay so do
you have any thoughts about the role of
communication and outreach for AI safety
when we talk to people about this is
there anything to emphasize or
de-emphasize
strategically word we want the public to
be yeah I have lots of thoughts about
this I would be happy to chat more at
office hours I think one really big
thing is to be aware of this dynamic
that's gone on where
you know the media likes to get clicks
that's their whole that's that's the
whole thing and so we've seen this whole
wave of articles talking about AI risk
in terms of the Terminator and in terms
of robots with guns deciding that they
hate humanity because they're evil and
coming to kill everyone so I think one
thing to be really aware of when you're
communicating about this topic to people
who haven't heard much about it before
is that their mind is immediately going
to go to that because that's the story
that people have pushed so I think it's
important too
kind of I often take the approach of
explicitly saying that's not what I mean
because it means that it preempts the
issue
and so I think it's important to be able
to disambiguate
the types of concerns that that AI
researchers actually have from this
concern that the robots are going to
wake up and come and kill everyone so
that's you know that's where we talk
about you know badly specified objective
functions
you know robustness reliability that
kind of thing
technical concern rather than angry
robots with guns yeah okay um so what
single effort would you recommend a UK
Member of Parliament to undertake right
now to affect change on this issue
gosh if you just give the solution that
would be great you
figured it out you guys
yeah I'm really I'm really not sure I do
think that
that on the weight you know when
thinking about the the kind of questions
that spring from what happens if
transformative a I of some kind just
developed in the next next little while
I do think that we're at kind of a pre
paradigm attic stage of things where we
really don't have very concrete well you
know by we I mean the sort of whole EA
community and the whole community of
people think about these issues don't
really have very concrete suggestions
for things that we should do at this
point so I think I'm gonna stick with
the very unsatisfying answer if I don't
know yeah and so it's one thing to say
do a PhD but there's no way to install
one does tech AI safety work it seems to
be restricted to top university PhDs
especially in reinforcement learning or
inverse RL what can be done to expand
the pool of talent yeah so open
philanthropy we're definitely trying
very hard to expand the number of places
where you can do this kind of work and
we do definitely hope we yeah we think
that our efforts along with the efforts
of others certainly have already helped
to make this make you know technical AI
safety work something that you know is a
legitimate thing to talk about at all
and we definitely hope to expand that in
the future I do think that and this is
something that that yarn like I said in
his talk at a global San Francisco
I do think that as you're getting
started out if you're interested in
doing this machine learning side of AI
safety it's probably more important to
get really really good at machine
learning and become a really really
talented machine learning researcher
that's probably more important than
trying to work on safety as soon as you
can so I think it's not necessarily a
problem if you you know if you get into
a program that doesn't have an
established you know a safety professor
who you can go and talk to I think
that's totally fine and you should just
try and try and learn as much about
machine learning as you can and then you
know while you're doing that sort of as
I mentioned be thinking about what kind
of research do you think could be
helpful on these issues and keeping up
with the the other AI safety research
that is going on and then at some point
either maybe the university you're at
will start to think about these issues
or you can switch to another university
or to somewhere like deepmind or open AI
who both have active safety active and
growing safety teams and so how would
you evaluate the merrier approach to a
safety compared with a more machine
learning based approach yeah so this is
a very difficult question to answer
briefly I think one thing I would point
to is my colleague Daniel Juiz who as I
said leads our work on the technical
side of things he wrote a long post on
the ei forum about his thinking on on
Marie's work
yeah I think it's I think it's really
hard to say I think people people
disagree a lot I think you know as I
said we're at the point right now where
we are interested in funding different
approaches from different angles from
many different people who we think are
you know smart and thoughtful on these
issues and I certainly think that we let
mary falls into that that bucket
yeah that's a longer conversation happy
to go into it at office hours okay great
thanks very much Helen could you join me
in thanking Helen first time
[Applause]


Preparing for AI: risks and opportunities | Allan Dafoe | EAG 2017 London

and so yes my great pleasure to
introduce Allen Defoe so Allen is an
assistant professor of political science
at Yale University and also a research
associate at the future of humanities
Institute at the University of Oxford
his research seeks to understand the
causes of world peace and stability
specifically he has studied the causes
of the liberal peace the role of
reputation and honor as motives for war
and the logic of provocation in crisis
escalation he has developed
methodological tools and approaches to
enable more transparent credible causal
inference for which he was recognized as
an emerging researcher with the leap
Lima Rosenthal prize for open social
science this work is published in the
american journal of political science
world politics Journal of politics and
science amongst other journals his
current research focuses on AI Brannon
strategy which requires clearly
perceiving the strategic landscape
emerging from transformative artificial
intelligence and helping humanity
humanity navigate safely through it he's
working with the future of humanities
Institute to build their AI politics and
policy group so please join me in
welcoming Allen
great we have a lot to cover so I will
begin and proceed quickly so we can
think of artificial intelligence as
simply machines that are capable of
sophisticated information processing or
more informally machines that perform
tasks that previously required humans to
do them that's an imperfect definition
but it suggestive we can break a eye up
into three components productively
there's these sensors an input that
provides some kind of data to the
algorithm
there's the algorithm itself depicted
here as actually it would be a deep
neural network which is sort of the most
common type of machine learning
algorithm at the frontier of AI today
and then you have some outputs when we
talk about AI we most often think about
the algorithm as being the sort of the
the component that most matters but it's
important to remember that the sensors
the input and the output are also
crucial components of machine
intelligence because that is what
enables them to be so influential so in
the past years AI has become a hot topic
as I'm sure you're aware we might ask
why that this is the case well one
structural factor contributing to it is
just this generalized Moore's law so
we're seeing continuing progress in a
range of computational characteristics
and particular exponential improvements
and this is driving a lot of the
progress but not only this there's now
more recently an influx of investment
dollars AI researchers are also growing
exponentially it seems this is
attendance at the leading a eye
conference nips I don't have a figure
for salaries but it also would look
explosive I like this
startups are pouring into the various
industries trying to deploy AI to
transform different sectors and I would
say I would offer this sort of AI 2017
conjecture which is that if we were able
to freeze technological progress so we
just have the algorithms we have
today we have the hardware that we have
today and we just built more Hardware
collected more data and deployed it and
reviser systems
I think AI of 2017 would transform
almost every industry that we can think
of in every area of society in politics
so we could at length go through each of
these most of the applications or the
potential seems to be positive
AI seems to be a boon to humanity uh
there's a lot of promise however there
are also risks and challenges and I will
focus on those not because I think most
of the probability mass necessarily is
on the risks but because there are
Prudential reasons for focusing on them
one of which is that the opportunities
the upsides tend to be I tend to
coincide with the profit motive so we
can sort of leave to capitalism to
identify those opportunities and I
deploy them whereas the risks often
involve externalities intergenerational
global externalities and sometimes have
a time window that might be closing and
so for that reason those of us thinking
about AI governance tend to direct a lot
of attention to thinking about what
could go wrong and what do we need to
get right soon to best harness the
bounty of AI this just shows a
distribution a sort of a credence
distribution of AI experts about
human-level AI and what it would do for
Humanity you see most of the masses on
things going well on balance good or
extremely good but there's still some
mass that it could go badly five percent
extremely bad on the order of human
extinction so we want to get that
probability down as low as we can okay
in the near term there's a range of
governance challenges and the interest
of covering a lot of content I'm just
going to collapse them into two
categories the first I will think of as
we want to think about the encoding or
the failure to encode human values
political values in our systems as we
deploy algorithms so as we start hiring
people using algorithms or us giving out
fellowships or loans or making policing
decisions and deployments or judicial
decisions or even just have our social
networks shaped by algorithms
this influences the extent to which our
values are represented in the world
around us and we need to be conscious of
that and look for opportunities to
govern our algorithms and incentivize
them and direct them so as to sort of
maximize the values that we would like
to see in the world around us so some
considerations are fairness
accountability transparency efficiency
and more broadly we might think of
ethics there's another big cluster of
near-term governance issues that I will
call safety and critical systems so this
is the systems in which people could die
lots of people could die
or the economy could suffer extreme harm
we want the deployment of algorithms in
those systems to be robust transparent
secure and for there to be appropriate
oversight okay there's a lot of work
that needs to be done in this space it
is increasingly being funded and work is
being done on this I'm now going to move
to some more systemic global and extreme
risks in part because those are
especially neglected and EA's are the
kinds of people most likely to address
those governance challenges
so here are four extreme systemic risks
that I believe are plausible given the
2017 AI conjecture so even if we froze
technological progress today I think
each of these four risks are serious and
need to be addressed very briefly labor
seems to be likely to be increasingly
displaced we might need to reform our
training systems our education systems
even how we think about work it could be
that mass unemployment will be a
phenomenon of the future and we need to
reform our welfare system in our notion
of self-worth to accommodate that
inequality seems like a candidate for
dramatic expansion which poses lots of
challenges so there's lots of work that
should be done in this some EA is
already thinking about this you don't
need to think or you don't need to be a
specialist on AI to work on inequality
unemployment though it helps to pay
attention to what's happening in AI and
associated domains okay - I'm going to
go terrible
brief here but I'm happy to discuss them
these issues later
it's plausibly the case that AI
industries and services are natural
global oligopolies or monopolies and
what that means is if you secure a
leading company and some service say
search say social network provision say
ecommerce it may be very difficult for a
challenger to enter right and there's
there's reasons related to the
digitization of these markets so the
marginal cost is low to network effects
say in social networks and to other sort
of economic factors but I think it's
plausible across all of them that AI
maybe unlike other kinds of services and
trade in that I you have now potentially
incentives for what's called strategic
trade strategic industry industrial
policy and that could change the whole
international political economic game
the world could become much more about a
sort of an economic nationalism
mercantilism
and these are extreme challenges for
global order 3 AI and associated
technology sensors robotics may enable
authorities with capital to have much
more capable surveillance ability to
profile individuals identify potential
potential political allies or dissidents
to persuade people at scale but tailored
to their individual psychology or
vulnerabilities and then finally deploy
robotic systems to use physical force to
maintain order this could radically
change the relationship between people
and governments and I think is something
we really need to be watching and
thinking about how that trajectory what
that could do and how we can make sure
it goes well or do our best to do that
and for there's this phenomenon called
strategic stability which is the idea
that what's the probability of two
nuclear powers falling into an
inadvertent on nuclear war so during the
Cold War there were a few moments the
Cuban Missile Crisis being one where it
seemed like we might it is possible to
fall in
to a nuclear war for a variety of
strategic reasons which I don't have
time to discuss and it's plausibly the
case that a set of technological trends
will reduce strategic stability so maybe
that's what's called secure second
strike will not be so secure submarine
ballistic missiles and mobile ballistic
missiles which are the two most
resilient arms of the nuclear triad
today I could become vulnerable because
of subsea sensor networks satellite
imagery inferences you can do based on
social network patterns and other data
all of which AI will help extract that
from that it from that data into usable
information couple that with hypersonic
missiles which shorten the retaliation
time from about twenty four minutes to
six minutes and the incentive thereby to
automate your nuclear retaliation and
you can imagine scenarios where a
conventional kinetic conflicts say in
the South China Sea over freedom of
navigation patrols could escalate on
machine timescales to nuclear war this
is something that serious IR scholars
and security experts are beginning to
think about and is also something that
we might be able to contribute to now
again all of this I think goes through
just with AI as it is today just with
some more sensors some more computing
systems but not not radically new
technological capabilities however we
can't it seems unlikely that AI is going
to stay where it is today on the other
hand it's plausible that we will start
seeing human level or superhuman level
machine intelligence in various
strategic domains and we need to think
about what that could mean now sometimes
when people hear this notion of human
level AI they think that the speaker or
the thinkers behind this have a naive
notion of machine intelligence their
anthropomorphize intelligence right
they're saying well humans have this
distribution of capabilities and so
they're thinking well machines are like
that times 10 that is not so the reason
we talk about human levels so often is
because it is a strategically level
relevant thresholds for narrow machine
intelligence or general machine
intelligence it is relevant for a few
reasons
one is what I call the substitution
threshold when machines are better in a
cost-effective sense than humans at some
task then you will start seeing the
substitution of machines for humans in
those domains that has obvious
implications for labour unemployment but
also implications for control if you're
taking humans out of the loop as it were
for power relationships between
Authority in capital and citizens also
for political backlash as people become
displaced from various roles that
they're used to they might mobilize to
oppose that so it's a politically
relevant threshold another relevant
threshold is what I call the performance
threshold this is when machines are
better than the best humans at some task
domain so machines are on track to
become better air combat pilots than
humans and when that happens in a
industry like defense that is very
quality sensitive right they're willing
to spend that extra dollar to get the
best combat pilot then you could you're
likely to see deployment of those
systems and it could change the whole
game right it's hard to know what it
would mean if we have superhuman math
machine intelligence or superhuman
engineering machine intelligence so for
the rest of the talk I'm going to focus
on these potentially really
transformative levels of machine
intelligence and this is all within what
we call the technical landscape in terms
of the broader research landscape one
point I want you to take away is that
the development of transformative
machine intelligence is again not likely
to be even or distribute in the way
human intelligence is distributed so
this shows deep Minds
dqn algorithm playing a range of Atari
games and you see that for some it's
superhuman and for some it's grossly
subhuman and I think this is a plausible
metaphor for what it will be like when
really transformative human-level AI
starts coming online it will be
radically superhuman in some domains
while still being plausibly subhuman in
other domains and which domains become
superhuman first could really change the
strategic landscape a related point is
that by the time machines are better
than humans at
every strategically relevant domain it's
not like the machine just got better
than the best human at that point it
just got better than the best human at
its worst strategic task and it is
probably vastly super-human or the
system's that it's part of our vastly
super-human and other tasks so we might
ask what are the narrow transformative
capabilities where we could see
superhuman AI coming on line soonest we
don't know but here are some heuristics
for thinking about it domains where
there's a lot of data especially domains
where you can simulate the environment
so this is why I mentioned math if you
think about alphago and alphago zero the
reason why it was possible to make such
rapid progress is because the universe
of go can be perfectly simulated so you
can really leverage reinforcement
learning to generate competent AI
systems three if the domain is narrow in
the sense that it doesn't touch on the
whole world it doesn't require a whole
world model which or these other sort of
notions of common sense that seem to be
like sort of a key mountain to what's
called artificial general intelligence
and for domains with high stakes right
if there's a lot of money or power at
stake you can expect a lot of effort to
be expend trying to get really
transformative or powerful AI systems in
those domains so here's some conjectures
where we might see these narrow
transformative superhuman capabilities
coming on lines first however it may be
the case that we don't just get a set of
narrow systems but we do get really AGI
artificial general intelligence that's
capable of what we don't really know
what it is but common-sense general
intelligence and so a question we want
to ask is what's holding us back from
that here are a set of mountains and
emesis words these are sort of
challenges facing cutting-edge AI that
seem like they're necessary to have AI
that's capable of doing the things that
humans do and there may be more
mountains or it may be that many of
these are actually share some common
factors some common capability and once
that's achieved then they'll all start
falling in rapid succession
I'm going to talk a little bit about
that very briefly here's a survey of AI
experts that I call collaborators
conducted this is not meant to be strong
evidence of when human level AI is going
to arrive this is the the question was
when machines will be better than humans
at every task rather it's one piece of
evidence which is the best we have so
far the main thing I would want you to
take away from this figure is that AI
experts if you look at the red curve
which represents the average do not rule
out that we could have really
transformative human level across the
board AI in ten or twenty five years so
that's ten percent or thirty percent
probability Quinn says okay I want to
show you a little allegory of how
progress in AI might go faster than we
expect if we're just naively
extrapolating so here's a figure or half
of a figure from iff metrics which is a
really nice webpage and it shows
performance on one of the Atari games
called frostbite and if we were in
mid-2015 and suppose frostbite was a
strategically relevant domain and we
asked by when will machines be super
human or least machines learning from
pixels I'd be super human we would draw
some lines out and conclude 2050 to 2100
of course
I selected this example that would be
grossly mistaken something clicked at
the end of 2015 and we went from
subhuman to superhuman within a year if
you look at all of the EF f metrics
performance graphs some of them look
like this whereas some of them look
gradual and so one important area of
research today is to try and make a
science out of this to try and identify
when we think we might be surprised by
progress and when we can extrapolate
from past trends or from other inputs
like compute talent resources spent
there are a number of reasons why rapid
broad takeoff is a plausible hypothesis
and that it might happen soon let me
just talk about the last two very
briefly you've probably heard of at
least especially the third one the
fourth point references the idea that we
might already have abundant key inputs
for advanced human level AI just waiting
to be untapped
waiting to be properly exploited so we
might already have sufficient compute in
the world to really deploy thousands or
millions of human level AI systems there
might be lots of insights just waiting
to be uncovered by a machine
AI sort of a research AI system that can
discover them more rapidly than humans
and I can give you an example of that
and we can also think about say data
overhang for example in the form of the
internet right the Internet has physics
textbooks history textbooks strategy
lots of conversations between people all
waiting for an organism that can read it
all and integrate it into a single world
model right we can't do that where we
were we're mortal we only live for so
many decades and we read pretty slowly
whereas a machine could plausibly read
the whole thing once it's able to
integrate that information and that
could be really transformative the last
point current with current machine
learning systems it costs a lot more
computing power to train the system than
it does to execute the system once
you've trained it on the order of a
million to 100 million times more costly
this means that once you've trained up
your first transformative AGI if you
choose to recycle that compute for
deploying it you might have around 10
million instances of that AGI so it's
not the case like it is with organisms a
biological organisms giving birth to you
know the next generation that they come
online kind of one at a time rather you
might go from 0 to 10 million AGI is
overnight ok I'm going to move really
quickly because I enjoyed talking about
those previous slides so a key question
is what the trade-off is between safety
and performance we just don't know and
it really matters if the trade-off is
stark if it's very costly to build safe
aligned systems then it makes the
governance problem much harder whereas
if it turns out it's relatively easy
then you know Humanity is in a much
better position so one thing we want to
do is think hard about what this
trade-off might be and perhaps some
systems some architectures face a less
steep trade-off than others I'm gonna
skip some slides so moving on to Dennis
and the AI race so Dennis points out as
have others that we want to worry about
racing and sort of impatient hurting to
the finish line when safety gets cut and
in particular this is a problem once
we're thinking about national
governments well it is the case today
that national governments are really
getting involved in the conversation so
Canada South Korea Japan the UK
increasingly especially China which is
has a seems to have a very deliberate
plan to be the world leader in AI by
2030 and they seem to be acting on it
there are also conversations in the US
but not yet at the sort of official
government level or rather the the
conversation hasn't changed the way it
has changed in some of these other
countries some other countries have
taken notice of the idea that AI might
be something that's powerful one leader
that got attention recently is our
neighbor a few countries over who said
whoever leads in AI will rule the world
he didn't in fact look like this when he
was saying and this is part of a little
message I want to communicate which is
that he actually said this in the
context of encouraging children or kids
in their science projects so there was
kids from around Russia doing science
projects about various you know sort of
frontiers of science and engineering and
Putin was encouraging them all and
saying everything like this is a future
of Russia this is really important he
did use especially hyperbolic language
for AI but this was embedded in a long
conversation however the world noticed
Putin's quote I I don't know what the
right metric is but one metric is a
Google News search of Putin rule world's
artificial intelligence gives over
25,000 news hits and it really you know
was was yeah many news sites were
talking about this and so this I just
want to point out is a challenge that we
face and doing research in this space
because while an AI race is something
that could be very dangerous and
important to think through
it's also very hard to communicate
publicly about because the the media and
the public
is prone to sort of being drawn to the
most extreme messages about it okay so
let me tell you very briefly about some
projects that we're working on and that
you might contribute to but there's
really a lot to do one category we call
AI politics and this refers to the set
of questions where we treat agents as
not sort of coordinated but pursuing
their notions of its self-interest
perhaps in a rivalry way so we have
worked on public opinion looking at the
relationships between government and
industry and researchers we're really
engaging trying to understand China and
and look for opportunities to cooperate
there a lot of case studies can be done
these are especially nice for a new
researcher to take on because you can
sort of pull together the literature on
a various relevant tech case study
something I think about a lot are the
abstract strategic properties of
technology and specifically what we're
likely to get from AI and how that could
change the economy military dynamics and
so forth in a range of other oh one
other off leg that Helen also mentioned
is the deployment problem you can think
of that as the hypothetical what if
there was an AGI breakthrough tomorrow
and you're saying an advisor to the to
the research lab or you're an advisor to
the government that hears about this
what do you advise it's a very hard
problem to answer but it's especially
motivating I think it's useful for that
reason another category of work that we
do we call AI governance and this now
asks if we're able to have a calm enough
conversation across relevant
stakeholders what is it that we want to
build so what are the institutional
structures that we want to employ what's
the constitutional foundation how do we
incentivize people to be part of this
governance regime and how do we make
sure it achieves the various values that
we have in mind again there's case
studies to be done some really
interesting technical work that I'll
flag miles Brundage's working on on AI
verification agreements so how can say a
principal in an agent you can
like the Board of Directors and the
researchers or two countries that are
trying to cooperate how can they verify
that the other side is building the AI
system in the way that they agreed that
it should be if you want to be involved
in this who can you talk to and join
there's a number of groups that are
thinking about this the two biggest ones
are the future of humanity Institute and
the open philanthropy project though
these other groups are also thinking
about transformative AI and have some
work thinking about strategy and
governance in particular the future of
humanity Institute has a not yet
announced or we'll call this the
announcement governance of AI program so
here it's depicted those of us who are
there on picture day and here are some
others working with us and we're really
trying to scale up as quickly as we can
to integrate all the talent in this
community to work on these problems
we're working on the career pipeline
it's a challenge and so if we're not
able yet to sort of get you into the
sort of career trajectory please just
keep thinking about this reading about
this writing your thoughts down and
building a community and joining the
community and last if you have any
questions or want to really try to enter
I think this community but especially
the government's VI program at FHI you
should talk to that guy Carrick Flynn
who is a core pillar in the program or
email this you know thank you
thanks very much Alan and those of
questions come through on that so on on
the stuff back end of your talk and so
what's your view on the probability and
consequences of a potential
technological arms race in the space the
probability consequences technological
arms race um so I think I'm going to
I'll say we don't yet know what the
proper way to discuss this is and and so
I will say it's instead of answering the
question I will say it's something that
we need to think about very seriously
it's sufficiently probable and
sufficiently consequential that it
deserves a lot of our attention okay and
so the lack of diversity in AI research
poses the threat of building our own
biases into AI a concern that's been
shown by assessing face recognition
studies what kind should we do to
increase diversity in this field to
avoid this yeah that's a hard question I
mean I think we should look throughout
the career pipeline to see where their
bottlenecks are and try to address it I
guess the way I think about addressing
it as being self-conscious about
possible obstacles may be reflecting on
your network so the extent that you're
recruiting or identifying people from a
network that you know it's correlated
with certain characteristics you try to
work against that I think yeah
fellowship programs and just encouraging
anyone interested in the space to you
know think about it and engage in it so
this is actually the same questions we
had in the last talk but they've
apologized and say it's still highly
relevant and so what what single effort
would you recommend to a UK member of
parliament to undertake right now to
effect change on this on this issue
yes so I was actually at Parliament the
other day
to show you a slide there we go so the
Lord Select Committee is is thinking
about this and there's been some reports
Wendy Hall in pinch NT I believe I wrote
a report on sort of building the AI for
UK industry there's sort of the
mainstream recommendation which is I
would say follow Canada's lead Canada's
doing a good job of adopting a sensible
sort of industrial policy with respect
to a I so if you think about AI is like
electricity or the Industrial Revolution
it means that we're going to have to
change our whole infrastructure to
really harness the benefits so this is a
like a mainstream near-term answer to
that question and is where most of the
governance attention is likely to go
however I would also say I'm thinking
about the longer term we want to start
building institutions today so that
they're compatible and ideally seed
cooperative possibilities in the future
and so to the UK Parliament and others I
would say start sort of committing to
the common good or common good principle
or something like that so that means
trying really hard to cooperate with
other countries in harnessing the
benefits of AI and making sure a AI is
being deployed for the common good so
there's lots of applications for the
environment for poverty alleviation for
addressing other inequalities and social
concerns that would be sort of a good
near-term direction for AI and this
start thinking about yeah these broader
problems of our values safety systems
safe systems and how we can reach
agreement about you know how we should
be building AI and then potentially
developing a sort of common institution
for governing AI in regions like Europe
or even the world and so you heat with
pretty early on that in that kind of
stage of development who said you think
kind of sort of best practice at the
minute Canada's best practice for at
least for UK's model for a Industrial
Policy I'll sort of promoting an AI
research or community
I haven't seen much commitment to the
common good not that they haven't but
that wasn't a prominent feature of their
strategic plan we are in early days for
institution building and you know global
institutions take a long time to build
though if you have a deadline sometimes
it happens quickly you know the UN was
put together and pretty short notice but
you need you know something to motivate
you so I do think we should be thinking
very hard about this today thinking
about what are the institutions we want
to build and how can we see those
institutions and you know maybe waiting
for the opportunity when other world
realizes that they need to build those
institutions so that we can give them
the blueprint okay great
fantastic so much can you join me in
thanking ironface time today