Musings on AI | Panel  | EA Global: San Francisco 2017

next up we have what is definitely going
to be one of the great highlights of our
program it's a panel on AI that we've
called musings on AI we've got five
great panelists and an awesome moderator
as well it's my pleasure to introduce
them all to you first up is Tasha
McAuley Tasha is a technology
entrepreneur living in Los Angeles her
current work with geo sim systems
centers around a new technology that
produces high-resolution fully
interactive virtual models of cities
this novel approach to reality capture
yields the most detailed VR environments
ever created out of data gathered from
the real world
prior to her involvement with geo sim
she co-founded fellow robotics a
robotics company based at NASA's
research park in Silicon Valley and she
was formerly on the faculty of
singularity University where she taught
students about robotics and was director
of the Autodesk innovation lab please
welcome
Tasha McAuley
yann Leica is next he has a PhD in
theoretical reinforcement learning from
the Australian National University where
he worked with Marcus hunter after
earning his PhD he joined the future of
humanity Institute at the University of
Oxford he was a research scientist at
deep mind and he is a research scientist
that deep mind presently and he is
working on technical AI safety please
welcome yamaneika owen cotton barrett is
next Owen has a PhD in mathematics from
the University of Oxford and he is a
research fellow at the future of
humanity Institute please welcome back
Owen cotton Barrett next up is Helen
toner she is a senior research analyst
at the open philanthropy project her
work focuses on policy governance and
strategy issues related to progress in
artificial intelligence across domains
including geopolitics national security
public policy and AI safety before
joining open philanthropy she studied
chemical engineering and Arabic at the
University of Melbourne while studying
she led ei Melbourne worked at a finance
startup called Veszprem capital and
interned with the Boston Consulting
Group in her free time she enjoys
Renaissance polyphony polyphony and
learning how to dance
please welcome Helen toner
[Applause]
darío Amadei is next he is the team lead
for safety research at open AI
previously he worked at Google and Baidu
he is the co-author of concrete problems
in AI safety which explores practical
issues in making modern machine learning
systems behave in a safe and reliable
manner and also of deep reinforcement
learning from human preferences which
develops a method for learning complex
human determined goals he sits on the
board of the partnership for AI a non
profit industry partnership dedicated to
ensuring AI technologies benefit society
previously he worked on prior to working
on AI safety he also helped lead the
project that developed Baidu's deep
speech to a human level speech
recognition system which was named one
of ten breakthrough technologies of 2016
by MIT Technology Review
please welcome dario Amadei and finally
to moderate this outstanding panel
Michael Page he is currently working on
AI strategy before that he earned to
give as a lawyer in Washington DC from
2012 to 2015 he then advised ceas 2016
restructuring during which time he
managed several research related teams
please welcome Michael Page all right
this thing working ok so the theme of
this panel is basically going to be
stuff relating to advanced AI that I
would find interesting to talk about
with this crowd and hopefully that you
would also find interesting I'm hoping
this will be a fairly casual back and
forth I'll ask some questions not
everybody needs to answer every question
so feel free just to pipe in if you have
a particularly interesting answer to the
question that I'm asking and also I just
learned that Game of Thrones started ten
minutes ago so thanks for coming here
instead of tuning in so when I first
heard about this panel idea it was I
think tentatively dubbed the the AI
safety panel and I wasn't really sure
what that meant people use the term AI
safety in different way
a lot different issues relating to AI
that this crowd might care about so I
thought I'd begin with a terminological
question I hear a lot of terms thrown
around in this space you have AI safety
the control problem the alignment
problem a nice strategy AI policy the
deployment problem what terms do you
guys like or not like so I guess first
state an obvious point you know I think
obviously the research that's being done
is a much more important than the
specific terms that we we we use to
describe it that said of the things
Michael has mentioned I mean in my own
experience you know we called our
problem our paper concrete problems in
AI safety because AI safety happened to
be the term that people were most
familiar with if we said the line Minter
control people would have thought we
were talking about something kind of
weird and different I'm not really a fan
of rebranding I think AI Safety's a
valid area and we should argue for its
validity directly so it's the term I I
typically tend to use but I I think we
shouldn't care that much about terms and
when people use other terms it it
doesn't bother me on the last two terms
a AI policy and AI strategy I think
these are important areas that are kind
of distinct from form AI safety and I
either term sounds fine to me
yeah I just note that in terms of
describing types of AI systems you know
sometimes gets thrown around like a GI
so artificial general intelligence super
intelligence and so on I found it really
useful in discussing policy and strategy
issues in this space to instead talk
about transformative AI which I guess is
you know very roughly defined as a
system or a set of systems that has an
impact
you know comparable in size to the
Industrial Revolution and we've just
found this really helpful to you know
when you're not focusing on the
technical details of building a system
like that when you're instead focusing
on what the policy responses should be
it's been really good to to get away
from debates of well does it need to do
everything a human can do exactly as
well as a human or can it just be most
things and doesn't need to be much
better than it you know it gets you free
of all of that stuff so we found
transformative AI to be a you know a
useful if rough concept to work with
I like transformative AI i also liked
the term AI aligned
I think you know we're going for
extremely reliable agent design and and
value learning and I think the idea of
alignment with human values is is useful
to keep in mind or at least as a
terminology for sort of describing what
the core focus of AI safety work might
be AI safety and the control problem
both have the tendency to maybe conjure
up the idea of an adversary or you know
a captive that you might need to control
which I think are useful perspectives to
take but focusing on the alignment of
values is is nice when talking about the
work that's being done and also probably
instills a little less fear in people so
so I use the term
technically a safety to describe you my
work I'm not quite happy with this term
because it kind of has this implicit
connotation that like other AI work is
somehow unsafe which is not true so the
the other terms that are floating around
like alignment I think alignment doesn't
quite capture some of the safety
problems that we think about like
particular robustness problems although
like some people use the word you mean
that control problems can be difficult
because there's a lot of technical
control problems out there so that gets
confusing I think I gave it Ari or that
it's like the precise term enews it's
not that important but I also think that
we're kind of stuck with AI safety
because it's kind of established and
it's really hard to change a lot of
terms is that actually there were
reasons to have want multiple terms like
it's a big complex world there's a lot
of actually different concepts that we
might want to be pointing to and some of
the time we might care about AGI and
then some other bit times we might be
thinking one of the consequences of this
and then the notion of transformative AI
is actually more useful so although
there's something nice about being able
to stick to a small number of terms I
think sometimes you want a new concept
and then it's good to actually reach for
the new concept rather than just use the
one that we've been building up cool
thanks
okay I want to talk a bit about AI
development both where we've been and
where we might be going so looking at
say the past year are there particular
developments in AI that have most
excited you so if I were to point to to
two areas and and these are things that
you know are not so much things that
make big splashes but I think research
areas that are starting to to blossom
and that I think should be watched we're
starting to see good progress on two
areas that I call um have been called
like hierarchical reinforcement learning
and meta learning so meta learning is
the idea that you kind of learn to learn
so you know often we we experiment with
different algorithms and say
reinforcement learning and you know
which we different researchers try
different techniques for how to learn to
play a game meta learning says no
instead maybe you should train an agent
where an episode of the agent is the
process of learning to play a game from
start to finish and then you train it on
a thousand games running each game from
start to finish knowing nothing to fully
learning the game a thousand times
then it learns the process of learning
the game and if you give it a thousand
and one thing then it will learn well on
those on that game as well
we the results so far on that are very
preliminary have been at the algorithmic
level but I think this is something to
watch on other areas I've seen a lot of
stuff on hierarchical reinforcement
learning which is basically setting
setting sub goals so if I'm a human I'm
walking around like you know if I think
about my day I think well I know I'm
gonna get up I'm gonna drive to work
then I'm gonna use my computer at work
then I'm gonna you know start writing
this paper I'm thinking at a very high
level about my tasks and our current
reinforcement learning systems don't
don't really do that but you know work
from open AI in deep mind and others is
uh is starting to think about that I'd
be remiss if I didn't mention the
development a couple a couple of days
ago where an open a I bought beat the
world world champion at dota which is a
popular eSports game dota 2
in in in kind of 1v1 minigames so we're
shooting for the for the full game after
that but that was a pretty pretty
exciting recent thing so I guess there's
also like so there's two things that I
found pretty interesting recently that
came out so one is the neural machine
translation that came out of Google
brain where you actually train your
networks to children translate from one
language to another and the other thing
was alphago and alphago mastered the
fact that you can actually train new
networks to Niska understand go better
than humans especially given all the
other parts that went into it so it's
not just deep learning but I think on a
high level one of the like things that
we learn from it is that deep learning
actually scares we scales really well
and we can put it on really big problems
and it will continue to improve and I
think that's that's quite interesting
yeah I agree with all of Yuans choices
as well any other parts yeah I guess I
just somewhat echo dario and saying that
I was interested to see a few
developments in the past year of yeah AI
systems doing well on games that were
not the kinds of games that that AI has
worked well for in the past so not games
where all information about what's
happening about the game and what's
happening in the game is available to
both players at all times so we had a
bot I'm not a yeah I guess about doing
really well at the World Series of Poker
I think in a specific variation of poker
but still that was a you know a kind of
development that that hasn't really been
seen before all right I say I agree with
fianc point about alphago and also it
was very interesting that it managed to
beat you know the best player in the
world a year ago and within one year it
was able to be you know the top 50
players in the world there was just an
enormous leap in one year looking
forward well so let's say the next five
years are there particular things the
a eyes might do in the world that really
excite you that you think might not be
on the radars of most of the people in
this room so I think we're gonna make a
lot of progress on manipulation by
robotic systems so everything you would
imagine kind of a you mean physical
manipulation of all usable manipulation
of objects not psychological
manipulation of humans are hopefully not
so just anything that you might expect a
household robot to do like picking
things up or or or moving things or or
or or moving furniture I don't know
about things actually being done in
physical robots and natural
commercialization happening because
actually those have very long timescales
that are often much longer than the
timescales of the fundamental
technologies needed to develop them but
at least in simulation and in principle
demonstrations of transfer of these
things to actual physical robots I think
will make a lot of progress on that in
the next five years and perhaps even the
next two or three years I expect that
area to progress very quickly another
area where I expect to see a lot of
progress to some extent unfortunately is
the use of reinforcement learning in
kind of like internet environments or
internet domains so people will begin to
use advanced machine learning to conduct
both attack and defense of cyber systems
so there may start to be kind of a
cat-and-mouse game or an arms race
between the attackers and the defenders
as we've seen in the last year cyber
attacks already have a huge impact on
the the world so you know well short of
AGI happening this is something that
could be you know if not catastrophic or
or or or existential could be a
near-term application of AI that really
has some effects on the world not not
all of which may be positive and then
finally this this stuff on meta learning
that I mentioned I I expect that this
you know people often discuss you know
big weakness of deep deep learning is
that it needs a lot of data to learn and
unreasonably more data than then is
needed for humans I think meta learning
is gonna progress in a few years from
now even a couple years from now no
one's gonna be able to say that anymore
I think so taking the cymbidium other
direction I think one thing that I'm
pretty excited about is Starcraft so you
might you might have heard like did mine
recently released like research
framework environment you for other
people to work on on Starcraft 2 and I'd
be very curious to see when we can be
the top pros on on Starcraft 2 from just
agents playing from the raw pixel input
and the reason why it's so hard to do
this is that Starcraft kind of requires
you to do long term panic and like kind
of hierarchical decision-making like of
the kind that are you kind of alluded to
earlier and also like the action space
in Starcraft is very complicated
compositional because you're selecting a
your units and then you're giving
actions as you selecting an action then
you're saying telling it where to
execute that action so that's that's
pretty complicated for these reasons and
love I love very small people are trying
to crack that problem quick question for
yon key people will beat Starcraft 2 or
dota 5v5 first unexpected start off
first
I think dota is how do okay maybe open
air will prove us wrong I don't know
it's it's implicitly a loaded question
because deepmind is working on Starcraft
and we're working on dota and III don't
think either is gonna work on the other
so I guess no comment it's announced
race I guess something which and seems
may be important in terms of practical
implications from developments over the
next few years is exactly things where
it doesn't need new physical
manipulators to have an impact and that
might come from things which are to do
with generating media there's been some
discussion like we already have some
ability to a spoof audio you can put
words into the voice of an arbitrary
person and and if we can spoof stronger
videos or maybe we'll have
things relating to like maybe just text
chat is gonna get a lot better I guess
one thing I'm kind of hoping to see on
Owen's point about fake videos and fake
audio I think a few people a number of
people are worried about effects that
that could have if people are taken in
by you know falsified audio or video of
from you know politicians or celebrities
making statements they didn't make one
thing I would love to see is for someone
to make a really great viral video with
you know famous people saying and doing
things they obviously didn't say or do I
would love to see that as a way to you
know spread around the idea that you
shouldn't trust video audio you know
unless it has certain qualities or
whatever the technical requirements
imply that's something you know so I'm
one of the audience wants to build that
I'd be happy about that
so one thing we talked a lot about this
community is the the risks associated
with developing advanced AI but
obviously they're also a lot of benefits
associated with developing advanced AI
so this is a bit of a cheeky question
but one way putting it is are y'all more
concerned about developing advanced AI
or not developing advanced AI I think
I'm deeply concerned about both so you
know on the on the not developing
advanced AI you know one one observation
you can make is that modern society and
particularly society with nuclear
weapons as you know only only been
around for about seventy years there
have been a lot of close calls since
then and you know it things seem to be
getting worse you know if I look at kind
of the world and geopolitics and in the
last you know in the last few years you
know China China's rising you know
there's there's a lot of unrest in the
the Western world a lot of very
destructive nationalism you know we're
developing biological technologies very
quickly it's not entirely clear to me
that civilization is compatible with
with with digital communication you know
it has really some subtle subtle
corrosive effects so you know ever every
year that passes is is a danger that we
face and
although although AI has a number of
dangers actually I think you know if if
we never built AI if we don't build AI
for you know 100 years or 200 years uh I
you know I'm very worried about whether
civilization will will actually survive
course on the other hand I mean you know
I work on I work on AI safety and so you
know I'm very concerned that you know
transformative AI is is very powerful
and that bad things could happen either
because of safety or alignment problems
or because there's a concentration of
power in the hands of you know the raw
the wrong people the wrong government to
who control AI so you know I think it's
it's it's it's terrifying in in in all
directions but uh not not building AI
isn't isn't an option because I don't
think civilization is safe I guess I
think I'm more concerned about building
advanced AI but that's just because I
don't think we will not build it bounced
AI and that doesn't seem on the table so
I think we should be focusing the
attention on making sure that building
goes right I think like I'm more
concerned with not building AI I think
AI has we great potential to help us
solve all the problems that like people
in the room here care about like global
poverty and Emily's animal suffering and
so on and less possibly new problems
that we don't really think about yet but
as as any new powerful technology like
you you have to be wise about deploying
it and be careful and I really think
about all the things that can go wrong
and in a way this is this is kind of
what technical is safety is like one
aspect of this of course like is similar
questions in air policy and i--i
strategy and I think we should really
take these seriously just to be clear
everything John said was meant to be
implicit in what I said so though it
sounded like we disagreed I don't think
we do I don't think we do no I agree
with Owens
you know point that we don't really have
a choice to about whether or not to be
concerned about advanced AI think it's
definitely coming I tend to think it's
more useful to take an optimistic
approach and then just exercise caution
when working toward an optimistic vision
one thing about humans were our
biological systems are so evolved to
understand events in the real world and
adapt to them and in this case I was
actually just watching a talk the other
day where Elon Musk was describing how
we have double Exponential's here we
have the exponential change in hardware
development with respect to you know
events advancing an AI and then we have
exponential change in the software
talent and even when just dealing with a
single exponential our intuitive systems
are not very good at predicting and
certainly in the case of double
Exponential's we're not going to be able
to to make proper predictions and also I
think whatever does happen we should
just be expecting that it's going to be
a shock one way or another and we're not
very well able to adapt to it
I liked the point about having
optimistic visions I think that if we do
have as a society more of an optimistic
vision of what a world might look like
with a great transformative AI that can
help us get on board with building it
together segue from your Elon Musk
comment so there's been a lot of drama
in the news recently between you and
Moscone and Mark Zuckerberg about like
when and how AI will tell us are you all
on Team master team Zuckerberg we don't
have our t-shirts I ordered them but
they didn't make any seed and Dario
you're welcome to take a pass on this if
you want it depends I'm happy to answer
it I mean I do in fact I do in fact work
for for Elon Musk I mean I uh so I guess
I guess I meant to pronounce but you
know I think my higher level
on things like this is uh you know my my
kind of own experience is you know at
the level of trying to work on specific
AI safety research ideas and I think
that while there's a lot one could say
on both both sides about this I actually
think it's probably good to pay less
attention to these kind of like Twitter
fights between extremely high status
people I think you know for many people
it's just kind of an excuse to - you
know get get interested in gossip and
you know fixate on people who are very
very very high status I mean you know
some people worry they say oh you know
Elon is saying these unreasonable things
like this will you know this will this
will turn people off of AI safety and
maybe that's a little true but you know
I've never talked to someone who said oh
you know I got interested in AI safety
you know because of this tweet Elon Musk
sent or like you know I was turned off
of AI safety because of you know this
tweet you on must sense I mean you know
what when I when I talk to people when
I'm trying to recruit people it ends up
being about the research the quality of
the research the the technical topics
and so forth yeah so I mean I guess I
guess I want to contradict that up just
a little in the sense that I don't know
I think I've heard from some people who
care about AI safety that you know maybe
Elon doesn't say things in the most
nuanced way but like at least he's
raising awareness and getting it on the
radar of policymakers and like surely
that's good right and I guess I just
feel a little cautious having been in a
couple of different situations where
policy makers and technical researchers
are brought together and there's a sort
of rough dynamic in the room of the
policy makers saying oh like I hear this
thing is really scary I see all these
news articles with terminator pictures
on the top like should we be scared and
the technical researchers put all their
effort into saying no no it's not
dangerous it's a long way away musk is
crazy don't listen to him
don't worry about AI safety is all fine
which i think is not a conversation that
would have happened if maybe if they'd
been less coverage in the media or if
the coverage had been you know less
sensationalistic so I guess I agree with
you know the bulk of what Dario said but
that's just an extra note that I would
add yeah so yeah I said this kind of a
young field become is the public
conversation around it is still kind of
shaping and my impression is that the
kind of Zuckerberg mass debate is kind
of polarizing it in an unhelpful way
that like you know forced to be there
Camp Zuckerberg I can't mask and they're
like good points on both sides and like
people should think about the short-term
consequences and they should think I
also think about the long-term
consequences about AI yeah I think yeah
I think more generally I think we should
work towards improving the public debate
around it and like as Han said like
please have less terminated pictures I'm
sure anecdote about this is a Nate
stories of Miri being quoted in an
article about AI risk saying how much he
hated it being quoted in articles as
terminate pictures and that article had
a terminated picture panna cotta I was
just gonna say I mean I was sort of
curious about when you were describing
polarizing people on one side or the
other I was I heard a term used by
Joshua Fox who I think used to be a Mary
he was describing himself not as a
techno pessimist or at economists but
like a techno volatility sort of saying
that any exponentially powerful
technology is going to have extreme
benefits or extreme risks one way or
another and that you might just have to
recognize that it's either on the side
of extreme pessimism or optimism and I
was I mean I guess we'll sort of get to
a question about optimism but sort of
curious how you where you fall so yawn
you mentioned something about improving
the public debate what is it about the
way that yeah is people just community
talk about AI develop into my safety
that most annoys all of you if you could
you know fix something about the way
that conversation is going on
what would you fix I think well
something I would love to see is more
science journalism around the eye from
people who really understand the subject
matter because there's a lot of
uninformed stuff out there and there's
very little billion fun stuff out there
so maybe that's a good opportunity for
people in this room to kind of look into
that I know so from the ei community in
particular I mean I think there's a lot
of things the a community gets gets
right about AI that no one else gets
right but one thing that I'd like to see
less of is that there's a particular
model I often see from not all but but
some people in NEA which is sort of that
there's there's like to progress bars
and one of them is the like AI
capabilities project progress bar and
the other is the AI safety progress bar
and if the AI capabilities progress bar
reaches the end before the AI safety
progress bar then we all die and if the
AI safety progress bar reaches the end
first then it's great I think this this
model is is kind of dangerous because I
think it's inaccurate and it it really
drives kind of the impression among
among AI researchers that AI safety
people think their work is evil or are
trying to hold back their work and you
know I think I think that actually does
does push against acceptance um the
reason I think it's not right is at
least in my own experience um a lot of
the safety work I do is made possible by
recent advances in capabilities work a
lot of the safety work I do also allows
you to do capabilities work in different
ways that's maybe safety compatible I
also think that relating to AG I in
particular some of the safety work that
we end up doing with respect to AGI a
lot of it may only be possible within
you know the last two or three years
before before it before AGI so there's a
lot of intertwine Minh two things and
you know while while it's true that you
know I think you know we should we
should work on safety research you know
right right away we shouldn't we
shouldn't wait until you know until
we're gonna build build AGI tomorrow and
that's why I'm working on it now
you know I think we should we should
maybe be less extreme and chill a little
bit about you know being scared about
the next you know the next the next
breakthrough in capabilities I mean it's
possible that we will have a hard time
you know doing all the safety research
at the end but there there's actually a
limited amount that that we may we may
be able to do do early on so the
situation is just not as not as binary
and I think that frame polarizes AI
researchers against AI safety
researchers and I think that's unhelpful
I agree with the fertilization issue
that although I do think like um often
there's a germ of truth in like simple
models and the idea that having more
safety research is pretty good
particularly relative to different
points on the capability track is the
germ of truth there and it's worth
acknowledging that as well but you can
frame it in a way that doesn't make
people pursuing AI research sound evil
yes and and ultimately air safety in
their research are not set as simple
things where it very much like it's all
just AI research that hasn't always been
the case correct I mean that could be
something that people just really maybe
haven't quite caught up to I mean yeah I
think I see this as a kind of open
question and as a question where
reasonable people can disagree
to what extent you know research on AI
safety is going to exclusively look like
the kind of work that that Daario and
yon do which is really you know machine
learning based it seems plausible to me
that you know all of the AI safety work
that needs to be done will look like
that as a non expert in the technical
safety and not expert in any of this it
also seems plausible to me that the more
kind of fundamental philosophical
mathematical foundations also need to be
explored I'm sure others in this panel
have have views on this question but I
see it I think yeah I think perhaps the
EI community is has not is not yet as
familiar with the machine learning based
research that is being done but I for
one at least I'm not not yet completely
confident that that will be the only
type of relevant work yeah I mean just
to be clear I'm a big fan of the work
that Mary is doing and I think it's
great that they're doing it but for me
that's still
yep relatedly go ahead on just something
else that I like I always annoys me a
little bit when I see people talking
using overly simplistic framings about
AI and I think this does happen in the
EA community I think it happens a lot
outside the EA community as well but I
think that we want to promote deep
understanding about this and so we
should try and hold ourselves to higher
standards perhaps than outside and that
means not claiming it as super
intelligence just means all-powerful not
framing it as its binary either the AI
is going to destroy the world on
everything it's gonna be great this gets
back to the techno philatelist
perspective related to that question if
you could instill one mean one a I
related meme in the EA community what
would it be
so is there like a pithy one-liner that
people here could like take home it's
like the thing they got at this
conference and maybe no is the answer
does it have to be one no
I can do two lines e2 is a I think one
kind of implicit assumption that I often
encounter is that people have this
mental model that the way will build as
powerful artificial intelligence is that
like somebody builds an initial system I
kind of seed AI and then it just ends up
self-improving very quickly and become
super intelligence or something I think
this is like a very misleading model I
don't think this is likely how it's
gonna happen and also like it's not an
implicit assumption you should be using
maybe don't anthropomorphize it's it's
very useful to work towards not thinking
of AI as a human but it's also an
interesting question of terminology as
well because you know we tend to use
even even the term artificial
intelligence gives people an
anthropomorphic vision of what might
happen and it's not maybe as catchy to
say you know a utility function
Maximizer people might not really know
what you're talking about but that type
of language is probably more descriptive
of what AI so maybe just finding ways to
enter the Mora is less so so here's a
one-liner a I doesn't have to learn all
of human morality so this is this
something actually that even even at
this point mere mary agrees with but you
know there's some some rut writing in
the past a lot of writing the past that
i think people are still anchored to and
in many ways so what what i mean by that
in particular is that you know i think
the things we would want from from an
AGI are to you know to kind of to kind
of stabilize the world to to end
material scarcity to give us control
over over our own biology maybe to
resolve international conflicts we don't
we don't want or i think need to kind of
build a system at least not build a
system ourselves that you know that kind
of is is a sovereign and kind of you
know runs runs the world for the
indefinite future and controls the the
entire light cone so of course you still
need to know a lot of things about about
human values but i
I still often run into people who are
kind of thinking about this problem in a
way that is I think is actually harder
than we probably need to solve
I guess one idea not a one-liner but
it's a one-liner once I explain it one
idea I'd like to seed tickets already
gets talked about some is interaction
risk so EA is often tend to think about
you know accident risk and misuse risk
where accident risk might be you know
the risk of misalignment the risk of
things going wrong and it's kind of on
the technical side is the system
designed correctly by the the AI
researchers and then the misuse risk
would be more on the geopolitical side
how will it be used will it be used by
dictators to oppress that people will
kim jeong-hoon get an AGI and i think
something we found it was something we
think could be important and we would
love to see more people thinking about
is your interaction risk which could be
the interaction between you know how the
system is designed and what the
geopolitical situation is so you know is
the is the system other really advanced
systems that are designed one day are
they designed in a context that is
peaceful that is internationally stable
where the designers feel like they can
take extra time to get safety right to
run double checks or is it is it you
know designed in some kind of race
situation or something like that
so yeah my one liner that is now a two
word a' is interaction risk there's a TV
show called the expense that is
basically about interaction risk
although it's not about AI it's about
another powerful technology um i guess
here's my one line app we don't yet know
all the answers but we also don't yet
know the important questions I think
that if you take a frame of like this
progress bar thing it's implicitly
assuming that we know what the questions
are it's how do we build safe D into our
a eyes in a way which implies that
safety is a separable thing and I just
think that that might well be false and
I think that like almost every other
question that we asked might not turn
out to be the important one there's been
a lot of discussion online for a long
time within this community about how
hard the control or
problem is and that's something that I
know you know Dario and Yun worked
directly on and some of the rest of you
less directly I'm kind of curious who on
this panel relative to the others thinks
it'll be most difficult and who on this
panel relatively other things will be
least difficult and I want you to to
kind of Duke it out so what's the most
efficient way of figuring out who thinks
it'll be most at least difficult I maybe
want to opt out and let the others argue
and then decide what I think so so maybe
let's have like a game where put your
hands high everything will be relatively
difficult and low can you'll be
relatively easy and we'll just see your
hands are I'm indicating I think there's
like I have a lot of uncertainty about
this I have a range of things that I
think applause Apple I take like yeah
I'm well I'm kind of like not answering
your question yeah okay so I'm gonna
nominate Dario for that it'll be easy
view point all right I will I will I'll
take it go with this so uh so first of
all I I don't know the the problem could
be could be really easy it could be
really hard that's a reason to to work
on it early I am relatively optimistic
though no one should should you know
take this as a reason to stop stop
carrying about about AI safety even if
there were only a 5% risk that something
catastrophic would happen to the world
because of it that would be well worth
much more effort than the world is
putting into it the effort of all the
all the people in this room and and and
many others it's still just ridiculously
under allocated that said one reason I'm
to reasons I might have for believing
that it might not be all that hard the
first is the one I just said before that
I think you know we may not need to
learn all of kind of human morality and
build a sovereign and basically built
something that decides what the best way
to set up human society is that problem
sounds really hard we may just need to
build artifacts that that perform
particular engineering tasks for us in
order to put the world in a better place
now they're very hard engineering tasks
that
we don't we don't currently know how to
do and so the problem is still hard
because you may still have to do large
searches over difficult strategies but
you know it's not not nearly as hard as
you know if you look at their early
writings of Mary Mary 10 years ago
people suspected that that that that it
would be so that that's one thing and
the second thing is I would I would
point out a lot about the meta learning
thing I said before where I'm a lot of
the time we're finding that it is often
more efficient rather than developing an
algorithm to understand you know how
something works in a fundamental
philosophical sense often that isn't
possible or maybe there there is no
fundamentally best algorithm for for
doing something so I often suspect this
about decision theories where there are
a bunch of different decision theories
that I'm you know each has different
pluses and minuses and you know people
people argue about the merits of them
and think about ways that catastrophic
things could happen depending on which
decision theory you you you use and you
know I think possible way around this is
just have an AI learn these concepts and
maybe the concepts are inherently fuzzy
just like you know it took neural nets
to you know tell the difference between
objects when the differences between
those objects are inherently fuzzy we
may be able to learn safety concepts
that way and a lot of the concerns about
safety are well there are all these edge
cases right what if we you know how do
we make a thing that makes that makes
paperclips and strangely it may actually
be the case that not trying to solve the
problem directly and taking one step of
abstraction up may allow us to solve it
at the same time there are still worries
that we may not have enough of that kind
of data we may not have the right models
to do it so even if that's right then
you know there's so many ways in which
we could have systems that are very
capable but that managed to be dangerous
in ways that we can't easily detect so I
don't I don't really know for sure but
you know those are those are some
reasons for optimism at least who wants
to take the hard viewpoint
I would want to add something you're
Daria's points if that's okay sure I'm
like on the on the optimistic side and I
think one one other point that makes me
kind of an optimistic is that I see more
and more just like traditional machine
learning researchers starting to think
about safety problems so for example
there is there was people that did mine
huge to work on safe expression kind of
unrelated to our technical area safety
team just because they realize that this
comes up in their day to day work and
they just have to solve this problem and
I expect as machine learning is being
applied to more and more problem that
this is going to happen more and more
and like one of my people kind of
naturally started look at I start
looking at AI safety problems and trying
to solve them
and I think that's a great thing and
okay I'll try and give a view for why it
might be hard and or why like the thing
that makes me personally think it's hard
it's just that some of the people like
that are the smartest people I know and
have looked into this a bunch think it
might be pretty hard and so I think that
the poll Cristiano has what looks to me
the closest to an approach which of all
the outline of an approach which gets
you to safety that he lays out in his
blog on AI alignment and he's not sure
if it's like will work and well I video
I think actually this looks pretty
plausible but he's spent a lot more time
thinking about it that I have and he's
not sure and wait I who was extremely
smart and has also spent a bunch of time
thinking about this things that probably
won't work so I I work with Paul
Cristiano and I probably say that I'm
he's probably about as pessimistic or
optimistic as as I am and probably for
similar reasons so you know I mean I
don't yeah I I don't I don't I don't
have exact answers here but we've we've
talked a lot and we have kind of similar
views on what will be hard and what
won't be hard and his agenda kind of
states things in a different way but I
think one of his themes is similar to
the thing I said which is you know he
wrote a post about corage ability in
particular where it's like you
there's all these paradoxes with you
know if you try algorithms for Quaritch
ability and maybe isn't that hard to
learn the concept of corage ability and
learn the concept of honesty and these
are fuzzy concepts and so if you kind of
try and define define them too you know
like our rule-based system or a very
simple machine learning system they
don't they don't make a lot of sense but
but but neither do it other advanced non
non safety related concepts that that
said I mean to give an object level
argument about why things might be hard
and kind of not not contradict but
supplement what I said before you know
people talk a lot about value alignment
and learning the right objective
functions I think we're just gonna have
a lot of very mundane problems with you
know you know you're developing a
self-driving car you know the training
data differs from a test data you know
you put it in the real world it doesn't
do the right thing and if it screws up
it kills someone with very powerful AI
if you're trying to you know make
breakthroughs in biology if you're
trying to resolve international conflict
messing up could kill a lot of people I
can even think of scenarios where it
could kill everyone and there's probably
a lot of those scenarios we'll be
dealing with a very new technology so to
me that's that's one way that
catastrophe could happen is Nate Suarez
in the audience
all right well ask him later what is the
the most important problem in an AI and
we'll say I feel like an altruistic
perspective that no one are almost no
one is working on
I think I think one kind of big question
that puzzles me is like what do we do
when we get there like say we build a GI
then what and this is kind of an
important question Wayne this is also
something that like we as a society have
to figure out and of course like hey I
might be a long way away but I think it
makes sense to kind of think about this
question early and kind of take all the
like the global the holistic perspective
our about our society like what doesn't
mean for a government what does the mean
like how do we get a structure society
what are we gonna do for work or what
what is it gonna look like I have an
idea how to approach that problem though
I don't know I like I feel like I have
the wrong background to really tackle
this question like I'm I commute
assigned as a mathematician by training
I mean it you know this is the kind of
work that Alan day fo at Yale and Oxford
was working on I think page you're
working on it some as well as alright
and yeah so I would echo you on that
this is absolutely something I would
love to see see more work on work on as
well I would yeah I doubt agreement to
that as well you know we've been calling
this AI strategy or AI policy I think it
relates to what Helen said with uh with
interaction risk just the idea that you
know suppose on one hand suppose we
succeed in making an AI system safe it's
still very powerful and they could still
could still lead to kind of very sharp
discontinuities and in in in who
controls what could upset the
geopolitical balance of power I think
that's uh that's that's pretty scary
people tend not to think about it
because they think about safety but I
think we should act as if we're likely
to succeed so if we solve safety then
we'll have to face this problem and so
we should think about this problem ahead
of time as well it's no good to solve a
safety problem if we've just if we just
totally you know don't don't don't think
about this this problem as well all
right we have about 45 seconds left so I
imagine a lot of people here are
thinking about whether they should get
involved with AI in some way any any
advice on how they should even think
about that question think about AI less
as a black box like actually think about
what you expect it will look like and if
you feel like that's kind of juicy to
you and you can maybe make some progress
on that and you can add novel thoughts
that might be a good sign that you can
make progress on some facet of this yeah
I would I would say I'm in the spirit of
wills opening talk for this weekend
which I talked about the portfolio
approach to choosing what to work on and
thinking about you know the EI community
or you know the world or the AI research
community as a portfolio and adding the
best thing that you can to that
portfolio I would just you know put in a
vote for thinking about what interests
you and what your strengths are and
following that because AI is a gigantic
space there's technical research there's
policy research you know all of the
organizations working on this need
operations support there's all kinds of
other you know ways that you could be
helpful and you're likely to be most
helpful if you're working on something
that you're good at and you're
interested in so I would advocate for
thinking more about that and a little
less about trying to figure out what the
absolutely most important aspect is any
parting thoughts
great oh that's it we're out of time
thank you guys
[Applause]


Generative Adversarial Networks (GANs) - Computerphile

So today, I thought we talk about generative adversarial networks because they're really cool, and they've
They can do a lot of really cool things people have used them for all kinds of things
Things like you know you draw a sketch of a shoe
And it will render you an actual picture of a shoe or a handbag
They're fairly low-resolution right now, but it's very impressive the way that they can produce
real quite good-looking images
You could make a neural network
That's a classifier right you give it lots and lots of pictures of cats and lots and lots of pictures of dogs
and you say you know you present it with a picture of a cat and
It says it outputs a number. Let's say between zero and one
and
Zero represents cats and one represents dogs and so you give it a cat and it puts out one and you say no
That's not right should be zero and you keep training it until eventually it can tell the difference right?
so
somewhere inside that
Network
It's... it must have formed some model of what cats are and what dogs are, at least as far as images of
images of them are concerned
But
That model really... you can only really use it to classify things
You can't say "ok draw me a new cat picture", "draw me a cat picture I haven't seen before"
It doesn't know how to do that so quite often you want a model that can generate new
Samples you have so you give it a bunch of samples from a particular distribution, and you want it to
Give you more samples which are also from that same distribution, so it has to learn the underlying
Structure of what you've given it. And that's kind of tricky, actually.
There's a lot of...
Well there's a lot of challenges involved in that.
Well, let's be honest
I don't think as a human you can find that tricky
You know if... if I know what a cat looks like but, uh, being not the greatest artist in the world
I'm not sure that I could draw you a decent cat. So, you know, that this is not confined to just
Computing is it? This...
Yeah, that's true. That's really true.
but if you take
Let's do like a really simple example of a generative model
say you you give your network one thing
It looks like this.
And then you give it another one you're like these are your training samples looks like this
You give it another one that looks like this, and then...
What are those dots in the systems?
Instances of something on two dimensions?
Yeah, I mean right now, it's literally just data. We just... it doesn't matter what it is
Just some... yeah, these are these are data points
And so these are the things you're giving it, and then it will learn
You can train it. It will learn a model, and the model it might learn is something like this, right?
It's figured out that these dots all lie along a path, and if its model was always to draw a line
Then it could learn by adjusting the parameters of that line
It would move the line around until it found a line that was a good fit, and generally gave you a good prediction.
But then if you were to ask this model:
"Okay, now make me a new one"
unless you did something clever, what you get is probably this, because that is on average
The closest to any of these, because any of these dots you don't know if they're going to be above or below
or, you know, to the left or the right. There's no pattern there. It's kind of random.
So the best place you can go that will minimize your error, is to go just right on the line every time.
But anybody looking at this will say: "well, that's fake"
That's not a plausible example of something from this distribution, even though for a lot of the
like, error functions, that people use when training networks this would perform best, so it's this interesting situation where
There's not just one right answer.
you know, generally speaking the way that neuron networks work is:
you're training them towards a specific you have a label or you have a
you have an output a target output and
You get penalty the further away you are from that output, whereas in in a in an application like this
There's effect... there's basically an infinite number of perfectly valid
Outputs here
But, so, to generate this what you actually need is to take this model and then apply some randomness, you say: "they're all
Within, you know,
They occur randomly and they're normally distributed around this line with this standard deviation" or whatever.
But a lot of models would have a hard time actually
picking one of all of the possibilities
And they would have this tendency to kind of smooth things out and go for the average, whereas we actually just want
"Just pick me one doesn't matter". So that's part of the problem of generating.
Adversarial training is is help is a way of
training
Not just networks, actually, a way of training machine learning systems.
Which
involves focusing on
the system's weaknesses.
So, if you are learning... let's say you're teaching your
Network to recognize handwritten digits.
The normal way you would do that you have your big training sample of labeled samples
You've got an array of pixels that looks like a three and then it's labeled with three and so on.
And the normal way
that you would train a network with this is you would just
Present all of them pretty much at random. You'd present as many ones as two as threes and just keep throwing examples at it
"What's this?", you know, "Yes, you got that right", "no. You've got that wrong, It should really be this".
And keep doing that and the system will eventually learn
but
If you were actually teaching a person to recognize the numbers, if you were teaching a child
you wouldn't do that, like, if you'd been teaching them for a while, presenting them and
You know, getting the response and correcting them and so on, and you noticed that they can do...
you know... with 2 3 4 5 6 8 & 9 they're getting like 70 80 percent
You know, accuracy recognition rate.
But 1 & 7 it's like 50/50, because any time they get a 1 or a 7 they just guess because they can't
Tell the difference between them.
If you noticed that you wouldn't keep training those other numbers, right? You would stop and say:
"Well, You know what? we're just gonna focus on 1 & 7 because this is an issue for you".
"I'm gonna keep showing you Ones and 7s and correcting you until
The error rate on ones and 7s comes down to the error rate that you're getting on your other numbers".
You're focusing the training on the area where the student is failing and
there's kinda of a balance there when you're teaching humans
because if you keep relentlessly focusing on their weaknesses and making them do stuff they can't do all the time
They will just become super discouraged and give up. But neural networks don't have feelings yet, so that's really not an issue.
You can just
continually hammer on the weak points
Find whatever they're having trouble with and focus on that. And so, that behavior,
and I think some people have had teachers where it feels like this,
It feels like an adversary, right? it feels like they want you to fail.
So in fact
you can make them an actual adversary. If you have some process which is genuinely
Doing its best to make the network give as high an error as possible
that will produce this effect where if it spots any weakness it will focus on that and
Thereby force the learner
To learn to not have that weakness anymore. Like one form of adversarial training people sometimes
Do is if you have a game playing program you make it play itself a lot of times
Because all the time. They are trying to look for weaknesses in their opponent and exploit those weaknesses and when they do that
They're forced to then improve or fix those weaknesses in themselves because their opponent is exploiting those weaknesses, so
Every time
the
Every time the system finds a strategy that is extremely good against this opponent
The the opponent, who's also them, has to learn a way of dealing with that strategy. And so on and so on.
So, as the system gets better it forces itself to get better
Because it's continuously having to learn how to play a better and better opponent
It's quite elegant, you know.
This is where we get to generative adversarial. Networks. Let's say
You've got a network you want to...
Let's say you want cat pictures
You know, you want to be able to give it a bunch of pictures of cats and have it
Spit out a new picture of a cat that you've never seen before that looks exactly like a cat
the way that the generative
adversarial network works is it's this architecture where you actually have two networks one of the networks is the discriminator
How's my spelling?
Yeah, like that
The discriminator Network is a classifier right it's a straightforward classifier
You give it an image
And it outputs a number between 0 & 1 and your training that in standard supervised learning way
Then you have a generator and the generator
Is...
Usually a convolutional neural network, although actually both of these can be other processes
But people tend to use in your networks for this.
And the generator, you
give it some random noise, and that's the random,
that's where it gets its source of randomness, so
That it can give multiple answers to the same question effectively.
You give it some random noise and it generates an image
From that noise and the idea is it's supposed to look like a cat
So the way that we do this with a generative adversarial Network is it's this architecture whereby you have two networks
Playing a game
Effectively it's a competitive game. It's adversarial between them and in fact
It's a very similar to the games we talked about in the  Alpha go video.
it's a min/max game
Because these two networks are fighting over one number
one of them wants the number to be high one of them wants the number to be low.
And what that number actually is is the error rate of the discriminator?
so
The discriminator
Wants a low error rate the generator wants a high error rate the discriminators job is to look at an image
which could have come from the original data set or
It could have come from the generator and its job is to say yes. This is a real image or no. This is a fake
any outputs a number between 0 & 1 like 1 for its real and 0 for its fake for example and
the generator
Gets fed as its input. Just some random noise and it then generates an image from that and
it's
Reward you know it's training is
Pretty much the inverse of what the discriminator says
for that image so if it produces an image
Which the discriminator can immediately tell this fake? It gets a negative reward you know it's a
That's it's trained not to do that if it manages to produce an image that the discriminator
Can't tell is fake
Then that's really good so you train them in a inner cycle effectively you you give the discriminator a real image
get its output, then you generate a fake image and get the discriminator that and
Then you give it a real so the discriminator gets alternating real image fake image real image fake image
usually I mean there are things you can do where you
Train them at different rates and whatever but by default they're generally to get any help with this at all, or is it purely
Yes, so if you this is this is like part of what makes this especially clever actually
the generator does get help because
if
You set up the networks right you can use the gradient of the discriminator
to train the generator
so when I
Know you done back propagation before about how neural networks are trained its gradient descent right and in fact we talked about this in like
2014 sure if you were a
You're a blind person climbing a mountain or you're it's really foggy, and you're climbing a mountain you can only see directly
What's underneath your own feet?
You can still climb that mountain if you just follow the gradient you just look directly under me which way is the
You know which way is the ground sloping? This is what we did the hill climb algorithm exactly
Yeah, sometimes people call it hill climbing sometimes people call it gradient descent
It's the same
metaphor
Upside down effectively if we're climbing up or we're climbing down you're training them by gradient descent, which means that
You're not just you're not just able to say
Yes, that's good. No. That's bad
You're actually able to say and you should adjust yours you should adjust your weights in this direction so that you'll move down the gradient
right
So generally you're trying to move down the gradient of error for the network
If you're like if you're training if your training the thing to just recognize cats and dogs you're just moving it
You're moving it down the gradient towards the correct label whereas in this case
The generator is being moved
sort of up the gradient for the discriminators error
So it can find out not just you did well you did badly
But here's how to tweak your weights so that you will so that the discriminator would have been more wrong
So so that you can confuse the discriminator more so you can think of this whole thing?
An analogy people sometimes use is like a a forger and
An expert investigator person right at the beginning, you know let's assume
There's one forger in there's one investigator and all of the art
buyers of the world are idiots at the beginning the the
Level of the the quality of the forgeries is going to be quite low right the guy
Just go get some paint, and he he then he just writes you know Picasso on it
And he can sell it for a lot of money and the investigator comes along and says yeah
I do I don't know that's right or maybe it is. I'm not sure I haven't really figured it out
And then as time goes on the investigator who's the discriminator will?
Start to spot certain things that are different between the things that the forger produces and real paintings
And then they'll start to be able to reliably spot. Oh, this is a fake
You know this uses the wrong type of paint or whatever
So it's fake and once that happens the forger is forced to get better right you can't sell his fakes anymore
He has to find that kind of paint
So he goes and you know
Digs up Egyptian mummies or whatever to get the legit paint and now he can forge again
and now of the discriminator the investigator is fooled and
They have to find a new thing
That distinguishes the real from the fakes and so on and so on in a cycle they force each other to improve
And it's the same thing here
So at the beginning the generator is making just random noise basically because it's it's it's getting random noisy
And it's doing something to it who knows what and it spits out an image and the discriminator goes that looks nothing like a cat
you know
and
then
eventually because the discriminator is also not very smart at the beginning right and
And they just they both get better and better
The generator gets better at producing cat looking things and the discriminator gets better and better at identifying them
until eventually in principle if you run this for long enough theoretically you end up with a situation where the
Generator is creating images that look exactly
Indistinguishable from
Images from the real data set and the discriminator if it's given a real image, or a fake image always outputs 0.5
5050 I
Don't know could be either these things are literally indistinguishable, then you pretty much can throw away the discriminator
And you've got a generator, which you give random noise to and it outputs
brand-new
Indistinguishable images of cats there's another cool thing about this
Which is every every time we ask the generator to generate new image
We're giving it some random data, right we give it just this vector of random numbers
Which you can think of as being a randomly selected point in a space because you know if you give it
If you give it ten random numbers you know between zero and one or whatever that is effectively a point in a 10 dimensional space
and
the thing that's cool is that as
the generator learns
It's forced to
You if the generator is effectively making a mapping from that space into cat pictures
This is called the lateness base by the way generally
Any two nearby points in that latent space will when you put them through the generator produce similar
cabbages you know similar pictures in general
Which means sort of as you move
Around if you sort of take that point and smoothly move it around the latent space you get a smooth léa varying
picture of a cat and so the directions you can move in the space
Actually end up
corresponding to
Something that we as humans might consider meaningful about cats
so there's one you know there's one direction, and it's not necessarily one dimension of the space or whatever but
And it's not necessarily linear or a straight line or anything
But there will be a direction in that space which corresponds to
How big the cat is in the frame for example or another dimension will be the color of the cat or?
whatever
so
That's really cool, because it means that
by
Intuitively you think
The fact that the generator can reliably produce a very large number of images of cats
means it must have some like understanding understanding of
What cats are right or at least what images of cats are
And it's nice to see that it has actually
Structured its latent space in this way that it's by looking at a huge number of pictures of cats it has actually extracted
some of the structure of
cat pictures in general
In a way, which is meaningful when you look at it?
So and that means you can do some really cool things, so one example was they trained Annette one of these systems
on a really large
Database of just face photographs and so it could generate
arbitrarily large number of well as largest the input space a number of different faces and
So they found that actually by doing
basic
arithmetic like just adding and subtracting vectors on the
Latent space would actually produce meaningful changes in the image if you took a bunch of latent
vectors, which when you give them to the generator produce pictures of men and
a bunch of them that produce pictures of women and average those you get a point in your
latent space which
corresponds to
a picture of a man or a picture of a woman which is not one of your input points, but it's sort of representative and
Then you could do the same thing and say oh, I only want
Give me the average point of all of the things that correspond to pictures of men wearing sunglasses
right and
Then if you take your sunglass vector, you're you're men wearing sunglasses vector
Subtract the man vector and add the woman vector you get a point in your space
And if you run that through the generator you get a woman wearing sunglasses
right
So doing doing basic vector arithmetic in your input space actually is?
Meaningful in terms of images in a way that humans would recognize, which means that?
There's there's a sense in which the generator really does
Have an understanding of wearing sunglasses or not or being a man or being a woman
Which is kind of an impressive result
All the way along
But it's not a truly random thing because if I know the key and I can start want to generate the same
Yeah
I'm so I mean that's about
Unfortunate is the problem with cryptography is that we couldn't ever use truly random because we wouldn't be able to decrypt it again
We have our message bits, which are you know naught 1 1 naught something different?
And we XOR these together one bit at a time, and that's how we encrypt


Careers in technical AI safety | Owain Evans and Victoria Krakovna | EA Global: London 2017

now very pleased to introduce a wine
Evans and Victoria Krakov now so a wine
works on AI safety and reinforcement
learning at the future of humanities
Institute at the University of Oxford he
also leads a project on inferring human
preferences with andreas stool ruler of
org he has published papers at nips and
aai and an online interactive textbook
at agent models org his recent
collaboration surveying AI experts is
forthcoming in the Journal of AI
research
Victoria is a research scientist at deep
mind she's working on AI safety she did
her PhD in machine learning and
statistics at Harvard on building
interpretable models she also co-founded
the future of life Institute a nonprofit
organization working to mitigate
technological risks to humanity and
increase the likeliness of a positive
outcome so please join me in welcoming
Orion and Victoria
thank you all for joining us today for
careers in a ICT session
first I will talk about the kind of
research problems that come up with this
area and some of the things we've been
working on and I'll hand it over to a
wine to tell you about how to get into
the field so
in terms of technical AI safety research
there are two major perspectives on the
research agenda
on the one hand we have the machine
learning perspective that's represented
by the concrete problems in AI safety
paper so if you haven't read that one
yet to how they recommend and this is
basically about
the kind of problems that are likely to
arise with advanced AI systems in the
future and are likely to be serious
problems that already come up in some
form for present-day machine learning
systems so we can have some sort of
feedback loop
that can empirical feedback loop between
supporting these problems on present-day
systems while keeping an ad for general
solutions that might generalize to
future systems that are much more
advanced on the other hand we have the
agent foundations agenda
which was developed by Miri and they're
focusing on really foundational problems
in aligning super intelligent systems
such as decision theory in logical
uncertainty
[Music]
both of these research directions are
really important but in this talk we are
going to focus on the machine learning
perspective on technically ICT
machine learning safety research
can be seen as
divided and to drop two broad categories
which is specification and robustness
the specification problems are about
making sure that we can specify human
preferences to air systems in a reliable
way and that if our specifications are
incorrect then our agents can still
figure out how to how to do the right
thing how to do what we want them to do
and they're just what we say that we
want them to do
robustness problems
are about
reliably learning to satisfy a
specification
so on the specification side you have
things like reward gaming where
basically if you have for example
reinforcement learning agent and you
give it a reward function then if your
reward function doesn't perfectly
represent what you wanted to do then
the agent might find a loophole and get
lots and lots of reward without doing
what you want
here unfortunately we don't have the
video but opening I had this awesome
demo where the agent was supposed to be
playing a boat race game so the boat was
supposed to go around the track as fast
as possible and complete the race when
instead it found that it could just go
in another circle and hit the same
targets over and over again and get lots
of reward without actually playing the
game this kind of thing is sort of fun
to watch in a game video but less fun if
your agent is actually trying to do
something important in the world
interrupts ability is another big
problem that has to do with being able
to turn our agents off
when
you know when we need to fix bugs we
need to change their objectives and so
on and the issue here is that
if the agent understands that is about
to get less reward if it's turned off
then if it's not turned off then it has
an incentive to
change its policy changes action to
avoid being shut off
one tricky thing about trying to solve
this problem is that we don't only want
our agents to
not
resist being turned off but we also want
them to not seek being turned off so we
don't want them to cause trouble just so
we turn them off so we have to kind of
align the incentives so that they are
exactly different to being turned off
which hard-
side-effect is the problem of having the
agent that she been objective without
causing unnecessary disruptions to its
environment for example if you have a
robot and you want to carry a box from
point A to point B then you're
implicitly asking it to carry that box
without breaking the bars in its path or
without
you know pumping it to humans without
scratching the furniture etc etc so
there are lots of these kind of
common-sense constraints that we want
the agent to satisfy but we don't want
to have to specify them exclusively and
we want the agent to have some sort of
general heuristic
about not causing disruptions
yes
the other hand we have robustness
problems so even if your specification
is perfect then unsafe things might
happen while the agent is learning to
satisfy the specification
one issue that comes up a lot is
distributional shift where your training
data must not be from exactly the same
distribution as your test data for
instance
maybe you trained you a robot arm to
pick up blocks but you're testing it on
picking up a mug and we want our agents
to feel gracefully in these situations
and see if acceleration is what happens
when the agent is trying to explore lots
of different states so they can find a
good solution but it gets into a state
that it cannot recover from
like this unfortunate room bonus
staircase
so basically current reinforcement
learning systems
often have to explore all the states
many times in order to find a good
solution but a practice that is really
undesirable
because some people not worth exploring
intracellular perturbations are a method
for for example taking the spanda image
and adding some tweaks to the pixel so
it looks exactly like a panda to a human
but the your neuron that will be really
really confident as a given
so basically you can do this
with not just neural networks but also
other machine learning systems and
you can take an image or a
sound file or whatever and tweak it a
little bit and make the neural network
misclassified with high confidence
and right now this has been a very
popular research area there's been lots
of work post on the attack side and
defense side but so far as far as I can
tell the attack side is winning
here are some of the recent work that we
have been doing at deep mind and FHI
on
these technical safety problems
the first one unfortunately we don't
have the video of the backflipping
noodle
deep reinforcement learning computer
preferences is
paper about teaching the agent some kind
of
complicated human preferences that we
don't know how to specify directly so
for example if you wanted to do a back
flip that's hard to design a reward
function to bring it back there
but it is something that we can know
when we see it so human can look at two
videos of the agent try to do back
surface a little smaller back foot
so this paper the use this kind of
pairwise comparisons to allow the agent
to efficiently learn how to do this kind
of complex task
this is a collaboration between deep
mind and open air
and we first know cloning with the crab
reward channel is a formalization of the
problem often specified or what
functions
as you saw on the previous slide with
the boat race
example
sometimes the observed reward that the
agency's does not match what we could
call the true reward which is some kind
of idealized reward function that
represents what we really want the agent
to do
so for example if you look at the
different trajectories that the agent
could take then for most of them maybe
the observed matches its reward but for
that Luke we're just kind of hitting the
same targets over and over again that's
how observed work with low - reward and
in this paper we kind of investigate
what sort of extra information we could
give the agent to enable it to figure
out what the true reward is even in the
presence of corruption
where corruption is basically a state is
corrupt if the true and observed rules
don't matter
if the interruptible agents is
on the interrupts ability problem this
is basically a formalization giving some
you know a definition of interrupts
ability and investigating
what sort of agents are more likely to
be interrupted for example they find
that
off policy reinforcement learning agents
are more likely to be interruptible than
on policy ones
and
yeah the collaboration between dick
Milan FHI and
last but not least we have trial without
error which is
the wine has been working on with
Stanford
and here
basically they want to prevent the agent
from taking catastrophic actions
while it's learning so at the beginning
there is a human that watches the agent
learn and interferes whenever the agent
is about to do something really bad and
there is a
classifier that's trying to predict when
the human is going to intervene and
eventually now in the class the
classifier is drained the classifier can
take over so the human doesn't have to
watch stage in forever
you know if you want to add anything
else about the peeper
wait so yeah these are some of the
things we have been working on in this
area and
if you are excited to do this kind of
work then no one is going to tell you a
bit about how to get into the queue
great thank you so I think the field of
AI is incredibly exciting at the moment
I don't think any field maybe in human
history has illuminated some fundamental
area of
of science in this case the science of
intelligence as quickly and as
excessively as AI and on top of that so
I think it's very exciting time for AI
but also the AI safety problem I think
is ever more important and so I think
there's lots of reason to get into this
field so these are some of the
organizations that are involved probably
by the time some of you finish
undergraduate or PhD there'll be more of
those organizations so this is not going
to be the end of it and many of these
orgs are represented in some form at
this conference
so how do you get into this field how do
you end up one of these organizations or
somewhere else doing AI safety research
so the fundamentals and the graduate
degree or if someone is going back to do
an undergraduate ii undergraduate degree
what you need to cover this is really
just a laundry list of courses standard
math background and some programming
some fundamental kind of deep learning
machine learning reinforcement learning
if you're interested in more theoretical
work the kind that Mary does but other
organizations are doing as well then
more pure maths and computer science
theory if you're interested in being a
software engineer or research engineer
as the sometimes called where you're
working from an engineering perspective
on machine learning very safety then
doing more CS and software engineering
in kind of general tips for the learning
the fundamentals
I'd recommend prioritizing harder
courses
acquiring research experience
as early as you can
so try try to get your name on a paper
and find a mentor supervisor doesn't
need to be someone famous it could be a
grad student or a postdoc
but find someone work closely with them
work on problems that they're excited
about and
lots of these tips are about getting
feedback early about whether you're a
good fit for AI research and whether
it's a good fit for you so whether it's
something you're going to enjoy so take
hard courses to find out if you can if
you can handle them if you enjoy them
okay so that's the fundamentals and then
how do you actually get into
doing research so
I think that the the best background is
still a PhD in machine learning for the
kind of research that people that do
mind or open AI or FA much of the
research that FA chai chai and so on
there are some jobs that that don't
require a PhD but this is a good default
option and
if you're doing a PhD
definitely don't be afraid to do
research outside air safety as a way of
learning developing as a researcher
building up a set of collaborators
PhD in Europe can sometimes be quicker
and
there are some very good places in
Europe so don't rule that out
there's general advice on how to do PhD
going to make go into the major
conferences definitely good ideas of
building a network both of our
researchers and people interested in a
Oh safety
and so well alternatives through a PhD
so there are alternatives one thing you
might do is try to get an internship at
one of the groups doing AI research by
our safety research you don't always
need a PhD for this and that can
sometimes kind of bootstrap your your
career Google brain residency is a great
program explicitly aimed at taking
people without PhD and getting them up
to speed on contemporary deep learning
research people have done great work
there so that's a good launching point
for your career if you don't have
PhD or master's research
engineering is an option especially at
deepmind or open AI and so it's going to
be closer there to the normal software
engineering backgrounds I can talk more
about that in questions there are some
machining startups and
Google open AI FHI some organizations
will hire people with PhDs in related
fields and so fields where you'll do
similar kinds of maths or
statistics or programming and so people
have come from all of these fields to
these organizations so if you do it if
you're fairly advanced in a PhD in those
areas then might just make sense to to
finish it okay
and I just want to we'll take questions
in a sec just want to highlight some
really great resources which should be
the first port of call
eight thousand hours has amazing
resources online and also well worth
talking to them in person if you're
interested in getting into this field
Piku has a list of recent air safety
fields and a review and you should apply
for jobs if you're interested FHI has
two areas a few jobs and I'm always
looking for interns and
there are jobs in deep mind as well I
guess it didn't if you want to say
anything that
yeah I
can apply to the safety team or also
apply for internships so we have a
couple of insurance this year we'll
probably have insurance of future years
great okay so we'll end there thank you
[Applause]
[Music]
yes you can still submit questions on
online
so yeah it's our first question quite
quite generalist but um
given the sort of wrist in the necks of
centuries or decades why why do you
think this is herbs of AIChE would be a
priority in your in your thought of
opinion
like why is a safety priority over the
collar causes
so
one thing about
sort of try to predict progress in AI is
that it's very hard and if you look at
surveys of experts like for example the
AI impact survey that came out recently
then you can see that there is just a
huge variance in expert predictions on
when advanced AI or
human-level AI might be built and
basically I think the takeaway from that
is that we really don't know and the
experts really don't know so
it could you know it could potentially
be
100 years it could potentially be
just a couple of decades away and
especially in the case that it might be
sooner like it makes sense to start
working on a safety and
especially since a lot of these problems
are quite difficult both on the
foundation side and on the machine
learning side we don't need to start
working in these now and not wait until
we are certain that
you know advanced AI is upon us because
that would be too late and
it also takes time to build a research
community around safety and
who integrated with the machine learning
community that's something that we're
doing now it's important to continue
doing it
what would you both consider is
something that's maybe missing in the AI
community currently
give me that AIE rather than a safety
sorry
I can throw some things out there I'd
say
so
so one thing is the
safety in some ways is a very difficult
thing to study because the ultimate aim
is to build systems that have where we
have some guarantee that the system is
going to be safe even if it's
capabilities like far beyond humans and
far beyond maybe its initial state
before it does any any learning or
something like that and
to get a guarantee you need to room out
like every possible thing that could go
wrong maybe you need some kind of some
evidence like that that you're sure that
there's not going to be some mistakes
you've made and so you really need to be
aware of the big picture and I think
there aren't maybe that many people
combining the big picture thinking with
the
modular machine learning research the
kind of thing that can make it into
papers so one thing is it's difficult to
write a paper where you address the
whole big picture issue given that it's
got to fit into seven or eight pages
but I think that
it's really important to keep in mind
both the big picture goal that you've
got to have a system that is is kind of
completely reliable and is able to deal
with all kinds of you know you make sure
all the possible holes are patched and
at the same time then doing incremental
research that is modular and that other
people can build on
I've just like to add to that that
yes it is in fact quite helpful for more
people to kind of keep the big picture
in mind while you can get the machine
learning safety problems and
it would be good to
build more bridges between example the
Foundation's agenda and the machine
learning agenda because right now
they're still quite separate and there's
not very much overlap but potentially
there could be problems that are sort of
somewhere in the middle and generally
there may well be more sort of unknown
unknowns out there that we haven't
thought about yet that are not on any of
our agendas and I think we need people
to think outside the box and look for
problems that may not have been
discovered yet
okay great and so given how many people
are working in AI safety currently how
much more do you think we need me to
scale this in the next decade two times
more people five times more people
yes well if it was ten times more people
I would feel much happier about the
state of the field I think
so far it's still quite small and
there's a lot of difficult problems out
there and I
think the existing researchers are
making good progress but it feels like
right now it feels like there are too
few of us to really kind of make forward
progress on all these things so yeah I
think they can order of magnitude more
qualified and passionate researchers
would be that would be a big step what
do you think yeah I completely agree so
it's still a tiny field so machine
learning is not even that big an
academic field it will grow a lot it's
much bigger at Google because they can
act quickly academia it's not so big but
a huge proportion of machinery is people
doing things like
labeling images so you know finding a
cat in the image and like draw a box
around the cat is a huge amount of
research in that problem which is you
know important and useful and so on
there's like a
huge amount of research on ad clicks
basically things like that predicting
when people will click on an ad and
there's really a tiny amount of work on
fundamental problems in air safety and I
think if there were more people doing it
it would be very clear that there are
lots of hard problems for those people
to sink their sink their teeth into and
lots of interesting directions so yeah I
think I think it can grow like grow a
lot yes so on that point what do you
think is currently being done to sort of
gain that order of magnitude growth in
in safety research
I would say open fill is bland three
projects doing amazing work funding
academic researchers and
groups like open AI
epic Chow is hiring and scaling up I
can't speak you know deep mind is hiring
I guess yeah I think we are still
looking for more people to work on
safety I
think yeah at the moment I see this as
being mostly a
qualified talent bottleneck so people
who are both both have the research
experience and qualifications and are
passionate about air safety
just to give a a plug I'm on the board
of a new a small area safety nonprofit
funded by of the Flannery project which
is called org it's going to be based in
San Francisco
websites not alive yet but if you want
to work on our safety have a software
engineering background maybe or a
research background one live in San
Francisco but with some great people
there are new opportunities there and so
will I'm working to it's trying to have
there be other centers of this kind of
research which I think it's important
the
future of life Institute had a grant
program at some point to fund a higher
diversity of
teams to work on these problems and
there should be a second grant round I'm
not sure exactly when but I
think it's going to exist so
like one thing that's important both for
the pipeline and for solving the
problems is to
have more different teams working on the
problems from different angles and from
different perspectives yes especially
with I feel that's a little bit
speculative in the sense that we are
trying to predict what future systems
they're about you like issues will arise
it's important that we
try to approach these problems from
different perspectives so that some of
us will be right hopefully yeah okay
great and if you want to pick their
minds at all both Orion and Victoria and
actually Andrew from the previous talk
will be hosting and all will be having
office hours now which I believe is
actually on this floor we have a break
now until 3:45 so if you could join me
in really thanking a wine in Victoria
for their time
[Applause]
[Music]


Experts' Predictions about the Future of AI

hi there's a lot of disagreement about
the future of AI but there's also a lot
of disagreement about what the experts
think about the future of AI I sometimes
hear people saying that all of this
concern about AI risk just comes from
watching too much sci-fi and the actual
AI researchers aren't worried about it
at all when it comes to timelines some
people will claim that the experts agree
that AGI is hundreds of years away
prediction as they say is very difficult
especially about the future and that's
because we don't have data about it yet
but expert opinion about the future
exists in the present so we can do
science on it we can survey the experts
we can find the expert consensus and
that's what this paper is trying to do
it's called when will a I exceed human
performance evidence from AI experts so
these researchers from the future of
humanity Institute at the University of
Oxford the AI impact project and Yale
University ran a survey they asked every
researcher who published in ICML or nips
in 2015
those two are pretty much the most
prestigious AI conferences right now so
this survey got 352 of the top AI
researchers and asked them all sorts of
questions about the future of AI and the
experts all agreed that they did not
agree with each other and Robert Aumann
didn't even agree with that there was a
lot of variation in people's predictions
but that's to be expected
and the paper uses statistical methods
to aggregate these opinions into
something we can use for example here's
the graph showing when the respondents
think will achieve high level machine
intelligence which is defined as when
unaided machines can accomplish every
task better and more cheaply than human
workers so that's roughly equivalent to
what I mean when I say super
intelligence you can see these gray
lines show how the graph would look with
different randomly chosen subsets of the
forecasts and there's a lot of variation
there but the aggregate forecast in red
shows that overall the experts think
we'll pass 50% chance of achieving high
level machine intelligence about 45
years from now well that's from 2016 so
more like 43 years from now and they
give a 10% chance of it happening within
nine years which is seven years now so
it's probably not too soon to be
concerned about it a quick side point
about surveys by the way in a 2010 poll
44% of Americans said that they
supported homosexuals serving openly in
the military in the same poll 58% of
respondents said
they supported gay men and lesbians
serving openly in the military
implicitly fourteen percent of
respondents supported gay men and
lesbians but did not support homosexuals
something similar seems to be going on
in this survey because when the
researchers were asked when they thought
all occupations would be fully automated
all defined as for any occupation
machines could be built to carry out the
task better and more cheaply than human
workers they gave their 50% estimate at
a hundred and twenty two years compared
to forty five for high-level machine
intelligence these are very similar
questions from this we can conclude that
Aix PERTs are really uncertain about
this and precise wording in surveys can
have a surprisingly big effect on the
results figure two in the paper shows
the median estimates for lots of
different a AI milestones this is really
interesting because it gives a nice
overview of how difficult a AI
researchers expect these different
things to be for example human level
Starcraft play seems like it will take
about as long as human level laundry
folding also interesting here is the
game of go remember this is before
alphago the AI experts expected go to
take about 12 years and that's why
alphago was such a big deal it was about
eleven years ahead of people's
expectations but what milestone is at
the top what tasks do the AI researchers
think will take the longest to achieve
longer even than high-level machine
intelligence that's able to do all human
tasks that's right it's AI research
anyway on to questions of safety and
risk this section is for those who think
that people like me should stop making a
fuss about AI safety because the AI
experts all agree that it's not a
problem first of all the AI experts
don't all agree about anything but let's
look at the questions this one asks
about the expected outcome of high-level
machine intelligence the researchers are
fairly optimistic overall giving on
average a 25% chance for a good outcome
and a 20% chance for an extremely good
outcome but they nonetheless gave a 10%
chance for a bad outcome and 5% for an
outcome described as extremely bad for
example human extinction 5% chance of
human extinction level badness is a
cause for concern moving on this
question asks the experts to read
Stewart Russell's argument for why
highly advanced AI might pose a risk
this is very close
related to the arguments I've been
making on YouTube it says the primary
concern with highly advanced AI is not
spooky emergent consciousness but simply
the ability to make high quality
decisions here quality refers to the
expected outcome utility of actions
taken now we have a problem
one the utility function may not be
perfectly aligned with the values of the
human race which are at best very
difficult to pin down to any
sufficiently capable intelligent system
will prefer to ensure its own continued
existence and to acquire physical and
computational resources not for their
own sake but to succeed in its assigned
tasks a system that is optimizing a
function of n variables where the
objective depends on a subset of size K
less than n will often set the remaining
unconstrained variables to extreme
values if one of those unconstrained
values is actually something we care
about the solution may be highly
undesirable this is essentially the old
story of the genie in the lamp or The
Sorcerer's Apprentice or King Midas you
get exactly what you asked for not what
you want so do the AI experts agree with
that
well 11% of them think no it's not a
real problem 19 percent think no it's
not an important problem but the
remainder 70% of the AI experts agree
that this is at least a moderately
important problem and how much do the AI
experts think that society should
prioritize AI safety research well 48%
of them think we should prioritize it
more than we currently are and only 11%
think we should prioritize it less so
there we are
AI experts are very unclear about what
the future holds but they think the
catastrophic risks are possible and that
this is an important problem so we need
to do more AI safety research
I want to end the video by saying thank
you so much to my excellent patreon
supporters these people and in this
video I'm especially thanking Jason hice
who's been a patron for a while now
we've had some quite interesting
discussions over a patreon chat been fun
so thank you Jason and thank you all for
watching I'll see you next
[Music]