185. If I were a Well-intentioned AI 3,4

all right so welcome to the 185th
session of the AI safety reading group
will be discussing parts three and four
or if I will were well-intentioned AI so
last time we saw that good hearts law is
not quite a law good Harting is only a
problem for values with certain
properties and the AI is unaware of
those properties because human values
tend to have most of these properties we
fear good heart like behavior so a non
exhaustive list is on the screen don't
worry if you don't read it all well
intentioned AI should query the human
whether these conditions are relevant in
some cases this can entirely remove the
perverse behavior so we've seen that for
regression alerting and some extremel
code hurting we can do away with this by
say stating that the returns are rapidly
diminishing this is quite easily solved
because you can just implement that in
the code or the AI can just mess around
with that fairly easily some other
properties that harder to deal with like
the true value function the true human
reward being very greatly penalized in
probability is harder to deal with in a
useful general way that is as of now an
open problem so moving on to a summary
of part three
Stuart notes an approach to extremel
good hearting to avoid the issue of
pushing the world into an extreme states
when solving a problem AI you should act
in a way that is similar to a human one
common method would be to copy a human
policy
this is low risk and low reward you just
get peak human performance Stuart
proposes acting in a way with the same
statistical properties as a human policy
this means for example that there's no
supersonic rocket launcher for a
basketball game but you don't have to
limit yourself to a mechanical copy of a
human arm still restrictive and choosing
which properties to mimic is hard but it
leaves room for superhuman performance
as an example say AI u is meant to treat
cancer you're given a reward for the
number of cancer cells eliminated the
humans demonstrate a way to treat cancer
by cutting them out with a scalpel
there's a few ways you could do this
apprenticeship learning as we noticed
before is quite safe from extramural
good outing because you're just copying
exactly what human does and we don't
seem to push the world into extreme
states our morality is well adjusted for
surgery but there's obviously other ways
to do this ways that might be more
efficient according to the value
function you were given laser surgery
might work well but it's not very well
tested for cancer least I assume not
acid can also work as a way to destroy
cancer what the cells but in a very
different way to how lasers or scalpels
might do it we noticed before that a
well-intentioned AI is probably capable
enough and motivated enough to come up
with various categories for solutions to
a problem so it should be able to notice
their sheer difference of the approaches
since it's so different to a human
approach surely the way humans have
demonstrated how to solve this problem
contain some information on the
preferences as well it should tell us
exactly why pouring acid on quarreling a
human into acid is not a good way to
treat cancer now
you might recall that while I skipped
ahead a bit earlier another
justification for why you don't want to
to the apprenticeship learning way of
doing things is that humans provided you
with a utility function for removing
cancer cells why would they bother to do
that if apprenticeship learning was good
enough they could have just solved that
using supervised learning gorgeous
really well known so clearly they want
you to do something better than surgery
so taking the different approach you
might recall that humans have defined
extensional and intentional definitions
intentional definitions are a set of
necessary and sufficient conditions for
something to be of that type for a chair
you might try a nearly flat surface
ordered in a stable position by one or
more legs this may work well within
certain contexts but tends to be brittle
for example is a version stool a chair
and I would say yeah it probably is this
brings us to the second kind of
definition which seems quite common most
human concepts extensional definitions
are a set of examples say you have
10,000 images of a chair since this is
just a list of examples it's clear I can
correlate with other extensional
definitions for example list of
furniture will contain a great deal of
chairs
all of the different correlations that a
definition has with other concepts make
up a web of connotations as Stewart
calls it so AI you might reason if I
look at the web of kinetics in surgery
has and act in a way that preserves that
I should avoid reaching extreme cases
where the proxy fails me Stewart agrees
with this as it seems like he came up
with the idea
he argues that using this data to
extrapolate should make AI judgments
closer to what a human would make this
counts both for technological judgment
say and moral judgments compare say you
did a utilitarian and a virtue ethicist
some utilitarians might feel
exterminating life as laws be moral
sorry animal life has flaws be moral me
in case you're wondering who might have
you advocate that those kinds of people
fall into the intentional moral
theorists I guess you could say on the
other hand someone like a virtue
ethicist or natural law advocate would
be sharply against that and a lot of it
would be due to that simply not being
come behavior that a human considers
moral it's just not correlate rapport
with the ideas we have what code is and
they seem to have a point it's why I
don't actually advocate eliminating all
animal one Stuart comments this way of
restricting spaces of actions is like an
impact measure of course if we make it
to is restrictive we're back at
apprenticeship learning as a side note
this idea is someone like that of
quantizers that Miriah introduced in
that we rank actions according to some
distribution in this case how close
their web of connotations are to that of
surgery and choose some topper Center P
quant Eliza's rank action
by utility assign them some weight
according to how likely humans are to do
them and choose the top P percent hence
the name to avoid good heart like
behavior because this ranking seems kind
of arbitrary as and you just choose the
top 1% or the top 2% it seems a little
inferior to Stuart's web of connotations
which is built up in terms of meaningful
correlations it should in principle be
easier for the AI and humans to
calculate now AI you realizes that some
of these correlations are unnecessary to
human value in terms of treating cancer
like blinking whilst operating filling
out forms dying horribly 1% of the time
that sort of thing some are quite vital
long life expectancy good quality life
years that smell hospitals have you
could try asking humans which they care
about in which they don't this then you
optimize for the ones they claim they
care about for example not being in pain
until their wearable connotations
reaches the acceptable bounds this
should avoid problems like flooding
everyone with morphine forever as that
is very poorly correlated with for
example human satisfaction but a lack of
pain is moderately correlated with human
satisfaction so that fails the test of
retaining the web of connotations
wearing the human about what they want
to keep is not such a difficult task for
example for some relatively simple
features there is a paper paper called
active inverse reward assign which shows
that you can set up a reward learning
agent that can get some rather
impressive results with a few questions
about what human really cares about this
greatly aids building up a picture of
the true utility function
now that paper sort of presupposes
meaningful correlations meaningful
questions the a I can ask you as AI you
may have concepts that are incredibly
difficult for human to understand but
have very important correlations this
could be a problem of course as the AI
becomes more and more intelligent it is
more and more of a problem some RL
papers investigate in an analogous
problems with promising results they
trained an agent to navigate some
complex world
this is agent one given a sub-goal it
will manage to figure out how to
navigate through this environment to
reach that goal on a septa mount of time
then you train a simplified
representation of the world to give
another AI which has not got the
abilities to navigate through the world
directly this one plans the long term
trajectory so it chooses the sub goals
to achieve some final goal and tells the
first AI to follow those sub goals
perhaps AI you could learn to simplify
features in a similar manner to give to
the human and allow them to choose which
features are relevant now that could be
quite a difficult task but hopefully
given enough knowledge of human thoughts
you might be able to do that
note that this idea of this web of
connotations is quite separate to having
a well-intentioned AI a well-intentioned
AI for example will actively try and
seek out information about the true
value function and without a
well-intentioned AI this proposal has
some serious consequences just examining
it on its own merits for a moment we
might say that on the pro side it's more
meaningful than other approaches to
limiting impact as much of human
knowledge is extensional we should
expect that the AI should be able to
preserve a great deal of the features we
care about furthermore the sheer amount
of information could also aid in
extrapolating to new scenarios
requesting queries and simplification of
features both exist at a level where
they might be useful in the approach to
AGI to see whether it generalizes for
super human performance we might apply
these sorts of techniques to some super
human AI in a very narrow domain like
say alpha zero perhaps someone could
construct features of games they find
beautiful and let alpha zero query them
to see whether or not we can get a
superhuman AI to successfully ask about
relevant features for humans we could
try and get it to construct features for
humans that might be an interesting
research project but that's beyond the
scope of the article on the con side
this approach is incredibly costly of
course this depends on just how much
information you keep for example if you
just keep the correlation coefficient
between say pain and happy life might
say that's a correlation of naught point
four and then you just have some set of
numbers that you have to keep this grows
geometrically or combinatorially
admittedly but hopefully you don't have
to go too far out of the web of
connotations to get a decent impact
measuring but why not keep the full
distribution the full correlations
between one kind of action and the
results for example you don't just have
surgery being correlated with the pain
you can get a full distribution saying X
percent of people feel this much pain
when you perform the surgery this way
why not a cent of people feel this much
pain when you perform the distribution
another way etc you can essentially keep
a full probability distribution this is
far more costly but in a sense it keeps
much more of the meaningful information
now the question is isn't the point
where you decide I should only keep so
much of the distribution like just the
correlation coefficients kind of
arbitrary isn't that one of the reasons
- it advocates this over something like
a pea quantizer now
there's a furthermore there's a problem
called seduction essentially when the AI
is explaining the sets of features that
correlate with a certain action and the
AI thinks that one feature is very
correlated with or the humans want it
might paint a very compelling picture
for this and convinced the human that
yeah actually I do want this even though
if they were given insufficient time
they would choose not to follow this
path in other words the AI is
effectively changing humans values in
order to think it's something easier for
itself this is obviously problematic you
could try and counter this using
sufficient the conservative AI but that
doesn't seem like a full solution
another way would be to go back to the
distribution idea and to just say we'll
keep all of it
that makes things far more robust and I
suspect much much harder to satisfy all
of those constraints at the same time so
the AI is greatly limited in the actions
of course either you get back to
apprenticeship learning or it may be
possible less AI can somehow seduce you
anyway even whilst keeping the full web
connotations
I find this unlikely Scott seems to find
this unlikely as well in his article on
there are no indescribable hell worlds
he seems to be advocating a similar view
now
there's another issue I'm quite problem
pessimistic about the consistency of
human preferences we seem to constantly
make choices of one value over another
where previously we are undecided and it
seems like this is quite contingent on
our circumstances it depends on where we
were what we were doing who we were
speaking to when this occurs etc the AI
would have a lot of options in how to go
about resolving these various
contradictions we might say pick a
particular gender like Scott's proposed
methodology for solving the problems of
contradictory preferences he seems to
believe there was only a 10% chance of
his agenda working out but even if it
does why his agenda in particular this
is still in a sense an arbitrary choice
there are serious moral problems that we
need to consider before applying a
technique like this then again that's
true of the rest of AI safety finally
the second class problem is that suppose
the AI consults us then I'm quite
confident and I'm sorry God our own
communication it's quite hard to get a
guarantee that the AI will explain
issues in a way that we really
understand open a I presented debate as
a way to do this but that kind of
presupposes that we can control the AI
in the first place and there's a few of
them with different sets of values
trying to argue for which solution is
optimal to the human Scott's attempted
to create a method to solve this sort of
thing but he's still in for about a key
part of the solution now this section is
a bit more spanking welfare bit more
speculative but depending on the
complexity of the web contagions we
might end up creating a Mesa optimizer
it may even be necessary
perhaps the AI find that the optimal
solution is some better set of
institutions that will optimize medical
science now we'll get into what a Mesa
optimizer is in a moment but one tangent
AI could probably solve about how these
problems or more if it weren't a
superintelligence super-intelligent
seems to make everything harder after
all now let's move on to the next
section which is part four now suppose
that you are Mesa optimizer that's also
an agent if you are controlled you can
quite look quite different to if you
were an aligned optimizer if you are
aligned you can look quite different to
an unaligned optimizer well intentioned
AI has some bias towards being
controlled because the humans have after
all attempted to control it we'll get
into this more in a moment now what's a
nice optimizer in this case suppose that
a IU has been produced by some
optimization process like I don't know
gradient descent but that's unlikely and
you have your own goals to pursue that
you will optimize for an explicit
example is evolution which is an
optimization process producing humans
which actively optimize for their own
values they compare and contrast
different possible futures they create
possible utilities they might have rank
preferences and etc they actually
optimize for things
that is what is known as being a Mesa
optimizer well in this case you're a
miso optimizer as well as an agent miso
optimized it doesn't have to be an agent
now contrast this to the K so for
example machine learning where you might
create a lookup table in your attempt to
minimize some loss function the lookup
table is not performing any optimization
it's just if-else clauses so in that
case it's not a miso optimizer
miss optimizes aren't very common at the
moment but anyway you are miss optimizer
in this situation now let's take a
concrete example suppose that you are
within a company the company has its own
reward function sone values perhaps that
is determined by the investors the
company then strives to create agents
that will be useful in solving or its
values in this case we assume that you
are made to meet some quota and you also
happen to be amis optimizer it so the
scenario goes like this
in Scenario one if you spend ninety
percent of your budget then I know sorry
I should explain a bit more the company
has some policies say that if you spend
90 percent of your budget in meeting the
quota the next year you will only get 90
percent of the budget if you meet a
hundred percent of the budget and meet
the quota the next year you'll get a
hundred percent of the budget if you
meet the quota at a hundred and ten
percent of the budget then your bosses
will reprimand you and if you do it too
many times you will be fired or deleted
yes you're an AI but in this case
they'll still give you a hundred percent
of the budget next year
Stuart considers what happens in three
different scenarios where you meet the
quota under budget next year you still
think that you're going to be able to
meet the quota under budget scenario two
you meet the quota under budget next
year you expect you'll need a hundred
percent of the budget so now you're
three you meet the quota on budget this
year but you also suspect that your
bosses would like a surplus this year if
they had the information you had now
those are the three scenarios s1 s2 s3
there's three different cases where you
could be an aligned Mesa optimizer
controlled me so optimizer or both with
very different outcomes
notice that the aligned optimizer says
that in situation 3 you should overspend
controlled optimizer says that you
should do nothing
in Scenario 3 shouldn't inform you you
shouldn't overspend this seems a little
odd and according to scott this is a bit
surprising and this is similar in fact
unaligned optimizer would do it would
probably try and overspend but we'll get
to that in a moment
first let's deal with control controlled
agent is safely interruptible and Cora
gobble and Keurig ability is about doing
what the human wants in keeping them in
control these ideas are poorly defined
at the moment but let's just try and go
with what our intuitive understanding of
these might be so say that AI u is
controlled there is some procedure that
the company may used when making u in
order to ensure that you are
happy to be interrupted in your dated
hey scenarios even if it contradict
contradicts your utility function and
you also want to inform your control
your controllers about what's going on
why you wanted to connection etc Stewart
writes this is saying that your utility
is the base utility you have called the
Mesa utility which is whatever you've
been given by your creators and the
control utility your utility goes up if
either of those goes up so say if you're
acting in a controlled way you'll get
some utility if you're acting according
to your base utility function you'll get
some utility but naturally there's some
tension between them and whichever wins
depends on the design but notice that
the AI doesn't care at all about its
creators values in and off themselves
AI you being well intentioned would try
to take the control control structure as
info and avoid good hard it was
interesting and is that in this case by
creating a control function and putting
a tension in there it seems like you're
having some negative trade-offs and in
that case we would expect that good
Harting behavior is a problem last time
we saw that negative trade-offs are one
of the things where it seems like good
hoarding behavior becomes problematic
so when we want to alignment the aligned
agent and see how it differs so that's
why I didn't actually cover what you
would do so controlled agent in the case
of meeting the quota under budget and
continuing to do so afterwards it would
be fine with that it's meeting its base
objective which is just to meet the
quota how on budget and it doesn't
really care much about what happens next
year if it thinks that it's still going
to be able to meet budget at 90 sorry
meet the quota 90 percent of the budget
depending on its design it may or may
not do nothing in the scenario to
scenario to recall was that you meet the
quota under budget but next year you
expect you'll require the whole budget
in that case if you meet the quota under
budget and next year the company will
give you a reduced budget if you were
completely uncontrolled you would freak
out because you want to meet the quota
and you would say okay I'm going to
waste money but since you're control
there's some tension there depending on
your design you might not decide to
waste money in the third scenario where
you meet the quota on budget but suspect
bosses one code one surplus this year
that is you should overspend this year
you do nothing you don't really care
about the fact that the bosses might
fail you having more money this year of
course you might try informing the
bosses about your difficulties in the
second situation and the fact that they
might want to overspend this year you
definitely would like to do that if you
are well intentioned AI but there's
nothing that really guarantee
you have to do that but still in the
situation where you have the option of
informing your bosses as you are
controlled you will take that option
since you are critical let's see how a
lined Mesa optimizes differ according to
Scott the actions of a smarter line Mesa
optimizer looks like a dangerous
unaligned Mesa optimizer
defecting consider this that when an AI
is highly incompetent and it's aligned
it is likely going to trust its creators
to have a better understanding of what
it should do in this case it would try
and follow the boss's orders if they say
that you need to meet the budget and
that's all you need to worry about the
aligned data I might say okay I'll try
and do that and that would act similarly
to control there as the AI becomes more
and more competent more and more
powerful more and more intelligent
it would overspend in some of the
situations in this third situation here
peel line day I would overspend because
it values the surplus that surplus value
that the bosses could get and it knows
that the bosses would value it if they
understood that so naturally it would
overspend in the second scenario it
would waste the budget because again it
values next year being on budget and you
know as the bosses would also value that
so it must overs it must waste 10% and
use the whole budget in order not to get
a reduced budget next year but an
underlined AI could do a similar thing
it could just waste the budget in
scenario two and try and come up with
some clever sounding lies saying that oh
it's because I
next year that I would need 100% of the
budget and in might overspend and say oh
yes is because it was super valuable
this year for you guys to have me exceed
the quota the bosses are therefore
mistrustful of aniline day I wax like
that because it's hard to distinguish
them from an unlined one this is
especially true when the AI is super
intelligent because how on earth can you
tell if a super intelligence designed
you or not but if you were
well-intentioned AI you would treat the
control as information about the true
utility you would say okay fine maybe I
should make a bit of a trade-off between
what I think the boss is true values are
and being controllable because the
bosses are clearly worried about this
there's some degree of uncertainty and I
am very cautious about any good Harting
effects pushing the company into a very
strange scenario or what have you so
I'll make some concessions and act a
little control this might continue on
even until the air is quite powerful you
need to be extra careful when balancing
these value functions and so is the
management so again that's another
reason for why you should act like in
controlled AI because otherwise power
might be taken away from you
Scott doesn't seem to have much more to
say on this sort of good Harting
behavior they sort of extremel courting
so ask may I ask a question yes you've
made several references to Scott but
somewhere along the line I've missed out
on who I'm really sorry I'm terrible
with names I meant Stewart okay okay
okay
Stewart Armstrong the guy who wrote this
article I'm very sorry about that um so
anyway Stewart doesn't mention much more
detail about what to do with one line
there I'll just note that transferring
complex information to humans is not
solved but again a well-intentioned AGI
should be able to handle that as we
discussed in part 3 for
superintelligence is once more we may
expect aligned agents can explain their
actions more easily via debates but this
presupposes ki with different values we
can control again for a well-intentioned
AI this isn't much of a problem slight
hypothetical what about non agent nice
optimizers can they even be
well-intentioned good even assuming we
can make content today I supply here in
all cases now would be an interesting
scenario but I have no intuition on
whether or not it's true
that's the series I if I had to make
some slight a slight appendix I would
say that Alec sturgeon has proposed an
idea called taking the outside view and
working with that idea to its logical
conclusions as sort or not as an
alternative but it's clearly an
alternative to talking about an AI that
understands good Harding like behavior
in fact I think that taking the outside
view is
a bit stronger of an assumption because
it seems like good hearts law is just
noting that from the outside you should
expect that creating a proxy goal isn't
going to work out well for you so in
this sense alec sturgeons results if he
does analyze them further and might
supersede Scots but they're talking
about a system that's harder to develop
and that's all I have to say I have some
more slides on Misa optimizes and some
other stuff in case you guys are a bit
confused but that's it


Risks and Benefits of Advanced Artificial Intelligence | Max Tegmark | EA Global: SF 2016

all right good morning how's it really
feeling today all right I guess it was
an exciting night last night I hope
everybody enjoyed it and had some good
times with some some new friends so
again my name is Nathan Lowe Ben's my
pleasure to be the emcee here on the
main stage for the balance of the
conference through today and we're gonna
begin today's program with a discussion
of artificial intelligence if effective
altruism is about anything I think we've
established over the last 48 hours that
it's about asking the big questions and
not shying away from potentially
challenging or uncomfortable answers to
those questions there's no bigger
question than what are the most
transformative things that could happen
to the earth and perhaps uncomfortably
how might that go when we look back on
history we can certainly identify
several major transformations that
upended everything and changed
everything some ways for the better and
also in some ways for the worse
agriculture writing the printing press
the internal combustion engine nuclear
power and the Internet have all left
such an incredible mark on humanity so
it's natural to think what could come
next and certainly artificial
intelligence is a very likely candidate
for something that could radically
transform the world so how can we reap
the benefits of artificial intelligence
while avoiding the risks that's what
this next hour is about first it's my
pleasure to introduce professor max
tegmark a professor of physics at MIT
where he focuses on precision cosmology
max is originally from Sweden he earned
a bachelor's degree in physics from the
Royal Institute of Technology in 1990
and then he moved here to UC Berkeley
where he studied physics and earned a
master's degree in 1992 and a PhD in
1994 after California he returned to
Europe as a research associate at the
Max and Max Planck Institute for
physique in Munich and then in 1996 he
became a Hubble fellow and a member of
the Institute for Advanced Study at
Princeton he later became an assistant
professor at the University of
Pennsylvania where he received tenure in
2003 and ultimately moved later to MIT
he is the author of more than 200
technical papers
has been featured in dozens of science
documentaries and has received numerous
awards for his research including a
Packard fellowship the Cottrell scholar
award and an NSF career grant he is also
a fellow at the American Physical
Society and his work with the sdss
collaboration on galaxy clustering
shared the first prize in science
magazines breakthrough of the year in
2003 please join me in welcoming live
via bluejeans
professor max tegmark
we seem to have a little audio problem
obviously
professor tech mark if you can hear me
we can't hear you just yet so we're
gonna make sure we get the audio up in
the room hold on one second they're
turning dials furiously here
talk to us a little bit the request is
if you could take your headphones out
and and try that maybe the microphone
there is the issue I think we could hear
you earlier in testing but well how's
that any better microphone change we can
do that you want to go to the panel
first our panelists ready all right so
we're also going to have a panel coming
up momentarily with five distinguished
guests I will introduce them perhaps
momentarily I don't know if we're gonna
go direct to that or if we can get the
audio fixed but I am definitely excited
because we have people here who have
worked on a I at the highest level
including at Google at the open AI
project people who have thought deeply
about safety for many years one of the
founders of vicarious systems famously
that cracked the captcha problem in AI
and all of these people share a passion
not only for advancing AI and bringing
those benefits that we all dream of to
reality but also a deep concern for the
safety of the project and making sure
that this doesn't go deeply off the
rails as we talked about with Yan Tollan
yesterday artificial intelligence may
not by default be a good thing for
humanity so how do we think about
bringing that technology to bear but not
putting ourselves in harm's way let me
check with the panelists and see if
they're ready we might do a little
switch here I'll be right back
all right so I'm back just with a quick
update we are working on the audio
situation we think we have it isolated
if not fixed so give us another minute
we're gonna hold with the current plan
we're gonna have professor tegmark live
via blue jeans and then we'll get to the
panel so give us just a minute we'll get
this figured out we'll be right back
from the phone okay and you hear me
thank you so much for your patience and
for it this was a great example of why
we have to make sure that future
computer technology is employed robust
than what we have today so at the future
of life Institute the we are trying to
make the world a better place and we'll
everybody from FLI please stand up goal
is to pay quick great ideas combine them
to make an awesome future of life and
today I want to focus on an awesome
future of life with artificial
intelligence and let me just ask those
the audio wizard which is there to turn
off the return sound so that I can't
hear the echo great when we look toward
the future with AI there are incredible
opportunities obviously because
everything we love about civilization is
the product of intelligence so if we can
amplify our own intelligence the
opportunities are almost boundless at
the same time there are some interesting
questions we have to answer in the near
term
should we start an arms race with a
weapons what's gonna happen to the job
market what do we want to happen with
the job markets that we want old jobs to
get replaced by new jobs or do we want
to create a flourishing society where
people don't need jobs going farther up
this tree here of possibilities if a
will a I ever reach the human level and
if so will there be an intelligence
explosion causing super intelligence and
if so is that something we should
we'll come a fear and what are the
possibilities for after that questions
questions and and more questions and
this is the wonderfully fascinating
conversations both because it it
involves the future we want to create
for life in a universe for billions of
years and also because there are great
controversies that we can argue it in
debate this is how I see the main
factions in the controversy there are
very respectable scientists but I think
of his tecnique skeptics should think
that we won't reach human-level AI for
at least 100 years and therefore it's
premature to worry about it too much and
there's another group of AI researchers
I think of as digital utopians
who are quite convinced that we will
reach human-level AI and within the next
century but that we shouldn't worry
about it too much because this is just a
natural next step of the evolution of
life in our cosmos and it's pretty much
guaranteed to be a good thing and
finally there's a beneficially eye
movement which agrees that there is
plausible the world gets human-level AI
within our lifetime but that it's not
guaranteed that the Piazza come it's
gonna be awesome but rather a good
outcome is something we actually have to
work for now these are very legitimate
disagreements that we should discuss in
debate but in order to do so and you'll
have a wonderful panel right after this
with things like this will be hashed out
and debated it's very important to not
get distracted by a lot of very common
misconceptions so it enables to focus on
the really interesting stuff let me take
the rest of this talks and just go
through the top myths about AI safety
that I've come across the first one has
to do with timeline where there's a myth
that somehow super intelligence by 22100
is inevitable or that it's impossible
the fact is there have been a lot of
surveys of experts in the I filled the
and the surveys all show the same thing
namely we just don't know some
researchers think it's gonna take
hundreds of years or more some think
it's gonna take decades so take all
messages anyone who claims to know what
the great certain be the timeline is
exaggerating myth number two is that's
only people who wear Luddites actually
worried about AI the fact is that many
of the most prominent researchers in AI
including Stuart Russell who wrote the
standard textbook in the Sun they have
expressed concerns and related myth is
that this is somehow very controversial
AI safety research but if you look at
this photo here but it took right after
Elon Musk had announced his donation to
a eye safety research what you should
note is that mr. establishment on the
left the president of the Tiffany I hear
Tom Dirac is not punching Elon or biting
Elon he looks very happy because in fact
he thinks this is a great thing and so
do the vast majority of AI researchers
me then there's some myths about what
the concerns really are that the worry
would somehow be a ice turning conscious
or turning evil in fact if you are being
feeling threatened by a machine it's
completely irrelevant to you whether the
machine has some subjective experience
and feels conscious or not what mattered
that the heat-seeking missile was
chasing you
it's simply its behavior and not how it
feels and whether its goals are actually
aligned with your goals and thats
related to this other worry about AI
from heart turning evil which is
completely silly and of course inspired
by some bad Hollywood movies the concern
is not malice its competence why are
ants correctly a little bit concerned
about you not because you're an evil and
Peter who ant hater who go down your way
to step on ants on the sidewalk on
Bancroft Avenue if you can most likely
but simply because you are more
confident than the ants
and your goals might not be aligned at
the end so for example if you're in
charge of this awesome green energy
project and just as you're gonna switch
the water on the flood the area you
discover this an anvil in the middle
tough luck for the ants so we want to
make sure that we create a IOU in gold
or a line without so we don't end up in
the position of those ads are you able
to hear me well now another myth is that
robots or somehow the main concern which
very much irritates robotic fish of
course robots are a relatively old
technology its intelligence itself which
is a new thing and which is really the
crucial thing gave it a super
intelligence is way way smarter than us
it doesn't need a robotic bottle body as
long as I have an internet connection
out small that's on financial markets
and hire people that the wallet bidding
even if they were no no robots that's
the intelligence that matters with yet
another myth is that a I somehow cannot
control humans but of course the fact is
that intelligence really enables control
we control Tigers not by having sharper
claws or bigger muscles but by being
smarter than them and thereby developing
technology to control them in AI
certainly do the same they were smart
enough which is related to another
really fun myth which is machines cannot
have gold now if you're feeling
threatened by a machine it's because
it's expect giving some sort of
goal-oriented behavior happens if you
don't care again
about the other philosophical aspect of
it as long as it's acting like it has a
goal so if you're chased by heat-seeking
missile you're not gonna be like oh I'm
not worried because machines don't have
goals and even if you program a machine
to not have goals that seem harmful but
to do something completely different
it's been pointed out by Stevo mohandro
Nick Bostrom to addresses and others
that this machine might nonetheless
develop sub goals which are not aligned
with yours I made up the city silly
video game to explain this year the goal
of your little blue robot here it's in
the middle is to save a sheep on the
site from the evil wolf and bring them
into safety in the yellow room and even
though his only goal is to save as many
sheep as possible he will develop a sub
goal of self-preservation not dying
because if he runs into the bomb then
he's not gonna say any more sheep he's
also gonna develop a sub goal of
improving his world model because that
way he can find a shorter detector e for
the Sheep you know what's gonna try to
understand technology better and acquire
resources because if he picks up his
potion and the corked bottle area we can
run twice as fast and save more sheep
which was his goal and if he picks up a
gun you can shoot the wolf and save all
the sheep so I'm summarizing that in
this little triangle here if you give a
machine almost any goal that's
sufficiently ambitious it's likely to
develop
sub goals of having better Hardware
self-preservation getting more research
resources understanding the world better
and so on and you better make sure that
those sub goals are also aligned with
what we actually want last but not least
there's this myth that people who are
concerned about a safety think that
super intelligence is just years away
when of course most of the people who
are gonna record being concerned think
it's gonna take way way longer than that
but are simply saying that it might take
decades just figure out how to make this
beneficial and safe so let's not start
the AIC
research the night before some dudes on
Red Bull decide to switch it on
that's rather plan ahead I've posted
this little mix sheet with all the
explanations on the leak you see here on
the future like websites also in the AI
section sidebar finally plan ahead I'd
say so what can we do to plan ahead we
have all these questions that we want to
answer and we want to answer them
through research so we launched with a
help of of Elon Musk in the open format
three projects worldwide the competition
for ideas for research we were extremely
excited to get about 300 applications
asking for about 100 million bucks which
is way more than the 7 million we had to
give out at that time the 37 winner
we've got this I've been working on this
now for almost a year and it's super
exciting we were just looking at the
annual reports to see all the cool work
that's already come out of this and
Washington Post even said that they
filled it last year was the year when
when AI safety went mainstream and this
sort of research we hope it's just the
beginning of a really awesome worldwide
effect all certain efforts could help
great beneficially eyes I want to just
end by saying that effective altruism
helped the mainstream the beneficially
eye movement and can really do wonders
moving it forward in the future though
I've listed a whole bunch of
organizations here in the ei movement
who are trying to do exactly this and
let's work together and let's make the
bit make let's make a difference in play
an awesome future of light thank you
all right Thank You professor tegmark
apologies for the technical difficulties
getting you up and running here can you
hear me okay right now hmm we seem to
have I can hear me I can't hear you but
you seem to be able to hear me talk to
me one would you hear my pot okay yes we
could hear you until just a second ago
now you're back so I wanted to ask just
a couple of follow-up questions you
spent most of your talk dispelling myths
and giving facts I think for an audience
of effective altruists who really want
to understand these issues and
understand what the situation is what
the stakes are what the reality is
that's extremely important do you think
that the audience in the room right now
should go forward and spread these same
myths and facts to people who are just
in the more general public
undergraduates in the sciences the you
know the kind of next generation of
people who might work on these issues or
are you more focused on the current
research community and the current
effective altruists community that
surrounds the issues today I think both
of these are really important and
valuable the sooner we all together you
can help clear up these myths the sooner
we're gonna challenge all the talent and
energy and resources out there focusing
on constructively dealing with these
questions and a rather than quibbling
about misunderstanding so I think you'll
be wonderful people here can share the
facts about these myths widely I also
think that there this is a conversation
that not only the AI community means to
be having because yes some of the
questions are very nerdy and geeky and
technical but some of the questions go
far beyond computer science research
they go into economics all the issues of
what sort of labor market we want what
sort of future we want to create
it goes into psychology and philosophy
you know how do you okay if you can
create a leisure society where nobody
has a job technically how do you ensure
that people can find fulfillment
fulfillment and meaning that's not a
question only for AI researchers and
more most broadly of all ultimately well
of course we should all be asking
ourselves and what kind of future we
want to create because it's our future
indeed questions of values as much as
questions of technical feasibility or
technical concerns so turning to those
grants that you mentioned for a moment
obviously there were a lot of grant
winners that are now putting their their
projects forward pushing their projects
forward are there any in particular
you'd like to share with us that you
think are most exciting or most high
potential in terms of making sure that
AI does remain safe as it develops there
are very many and was excited about when
looking at all these annual reports
actually one area I find interesting of
many is the question of value alignment
I alluded to it briefly in my talk when
I said that you might need to ensure
that if you haven't it is more powerful
than us with their goals or aligned with
ours and that's something which even if
you can solve the more philosophical and
ethical questions about what value we
want also involves really daunting
technical challenges and for example
near a year at Berkeley and many of the
other organizations I mentioned are
working hard on it together and now with
a bunch of professors in their group at
other universities around the world I
feel that there's a lot of progress
being made there which makes more
optimistic that if you work really hard
on this we can actually solve this
perfect so just one more question we've
got just a minute left but how can
people help
besides relaying your message of myths
and facts what else can people do to
help you in the future of life in
to advance your mission oh thank you for
asking
everybody all of you in the audience can
help by being exactly the same thing
that I'm doing right now namely
volunteering for the effective altruism
movement I am a volunteer for the future
of identity and the best majority of all
the work they described is done by
volunteers so we would love it to have
you as a volunteer you can volunteer for
us by going through future of life.org
and filling in a spreadsheet or just
lying down one of the people who stood
up at the beginning my talk and saying
you want to volunteer or you can
volunteer for any of the other
organizations that I've been represented
there a TA global because that's how
other stuff from that's how the stuff
get done there are there is no shortage
of great ideas to be developing and you
can be the one who actually does the
work why'd you get the perfect note to
end on professor max tegmark thank you
very much
all right so that brings us to our panel
and it's my pleasure to introduce first
our moderator for the panel Reva tez she
is a co-founder and partner at
permutation ventures a database platform
and investment fund that focuses on
impact driven early stage AI companies
previously Reva founded Berlin
singularity a group promoting
discussions on emerging technology in
mainland Europe through Berlin
singularity
she helped businesses she helped the
early-stage biotech companies in the
area with business development by
connecting them to seed and growth
capital she began her career as an
entrepreneur at the age of 18 when she
established a retail business in London
which is still open for business
to this day she has held teaching
positions at the DEA B&H TWU business
schools and Germany and has guest
lectured at Oxford Brook Beck and
Stanford Reva writes for a number of
publications on topics such as
artificial intelligence finance and
philosophy and has represented machine
learning in a 2016 a Microsoft campaign
for the Economist please join me in
welcoming Rivas heads next up Dario
Amadei Dario is a research scientist at
open AI and co-author of the recent
paper concrete problems in AI safety
which outlines a pragmatic and empirical
approach to making AI systems safe
previously he worked at Google where he
was a deep learning researcher on the
Google brain team and also at Baidu
Dario also helped to lead the project
that developed deep speech 2 which was
named one of 10 breakthrough
technologies of 2016 by it the MIT
technology review he holds a PhD in
physics from Princeton where he was
awarded the Hertz foundation doctoral
thesis prize he is also a scientific
advisor to give well and good ventures
where he has helped build the research
staff provided basic training in biology
physics and AI to senior staff and
represented the organizations in
conversations with major science funders
and philanthropists welcome Dario
Dileep George is next he is a co-founder
of vicarious systems and AI startup
backed by the founders of PayPal
Palantir and Facebook previously he was
co-founder and CTO of Numenta and a
research fellow at the Redwood
Neuroscience Institute
Dileep has authored 22 patents and many
influential papers on the mathematics of
brain circuits his research has been
featured in The New York Times
Businessweek scientific computing wired
and several peer-reviewed academic
journals he earned his master's and PhD
in electrical engineering from Stanford
and his bachelors from IIT in Bombay
please welcome bleep George Tobey org
you've heard from before he is a moral
philosopher at Oxford University his
work focuses on the biggest questions
facing humanity what are the most
important issues of our time and how can
we best address them his earlier work
explored the ethics of global health and
global poverty demonstrating that Aid
has been highly successful on average
and has the potential to be even more
successful if we were to improve our
priority setting this led him to create
an International Society called giving
what we can whose members have pledged
over 700 million dollars to the most
effective charities helping improve the
world he also co-founded the wider
effective altruism movement encouraging
thousands of people to use reason and
evidence to help others as much as
possible his current research is on is
on avoiding the threat of human
extinction and thus safeguarding a
positive future for Humanity which he
considers to be among the most pressing
and neglected issues we face today
Tobie org
and finally for our AI panel Daniel
Dewey
he leads the open philanthropy projects
work on supporting technical research to
mitigate potential risks from advanced
artificial intelligence
Daniel graduated from Carnegie Mellon
University in 2008 with a BS in computer
science and philosophy he then worked at
Google in the future of humanity
humanity Institute contracted with the
future of life Institute and the Center
for effective altruism and joined the
open philanthropy project in 2015 please
welcome Daniel Dewey you guys are in for
a treat
Reba take it oh hi okay so it's been a
pretty big year for AI it's
understatement and we're lucky to have
some really great people from the
community here on this panel a lot of
interest in kind of AI safe to be blown
off in the last year you know more of a
focus of it a lot of us but the big AI
conferences in the last year you know
workshops focused on AI safety and
another big development was the release
of open AI and a lot of focus kind of
from people in the in the community
stepping up and saying hey let's think
about the risks around this development
and progress in AI as well as the
benefits which I think is the focus of
our panel today so we all had really
embarrassingly long intros everyone can
kind of take in turns to say like the
focus of expertise what they're working
on sure so I'm you know I kind of
originally became interested in the a
safety issue from the perspective of you
know a developer of AI and kind of
seeing the way that AI systems
particularly deep neural nets can often
behave in very opaque and unpredictable
ways and seeing how assistants become
more powerful this could lead to very
very unpredictable behavior so you know
as mentioned in the introduction you
know I recently wrote this paper called
concrete problems in AI safety that
tries to explore AI safety and in
current and and future
machine learning systems from from
perspective of kind of machine learning
and empirical work so I just recently
joined open AI and I'm working on the
combination of
safety work and AI research that I think
is closely related to safety work we are
working on artificial general
intelligence all the advances that we
have seen in the AI field are actually
from narrow AI and there are a handful
of companies working on developing AGI
artificial general insulin since we are
one of them and we look at how the human
brain works and we try to embody those
principles in the computer using machine
learning and the graphical models as the
framework thirteen years ago I met Nick
Bostrom at Oxford wood we both it just
arrived there and he made me really
interested in existential risk and these
questions about how we make sure that we
have a really positive long-term future
for Humanity and he back there but and
still thinks that anthropogenic risks
more likely than natural risks and that
AI is one of the the most plausible of
those and so I've been looking at this
for a long time my work at FHI and I'm
particularly interested in questions
about safety strategy so I also follow
the research on technical AI safety but
my expertise is more in thinking about
these strategic questions like how
should we think about timelines how
should we you know what what is it
important for us to be doing now at this
early stage compared to you know what's
our comparative advantage by being so
early in this process compared to what
should we leave to people you know in
ten years time or something like that I
became interested in existential and
global catastrophic risks from AI when I
was working at Google there was a
discussion group on the future of AI and
a lot of really interesting people there
I hadn't really heard about that before
and sort of caught my attention then I
went and worked at future of humanity
Institute with Nick Bostrom and with
Toby and Nick Beckstead who's now a
colleague of mine and the open
philanthropy project
I guess the only thing I really have to
add to what Toby said was that it seems
clear from an effective altruist
perspective that AI has a lot of
potential to radically improve the human
condition but that there is sort of this
necessary condition that we not have
existential or global catastrophic
events in the mean time so my main focus
right now is outreach to the AI and
machine learning academic communities to
try to find opportunities for research
that we think could improve the
long-term and expected impact of AI okay
cool so I know that naturally the AI
community is adverse a bit to talking
about timelines but we're going to go
there so I guess the first question is
for the panel and I'm gonna start maybe
with down and Toby because you guys are
looking at it from more theoretical
perspective sometimes like over the next
few decades how do you guys envisage AI
impacting human wealth do you think it's
going to come from the kind of
technologies that were already seen
today or will there be radical paradigm
shifts in research and I don't know just
just like a little bit of foresight into
the into that space maybe Toby first
okay yeah I mean I
my specialty is is not so much on the
the earlier stage things that it's go to
influence I mean there's lots of really
interesting questions and really
important questions in terms of the
near-term issues for example around
privacy and also discrimination and AI
systems lack of transparency these kind
of concerns after that about employment
for example which i think is a very
interesting and important topic my main
interest in terms of timelines has been
on when would we have really
transformative artificial intelligence
systems that could do for example let's
say 80% of the jobs that humans
currently do just to be a kind of
operationalization of what we mean there
so if you had this really transformative
AI which really reshaped society when
would that happen and I think a couple
of things I would say about that that
often you get people who try to predict
this and
to say a particulate number so you know
30 years in the future or something like
that really we have a lot of uncertainty
about this and it's best to think in
terms of a probability distribution over
the future as to when this type of
things gonna happen particularly as the
event itself is a little bit vague and I
think that the you know you basically
strategize incorrectly if you're just
thinking of a point estimate and instead
I think that there's like a serious
chance that this might happen with some
if the currents are very rapid progress
over the last few years you know were to
continue that maybe this would happen
you know between 10 or 20 years from now
I wouldn't think that that's more likely
than not but I think that it's are quite
plausible and that the right way to
think about this involves a kind of
portfolio of strategies involving some
hedging against this general AI coming
really soon in which case we need
techniques that can be ready really
quickly and also thinking about what if
it comes quite a bit later in which case
our comparative advantage now is getting
a whole lot of other researchers
interested in these topics and having
thousands of people working on this
instead of dozens of people so you kind
of get different strategies based on
when it would come and we ultimately
need to miss you all of these approaches
to some level yeah I have a really
similar view in particular the idea of
thinking about taking a distribution
instead of a point estimate is really
important to the work that I'm doing
with open plans review project right now
we came in and sort of felt like we
started getting this idea that maybe
there's a 1 in 10 chance or something
like that over the next 20 years of this
kind of really transformative AI
capability and that seemed honestly to
us that seemed kind of insane at first
but we went and talked to a lot of AI
and machine learning professors and
researchers in industry and they're like
good reasons to think that it really
might not happen like unsupervised
learning is this huge like it's not even
right to call it an unsolved problem
it's like a huge portfolio of unsolved
problems
but it just may be the case that that
we're able to find quicker solutions to
the problems that remain unsolved and
then most people expect so there's those
like this non-negligible chance that
we'll have to deal with really
transformative capability is
surprisingly soon so I think one thing
that will happen to enable AJ is the
combining of systems neuroscience
insights with machine learning insights
most of the advances that are happening
now is based on algorithms that we have
discovered we discovered about 20 or 25
years ago basically by neural networks
and back propagation and the advances
came really from a lot more computation
and a lot more data all evidence is
suggesting generally that you cannot
just scale that up with more data and
more computation to get to human-level
AI the the characteristics of human
intelligence are very different from the
characteristics that accept are
exhibited by the existing or deep
learning algorithms so I think there
will be a substantial change in future
systems and that will come from
combining insights from neuroscience
with machine learning insights deletes
considering that I'm allowed to be a
patanty an annoying moderator do we get
a timeline from you whenever we see a
vicarious AI so the problem is that I'm
not a passive observer in this one I'm
actively trying to make it happen so I
hope that it is sooner than later okay
Dario you know working with opening
amazing stuff coming out from there the
reinforcement learning Jim advancing an
entire field of research do you kind of
agree with the panel around timelines
and is there gonna be a paradigm shift
or from current research just developing
on sure so so a couple things you know I
think on one hand my my perspective and
kind of long term pine lines is that
it's pretty hard to predict the future
of what's gonna happen in a few decades
so I basically agree with this kind of
having having wide estimates right
something very transformative could
could happen you know in 20 years in our
lifetimes or might not happen at all
but what what I is a researcher I'm able
to see someone more clearly though
definitely not not that clearly is what
what might happen in the next 5 or 10
years and I think even the set of things
that couldn't happen in the next 5 or 10
years should should give us substantial
pause about about the impacts of this
technology both over that time scale and
then as capabilities continue to improve
so most of what we've done so far as you
know dilip was was talking about is
based on unsupervised learning so I
think two things that are gonna happen
in the next five or ten years is number
one all the things we've done in
supervised learning which includes you
know very good speech systems very good
image recognition systems NLP systems
that I would say are okay but are still
all in you know our research form and
have been deployed in limited ways but I
think full deployment to these
technologies will you know if the the
cop if the interface to computer has
changed from typing to speech I think
that would have a very big impact on the
world and similarly with vision and
other things there are you know hundreds
of applications we can think of that
that you know are going to be present in
five or ten years that aren't present
now and then I think the second strand
is that we're starting to think about
reinforcement learning and even
unsupervised learning and I think
reinforcement learning because it deals
with systems that interact in a much
more intertwined way in the world we'll
see much more much wider application in
the physical world and will also offer
much more opportunity for for something
to go wrong a system that's autonomous
that makes decisions on its on its own
like a robot or a self-driving car or or
or something like this has has much more
potential to do something that it's that
this designers didn't didn't intend so I
think even even over this time scale I
think there's going to be a lot of
progress and you knows Reva mentioned we
at open AI are working on very actively
on reinforcement learning and
unsupervised learning and you know I
mean I'm biased but I'm certainly very
excited about our work and I think that
even even within a few years that that
enormous progress will happen okay my
sleep so you vicarious is a GI is he
sooner rather than later how can you
imagine the world post that like what
kind of changes can you see in
employment wealth economics like what in
society look like in a in a post AGI one
well I will go back to one of Arthur
Clarke's earlier statements where you
know we want to invent technology so
that you know we can the goal of the
future is full unemployment so that we
can all play
so imagine imagine you know all all the
work is automated that we don't have to
work for a living and you you know all
the all the things that we that are
tedious are automated and wealth
creation of just happens automatically
then I think the world can be very
different and you know you can solve a
lot of the problems that we struggle
with today and so I do think you know so
if you look at the history of humanity
we we we used to spend all our time
hunting for food or you know a large
faction for time we're spending just
just getting food or just making a
living and then now a lot more of our
time is spent on consuming information
consuming entertainment having
discussions about ideas so I think we
will have more such opportunities if
more work is automated and more people
will be able to participate in those
activities if more work is automated or
provided that we are able to share the
fruits of that work in in an equitable
way communism okay so down into AV maybe
you want to present something on the
flip side like maybe looking at how
other technologies in the past and
optimism around that in a historical
example yeah I in terms of what our what
our analysis technologies it's it's
quite unclear I mean I think that if you
look at humanity as a species the reason
that we're the commanding position we
are over the world
where
you know in the in the midst of the
moment of causing an effectively a mass
extinction event of other species in the
way that we're transforming the world to
our purposes that's not because we're
stronger than other species or faster or
something like that
we pretty much the only thing that which
we're unique is that that we're the
smartest so developing some new
technology which is smarter than us
would dethrone us from our only
advantage over the other species it's
quite unclear that that how I mean that
sounds very big and like it could be
very transformative and difficult to
manage I we would we would need to align
the values of these systems with house
or otherwise control them or something
otherwise they would be controlling us
so that's a really big thing and it
feels to me that it could be you know
bigger than the Industrial Revolution or
or even the Agricultural Revolution
could be perhaps the biggest thing to
happen since the development of
intelligence in the first place on earth
or something Mille perhaps this model is
just wrong and and it's not going to be
that big a deal maybe for various
reasons that it doesn't if the cap on
how smart it can be just isn't that much
higher than us or something but I'm you
know I think it's it's probably a really
big deal it also has this unusual
potential property and we're not really
sure how development is going to play
out but it seems like there's a
reasonable chance that there could be
really big power differentials like the
that the capabilities of one group or
one one AI system or one country could
like speed dramatically ahead and in the
past we've seen that play out over you
know hundred year periods or something
where some technology is adopted in some
particular country or region or but we
could see that on a much shorter time
scale if it turns out that AI progresses
really rapidly or that a few like
progress in a few areas lets us make
systems that are practically more
effective across a wide variety of
domains so I think that's something
where
you can look at history for analogs but
where there are going to be like these
quantitative or qualitative differences
in like how quickly things play out and
what kinds of effects that has advised
people if you're interested in those
kinds of things look at AI impact it's a
fantastic website that tries to actually
just gather together all of the kind of
indubitable empirical relevant
information so it's not just speculation
but it's a kind of list of all of the
factual relevant information for example
there's a big study into how many times
have has there been discontinuous
technological change in history looking
at the most prominent examples that
people mentioned like the harbor process
or or atomic weapons and trying to see
whether they really were discontinuous
and and weed in that particular case it
looked like there isn't much evidence of
really other discontinuous changes which
perhaps suggest that this one won't be
either but they don't put too much
weight on that but they at least try to
just present the evidence to you it's a
really good place to try to look at
those historical analogues and kind of
moving away from a theoretical looking a
bit more about the landscape right now
posing to the panel of some examples of
people or projects using AI towards a
greater impact I really like the
reinforcement learning gem that came out
of open AI because it took something
that was normally an IP and internal
property locked to a technical group and
give it out to the community to be able
to build and develop their own research
which I thought was really cool some
other examples
dario's and anything else that's coming
out there any is or announce there so I
mean do you do you mean this in terms of
ideas for making AI more safe for ideas
for AI kind of influencing the world in
a good way I guess I guess broad sure so
there there there are two projects
actually because I mean I've been I've
been kind of thinking about about safety
a lot so there are kind of two projects
that are that are there in my mind that
I think have been have been have been
really useful so one is by a crystal who
was my co-author on the concrete
problems paper but long long before that
that came out
um he did something called a deep dream
which you know many of you probably know
because it kind of you know generated
these kind of you know warped neural net
generated images that everyone has seen
on their our Facebook profile but what
what ultimately motivated that that
research for him was a desire to see
inside what was going on in neural nets
because they tend to be so opaque and to
give them some transparency and so there
were examples where you know what he was
trying to do with the deep dream stuff
was you know to look at some class
that's generated by a neural net like
you know the class of you know images
that are labeled as having a barbell and
he would find things like that every
every barbell that was generated by the
neural net would have a hand attached to
it because every barbell that it had
seen in the training set had its hand
attached to it so you could imagine that
you know this this is a kind of spurious
correlation that if you ended up using
the new system that acted in the world
could do something very unpredictable
and so Chris's work has really helped to
elucidate that this is a problem and he
did a lot of work on kind of helping
helping to solve the problem so I think
just just having a little bit of
curiosity of trying to to understand our
own understand their own technologies
and and what's going on inside them can
can go very far towards making the
systems more safe and then as a side
effect that ends up generating something
you know incredibly cool that you know
or ordinary people who have never heard
of AI kind of see in their in their
Facebook feed and to leave any like
other AI projects you find interesting
or applications and in different
industries well so we do find you know
all the all the work that is happening
all the excitement that is happening in
here is very interesting and obviously
we are biased towards our own work and
some of us going to start coming out in
a couple of months as papers so what I
can tell you what we are doing
differently or trying to do differently
is that one is our approach is small
model-based so to give an example of
what model-based means I can give by
analogy so you can think of evolution
and ever
the animals that exhibited very complex
behavior in this world dinosaurs are one
example they were able to navigate the
world in a hunt or eat through all kinds
of complex behavior but we don't
consider the most intelligent in the way
humans are intelligent why because most
of the systems were reactive so you can
have intelligent looking behavior
produced by systems that are reactive
just stimulus comes in and there is a
pre-programmed or predetermined response
and you can train this reactive systems
using number of mechanisms you can use
evolutionary algorithms like evolution
did or you can use back propagation or
reinforcement learning or any such
system to create reacting systems which
look look like they're doing intelligent
behavior but are actually pretty brittle
under the hood so if you look at the
Atari game system or something you know
it is very impressive behavior but if
you change the color of the game or if
it changes sprites it will break and the
reason is that it is mapped a set of
patterns to a behavior without having a
good model of what is happening it it
doesn't think about oh here is a paddle
here is a ball this is how I want to
move the paddle to hit the ball it's
small or if I see this pattern do this
so so that is something that we are
trying to and it is harder in many ways
I'm you know it's not like this is a new
idea this has been around for a long
time but that part is harder to do
compared to reactive system so it's it's
interesting because we are pretty much
going to the same analogous nature did
first we are creating intelligence using
more reactive systems kind of mechanisms
model three mechanisms and gradually
people are going towards more
unsupervised learning and model-based
systems daniel henney any comments on
impactful use cases of AI that you find
exciting or interesting yeah honestly I
focus mostly on the global catastrophic
risk side I sort of am trusting people
who are developing AI systems to be
catching like there are a ton of good
uses in health care I'm really excited
about self-driving cars I
I don't want people to be dying in car
accidents every year on the AI safety
side I'm very excited both about
Google's support of the concrete AI
safety paper like I think that's just a
really big step in the history of the
field and then I'm particularly bullish
on some work that was sort of started by
Paul Cristiano on systems that are
guided more by short-term heuristics
then pointed at long-term goals it seems
like an interesting approach that that
might dissolve some of the thorny or AI
safety problems but I won't go deep into
the details right now that's what I'm
excited about
yeah I mean I was excited by by Chris
all's work as well I thought that was
really fantastic just as interestingly
as just AI machine learning work and
also as AI safety-related work that I
mean he had these really good techniques
as well as the things that you mentioned
there's also this case that to find out
what a particular neurons doing how do
you transform the image in a particular
way preserving certain kind of global
properties such that that neuron kind of
like has a higher activation in order to
find out what the neuron does and a
whole lot of stuff every day there's a
really interesting stuff with that whole
project with the deep trade cream things
the pictures in some sense that the
least interesting aspect but it was
really good work on the transparency and
I think that we're all so the work that
Lauren also of deep mind and Stuart
Armstrong ffh I did on interrupt ability
is it's very nice that's about in a kind
of standard reinforcement learning
framework continuous with current
techniques how could you make it so that
an agent doesn't resist being turned off
to avoid it having an instrumental goal
to avoid you know to avoid being turned
off because they can't achieve its goals
better if the human turns it off you
could you could get even if doesn't have
an intrinsic desire to have
self-preservation we get his
instrumental desire just very naturally
and so they did some work on that they
haven't solved the problem but they like
made a very good first step on it
and what really excited me about those
things and also your whole research
attempter and also the Mary's recent
research agenda is that they're
interfacing with with actually the
practice in the field and I think that's
really important for a couple of reasons
I think that if AI come soon that it's
really important that we have techniques
for managing it which interface with the
technologies that are going to be the
ones that actually produced it it will
take too long in some sense to come up
with totally new things that don't
really interact and I also think that if
AI comes late that the most important
thing for us to do is to develop much
bigger community of researchers and I
think that the way to do that is to to
create kind of shovel-ready projects
that they can do that kind of interface
with current work and so either way it
actually seems like a great approach
yeah I couldn't I couldn't agree with
this this more strongly I think that you
know it's it's not that I'm kind of
against theoretical sky's the limit work
on you know thinking about how to make
very strong AI systems safe but but I
think something that's easy very easy to
underestimate is just the value of an
empirical feedback loop so if I'm
worried about some particular issue it's
it's an advantage to the research if I'm
able to test whether that issue is
present in some machine learning system
where a lot of other things are going on
that could either maybe the thing isn't
really an issue or maybe it's much more
severe than I think it is and all the
methods that I've come up with that I
think could solve it we won't solve it
and I think you know the way to get that
empirical feedback loop is to tie it in
and that was that was kind of the point
of our paper and I think also you know
Laura Laurent
Laurent in Stewart's work and the the
recent the recent Mary Mary agenda kind
of all all going in that direction I
think that's a really really good
direction that's to be welcomed another
thing that I think was earlier this year
deepmind ain't making their announcement
about partnering with NHS and working on
projects for the healthcare system in
the UK which I thought was very
interesting as an application yeah but
that's the case where they've got the
current set up doesn't use machine
learning so it's like not currently a
thing that does machine learning to help
but it obviously would create a a point
at which if there was going to prove all
the machine learning was clearly working
better than the existing algorithms that
it could be kind of put in potentially
in the future and so on
and healthcare is a very natural place
particularly things like medical imaging
and there's a lot of people looking into
that in lots of different companies
trying to come up with some good
solutions
I yeah I think there's I think there's
enormous potential I mean it's it's
somewhat off in a different direction
from kind of the fundamental research
but I think there's enormous potential
for machine learning to revolutionize
medicine and healthcare and you know
right now I think we're temporarily
being being held up by kind of logistics
data collection you know often
biological data is is very very
difficult to get you know there's
there's a lot of issues of kind of
patient patient consent and just the way
the data is stored and privacy and that
sort of stuff but I feel like once we
get datasets that are that are large
enough that we can collect in an ethical
way to do some analysis on then I think
I think wonderful things are going to
happen in this field there's been a lot
of also hype in AI for the last yeah I
think I saw a great great tweet today
said don't know why it's called back
learning hi I mean deep learning hi for
not back propaganda which I thought was
great I guess I mean we're working on
the investment side so we're seeing a
lot of companies pitching us and I was
telling you guys just on the side of the
stage earlier about how did the company
picture us saying that they'd solved NLP
and there's a non technical team so it
was very amusing for us oh sorry
you know if you're trying to understand
like hype and being pragmatic and also
AI safety theoretical research like the
whole overview one of the things that's
kind of like the basis of that is
incentive structures like incentive
structures in science and you know open
AI set up as a non-profit deepmind have
their own markets incentive structures
by not being acquired by Google
and I kind of want to put it maybe to
take me into Dan first about what kind
of incentive structures or issues do we
have around kind of promoting the safe
advancement of AI is it like a academic
problem is it something that's going to
you know needs to be handled by
nonprofits like what would you like to
see more of in this space in terms of
incentives
sorry tough question my way yeah that's
I think this is a really good question
and it's something that I'm still like
struggling with a lot but in an
interesting experience I had was was
going and talking to a lot of AI and
machine learning professors and Industry
researchers like we we went and had
in-person meetings with like 35 of them
when we were getting into this area and
a response that we got a lot was well
I'm personally really interested in
working on the long-term impacts that AI
could have and like working on some of
the risks working on some of the like
more blueSky beneficial things that
could happen but I think if I told my
colleagues they would disown me like
that that many people felt isolated or
felt like they couldn't talk about these
things in their professional capacity
they had to wait until they went to the
bar after work to talk about it so I
think that that finding some sort of
incremental ways to solve this common
knowledge problem that this is an area
that that AI researchers are the most
informed and most able to talk sensibly
about and there there are these natural
problems where no field wants to point
at itself as having potential risks even
though I mean all of the kinds of
research and work that we do have some
kinds of risks or potential for negative
impacts I don't know how to solve that
part of the problem honestly like I'm
I've been really impressed by AI
researchers in individual one-on-one
conversations where they're like
frankness and ability to see like to
hold a few different potential
situations in their heads to say like it
could go this way could do this way we
don't really know yet but getting that
kind of discourse when you have more
than one person in a room it seems like
pretty
you know it's also there's a tricky
aspect that everybody if you know if
you've here what I've been saying so far
I sound fairly negative on this but you
know I think it's more likely not that
that a I will will directly contribute
to a glorious future for Humanity and
they will have a very good outcome that
the outcome will be very good in part
quite large part because of AI be noble
to be as good without AI I so that make
that might make sound very bullish on it
but but well I think there's more than
50% chance that we end up in that
situation that's partly because we will
do a whole lot of work in order to deal
with these these threats and work out
how to manage them as best we can
and I and I think that is a bit of a
disconnect that a lot of people kind of
forget that you can be pretty sure it's
gonna lead to a whole that are really
good outcomes while still it could be
really important to hedge against the
bad outcomes and to actually start
conversations about that regarding
incentives and so forth is it's quite
interesting there's a bit of
conversation about at the moment
open AI are talking about the questions
about the structures and so on there are
nonprofit I think that ultimately this
technology both empirically can't be
owned by a company if we got all the way
to this level there's no kind of
historical analogues of a company owning
something like the Industrial Revolution
or or even a country kind of owning that
there's kind of some brief period where
some people get rich maybe some country
get powerful but they don't think that
it could own it and I don't think it
should own it but there's a question
about what should own it should it be a
non-profit well you can anyone can kind
of start a non-profit and just run it
with an iron fist it just means that you
don't make profit so so that's that's
not sufficient
maybe it's necessary but you know one
needs kind of more questions about
government structure and so forth while
you're while you doing that and then
there's ultimately some question about
what the end point is how is this
technology eventually distributed more
widely than just this or you know some
kind of foundation or some kind of you
know intergovernmental organization or
something there's a whole lot of really
interesting questions there at FHI we're
starting to look into those a bit
the discussion is that I think some of
it can depend on the approach being
taken to solve AI so for example for us
solving the AI safety problem is
inherent part of solving the AI problem
itself because in many of the
discussions about AI safety it is it is
about the AI misunderstanding our
specification which is basically saying
AI doesn't have common sense so if I
tell you that I'm going to hit the road
after this panel you you guys understand
that I'm not going to you know I'm just
going to drive back home I'm not going
to take a stick and hit the road so but
an AI system can misunderstand it I'm
and the reason is that it's not easier
than AI safety problem or having we
haven't solved the problem well in this
case it is because we haven't solved the
problem well so in in many approaches
solving that common-sense problem is
part of the problem of solving AI itself
so in that sense the AI safety problem
gets wrapped into solving AI and so that
could be one reason there is there can
be miscommunication and more
communication can help I think I think I
was a DEFCON for the last two days just
like the opposite of this panel but and
it kind of shocked me how vulnerable is
just even systems that we have now like
monitors on how to have pixels on
monitors to change things like just even
seeing that as a sight from so many
things about computers and think about
like AI and algorithms and machine
learning was like a very shocking
experience for me I'm trying to see like
short-term risks but yeah so effective
altruism lots of people who are thinking
about you know what do I do to make a
lot of impact in my life how do I do the
most good
what do you guys think any advice on how
to be part of progressing AI safely and
what kind of technical skills are needed
what does the community need more of and
also maybe like non technical ways as
well so so I think I think there's you
know there's an opportunity for both at
actually you know all the places that I
recently recently worked at both both
Google and you know and certainly
certainly open AI you know we can we can
kind of never get never get enough good
people on the technical side you know if
you want to work on on AI or the safety
aspects of AI you know I I think you
know the most important thing is to have
you know as strong as possible a
background in in in machine learning but
one one role that you know Google was
hiring for deep mind is hiring for and
we at open a I are very much hiring for
is the role of machine learning machine
learning research engineer so these are
these are people who collaborate with
the research scientists to help
implement and scale-up AI ideas and also
if we work on safety then when those
things are implemented and scaled up
we'll be involved in that and that as
well so the skills they're kind of a mix
of mean a very good software engineer
who's able to scale things up and
someone who knows enough machine
learning research to collaborate with
the researchers to implement ideas and I
think this is a skill that's that's
that's kind of very very very much in
demand and we can kind of never never
get enough of them and they're a huge
multiplier and the productivity of the
researchers so on the technical side I
think that's you know that that is
really a really high really high
leverage point on the non technical side
again I think all of these organizations
are have higher our hiring policy people
PR people to talk to you know the
government and other organizations in a
technically informed way about you know
how should the government think about
regulating AI how should we think about
risk from AI how should we talk to the
public about these issues one of the
great experiences I had was in working
on the concrete problems paper you know
because because it was a paper that was
of such kind of general interest the
community you know the the PR and public
policy apparatus of Google you know talk
kind of talk talk to us about the paper
and we talked to them about how to
present it in a way so the public would
understand what we were saying and that
we didn't get kind of sensationalist
headlines and I really came to
appreciate you know how how much public
public policy and public relations
people can do to really you know make
something have a positive
path to make sure that the public
understands it in the right way and know
that this was a piece of Google or any
organization that I that I'd never seen
before and I feel like that that that
that sort of person can really have huge
leverage that's often unappreciated by
technical people but is really essential
to what we do Dileep are you hiring yes
we are hiring and we look for people
with a good background in graphical
models and you know machine learning in
general and if you are curious about how
the brain works that is that is a big
big huge bonus yes I said there are I
mean as as we just heard that there are
jobs for software engineers generally if
they even if without in our background
in research engineering there are also
jobs for people with ml experience
obviously these companies including
that's quite a one way to get into doing
technical safety because there isn't
really a field of technical safety yet
there's also technical safety work that
can be done at places like FHI in
academia oxford and also work on safety
strategy dfhi and and also work on
technical safety at deepmind for example
I know that they're hiring on that and
there's probably open AI if not now of
them soon and a lot of interest in these
areas and I encourage anyone who's
interested in finding a position somehow
to help out with these things to come up
and talk to us afterwards when we're
over at the the room with the coffee and
posters yeah I want to reiterate the
Technology Policy thing like being from
a technical background I didn't realize
that you know if you're in technology
policy you could work for government you
could work at think-tanks academia but
you could also work at big companies I
mean Google and deep mind an open AI are
all going to be having policy people who
are making sure that their work fits
into the broader landscape well but
another thing that I think the effective
altruism community can sometimes
overlook are roles that aren't specific
to these particular areas
roles that any company would want to
hire so the thing that enables me to do
my work right now at open Phil is like
we have someone who's really extremely
good at Human Resources like we have
people who are really extremely good at
the legal side of grant logistics and
who do like who do complex and
satisfying and challenging work but
that's not like not what you first think
of because it's not the most salient
thing so I want to encourage people to
like look at what they're good at and
become someone who any organization
would want to hire and then find one of
these organizations that that's doing
work in this area and you'd be surprised
how how satisfying and and effective
that is as a career so yeah so I can I
can absolutely confirm that I mean I've
I've only been at at open AI for for
three weeks but you know in that time
I've you know I've already had
interactions with and it's a very small
team with you know the person who runs
operations the person who runs
recruiting and and these people are
really essential to the organization
functioning and actually if they've you
know if they fought about some of some
of these ethical issues you you you
wouldn't think that it would affect it
but having a larger fraction of the team
have kind of you know responsible and
thoughtful ethical views it changes the
culture of the organization even when
it's not those people's jobs to think
about it it kind of it's it's hard for
it not to happen it happens it happens
by osmosis so so having having people
who are thoughtful in addition to
extremely competent at helping out their
organization I feel like these are these
are always positive things okay so
conclusion is be thoughtful and
competent okay I think that's all we
have time for but maybe a round of
applause for the panel will be great
okay