Vael Gates: Researcher Perceptions of Current and Future AI

welcome everyone to the hai weekly
research seminar i'm vanessa parley and
i'm the interim director of research
programs at stanford's institute for
human-centered artificial intelligence
i'm thrilled to introduce vail gates
vale is a postdoctoral fellow at
stanford based in the center for
international security and cooperation
and h.a.i so convid dual fellowship
program
they'll receive their phd in
neuroscience from uc berkeley working on
formalizing and testing computational
cognitive models of social collaboration
and in today's seminar vail will discuss
a new project they are working on to
better understand how the developers of
artificial intelligence and ai
researchers perceive the benefits and
risks of their work
they will present results from over 70
interviews with ai researchers asking a
wide range of questions
and we are looking for a good amount of
audience participation in this talk more
towards the end of the talk so be ready
to respond in the zoom chat
before we begin a few logistics as
mentioned you can use the zoom chat to
message the group but please use the qr
code on the screen to import your
questions through slido
you can also click on the link that will
be in the chat shortly and after the
presentation i'll be choosing questions
from slido and cyto has that nice upvote
feature so i can choose questions that
all of you are most interested in
closed captioning has been enabled so
simply click click the cc feature on
your zoom screen to show captions
throughout the
hour thank you so much for joining us
feel free to share your screen and begin
great
all right hopefully you all can see that
thanks so much for showing up everyone
uh my talk today is called researcher
perceptions of current and future ai
though it could also be called
researcher perceptions of risks from
advanced ai as my talk
is actually focused on risk from
advanced ai
so the structure of this talk is as
follows i'm going to give some context
for the study i did i'll talk about the
development of ai
the concept of agi and the alignment
problem and existential risk then i'll
go on to the research methods i used in
this study some of the research
questions i asked researchers
and the interim results finish with some
including concluding thoughts and then
we should have about 10 to 15 minutes of
q a if by time this right
so let's start with some context where
are we in ai development
and so here's some history from
wikipedia started with some precursors
birth of ai like 1952 symbolic ai we had
an ai winter boom cycle um and second ai
winter and 1993 2011 and here we are in
the deep learning paradigm which is more
2011 to present with the with alex net
and deep learning revolution
so we have some components of the
current paradigm that we wouldn't have
necessarily have expected in the 1950s
um we have black box systems we're using
we're using machine learning neural
networks uh the compute compute is very
important the computing power data
algorithmic and advances and some of
these algorithmic advances are kind of
aimed at scaling so if um methods that
are very general
that you can throw more compute and data
in uh to get better behavior
there's also something of sun's bidder
lesson here which is the idea that
general you need to general methods that
leverage computation are ultimately the
most effective and by a large margin
compared to human knowledge approaches
that were used earlier on
here's a quick comment trying to
illustrate that lesson
so in the early days
of
ai we used something like statistical
learning where like you would know a lot
about the domain you would be very
carefully using methods and then these
days um there's the idea of stack more
layers throw more computing through more
data in and you'll get ever more
sophisticated behavior
so it's worth noting i think that we've
been working on ai for less than 100
years and the current paradigm is you
know around 10 years
and we've gotten pretty pretty far in
that time or some people thought that we
think that we should be much
further so let's move on to where we are
in the present i think a useful
distinction between uh in the current
paradigm is the difference between
narrow ai or a machine learning and more
general methods so historically it makes
sense to start with narrow eye ai to
just specific tasks these tasks include
things like self-driving cars robotics
translation image classification off the
go plays go alpha fold is protein
folding codex does
this coding
but increasingly we've been seeing a
move towards more general models these
large language models
in stanford known as foundation models
an example would be gpt3 or more
recently palm gato
indeed in fact in april and may 2000
like this year uh we've seen a number of
papers come out so we sell some
chinchilla and palms big language models
dolly too here's an example of dolly too
you can write in text like teddy bears
mixing sparkling chemicals as med
scientists in the style of steampunk and
you get images that are very beautiful
like this and you can use many different
prompts and imogen
which has came out shortly after dolly
and is in fact even better than dolly
socratic models flamingo seiken and gato
here's some things that gata can do gato
is as described a deep mind is
the same network with the same weights
can play atari caption images chat stack
blocks with real a real robot arm and
much more deciding based on its context
whether to output text joint torques
button presses or other tokens
so we were coming from a place in ai
where
ai and ml applications were very
specific and now we are
com going towards having models that are
able to do more tasks at once more
general
and a question here is whether we're
going to be able to continue scaling in
the future
um you we we saw that we've seen
surprisingly that scaling does continue
to work in some sense that these that
more and more people are using these
large language model models and so one
can ask in the future
whether we'll get even more general
there's some trends in this direction
here's a
a figure showing time on the x-axis and
we've got a logarithmic scale on the
y-axis of a measure of compute and you
can see that
compute is definitely increasing there's
a question of like well we're going to
run out of compute eventually but we
have models like chinchilla that show
that sometimes you can substitute data
or compute
in various ways to
help correct this
and so maybe in the future we will see
something like artificial general
intelligence which as defined by
wikipedia uh agi is the hypothetical
ability of an intelligent agent to
understand or learn any intellectual
task that a human being can
so whether or not we see adi
specifically it seems like we are moving
in a direction where we have agi-like
systems or systems that in the future um
and again we've only been working in the
deep learning paradigm for 10 years and
ai for less than 100
may have maybe applicable to many more
types of tasks at once
so if we do see these systems when will
we see them
but there was a study done in uh 2018
that surveyed people uh who
submitted paper who got paper submitted
at icml or nureps which are two major
machine learning conferences in 2016 and
they asked about high-level machine
intelligence
and they described define this as
high-level machine intelligence is
achieved when unaided machines can
accomplish every task better and more
cheaply than human workers
here are the results so you can see that
uh the median 50 probability of
high-level machine intelligence uh was
about 45 years from 2016
with 352 researchers responding to this
survey
so that's within within many of our
lifetimes um and coming kind of soon
just interesting
here's another source that
has been trying to aggregate these so
there's a
platform called mitacuculus which is a
prediction solicitation and aggregation
engine so sort of like a prediction
market where they have a bunch of
forecasters who are trying to answer all
sorts of different questions they've had
some success on predicting things like
covid and russia's
invasion of ukraine
and here's a question that they asked
the date that weekly general ai is
publicly known
and it's very hard to define what is
general ai what is weekly general ai and
so they have a whole bunch of
descriptions here
so it needs to be able to reliably pass
a turing test of the type that would win
the lobe nurse silver price uh needs to
be able to score 90 or more on a robust
version of the winograd schema challenge
be able to score 75 percentile on the
full mathematics section of a circa 2015
to 2020 standard sat exam you'll be able
to learn the classic atari game
montezuma's revenge and you can see that
as you go along you've got some amount
of so guesswork here
and then there's this drop in april and
may when all the when all the new papers
came out and they're currently at a
community prediction of 2030
which again is is quite soon
so
systems are getting more powerful we
are
using large language models eventually
we may get to things that are more agi
like future
what are some of the risks of these
models
turns out people are very concerned
about risk so this is skipping a little
bit ahead to some of my results uh from
my work but when i asked people what
they were worried about in terms of
risks from my eye
people mentioned all sorts of things so
one thing people mentioned is the idea
of trustworthy eye or ethical ai ethic
ethics ai
and things within this include things
like fairness algorithmic bias privacy
surveillance transparency
interpretability explainability worries
about manipulation this is more like the
social media vein
military applications algorithmic
attacks so systems that aren't very
robust
and misused by
industries by nations
however i'm going to focus a little bit
more on risks that i see arising
specifically from very from very general
ai
and
one of the problems that people most
talk about is called the alignment
problem
and this could possibly as some
researchers think lead to existential
risk and is relatively neglected
compared to the amount of risks from
narrow ai
do people so i've been talking about uh
risk from general ai and existential
risk which is the loss the death of all
humanity seems pretty extreme are people
even worried about this
so
here we're referencing again the study
that i mentioned earlier from grace doll
surveying the icml and nurbs researchers
in 2016.
and they were asked that the chance of
high-level machine learning has a
positive or negative long-run impact on
humanity and here are their meeting
answers again you can see that many of
them think it's going to be good so now
it's good or neutral and there's some
proportion you think it's or sorry um
they have some percentage on it being
bad or extremely bad could lead to human
extinction
so uh
so here's the text hm high-level machine
intelligence is seen as likely to have
positive outcomes but catastrophic risks
are possible so they had a probability
of five percent on an outcome described
as extremely bad like human extinction
so that's a that's a dice roll um and
it's pretty interesting that researchers
think that there's a 5 chance median
that
the
end result of their work will result in
extremely bad outcomes like human
extinction
so even though that probability is not
the most likely uh it's pretty small
five percent is still way higher than
i'd like to gamble on um and it might be
worth putting attention on the
possibility of extremely bad situations
so what is the problem exactly what are
people concerned about um and the
challenge that people often talk about
is called the alignment problem which is
essentially the challenge of building
systems that are aligned with human
values that do what humans want
that align with what people want
so this problem occurs in the context of
current day systems as well and so i'm
going to walk through one of those
examples
here is a boat racing game called coast
runners
and the goal here is to train this boat
to learn to win the race you can see
there's a little bit of an obstacle i
mean there's a course that they want the
boat to go along
and you can see in the top
left corner here uh there's a bunch of
other boats racing along and
this boat has been trained to
to the goal was to train the boat to
learn to race well and win the game
however the
designers were training it on a reward
function of points which you can see at
the bottom here
and so in the end they got a boat that
did exactly what they wanted so a boat
that optimized for the number of points
and what the boat did was found this
little corner of the of the course where
it could go around in circles and wait
for these little turbocharged uh point
things to develop and then just go
around and collect those and this was in
fact the strategy that earned the most
points even if it wasn't racing the game
um so or winning the game so that's an
example of alignment problem where the
designers wanted something
uh from the ai and so they set a reward
function to try to get it to incentivize
it to achieve that reward but in fact
the thing that they got out of the
behavior they got out wasn't what they
wanted
and so you can imagine that this
challenge gets even harder as you get to
more and more general ai as you have ai
that's acting in the world uh
that is dealing with increased
complexity
also human values are very hard to
systematize and write down um they also
differ between people they differ across
cultures they differ over time things
have changed since the 1800s now and so
you can imagine that it would be very
hard it may be tricky it could be
automatic but it may be tricky to try to
get an ai that performs exactly as
humans intended given that we have to
speak to it in kind of this machine or
mathematical language that is important
in programming
so as we approach agi and in general our
powers is powerful systems you may
expect the alignment problem to become
more difficult
but you know we have solved everything
so far wow humans haven't blown
themselves up yet and we have this thing
called try trial and error where we can
try a system and if it doesn't work then
we'll just fix it and send it out again
unfortunately something happened that's
very tricky with very general systems
and that is an idea called instrumental
incentives
so
in the words of nick bostrom who's a
philosopher
artificial intelligence agents may have
an enormous range of possible final
goals nevertheless according to what we
may term the instrumental convergence
thesis there are some instrumental goals
likely to pursue likely to be pursued by
almost any intelligent agent because
there are some objectives that are
useful intermediaries to the achievement
of almost any final goal
so you can maybe try to guess what those
would be what's what are some of these
uh
instrumental incentives that would arise
as an agent trying to achieve any final
goal
and some of the ones that foster
outlines are self-preservation
acquisition of resources and
self-improvement
which are all different sub-goals that
sort of arise or incentives that arise
as you're an agent trying to achieve
anything
in the words of stuart russell professor
stewart russell who's a professor at uc
berkeley um you can't fetch the coffee
if you're dead so if you're a coffee
fetching robot which is some arbitrary
goal then you definitely can't fetch the
coffee if you're dead and so you have an
incentive to make sure that you stay
alive um so that you can do the goal
that you've been assigned um you also
might have an incentive to acquire
resources or make sure that you have
enough like self-improvement maybe
you're smarter uh maybe you have um
maybe you have good better access to
power than you would initially so that's
described in the book human compatible
and also in the book the alignment
problem by brian christian
so this problem of instrumental
incentives means that if you have an ai
that is sufficiently smart that is able
to act in the world that is able to plan
ahead
that it could have an incentive to make
sure that it is
self-preserved that it stays live that
it doesn't get shut shut down um and if
that's true then we may only have one
shot at developing um an ai that is
fully aligned with human values because
if it's not fully aligned with value uh
human values on that first shot then
it's going to have an instrumental
incentive to
not be shut down and then you're kind of
stuck with whatever we've got
so i'm just going to lay out that
argument structure again one time so
this is um so this is the logic that
underlies the idea that very powerful
systems not perfectly aligned without
human values could lead to existential
risk um so you can evaluate kind of the
arguments for yourself whether you think
that this makes sense
so agi is general intelligence it can by
definition think outside the box unlike
narrow systems it has capabilities that
can affect the real world even if maybe
only initially through through text
it has capabilities so it can duplicate
itself much easier than humans can
it could use the internet to buy or and
sell goods so it could earn money
electronic money and it can consume
information and data and produce and
send information over the internet we
already know how powerful these things
are through for example social media
ai could augment itself so it could buy
more compute it could construct data
sets
it could refine its code for higher
efficiency and write new code and we
already have some of these like some of
the beginnings of these capabilities
through things like codex
ai will be likely to be able to design
and implement any number of ways to kill
humans if incentivized to do so so for
example it could use synthetic biology
to create pandemics or it could
otherwise take advantage of humans being
biological organisms while it isn't
it also has
instrumental incentives which may arise
through it being an agent aiming towards
any goal and these include things like
self-preservation acquisition of
resources and self-improvement and maybe
this doesn't happen or maybe it does but
there's a possibility that maybe
instrumental incentives would arise if
it is sufficiently agent-like
um and that would mean that humans
consuming resources or trying to shut
off or modify the ai from its original
goals
humans would then be obstacles to the ai
achieving its original programmed goal
and so if not perfectly aligned with
humans ai is thus incentivized against
humans which is a problem given it's a
very that it thinks as well or better
than is good at reasoning better
reasoning than humans or human level and
so this is not a story about malicious
ai so
as popular in the media uh things like
the terminator this is an ai that is
indifferent with alien values where
humans are in the way sort of like ants
are often in the way of humans and so we
step on them and it's not like we hate
ants it's just that you know
we're trying to accomplish things and
they are in the way
and worse maybe we only get one shot at
an aligned ai if it is incentivized to
not be shut down through this
instrumental incentive
so one could say well why don't we just
make sure that ai's are aligned with
humans then we'd avoid all this all this
and i think that's right i think if you
get a perfectly aligned
ai with humans then we may have very
amazing futures um ai can help us with
many of the sort of things that we would
hoped it would like solve cancer and all
sorts of things um but there's the
possibility that leading ai companies
may not by default create powerful
systems that are perfectly aligned with
hard to specify and the change over time
and between people human values so i
don't know that the economic incentives
are in place such that ai companies
would by default be trying to make sure
that these systems are perfectly aligned
with human values
so this is the story of how this sort of
thing could lead to existential risk
so one may ask who's working on this
it's a problem then surely there's some
people working on it and there are um so
this is called ai alignment research or
long-term ai safety research although
perhaps not as long-term as we would
necessarily hope
or ai or ml safety that scales to
advanced systems and this is in contrast
to more near-term safety areas which are
also of course important
um and it has expanded since 2015
there's now books and conferences and
research publications in industry or
nonprofits there are the deepmind safety
team open ai safety team anthropic
uh redwood research alignment research
center machine intelligence research
institute and there's a number of people
in academia as well so the center for
human compatible ai shy at berkeley
it's quite near us uh we also have the
cooperative ai foundation
and a bunch of individual researchers at
various locations so people at uc
berkeley nyu oxford and at stanford we
also have researchers at the stanford
center for ai safety as well so people
quite near home
in fact in this sample so again skipping
forward a little bit to my results i
asked my researchers have you heard of
ai alignment and something like 39 of
them said they had they wouldn't
necessarily be able to define it but
they'd heard the term before so this is
an idea that is somewhat around
this is in contrast to have you heard
about ai safety something like 76
percent people said they'd they'd heard
they'd heard of this
however the ai alignment community is
growing slower than capabilities so the
people trying to work on making sure
that
advanced ai is safe is smaller than the
number of people who are working on the
very difficult and tricky problem of
trying to make uh ai do what we
um
hey i have more capabilities ai be able
to
work on all different sorts of
applications work in many different
contexts
um
and so it's this kind of discrepancy and
how fast both of these are communities
are growing that concerns me
so that's the context for this study
which is understanding how ai
researchers perceive these risks from
advanced ai and specifically i presented
ai researchers with some of the claims
used to describe this possible
existential risks and explore people's
responses to this aspect of safety
so that was that's all the context for
my work um and now at this point let's
talk about what i actually did talk
about research methods
so um the first thing i did was i cold
emailed researchers with papers accepted
at nurips or icml 2021
and i did semi-structured interviews
over zoom with them and they were 40
minutes to 60 minutes long
interviews were conducted during
basically february so from february 2nd
to march 4th um so this is before the
suite of new papers in april and may
uh fact uh russia invaded ukraine during
the interview period
and researchers did mention this along
with any new papers that had come out at
the time and i think this might have
influenced the the degree of like the
world was unstable existential risk
concerns
in my sample
a later half of my sample
and i did zoom calls with 86 researchers
and i asked about their general views of
ai and several arguments for risks from
advanced ai to understand their
perceptions of these arguments and risks
the initial email
did not mention risk from advanced ai to
avoid biasing the sample for people who
were particularly interested or not
interested in that um and it was a very
dialogue style
exploration of opinions and arguments so
i would ask them questions they would
respond i would maybe give a response
and then they would respond back and
we'd go back and forth
the data analysis is unfortunately still
in progress so while i have a bunch of
data for you i don't have numbers on
them i want like statistics and
percentages of which researchers are
saying what i don't have that for you
yet unfortunately
let's move on to the questions i asked
researchers so i started out with this
question what are you most excited about
in ai and what are you most worried
about what are the biggest benefits or
risks of ai
then i asked in at least 50 years what
does the world look like so looking a
little bit forward
what are your opinions about policy
oriented
ai what opinions would what opinions and
beliefs do your colleagues have and how
would you like those to change or not
change and what opinions do the public
have do you think the public has on ai
and how should that change or not change
i then got into my core questions so
my first core question was when do you
think we'll get agi or capable or
generalizable ai or have the cognitive
capacities to be a ceo ai if that
happens in the future so this is my agi
question
i then went on to this next question
what do you think of the argument highly
intelligent systems will fail to
optimize exactly what their designers
intended them to and this is dangerous
so this is my alignment problem question
and then i went on to this question what
do you think about the argument highly
intelligent systems will have an
incentive to behave in ways to ensure
that they are not shut off or limited in
pursuing their goals this is dangerous
this is my instrumental incentives
question
and so i walked through the questions in
basically this order so what are you
most excited about in 50 years and then
got into my poor questions i asked the
questions about policy a little bit
later
and at various points we would talk
about depending on who i was talking to
sometimes we would get through all these
questions and sometimes we wouldn't we'd
spend a lot more time in the beginning
so this is what i asked researchers and
for this next part i am going to present
some of the results and then have you
pretend to be a research researcher
since this is uh probably not a super
dissimilar sample from
who i talked to and see what your
responses are
so the first question i asked is what
are you most excited about now and what
are you most worried about biggest
benefits or risks so i just want to put
up some results here um so these are the
the tags that i used to um to describe
the data there's i have a whole bunch of
transcripts i have 86 transcripts and
i've just been labeling them with
people's responses
so qualitative research so
for benefits people said said things
like health self-driving cars came up a
lot uh most of the responses were in
this large category of increasing
productivity convenience better machine
learning or ai automation applications
some said reducing physical risks some
said agi
there was a whole bunch of people were
very worried about many things they had
many different risks
so i'll just give you a couple of
seconds to just take a look at what's on
this screen
so you can see there's a variety of
different uh worries on here that that
span from things like fairness bias to
academics being pushed out or another ai
winter
i then asked them to focus on future ai
so putting on a science forecast sci-fi
forecasting heart hat say we're 50-plus
years into the future what does that
future look like
um and a variety of answers here most
people
hadn't it sounded like most people
hadn't thought about 50 years in the
future just because i mean there's no
there's no particular reason to people
aren't incentivized in this direction
and
many people told me that this is a very
difficult question which seems right to
me so a variety of different types of
answers some people pretty like very
non-committal
machine learning more autonomous
robotics
agi not agi
ai assistance just a variety of
responses
and then i moved into my core question
so here is a
excerpt from a transcript that i used to
talk to most of the researchers and i by
the end of it i had a pretty i had a
spiel that i just gave them so i'm going
to give that to you now
all right so now i'm going to give a
speed so people talk about the promise
of ai which can mean many things but one
of them is getting very general capable
systems perhaps with the cognitive
capabilities to replace all current
human jobs so you could have a ceo ai or
scientist ai et cetera and i usually
think about this in the frame of in 2012
we have the deep learning revolution
we've got alex and gpus 10 years later
here we are and we've got systems like
gpd3 which have a kind of really merging
capabilities they can do some text
generation and some language translation
and some code and some math and one
could imagine that if we continue
pouring in all the human investment that
we're pouring into this like money
competition between nations human talent
so much talent and we're training all
the young people up and if we continue
to have algorithmic improvements at the
rate we've seen and continue to have
hardware improvements so maybe we get
optical computing or quantity then one
can imagine that eventually this scales
to quite general systems or maybe we hit
a limit and we have to do a paradigm
shift in order to get to the highly
capable ai stage regardless of how we
get there my question is do you think
this will ever happen and if so when
so at this point i want you to type
things into the chat and let's see what
those responses are
i'll give you at least 30 seconds
all right so we have 20 45 yes 20 90 yes
no i don't think that will happen yes in
25 years not the next 50 years 100 years
50 to 20 years 50 years next 10 years
30 years 50 years 25 years 20 50
50 years
awesome 20 30 10 years not have been
impossible in this century 10 years if
climate change doesn't cause all of
human collapse 2040
great yeah 20 25 years 100 years um so
really a span of responses here
nice
yeah so let's take a look at what the uh
what the researchers responded
so these were the buckets that i put
things in um
uh some people said yep world definitely
happened uh there was some people get a
very wide range i didn't see a lot of
wide ranges in the chat here some said
under 50 saw a bunch of those in the
chat um 50 to 200 was also pretty common
i also saw that in the chat some people
said 200 years plus um
a number of people said this well this
will never never happen uh and we and
the reasons they gave me were that they
uh so again these were these were all
just tagged uh my tagging types of types
of responses um
so some responses is uh i can't see it
based on current progress i don't think
we're progressing fast enough i don't
think we're gonna have anything like
general ai um some people said that
people
would
like we we would be on the path towards
developing something like agi but then
people would see that it would be
dangerous and would stop progress they
would just shut down um the system i
felt i feel a little bit i often felt a
little bit skeptical of this argument so
i'm like hmm how good are humans at
stopping something that has huge amounts
of money and power and people behind it
uh but we would we would dialogue about
something like that um it's very common
that people express frustration with
public perception uh
people as ai researchers seem ai
researchers seem to encounter a lot of
ah but you're like the terminator isn't
this bad uh and they were like well this
is not like a terminator situation here
that's not gonna happen so there's a lot
of frustration there um many people said
they would need to see more progress
before they could consider that adi
might happen in the future
and there was also this argument that
humans are special um something about
biology is special um true artificial
intelligence isn't possible it seems
very
um it seems very weird to imagine that
something as smart as humans could make
something smarter than that that
wouldn't necessarily make sense and that
seemed to be something or there's
something about creativity that humans
have that that wouldn't previously exist
uh or that we we couldn't create and i
ended up dialing blogging a lot with
those people being like i don't know
evolution made humans it seems like if
we waited long enough we would maybe get
it that way and like humans are more
efficient many ways at engineering than
evolution is um
and yep so we talked about a lot a lot
about that sort of thing i saw some
people in the chat also who um perhaps
these are some of the reasons why they
thought it is impossible um
yeah uh i also saw one of the reasons
which is like where unless we get if we
get it wiped out first that can also be
a reason a number of people thought that
we needed some sort of embodiment so we
wouldn't be able to get uh something
like agi unless we had something like
robotics and sometimes robots that are
growing up in the context of humans were
a number of people said perhaps but we
need a paradigm shift it's not going to
be the deep learning systems uh although
again i think we'll have a little bit
more evidence that
the deep learning system scaling idea is
working with the april and may
results than we did than we did
previously but yep still unknown future
is very hard to predict number of people
wanted to talk about consciousness
which i had plenty of opinions about but
i tried to steer people away from in
this conversation i think you the danger
still exists even if you don't have
conscious ai uh and another group of
people also said only if we understand
the brain will we get agi
so the next question i asked people uh
was about the alignment problem and i'm
gonna ask you to put in your replies to
this as well
so here's the text all right so these
next questions about these highly
intelligent systems so imagine we have a
ceo ai and i'm like all right ceo i wish
for you to maximize profit and try not
to explain people i don't run out of
money and try to avoid side effects and
this might be problematic because
currently we're finding it technically
challenging to translate human values
preferences and intentions into
mathematical formulations that can be
optimized by systems and this might
continue to be a problem in the future
so what do you think of the argument
highly intelligent systems will fail to
optimize exactly what their designers
designers intended them to and this is
dangerous
so i'll give you another 30 seconds for
that
seems like agree very true correct three
very reasonable true agree
highly likely
i agree yes i think it's a strong
argument corner cases are hard to
project yes they will do it as the
designers intend them to agree
yes they will fail to authorize exactly
agree group wow y'all are so agreeable
um
disagree ah i first disagree agree
seems like the type of seems like it
depends on the type of society are um
this is based on contemporary values and
principles asthma obstetric laws agree
true don't think it's
i don't think it's dangerous um uh
disagree humans have to align their own
values for those values yeah all right
so um so i think you were more agreeable
than my sample on the whole uh but many
many researchers did agree with the
statement partly because
we see these sort of problems i think in
contemporary issues uh so
this felt like reasonable to people a
lot of the time many people said it was
some form of a valid argument with
various degrees of agree or disagree um
it was pointed out that many people
thought that in the long or in the long
i don't know if many many people people
said um that in the long term reward
functions probably won't be designed
this way or that in the long term we'd
probably test it first so it would be
okay
a number of people thought this was an
invalid argument
for example that the alignment problem
would be solved
automatically in the course of normal
progress and this remains a point of
confusion
for me actually after talking to some
number of researchers who argued this uh
argued this to me because in some sense
like it could be that
as we try to make progress on ai doing
exactly what designers intend we just
get better and better at that and it
just kind of follows naturally that it
will be perfectly aligned with all of
human values i don't
think that this is true in the whole or
i think that there's a possibility that
they that they won't be uh in the course
of normal progress depend um given who's
developing these and what their goals
are
but
there were various kind of side cases of
how this was described to me where i was
like i'm not actually sure about that
number of uncertain and unusual
responses
i heard from the number of people that
we need to stop worrying because perfect
alignment is not needed um and then
similarly humans have alignment problems
too
for those uh for those points that
people brought up i often wanted to talk
with them about what i think are the
difference between humans and very
advanced ai which is uh humans do have
plenty of alignment problems as well
like people argue with each other
they're not perfectly lined with each
other um and like they argue with
children for example and disagree with
children so i think this is different in
a few ways one is that humans are on the
whole less powerful than i imagine these
general ais will be in that if you lead
a country you have a lot of power as a
human that can affect many people but
most humans are not in that position
we also have the opportunity to make
sure that these systems are aligned with
us in the way that humans and humans and
other humans don't have
because we're creating new systems and
then i think a third point on this one
is that humans are in fact
somewhat quite like more similar to each
other is what i'd argue uh compared to
ai so humans have the same biology uh
and come from similar cultural contexts
have similar priors on the world um
because we're from the same species
versus ais can be are kind of exploring
the space of reality and they're being
designed in the space of reality which
is quite large and so i would expect
that um you you just much more get alien
values than you would with humans and so
the alignment differences would would be
more different than between humans so i
would discuss with these and see if
these were convincing to people
i then move on to our my final core
question uh which is uh about
instrumental incentives so
all right next question is so we have a
ceo ai and it's optimizing for whatever
i told it to and it notices that at some
point some of its plans are failing and
it's like well hmm i noticed my plans
are failing because i'm getting shut
down how about i make sure i don't get
shut down so if my loss function is
something that needs human approval and
then the humans like want a one-page
memo then i can just give them a memo
that doesn't have all the information
and that way i'm going to be better able
to achieve my goal
so this is not positing that the ai has
a survival function in it but as an
instrumental incentive to it being an
agent that is optimizing for goals that
are maybe not perfectly aligned it would
develop these instrumental incentives
so what do you think of the argument
highly intelligent systems will have an
incentive to behave in ways to ensure
that they are not shut off or limited to
pursuing their goals and this is
dangerous
so this is uh the question i asked um to
address some of the things here about
but in the long term we'll like test it
but long term it probably won't design
uh designed that way sorry it doesn't
actually fully apply that argument um
but the idea that like we might actually
not be able to test it too much if we if
we deploy it because of this
instrumental incentives argument all
right so put those in the chat
response agree agree agree
lots of agreement agree totally agree
anything that is the powerful never die
is too powerful agree agree yesterday
and said unsure on the danger
yep
absolutely
virtually certain
agreed
um again so uh
thanks everyone um yeah so my
my sample my the researchers i talked to
were much less agreeable on this
particular question compared to the
previous ones uh partly i think because
of the implication it implies that
things are things are in fact quite
dangerous and they might be very
dangerous as we get to this level
um
and so there's a number of people
thought it was a valid argument
so that included various levels of
uncertainty like yeah that seems like it
might happen i'm not sure i'm not a lot
of that uh and a number of people who
thought it was an invalid argument so
arguments like reward functions don't
work this way or they won't be designed
this way in future um weird arguments
like we will be able to physically stop
the ai that one i often pushed back on a
bit because i
think that if you have a you kind of
have to imagine this thing as having
something like human level intelligence
so uh
because because i think we will get to
something um at that level at some point
and if you have a huge if you have
something that is very good at reasoning
that may know that you intend to
physically stop it so they could build
up barriers in the way in front of its
uh stop function or it could um get
itself uploaded to the internet in some
way and then it's much harder to
physically stop for example
um some people said that we did human
oversight or ai checks and balances so
that's the idea of like an ai monitoring
another ai um i highly
i would be very excited if we had good
human oversight i think it's it's
trickier than just putting a human in
charge of watching um the ai decisions
because maybe the ai thinks better than
the human
and is able to trick it but i think
human oversight seems pretty key um uh
we will not tell the ai about the fact
that it can shut down i would then argue
to people something like well
if the ai is operating in reality a node
and
just can't see the fact that machines
can't get shut down or humans can't die
or things become broken if it's sort of
missing that concept i imagine it will
kind of figure it out on its own just in
the course of trying to navigate
reality uh nothing like this happens in
current systems uh that seems absolutely
true to me um it seems true to me that
we don't have these instrumental
incentive things arising in current
systems because i don't think they're
good enough at reasoning and operating
in complex enough domains
consciousness won't happen
and it's hard to predict the failures of
future systems wait until we're closer
i'm i'm quite sympathetic to this
argument i think that if
uh like if people in the 1950s were
worried about uh
you know ai
taking over our ai the ai alignment
problem but they didn't know about deep
learning they would hardly be able to
make a lot of progress on this um and so
if we're like several paradigm shifts
away if this thing isn't happening until
like 300 years then we should probably
wait until we're closer before getting
real worried about it i do think we have
more evidence about agi coming sooner
than we were expecting um and so i'm i'm
pro people working on this uh because i
worry that we're not working enough
ahead
given that it might be coming in you
know 10 20 30 40 years uh but
uh that i think this argument makes
sense if your timelines are long
and some other things people brought up
is that they would related to the
previous one they would need to know
what type of agi for safety you need to
know what you're working on um a number
of people thought misuse was a bigger
problem so let's focus on issues of uh
not the systems being designed badly and
um but rather just people using systems
in very dangerous ways
bad ways um and some people said this is
not as dangerous as other large-scale
risks that we're currently humane is
currently facing again i think this is a
little bit influenced by uh the um
the russia's invasion of ukraine people
are concerned about nuclear and other
risks as well
great so that those are the things that
i walked my researchers through i spent
um a lot of time on this this last
question often with people but some
people i never got this question we
talked quite a lot talking about the
first two questions
so i want to give some including
thoughts
so why do we care
so i think it's likely that agi will
occur during our lifetimes that the
alignment problem could be non-trivial
so we need a lot of research effort into
it and there's some probability of
existential risk from the alignment
problem or other risks from very
powerful ai systems
and i think we're currently ill prepared
for these risks and not investing enough
in technical advanced ai safety research
so we're investing a lot like a lot of
investment in capabilities from
governments from industry
for talent um and then we're investing
some amount in current day risks um
things like various privacy
and robustness a whole bunch of
different issues and i think that's
definitely good um and also some of the
uh current day sort of research may
apply and may scale to very advanced
systems however i think most of the work
today on current day risk doesn't scale
very well to advanced ai systems so i
think work on robustness maybe scales
better than things like
current day fairness just based on like
the type of research that is being done
on the type of system and how custom
built it is for on that particular
solution and so there's much less
investment in advanced ai risk um
and i hope that if people agree with the
arguments detailing why there might be a
small probability of risk with great
potential downside so that
this is not only the
like the extinction of of the current
humans but also you know the lack of
children lack of grandchildren
lack of people
like the whole of possibility of humans
existing in the future millions like
very like very long time scales but even
just like today's people and
the lack of the next couple generations
um
that they may invest more in advanced
safety research or in governance aimed
at advanced ai
and so this research project was meant
as an exploration for how experts relate
to arguments describing risks from ai
so i'd like to acknowledge my
collaborators um sam huang mary kelly
wilkes my faculty mentor so toby
gerstenberg
people who helped with transcripts
angelica kitt and shannon thank you very
much to hai and cezak um especially for
hei pitching and additional money to
help with the transcripts i also have
additional slides available on the
following questions so if you could
change your
colleagues perceptions of ai what do you
think the public and media worried about
how much do you think about policy and
what we can convince you to work on this
and the slide i want to end on is
further resources so uh if you are were
interested by this talk there's a number
of further resources that you can use to
learn about uh learn about more so first
thing definitely contact me i'm bill
gates my email is vl gates at
stanford.edu
definitely feel free to email me at any
point including
people who watch the reporting later
and here are some readings that i think
are
quite illustrative so
if you go to tinyurl.com existential
risk from ai you'll reach an article by
kelsey piper
an article in box which kind of lays out
the arguments here something like what i
was doing with researcher quotes um
there's also the books the alignment
problem and human compatible uh by
stuart russell o'brien christian and
stuart russell um and note that the
above are introductory and public facing
so if you want if you're a researcher
and you want more technical readings uh
feel free to contact me i can send you a
whole list of work in this area
and there's also the opportunity to
connect with other researchers at
stanford so we have the center for ai
standard center for ai safety which is
working on some people that are working
on some of these issues
we also have the stanford existential
risk initiative
so stanford has a lot of resources and
energy kind of in this space
and there's also funding for students
and faculty interested in doing this
sort of alignment research
from
open philanthropy uh from the fdx
foundation future fund and the ltf
and i will include i think all of those
links will be posted online at some
point but uh if you'd like it in the
meantime please email me bailgates the
lgates at stanford.edu
and so with that i want to thank you all
for listening and participating and we
can move on to the q a
great thank you so much bill um we have
a lot of really great questions
um are you able to go maybe just like a
few minutes over time
okay
um
all right so let's see so the first one
i want to ask is from daniella
um how sensitive are ai researchers
about colonialization issues and how ai
replicate sustain um
these issues did you hear about any of
that in your conversations
yeah
um
so this wasn't so uh ai researchers were
concerned about a number of risks
and a lot of them were
sort of in the domain of
you know treating people fairly
making sure that people's like rights
are preserved in various ways
this was mostly shown in my the question
where i was asking people about risks
may i i think at that point i pretty
quickly swiveled to the the core of my
questions which were more about um like
existential risks and a very very
general ai but people did mention
things in this vein
more
maybe not specifically about colonialism
but a lot about rights
thank you
um
let's see
yeah so kind of related um
cassius how do you align ai with human
values when human values vary by culture
and what happens when the competing ai
systems have opposing values
yeah uh so
um
for for the part that says computing ai
systems have opposing values
uh
ideally what we want is an nai so we we
get to control what happens with the eye
we are making these systems
and so ideally we want to instill in
them uh
an ability to
adapt to different types of human values
and values that change across different
people in different areas and so some of
the research areas that people are
working on
in the alignment space are things like
human interaction with with ai so having
a human oversight having ai instead of
by default just pursuing whatever goal
gets put into it instead having it have
uncertainty over what the reward is so
what the goals are such that it's
incentivized to ask humans what it wants
at any at any juncture and so then you
still have the question of like who are
the humans that it's asking but you can
imagine like various types of systems
for aggregating responses across numbers
of humans um you also maybe want to ask
humans over like a long time period so
like if they've had a lot of reflection
time or you want ai maybe to help um
uh or
do you want ai to kind of imagine what
humans would do like in the future but
we're kind of at the very basic level
where we want ai's for example not to
just uh do whatever we've programmed
into it initially any whatever the
design is programming but even be
incentivized to maybe have uncertainty
and ask any human what they would like
um so i think we're at a pretty basic
level in terms of what research exists
and how we're getting it aligned to any
humans at all yeah yeah good
um and then also somewhat related um
elizabeth talks about uh how
many researchers
often these types of conversations are
eurocentric
um do you like how what
uh your sample of researchers do you
feel um it was like appropriate global
uh sample do you feel like there's gaps
yeah so uh my sample was so i kind of
reached out to just the sample of people
who had gotten paper submitted generics
and icml and that isn't that definitely
isn't just a fully representative
example of like the people that exist in
the world it's
biased in the ways that you would expect
it to be biased um so like most of my
people i talked to were male
for example and most of the people i
talked to were upper level grad students
so like fifth or sixth years in the us
in europe um there was a couple
uh from from different places um but i
think my sample was pretty much what you
would you would expect given the
distribution of
people who work in this space yeah good
good thank you
um
yeah so then one thing i thought was
interesting um i think you said that
about 25
of the researchers were not aware of the
alignment problem
do you have any thoughts or reactions i
re i was surprised by that um yeah it's
funny because uh so most of the people i
talk about here i'm like oh yeah you
know twenty-five percent everyone
learned the alignment problem they're
like what but everyone here talks about
that um but yeah i think that that is i
think it's revealing that there are just
many different types of bubbles uh
in in the different communities and in
fact it doesn't feel particularly
surprising to me that people who are
working in like at startups working on
ai applications maybe haven't heard
about this um it does feel a little bit
more surprising to me
um that or in fact people who worked in
like very like
open my uh open ai deep mine kind of
like the the companies that are really
pushing um or have it explicitly in
their mission statement they're working
on agi likes or trying to get to
something like aji like systems have
heard of this um so it sort of just
depended on the distribution um but yeah
people sometimes that were like oh yeah
i saw that on twitter once they're like
yeah i guess i had a colleague who's
maybe working on that so there's there
was a variety a variety of different
types of responses there yeah yeah
interesting
um
cool let's see
all right so this question is from roger
um ai researchers have big incentives to
be optimistic about capabilities
strengths and downplay weaknesses
including high dollars enthusiasm and
press culture
um
[Music]
did you measure or adjust for this in
your research
yeah yeah so um okay so the idea is that
ai researchers
um
yeah have a lot of reason to be
optimistic uh there's so yeah there's a
lot of different kind of researchers
incentives here i was frankly surprised
by the number of people who thought that
agi would never happen um or who had a
pessimistic outlook on ai i think
there's something like uh
or like you saw the number of people the
the number of risks that people were
bringing up um so i do think there's
certainly there's like and there's a
prior baseline of technical optimism
given that they're working this there's
people work
given that they're working in this space
but i think there was also maybe what
you would expect to be a surprising
amount of people being um kind of
worried about what was going on there um
did i adjust for this not particularly i
think i was just trying to get a survey
of what people
thought on these particular things
although also keep in mind that like
many of them had never the questions i
was asking um uh about the alignment
problem that they're instrumental in
like very few people knew the details of
those so this was often quite new to
them um and perhaps this would mean that
they had uh well
um i guess you would know you would
still see a a biased words like oh no
but my my the thing i work on can lead
to that um
and yeah actually that makes me want to
skip a little bit to what would motivate
people to work on this so i'm just gonna
stick to one of those slides
um
uh yeah so i actually asked people
whether they'd be willing to work on
these sort of alignment questions and a
number of people kind of said yes they
would they would work on long-term
safety now that's been descended i think
it's important they would need to learn
more um but a number said look there's
not any
i would be like this this field seems
somewhat pre-paradigmatic that seems
true to me in that like we still need
help trying to figure out what the
questions are and like what directions
we should go and so they would need to
have a specific problem or incentives
although people are working on making
this uh
be like a more outlined problem so that
people can be funneled into space um a
number of people are like oh yes i would
work on current day safety so if i kind
of missed my question about more
alignment um and other people and then
some i think maybe one person had tried
long-term safety work but found it kind
of depressing that they hadn't found
um the rest of the i think it really
helped this person seem to have not
connected with the rest of the community
working on this which i think is
actually very important when you're
working on this
kind of uh work because it is you are
thinking about like the death of all
humankind
um
yeah and then some people were like no
um i need examples in current systems or
i'm not an ai or i'm not at fort in the
forefront or various uh kind of
perspectives on it's not being my
problem
um
so this is like most people again we're
not like very interested in switching
fields which makes sense it's a lot of
work uh but um yeah there's there's
different kind of vices and and uh
thoughts here and beliefs here which
makes sense
yeah yeah good good so um yeah another
question kind of related to i think one
of the questions you asked but didn't
quite share yet um from jennifer and a
few others are wondering about
what kind of public policies need to be
implemented to curtail these risks um
yeah and if you got any any feedback or
if you have thoughts on that and what
needs to be done
yeah i think that um so we have a
problem in the policy space where uh
um ais and emerging technology is moving
incredibly fast like the number of
papers that came out in like just april
may but even before that is just it's
absurd um and so government isn't set up
to deal with that isn't set up to like
follow the pace of technology um and so
that is one of the problems that is
coming up and also
not many people are future looking so i
think it's actually very important what
i always want in policy is for people to
be looking
ahead like you know to these very
advanced ai systems and trying to figure
out what we can do about that um and i
know that the center for security and
emerging technology does a little bit of
this uh in in the us and there's a few
other places as well um the ai
researchers expressed things like okay
oops here we go um that they don't know
the space well it's not relevant to my
work policymakers don't know enough more
education is needed scrutiny should be
done by specialists somehow it's too
slow
um we should they should work on
short-term issues rather than long-term
issues it's uh um different from what i
argue um we need regulations but don't
slow down research this is this was a
common point of contention um we needed
maybe some people said we need to
regulate agi worldwide market regulation
market incentives but generally people
i think people hadn't thought very much
about policy stuff and they were
confused about what to do because i
think it is a very difficult problem
this is why i think we need more
researchers working in this space um and
policymakers in the space because it's
difficult to know what to do um the even
trying to figure out what to do for the
safety aspect uh the technical safety
aspect is difficult and government's
governance i think even more so
yeah yeah yeah you mentioned in your
talk that you know
uh researchers aren't incentivized to
think like 50 years in the future
and i think i saw someone in the chat
mentioned that that's quite scary um
yeah yeah there's so a lot of people i'm
like have you ever thought about this
they're like well i guess sometimes like
around the water cooler like sometimes
occasionally but mostly talk about work
and i'm like yeah that's right people
talk about the work that's their day to
day um and yeah there's not very much
reason that any of us have to think very
far in the future at least in terms of
our context so yeah yeah
good
um all right i'll ask just one last
question
um
so
turning the questions on to you a little
bit um that you asked the teachers what
are you most excited about in ai and
what are you most worried about
all right i am i am definitely most
worried about uh this existential risk
problem i i personally definitely do not
want to die early
um uh and i also am
so i'm excited for the near-term future
um
okay so so my belief structure is such
that i think that there's actually a
very good possibility that will die
kind of early um from agi and this is
um like very worrying to me uh so i'm
really looking forward to the bit before
that when we have all sorts of really
convenient cool technology uh that lets
our lives be like very fun and
interesting and convenient um and
i'm really not looking forward to if
this problem doesn't get solved and uh
if it does get solved however i think we
have a really great feature ahead ahead
of us um so if we have an ai that is
like more intelligent than humans that
is perfectly aligned with human values
it can kind of help us achieve whatever
we want so i look forward to things like
you know extended lifespans um like you
know many generations to come um it
could probably help us getting get
control of the other existential risks
that exist so maybe we wouldn't be in
danger of nuclear anymore or or
pandemics or climate change
um and it can do like more wild things
so maybe for people who are
if if this is desirable for humanity um
we could
do
like yeah it's kind of kind of so many
things are are available in the future
if we're interested in this it's more
sci-fi stuff as well um so i really hope
that we get more researchers working on
this uh and are able to
reach those good features
good good
all right well thank you so much um
i will let you go and let our audience
go um thank you to our audience for your
participation and your questions
and the hia seminars will be on break
for the summer and will return in the
fall so september be on the lookout for
announcements of speakers thank you so
much fail for for speaking thank you


Demo Day 14: AI safety at Faculty talk

Thanks thanks everyone
tough tough acts to follow I will talk
to you guys about some of the work we've
been doing the last year or so and a
nice safety I'll talk for about ten
minutes and then you can go get drinks
and hang out and rest for another ten or
so excellent talks so my name is Ellie
oh no I was trying to use the laser yeah
there's anyway um my name is Elia um and
I'll just jump right in so faculty often
says our mission which is to make AI
real in enhancing products improving
services and saving lives but making a
r-real means means making a nice safe so
first let me just kind of coalesce a
bunch of ideas of what AI safety is so
you guys might might think about unsafe
AI in a bunch of different ways from
killer robots too biased algorithms and
it's probably like these are all AI
safety issues but to kind of put them
together in some in some kind of
framework we like to break the space
down into the autonomy of the algorithm
so something like a classifier that
makes a business decision or something
like a autonomous agent that learns as
it goes versus the intention of the
practitioner so a practitioner who is
just on a data science team at a company
trying to build a useful algorithm
versus a kind of a malicious actor and
indeed these these examples that we saw
kind of land across the spectrum with
some of the kind of longer-term
existential issues higher up on the
spectrum as as agents get more
autonomous so this is this is AI safety
except there's tons more examples so
here are here are many of them this is
definitely too much text on a slide but
anyway there are lots of examples some
of these are
are technical like unsafe reinforcement
learning so how do you make
reinforcement learning safe of as well
as things like AI generated media how do
you detect a I generated media and some
of these problems are our policy
problems like you know mass unemployment
so we've been kind of working across
this space kind of excitingly last week
we were at the Copenhagen democracy
summit demonstrating this this fake
Donald Trump algorithm where you can
where you just get a video of him
speaking in his own voice of things that
he never said before as a demonstration
of of what the kind of what the current
cutting edge of AI can do to to wealth
to a bunch of world leaders there we've
also done some some just pure academic
research in reinforcement learning
inspired by how parents train
intelligent agents to you know be
relatively safe in the real world and
this is just some research in the last
year that's come out but what I want to
get into details on today is is this
bottom left quadrant which is which is
really what companies companies in the
economy right now it should be should be
thinking about so anyone who's deploying
AI these are these are relevant
questions so things like bias well
actually a useful framework for for
these things is is fairness robustness
and explain ability and you can think of
fairness as kind of generally will my
model do the right thing you can think
of robustness as will my walk model work
the way it works in the lab in in the
real world or in deployment and you can
think about explain ability as how do I
understand my model kind of
independently so I'll go through each
one of these things and and kind of
present some of the some of the work
we're doing on this front and I guess
the the high level is is yeah you can do
something here like there you can
actually make
progress on each one of these fronts and
and and deploy safer AI basically
already so firstly fairness you can I
mean we we have tools that can that can
automatically test for and enforce kind
of flexibly enforce fairness for once
you have a precise definition of of what
what fairness applies so there's still
the difficult ethics question but given
that you can then solve the machine
learning problem and this curve shows
shows a model that has kind of a gap gap
to optimal performance so if you go up
on this graph your your performance is
best and if you go to the right you're
the fairest and as you'd probably expect
if you if you thought through there
there's a trade-off here but the
important thing is if you just train a
model with with kind of current current
open source techniques you you you
always wind up right here in the upper
right so this is just you know model
that decides who gets alone but if you
kind of use the right approaches you can
you can not trade off too much
performance and and and more than have
the gap of fairness so this is kind of
well I guess the thing you should take
away from this is is is define your
fairness and then and then without
losing a lot of performance if you use
the right techniques you can you can
well do something to to improve improve
the bias in your models but certainly
all all approaches to this aren't aren't
created equally for explain ability this
is a massive massive topic so we've been
doing a lot of a lot of research on
explain ability I hear a couple papers
that came out in the last year one of
which was that I say about last month
and one came out last week with a kind
of a state of the art model that was
also fully interprete believed been
turning kind of that that research into
into tools as well so here is a kind of
a snapshot of an explained ability tool
that we're working on and just you just
start in the upper left you get you get
kind of a look at how the model performs
overall so this says
what were the most important features
that went into my model this here is a
just a neural network model that decide
decides who's going to default on a loan
on some kind of real world public public
data and the most important thing is
like the grade that the lending platform
gives to the loan and the third most
important thing which you can't really
see is the credit score and actually one
one kind of cool thing is this these
like bar chart structures kind of look a
lot like what David was showing for the
marketing attribution stuff so the like
statistical techniques that kind of
optimally attribute how a model is using
its features are the same ones that we
use in marketing attribution so there's
kind of a very cool synergy there but
anyway so this gives you an overall look
at how your models are your models
performing in a kind of nice digestible
way with all the good stats and
engineering under the hood there but you
can you can then just like click on that
credit score and then you get over here
where you can where you can see over my
whole data set how how does the model
use each individual or look at each
individual person and and decide how to
how to use their credit score I mean in
the bottom right you see kind of a
scatter plot where the kind of green
curve goes up to the right that's people
with higher credit scores are more
likely to pay their loan back as you'd
expect and the red curve goes down and
to the right people with lower credit
scores are more likely to default as
you'd expect but but this tells you kind
of so much more than that you're looking
at the whole data set right these points
are all your all your customers and the
the slope tells you how your model is
responding to credit score in the way it
makes predictions the the turnover point
out about 690 tells you that a credit
score above 690 roughly your model is
going to start calling that a positive
indication of paying back the loan but
below 690 it's not and you can see that
near that turnover your data's really
tight so that your your models pretty
confident what to do there but if you go
out into the like high credit score
region it
spreads out a lot so the models kind of
uncertain about how to use that feature
out there and and so so that's kind of a
lot of global information about how your
models using this feature you can go
through every single feature like this
but but what's what's cool is you can
you can you can do more you can click on
one of those individuals this particular
that's kind of big you can click on that
individual because it's a weird outlier
and see what's going on and indeed this
person is someone with an 850 credit
score who who the model thinks is going
to default on their loan and you can go
ask why does the model think that and
that's what you get in the bottom left
here and the top the top red bar is the
model says because this person jointly
applied for their loan the model says
Matt they're gonna default which is a
little bit surprising the the second
thing which is about half the size is
the grade that the platform gave and
that the grade is telling you that this
person is going to pay their loan back
as you'd expect and then the third thing
is the state that the person is from
this is a u.s. data set so somehow this
model is saying well like the person has
like a massively high credit score but
they're from Alabama so so we're going
to we're gonna say default on their loan
and they apply jointly and and the
reason for this is most likely because
this person comes a REIT from a region
of the data distribution where there's
just not a lot of data and the model
just doesn't know what to do over there
so anyway III could certainly go on and
on but this type of tool allows you to
just just probe and probe and probe into
how you're like black box and real
network is working and understand its
shortcomings so this is this is kind of
what explained ability is is is is
shaping up to look like and then just
the last thing is robustness so there
are tons of issues with robustness but
this is maybe one of the most the most
shocking ones here are five models in
coral that are trained on five different
data sets the coral bar gets up to it 95
percent accuracy of the model so those
are good models they're all neural
networks and this is all the test set
performance so it's
of best practice um but if you go to
that model and you're allowed to like
tweak the data just a little bit just
make tiny changes the data you can
completely destroy the models
performance so the gray bar is how well
the model performs on data that's trying
to trick the model so tiny little little
changes and for like complex data you
can just get the Akshay down to zero so
the model makes a mistake every time and
then but even for data that has ten
features in two classes like the stuff
that ever simple stuff you can cut the
you can cut the accuracy kind of down to
two thirds and this is this is not that
esoteric imagine you're a big bank in
Europe you're a compliance function in a
big bank you've used because you're
working on Prem you've you've grabbed a
speech transcription model online you're
transcribing your employees calls to
make sure that their their behavior is
is is kind of authorized um I can just
go grab that model online as soon as
long as I know what model are using and
I can find out how to trick the model
and then every time I have my call I can
just insert that noise and the
transcription engine is going to just
transcribe gibberish I so I've just
completely fooled the compliance
function of this Bank that's totally
totally doable so these are the types of
things that that that you need to worry
about and and indeed you can so there's
a way it ways to mitigate this it just
doesn't happen you know for free cool I
did I didn't mean it like that I didn't
mean that I'm a technical person I don't
know about sales anyway so this is how
badly model works if someone's trying to
try to trick it cool so I'll conclude
just just coming back to the main point
so this is this is a quote from Stuart
Russell who who we were talking to
yesterday and he said these words and
engineers don't don't distinguish
between bridges and and bridges that
don't fall down there's just just
bridges and and that's how we should be
thinking about AI so so making a real
means making AI safe and well you know
that's that's all I have cool have a
good break see in 20 min
[Applause]


AI Safety Reading Group (Session 42)

hello and welcome to the 47 40 second
session of the AI safety RK reading
group and today we will talk about a
article in nature called robotics ethics
of artificial intelligence by Stuart
Russell Russell man and manuel de
Villota nature is probably the most
prestigious international journal of
science and this is an article from two
years ago where with it with a headline
for leading researchers share their
concerns and solutions for reducing
societal risks from intelligent machines
and of this I we're only reading three
of the four because they're being how
has a lecturer in robotics it's not
exactly talking about ethical problems
but more about solutions in particular
startup Robo hot dog and this is not
super interesting for us much more
interesting is the article by Stuart
Russell professor of computer science
from Berkeley Russ Altman professor of
bioengineering and computer science from
Stanford men we love n minus a libelous
professor on computer science from
Carnegie Mellon of course I sorry I
forgot to share my screen in a moment
you should be able to see my screen so i
hope you see great so we'll start with
Stuart Russell who talks about
artificial weapons in particular that we
should take a stand on them because we
in this case is the people working as
official intelligence and robotics we
need to decide whether we want to
support or post in the development of
lethal autonomous weapons system
abbreviated laws so we are talking here
about robots and other kinds of systems
that cure
who to engage choose who to who to kill
really without a human in the loop this
is something that the shiraz business
will be feasible recently soon because
all the elements that are required to
have legal alternatives where the
systems have been developed but in
isolation so they just need to be
combined and of course building these
things take some time but data is
working on it right now the Defense
Advanced Research Institute from from
the United States and what it looks like
these legal autonomous weather systems
at least in the near future they will
probably have a form somewhat like armed
quadcopters like these drones you see
some from time to time and not remotely
piloted drones like the Predators there
are of course
and I think the Eagles the system is
probably recently clear in who has the
plane if a lethal autonomous weapons
system fails because if say the Danish
military feels some lethal autonomous
whether the system and this by accident
kill someone then I believe the Danish
military will obviously be P to blame
from a legal point of view in much the
same way as if they you know fire a gun
or artillery and kill someone like that
there it will be probably a lot of the
things and I'm problems with autonomous
weapon systems will be recently close to
the ones we are looking at right now and
right now there are a number of rules
and laws of war in particular there is
the Geneva Convention which has very
roughly four parts the first is that the
military means that I used must be
necessary the second is that should be a
discrimination between innocence and
people I to actively fighting and there
should be some kind of proportionality
even if there's one enemy in a village
it's not acceptable to annihilate the
entire village because there is not a
proportional response even though it
kills one enemy if it kills too many
innocents the fourth is a principle of
humanity and somewhat more Bay and in
particular some of these obeying these
international boss is half I mean it's
hard for humans and we expected to be
much harder for AI in particular in the
beginning something like discriminating
between combatants and non-combatants is
something that will be really really
hard for AI to do right now and Stewart
Rosen writes that if the international
laws are not amended in some way to
account for artificial intelligence the
alternative will in
evitable an arms race I a couple of
times ago we chopped we had an article
about arms races and got into more
details about what is an arms race
actually and from that I must say that
to me it's not clear at all that we will
have an arms race based on legal
autonomous weapons systems it might be
that the Americans make them and the
Russians create something that is really
good against drones and then we don't
have an arms race at this not an arms
race as defined in the in the previous
articles but this is probably a minor
point so the international status is of
course that right now the lethal
autonomous weapons systems have not been
banned but a number of people in the UN
are trying to ban them in particular
Germany and Japan I against leave all
tournaments organ systems and United
States United Kingdom and Israel are for
this if it's bent it will probably to an
extension of the convention of certain
conventional weapons that's a really a
wonderful tool title contention on
conventional weapons but the the
argument that is used against the
terminus women systems is that the
countries that are deploying them
already have internal review processes
to ensure compliance with international
law at least that is how your Brussels
frames their argument I'm not sure it's
exactly a good summary because saying we
if some people I'd say arguing that we
should change the law then saying oh we
shouldn't do that because we are already
compliant with the law it's kind of a
non sequitur it doesn't follow in any
meaningful way there is under broad
strokes however an international
consensus that that should at least be
meaningful human control with these kind
of autonomous weapons but unfortunately
the word meaningful is undefined and
not meaningful so this means that it's
something vague that will that are not
restricting anybody from doing anything
in practice there are more arguments for
or against these kind of weapon systems
one is that if AI turns out to be really
effective and they might also be more
selective and if they are more selective
than humans they might minimize civilian
casualties like you will go back to the
Geneva Conventions if the discrimination
of combatants and non-combatants can be
done better by an artificial
intelligence and by a human then it
might be more ethical to have lethal
autonomous weapons systems and I would
like to make a point here that this is
something that has been argued very much
about remote weapons and not autonomous
but remote weapons here for instance in
particular the American Predator drone
the program which has killed a lot of
people and where where people argued
because this weapon allows people to
take decisions while they're sitting in
an air-conditioned room a thousand
kilometers away and not in the heat of
the moment they will make better
decisions and there will be less
collateral damage and this is something
that is very contentious Pakistan claims
that 50 civilians are killed for every
militant while so like a huge amount of
collateral damage the United States
claim that there has never been any
collateral damage at all which is also a
very fantastical claim I will probably
get the truth is somewhere in between
and so that is one argument that also
needs to be considered done more because
if legal autonomous weapons systems
turned out to be really effective they
might lower the threshold for going to
war which just like our power made it
more
people the American and the United the
European Union were more active in Libya
because they could use air power instead
of having boots on the ground and this
kind of threshold theory is really
important for when countries go to war
another problem with the autonomous
robots is that they if terrorists get a
hold on them it might be something that
is very easily easy for terrorists to
repurpose to to attack civilians and
that might be really bad it might also
be something that peacetime policing
functions could suddenly start to use a
lot if them if it developed very very
much by the military and the last it's
one I have a bit of a problem there a
man a while utilitarian consequentialist
but many people are not and solve the
people who are not to say that lethal
autonomous weapons violate a fundamental
principle of human dignity because the
machines choose who to kill and this is
a problem too many people and I of
course respect that and not exactly sure
eyes can emit is in DC at least why it's
so important who decides to kill
civilians I think the more important is
to avoid civilians are killed but but
this is something that many people care
deeply about the Stuart Russell's last
point of maybe second vast is that we
should consider the end point of the
trajectory meaning that we should
consider a world where drones have been
developed fully where the artificial
intelligence is a solved problem where
the weather systems are limited by
physics and not like the capabilities of
artificial intelligence in this case it
looks like we're going to have very tiny
flying robots extremely maneuverable and
thus very hard to target that carry just
a one grain shaped charge that is enough
to kill just one human but
applied in millions and this kind of
thing were opposed to be very hard for
humans to defend against and this is not
a desirable future so
that is of course the people who develop
this would say yeah we will have these
tiny flying robots and they will only
carry one grand shaped charges but they
will be able to but they will be able to
distinguish between combatants and
non-combatants and maybe even important
competence and not important competence
and only target the leaders for instance
and this might be true but it's also
something where unprepared humans at
least are utterly defenseless and that
sounds like Stuart Russell strongly
believes this is not a desirable future
and I probably think I think that's
probably a reasonable common position
though not self-evident but but you're
right of course that in theory it could
be used only for good it could be all
used target exclusively the Islamic star
state and al-qaeda leaders and then it
would only have good have good effects
if it's used only like that but this is
something that we need to make decisions
on we need the people who are working
with artificial intelligence and
robotics need to take a position think
about this maybe organize the basis
starting the arguments right positional
tables and vote of course also in their
respective organizations and maybe in
the government's because if we don't do
anything then that's a vote to continue
to build and deploy these weapons so
that is your Russell's hope of course
that that people will take take actions
against this I don't have a good view of
the political structure my intuition is
that there's not going to be a ban on on
any for Thomas weapon systems at least
as it looks now but this is not
something that I really know a lot about
so action is definitely need if you
believe that this endpoint is a really
bad impact moving on from Stuart Russell
to just briefly Sabine Howard why I
chose to not include ur just made this
robocop about how to shape the debate on
artificial intelligence and how to
ensure that different actors that are
working with artificial intelligence
have some kind of coordination and unity
of message and things like that and that
might be be a good thing and might be a
bad thing depending on well whether you
agree with them disagree with them but
it's not something that has a lot of
ethical significance Russ Altman however
has he started his article by having
three paragraphs of just our listing all
the wonderful things we could do with
artificial intelligence this is not
really an ethical statement because all
these wonderful things AI could do are
just good so that's not really an
interesting ethical article but it does
have to Ithaca concerns one is that
there will be a greater difference
between the health care that people in
the rich world receive and people in the
pool world receive in rich people and
poor people and you strongly believe
that if there is a two-tire system where
rich people can benefit from powerful
medical algorithms and poor people
cannot that would be unjust and unfair
and he believes to avoid this is the
responsibility of both the government
this is the responsibility of the
government is probably reasonably
uncontroversial but in particular it's
also the responsible guilty of the AI
researchers to ensure that the AI
technologies are distributed equally so
this is one of his concern the other
concern is that the result of artificial
intelligence can be really hard to both
understand
explain if you have something like
Beijing models you can to an extent
understand it and you can explain it but
deep learning and neural networks
notoriously difficult to understand and
explain and that would be kind of
difficult if you have a patient and the
algorithms say you should operate on him
and you can't explain to the patient why
you should operate on him that sounds
quite ethically problematic the second
the last part is Manuela bullosa who
says we should embrace a robot human
world because robots as they are now and
humans are different in that humans have
much better perceptions much better
cognitive ability and much more better
actuation we can do much more things
with our hands than robots can do and
she believes this may always be the case
this is exactly the opposite of Stewart
Russell who believes that the ethical
point was to look at the end point of
the trajectory and this caused her to
believe that robots will complement
humans not supplant them and here the
problem is to enable robot to ask for
help if it doesn't understand the
situation if it's in doubt about
something or it can't do a particular
thing it would like to do and to figure
out how do have a question
and that this is a good question in the
article Manuela bellows it does not give
any supporting argument for she just
rice this may always be the case without
anything in supporting things i think i
would expect when Manuela below that it
says always she doesn't actually mean
always but has just a more narrow
horizon and so she says this may always
be the case but it's actually meaning
within the next 20 years or something
possibly because if we are trying to
influence ethical matters then we might
be able to influence what happens within
the next 20 years and we cannot really
influence what happens more than 20
years in the future so for practical
purposes always might be correct I would
this is completely my guess about what
Manuel villosus believes I don't really
have I haven't read anything else just
written apart from this very short
article but that's basically all I have
so thank you for watching and see you
again in one week