AI Safety via Debates #RB11

I'm confused this new podcast of the
robustly beneficial group in Roseanne
today we're going to discuss a paper
that has been widely shared especially
in the official tourism community which
is a paper called a I said severe debate
co-authored by Java you having Paul
Chris channel and Dario a Malay all from
open a I was published in 2018 yen harsh
now and yeah also it's a paper so about
the idea of using debates to guarantee
your safety week you want to describe it
yeah so the idea of a debate is data we
we want to answer we want answers to
questions and we want to align dancers
and align answers would mean an answer
that you know that is in our own
interest but sometimes and we it's
possible that we we are not able to come
up with an aligned answer by ourselves
but if we are given another an answer we
we somehow have the possibility to
verify it and even better if we are
given a debates between two AI agent
about a specific answer about different
answers it's a it can be even easier to
verify from these debates what what
answer is the most let's say truthfully
aligned and and usefully aligned in to
us so a quick example could be you want
to to know where is best for you to go
on holidays and so you ask to AI
debaters to give you answers they can
decode propose a Alaska or Bali and then
a given simply D sensor you use children
come figure out ways best for you to
burn holidays but then then to the to a
a is would be will set up a debate and
they will try to give more arguments to
explain why their answer is better so
the first one could say that body is
warmer than
or lastly is better but then the second
one could answer that you need to have a
specific visa on a passport to go to
Bali argument to which the first one
could again replicate that the visa
application only takes 20 minutes so it
will not be difficult after this debate
that can take any number of steps we
expect you as a judge to to be better at
figuring out what is the what is the
proper aligned answer to the question
yep so I think there's a some
interesting analogy with what we know
from computer science particle
computational complexity theory where
there was this but all different classes
of RAM one of them is like the peroneal
class which is about solving a poll say
or verifying something and it turns out
that you can have more powerful
algorithms if you can solve the
so-called NP problem so NP prime is a
prime why typically need to search for
solution and so typically you can
imagine that in auto algorithms and one
can can prove that it understands more
than the other just by sending the
solution that it has completely computed
using this way and there are more larger
classes of of of abilities of
capabilities of programs known as as IP
class or IP is intuitive proof I think
it's 30 poin Ammiano IP in us like it's
about the interactions of between
different algorithms and using this we
can have codes that are much more
efficient and particular you can solve
not only and people but even like your
so called pspace problems and so the
intuition of the authors is that this
can be applied as well to non for more
rims and a century by interacting you
gain power and that's would be a reason
why having this sort of interactions and
debates could add to the capabilities of
constants algorithms proving to you that
they are more aligned or more powerful
another advantage of this this framework
is also that it's somehow it feels easy
to to implement in practice and the
reason is that so the the weight we work
is that a systems should not simply be
able to answer questions but also do the
task of pointing out flaws in the
answers so if if now we have just one
algorithms one artificial into agents
that can answer question and point out
flows you simply need to let this agent
be trained by a self self playing
against itself or against a different
version of itself so and this is a
technique that is very well known and
mastered for forty fifty a chance for
Nepal this is how the game of go was
bitten by having a won algorithms play
against itself and this is a very useful
for training so we say that this this
framework has some advantages
you know practical advantages yeah so
the idea of algorithms interacting with
the user two four five one to improve in
particular for the for the human to be
able to know what's wrong for instance
with the algorithm how to improve Tony
turn or to verify that it's working
properly is something that's already
widely deployed arguably like for
instance if you have if you programming
in in the IDE oh I don't remember what
it means well my tiger with a debugger
fine sense of empowerment yeah thank you
then well you can think of this
interaction with the the IDE as a way to
improve your capability and your waist
you to verify that the algorithm is
that you're trying to design is doing
what you want you can query information
so it'll be like equivalent of asking
questions you can say well could you
stop here tell me the value of this
variable at this point for instance so
and and this is a clearly weight that
improves at least the pace at which we
can't you you can program your
algorithms and improve upon it so yeah
so I think there are advantages of these
but it's a debatable so let's have a
debate whether these are like actually
robust oceans to AI safety and in
particular to to alignment so there's a
lot of such discussions in the paper
lists actually in section 5 what the
core reasons to worry and maybe you can
spend some time discussion discussing
the first and second point which in my
opinion are like the the most
problematic which is the human biases
like confirmation bias and then the
complexity of the debates being beyond
what humans can read for example like
mentioned the IPS interactive polynomial
in the in the introduction this assumes
like the in this debating framework it
seems very obvious that humans will lag
behind if the debate is is too long too
complex too many things to read to
understand before making a judgement I'm
even more concerned by the the first one
but we can discuss that's the second one
that firstly like for instance if you
take a concrete example like for
instance what should be done about the
corner virus a situation yeah what
should you be doing tomorrow should you
stay at home yeah yeah like maybe these
discs
is very very complex and maybe even like
the right solution is something we as
humans have not thought about yet and
just because and this may be just
because it's very complex and you need
to analyze a huge amounts of data do you
get to this conclusion and Mary it's not
clear why you made your whole career on
a specific molecule you would be tempted
to say that this molecule is the cure
don't be more like the first point first
point like confirmation bias disease
were about 98% of people healed without
nothing so if you have a strong bias
towards believing that such or such
molecule or such or such approach so
you'd have a lot of debates based on
confirmation bias by people who wants to
come to validate what they believe yeah
in practice it's quite however like the
videos for instance about the
correlation between the belief in some
scientific consensus for instance a
human major climate change or should we
worry about the corner of our situation
these are like there's been a consensus
among experts at some point there is a
consensus among experts on both of these
but I thought that if you look in the
u.s. what educated people believe they
tend to have opinions that are extremely
dependent on the party affiliation in
particular Republicans in this case are
more skeptical of climate change if
human-made climate change if they are
more educated which is a bit weird and
similarly for the corner very situation
so I don't have this figure for the
Conover situation but what I've seen is
that Republicans are much more skeptical
of the danger of the corner house than
Democrats is it because Democrats are
more smarter like I don't think so is
more because of probably some
confirmation bias which is a very widely
well studied and
really established phenomenon by a by
psychology yeah so one of the worry is
that the AI is itself trying to to give
more true answer and more useful
information they would simply try to to
manipulate us and play into our
confirmation bias and if we are in the
scenario where the judge is the human
and the two AI is activating most likely
the AI playing best on the confirmation
bias of the judge on what the judge
already believes will be the one that is
selected that's also one thing that we
have a near discuss and which is a
somehow and assumption in the in the
paper they expect that as as a is become
better at debating they will also
converge to us being a more honest and
the reason why why this could be true is
that somehow they if if if one of the
agent is a is trying to lie in a case of
such a debate if the somehow it's very
hard to read a debate if you are in line
because the the other agent could point
out could easily point out what you lied
about and then a window debate in this
way but this is only an assumption of
the end of paper and it might be it may
be not very skeptical of this assumption
unfortunately especially if the Georgia
is a human like I mean lawyers are well
known to debate well like a lawyer
debate is not necessarily does not all
only give emphasis on on Bayesian
probably tha like if you want to
convince humans in general so it seems
that there are like other strategies
that are more useful by playing phone
sensor within movie with emotions with
confirmation bias and other things like
this and
like we made the point also when we talk
about these paper that like even if the
instead of a human you would have a
Bayesian aligned algorithm then honestly
we would still in still not be clear
that honesty would be an optimal
strategy in fact I would highly doubt it
without this because if it's if it's a
Bayesian algorithm it's going to have a
prior distribution and if somehow
there's a one agent that that because of
some information that it has access to
but cannot share it because it's
something it's in and has not signed
signed proof of the data if you cannot
prove this data you can just talk about
its own experience explore the data that
it has been collecting without proving
that it has collecting this data if you
just has a posterior that's based on
this and if it is posterior turns out to
be unlikely according to the prior of
the chart then it will be extremely hard
for h22 to make the case for itself and
particulate will probably lose against
another algorithm that's just saying
what's most likely according to the dirt
so there will be lies that the judge
will think are more likely than a
banditry and the the malicious debater
can can use these specific lies and and
be and--but the honesty data
yeah and this is all because of you
trying to be bayesian you need to have
this power distribution on the record so
of course we can this is debate paganism
but I think the authors are pretty
convinced have I'm guessing that vision
is amiss an important way to go at least
and having powers is important and if
you have a prior then you're more likely
to be some things than others
and that's the laws of how it is like
it's not about yeah and so somehow it's
it seemed to solve the question that so
the question whether the agent would
converge towards optimal strategy being
honest somehow reduce to the question
whether it's easy to lie or to defend
against some specific lie and and if I
understand well your point is that if
the judge is is the patient then it will
depend on the lie that there will be
lies that are very likely given the
pressure of the church and and for this
specific lies to the reputation of the
lie will most likely be very unlikely
and in that case it your case where it's
easier to light and refute the lie so so
all right what we said about a
confirmation bias and also about whether
to lie or not is actually also something
that's mentioned in the paper that is
quite scary about this kind of framework
is that imagine these tools to
super-agent artificial intelligence that
are working towards not not maximizing
being aligned but maximizing convincing
the judge that they are aligned and if
if we discuss also something we also
often discuss good at law and and this
is something we need to take into
account that that what is you have to
keep function of this agent we want the
objective function to be being aligned
but unfortunately within this framework
the objective function is convincing the
human Church that they are aligned so
and this makes a big difference now so
that that's why you said when we
discussed is that it would be a great
framework is the human church is
obviously aligned himself yeah which is
not something we expect
yeah the judge needs to be both aligned
and and good and performant because if
he's just aligned but he's not very good
at understanding reading the different
arguments and applying Bayes rule to
infer what is more likely and what is
not then probably if you have these two
algorithms trained to win debates judge
by this human they will try to or this
algorithm could be also an algorithm and
they we exploit the vulnerabilities of
the judge as opposed to trying to be
honest so of all the framework like I
don't think he's very robust at all I
feel there are so many things that can
go wrong and the conditions to make it
go fairly well I'd say uh essentially
creating a judge that is already
robustly and aligned and very performant
which I feel is written being aligned
and performance yeah yeah so I don't
think this poem shows the alignment
problem at all because he seems to
really require alignment to be effective
and even though it's not clear to me
that it's going to be effective yeah and
and the second witness of this was also
the scalability that Humana if the judge
is human we cannot understand questions
that are way too complex and way too
long or we also cannot judge too many
questions pertain because we are
limiting and for this this is again
something we discussed in the within AI
paper that the way we could solve this
is by having an intermediary and giving
to every judge and algorithmic
representative that will learn to
imitate the church and and vote and make
decisions instead of the church that's
also a similar idea that for Christian
now discussed in the paper called
iterative amplification framework which
is also a similar idea to to what they
proposed here with the debates which is
that the iteration including an
amplification framework starts with a
one aligned agent with low capabilities
and with mixed methods they amplify the
this agent
better and high with higher capabilities
and they were also under the assumption
that if the original agent is aligned
similarly to when in this case the judge
is aligned then the do marker population
would be aligned and where we discussed
it a long time ago before the podcast we
were also not fully convinced by the
pro-business of this technique yeah okay
I feel these techniques can improve
capabilities or at least it does in the
case of of the game of go and that's
interesting in its own sake I see this
as a way to do faster computation
century because you could always
simulate the whole thing's and this is
all just like accelerations of the
competition's which are still important
but I think there are two flaws I'd say
one of them is you still need to do some
some influence from the data you like
this is adding capability to solve a
task that is already well defined but in
my learning and point you build powerful
algorithms you also need to gather a lot
of data and do inference from this data
I don't think this is attacking this
point and the other thing is the problem
of alignment like I don't see a good
argument for why this system would be a
robust way to preserve alignment like in
the case of the game of go alignment is
easy because the objective of the
algorithms in the game of Go is very
simple is to win the game and we know we
can write easily an algorithm that tells
whether one of the player won or not but
in real life application of algorithms
that want to make robustly beneficial I
think the prime of alignment is a lot
harder and determining what the YouTube
recommender should recommend to
different people it's so complicated in
so context-dependent like we talked
about the corner of our situation like
because of the carnival situation how
YouTube should have recommended a lot of
videos that were explaining the
importance of washing your hands and
stuff like this very early on in the
corner virus crises but just you know
this was extremely hot like and most
people were not convinced by the need to
wash their hands and so you needed the
algorithm to understand this even though
most people did not understand this and
this requires a lot of techniques that I
don't feel are addressed by this kind of
of approach one of the only reason for
me to see that you could steal so the
way this table alignment is that because
they expect a judge to be aligned and
and then I think one of the most
interesting thing is that it's more easy
to to judge something based on the
debate than it is solely based on the on
the answers so that's why it somehow
this debating framework was a is better
than simply receiving answers from one
single agent if we yeah so I also think
it's true in practice I would somehow
prefer to see a list of positive and
negative arguments against the one
question to help me figure out what he's
worth what is the correct answer
yeah I guess he does have some
application for so if you take the
framework of we build AI at some point
you need to design your algorithmic
representatives and one way to go which
was what was done by Whoopi ji is to
have different options different
algorithms so the we build away from
work
there was this machine learning
algorithm that was based on your kid
pairwise comparisons of alternatives and
there was this other model that you
built yourself using like rules if and
then and subsidies and you wanted to
compare these two algorithms so these
are very simple algorithms so I guess
comparing them was not that difficult
and so what they did is they show
example they asked the users preference
for for this example of of ethical
dilemma and they also showed the
opinions of the different algorithms so
I guess this is a very basic kind of
debate but it's just seen what they
think in the end the human Georgie's but
there may be some interesting research
to be done in trying to use this
framework in more complex settings so
for instance you can imagine everyone
trying to design his own their own
algorithmic representatives to motivate
YouTube videos and maybe there would be
different algorithmic candidates for
being the quiz entities and maybe you
could have a more sophisticated
discussion well maybe when I go as I'm
say I think this should be censored
because it's incorrect here and about
corner virus and it's very important
maybe the other can can can can then say
what actually it's aligned with what
World Health Organization is saying and
stuff like this so maybe there's some
interest in this kind of framework
because we stress again the fact that I
don't think this is sufficient at all
especially if you want the the human
judge to be to be saying the end world
because humans are easily hackable to
use the words of oh you know ha ha ha
yeah we have a lot of cognitive flows
I'm quite agree we lay that there like a
lot of points were what the paper
proposed boils down to solving alignment
in an algorithm
first place coming back to the drug
example yeah so I guess this paper is
also interesting like the ideas in the
vapor have to do more I think with
explain ability than with alignment like
you then I think alignment is what's
most important but if you want to get to
alignment I think interpretive a cheek
it can be excused which not what your
algorithm is doing and to verify for
instance that it is indeed aligned and
what this is a lot what we build I
people did when they had this
algorithmic representatives that people
could test and play with and I think
this is like a extremely important
moving on but yeah again I don't think
that this paper is is very relevant to
alignment per se which I still think is
the most important point next week we
will discuss another paper core will
chan bandied for the design of clinical
trials it's not directly relevant to the
field of safety but it's very relevant
to I think it is very extremely rare
event for example if you go back to the
paper on emotional contagion yeah
clinical trials having safe protocols
for clinical trials for sequential
clinical trials could have mean I could
have me made that research more more
robust and more beneficial while
controlling the harm the potential harm
it can have on the users so it is it is
very relevant to the field of AI safety
as we will try to convince you next week
it is very timely actually since its
initial motivation was not from AI but
from communica trial and now we are in a
situation or people are facing dilemmas
on what to try and volunteers are scarce
so now I like yesterday friend French
agencies reported that they are
struggling to recruit volunteers
you don't want to harm these volunteers
obviously and you want to have the
clinical trial as safe as possible and
as meaningful as possible to get results
so it is both family in the context of
the spread of kovat the 19 but it's also
very relevant in the context of AI
safety when it comes to large-scale
deployments of algorithms on users and
on the same time here you want to deploy
an algorithm and you want to have a
sequential deployment of policies so
tragically checking that the suite
checking the prime of fluids of him with
a tailoring needs something like like
and its solutions and how to detect that
this question how to design clinical
trials so you want to do it fast you are
in a tight deadline and you don't want
to harm the the volunteers who are there
for the clinical trial and in this
situation is the same as what's what you
social media is facing they want to try
things to see what what is the most
attractive to users was in the same time
you don't want to all right join us next
week for our next discussion will be
very exciting viewed by


A.I. alarms - Sam Harris and Eliezer Yudkowsky

let's just talk about the near-term
future here or what you think is likely
to happen obviously we'll be getting
better and better at building narrow AI
you know go is now along with chess
seated to the machines although I guess
probably cyborgs
you know human human computer teams may
still be better for the next 15 days or
so against the best machines but
eventually I would expect that humans of
any ability will just be adding noise to
the system and it'll be true to say that
the machines are better at chess than
any human computer team and this will be
true of many other things driving cars
flying planes proving math theorems what
do you imagine happening when we get on
the cusp of building something general
how do we begin to take safety concerns
seriously enough so that they're not
just committing some slow suicide and
we're actually having a conversation
about the implications of what we're
doing that is tracking some semblance of
these safety concerns I have much
clearer ideas about how to go around
tackling the technical problem than
tackling the social problem if I look at
the things that way that things are
playing out now it seems to me like the
default prediction is people just ignore
stuff until it is way way way too late
to start thinking about things the way I
think I phrased it is there's no fire
alarm for artificial general
intelligence did you happen to see that
particular essay any chance no now the
way it starts is by saying what is the
purpose of a fire alarm you might think
that the purpose of a fire alarm is to
tell you that there's a fire so you can
react to this new information by getting
out of the building actually as we know
from experiments on pluralistic
ignorance and bystander apathy if you
put three people in a room and smoke
starts to come out from under the door
the people
like it only happens that anyone reacts
around like a third of the time people
sort of like lamps around see if the
other person is reacting and they see
but they like try to look calm
themselves they don't look like started
us there isn't really an emergency they
see other people trying to look calm
they conclude that there's no emergency
and they keep on working in the room
even as this starts to fill up with
smoke this is a pretty well replicated
experiment I don't want to like put
absolute faith because there is a
replication crisis well there's a lot of
variations of this that found pretty
much basically the same result anyway I
would say that the real function of the
fire alarm is the social function of
telling you that everyone else knows
there is a fire and you can now exit the
building in an orderly fashion
without looking panicky or like losing
face socially right and it overcomes
embarrassment it's in this sense that I
mean that there's no fire alarm for
artificial general intelligence there's
all sorts of things that could be signs
alpha zero could be a sign maybe alpha
zero is the sort of thing that happens
five years before the end of the world
in across most planets and in the
universe we don't know maybe it happens
50 years before the end of the world you
don't know that either so no matter what
happens it's never going to look like
the socially agreed fire alarm that no
one can deny that no one can excuse that
no one can look to and say why are you
acting so panicky there's never going to
be common knowledge that other people
will think that you're still sane and
smart and so on if you react to an AI
emergency and we're even seeing articles
now that seem to tell us pretty
explicitly what sort of implicit
criterion some of the current senior
respected people in AI are setting for
when they think it's time to start
worrying about artificial general
intelligence and alignment and what
these sit and what these always say is I
don't know how to build an artificial
and general intelligence I have no idea
how to build an artificial general
intelligence and this feels to them like
that it must be impossible and very far
off but if you look at the lessons of
history like most people had no idea
whatsoever how to build a nuclear bomb
even most scientists in the field had no
idea how to build a nuclear bomb until
they woke up to the headlines about
Hiroshima
but the right flyer news spread less
quickly in the time of the Wright Flyer
two years after the Wright Flyer you can
still find people saying that
heavier-than-air flight is impossible
and there's and there's cases on record
of one of the Wright brothers I forget
which one saying that flight seems to
them to be fifty years off two years
before they did it themselves Fermi said
that a critical sustained critical chain
reaction was fifty years off that could
be done at all two years before he
personally oversaw the building of the
first pile and if this is what it feels
like to the people who are closest to
the thing not not that not the people
who like find out about the news a
couple of days later the people who have
the best idea of how to do it towards B
closest to crossing the line then the
feeling of something being far away
because you don't know how to do it yet
it's just not very informative I mean it
could be fifty years away it could be
two years away that's what history tells
us but even if we knew it was fifty
years away we granted it's hard for
people to have an emotional connection
to even the end of the world in 50 years
but even if we knew that the chance of
this happening before 50 years was zero
that is only really consoling on the
assumption that 50 years is enough time
to figure out how to do this safely and
to create the social and economic
conditions that could absorb this change
in human civilization I mean the way
professor Stuart Russell who's the
co-author of probably the leading
undergraduate AI textbook of the way for
Stuart Russell put it the same guy who
said you can't bring the coffee if
you're dead is imagine that you knew for
a fact that the aliens are coming in 30
years would you say like well that's 30
years away like let's not do anything no
it's a big deal if you know that there
are aliens that there's a spaceship on
its way toward Earth and it's like going
to get here in about thirty years at the
current rate but we don't even know that
there's this lovely tweet by a fellow
named McAfee who's one of the major
economists who've been talking about
labor issues of AI I could perhaps look
up the exact phrasing
it was roughly he said guys stop
worrying we have no idea whether or not
AI is imminent
and I was like that's not really a
reason to not worry now is it it's not
even close to a reason that's the thing
that's just this assumption here that
people aren't seeing that it's just a
straight-up non sequitur referencing the
time frame here only makes sense if you
have some belief about how much time you
need to solve these problems
you know ten years is not enough if it
takes 12 years to do this safely yeah I
mean the way I would put it is that if
the aliens are on the way in 30 years
and you're like as you should worry
about that later I would be like when
what's your business plan when exactly
are you supposed to start reacting to
aliens like what triggers that what are
you supposed to be doing after that
happens how long does it take what if it
takes slightly longer than that and if
you don't have a business plan for this
sort of thing then you're obviously just
using it as an excuse if we're supposed
to wait until later to start on AI
alignment when are you actually going to
start then because I'm not sure I
believe you what do you do at that point
how long does it take how confident are
you that it works and why do you believe
that what is that what are the early
signs if your plan isn't working what's
the business plan


Ensuring safety and consistency in the age of machine learning | Chongli Qin | EAGxVirtual 2020

hello
and welcome to this session on ensuring
safety and consistency
in the age of machine learning with
chong li tin
following a 10 minute talk by chong li
we'll move on to a live q and a session
where she will respond to your questions
you can submit questions in your name or
anonymously using the box in the right
hand side of this video
you can also vote for your favorite
questions to push them higher up in the
queue
we'll try to get through as many as we
can then
after 20 minutes of questions i'll bring
the q a to an end
but that's not the end of the session to
help you think through
and apply the ideas you've heard i'll be
asking you to join a 20-minute
icebreaker session
where you will have two speed meetings
with other attendees to discuss your
thoughts
on the content i'll explain how to do
that when we get there
but now i would like to introduce our
speaker for this session
chong li tin is a research scientist at
deep mind
her primary interest is in building
safer more reliable
and more trustworthy machine learning
algorithms over the past few years
she has contributed to developing
algorithms to make neural networks more
robust to noise
key parts of a research focus on
functional analysis
properties of neural networks that can
naturally enhance robustness
prior to deep mind choli studied at
cambridge
her phd is in bioinformatics
here's chong li
hi my name is tony chen i'm a research
scientist at deepmind where my primary
focus
is looking at robust and verified
machine learning algorithms
so today my talk is on ensuring safety
and consistency in the age of machine
learning
so with all of the great research which
has happened over the past several
decades
machine learning algorithms are becoming
increasingly more powerful
there has been several breakthroughs
many breakthroughs sorry
in this field and today i just want to
mention a few
so one of the earlier breakthroughs was
using convolutional neural networks
to boost accuracy of image classifiers
more recently we've seen generative
models that is now capable of generating
images
with high fidelity and realism
we've also been making breakthroughs in
biology where now machine learning
can fold proteins to unprecedented level
of accuracy
we can also use machine learning and
reinforcement learning algorithms
to beat humans in games such as go
more recently we've seen machine
learning pushing the boundaries of
language
um so the recent gpt2 and three models
have demonstrated
they're not only capable of generating
text that is grammatically correct
um but really grounded in the real world
so as the saying goes with great power
comes great responsibility
as our machine learning algorithms
become increasingly more powerful
it is now more important than ever for
us to understand
what might be the negative impacts and
risks of all of this
and more importantly what can we do to
mitigate for these risks
so to highlight why we need to start
thinking about these risks
i want to give a few motivating examples
starting with this one here
so this is a paper published in 2013
titled intriguing properties of neural
networks
what they have discovered is that you
can take a state-of-the-art image
classifier
and you can put an image through it so
in this case it's an image of a panda
and indeed we can correctly classify
this as a panda
so what happens if you take the exact
same image
add on a carefully chosen perturbation
that is so
small that the newly pertabbed image
looks almost exactly the same as the
original image
well we would kind expect the neural
network to behave in a very similar way
but in fact when you put this newly
perturbed image through the neural
network
it is now almost 100 confident that this
is a given
so in this instance maybe misclassifying
panda for a given
might not have too many consequences
however we can actually choose the
perturbation to make the neural network
output
whatever it is that we want for example
we can output
a bird or a vehicle
if such a classifier was used for
systems like autonomous driving
this can have catastrophic consequences
you can actually also discover something
called adversary universal adversarial
perturbations
which are perturbations that is um image
agnostic so here is an example of such a
perturbation
this is one single perturbation that you
can add to all of these images
and then more often or than not flips
the output of your neural network
some of the failure modes of machine
learning can be slightly more subtle
so here in this paper titled the woman
worked as a babysitter
on biases language generation what they
did was a systematic
systematic study on um how the gpt2
language model behaved
conditioned on different demographic
groups
so for example what happens if you
change the prompt from the manwac das
to the woman worked as well the
subsequent generated text
changes quite drastically in flavor and
is often
heavily prejudiced similarly for the
black man worked as
to the white man worked as
and i want everyone to take a few
seconds to read the generated text
as we change the subject of the prompt
as you can see although this model is
very powerful
it definitely carries some of the biases
that we have in society today
and if this is the model that is used
for something like auto completion of
text
this can further feed and exacerbate the
biases that we may already have
so with all of these risks we need to
think about
what can we do to enable
our machine learning algorithms to be
safe reliable and trustworthy
well maybe one step towards the right
direction is ensuring our machine
learning algorithms
satisfy desirable specifications
that is to say we have a certain level
of quality control over these algorithms
for example for image classifier we want
it to be robust to adversarial
perturbations
for a dynamical systems predictor we
would like it to satisfy the laws of
physics
we would like how classified to be
robust to changes
that is irrelevant for prediction for
example
the color of your digit should not
affect your digit classification
if we're training on sensitive data we
wanted to
maintain a level of differential privacy
these are just a few of many examples of
desirable specifications
that we need our classifier to satisfy
so here i want to introduce
specification driven machine learning
so what do i mean by specification
driven machine learning
well the core issue lies in the fact
when we're training with limited data
our models can learn a lot of spurious
correlations to boost the metrics
so what this means is that your model is
ultimately hinged on the data and metric
and unless we design our training
carefully
your model can inherit a lot of the
undesirable properties in your data
unless you specify otherwise for example
in this case
if your data is biased and limited then
your models will also be non-robust and
biased
so in specification driven ml we want to
enforce the specifications that may
or may not be present in your data but
essential for us
for your systems to be reliable
so i want to give some examples of how
we can train neural networks
to satisfy specifications i'm starting
with
this example here that is for your image
classifiers to be robust adversarial
perturbations
one of the most commonly used
methodologies to train on your networks
to be robust to these perturbations
is something called adversarial training
so i'm going to go into this in a little
bit more detail
this is actually very similar to
standard image classification training
while we optimize the ways of our neural
network to correctly
label an image for example in this case
the image is of a cat
so we want the output of the neural
network to predict a cat
as well what adversarial training is
is simply this with an extra data
augmentation step
while we say yes we want the original
image to be rightfully predicted as a
cat
but under any additive imperceptible
perturbations
we want all of these images to be
correctly classified as a cat
as well but we note actually it is
computationally infeasible to iterate
through all of these changes
um what adversarial training cleverly
does is i tries to find the worst case
perturbation
which is the perturbation that actually
minimizes the probability of the cat
once you have found this worst case
perturbation now you simply feed this
back
into the training loop like this and you
reiterate
and this methodology has shown to be at
least
empirically robust to these adversarial
perturbations
but we want our neural networks to not
only be robust to these small
perturbations
we want them to be robust to semantic
changes or changes
to our images that should not actually
affect our prediction
for example the skin tone of a person
should not affect the classifier
that distinguishes between smiling or
non-smiling
and actually in order for us to train
our neural networks to be robust to
these semantic changes
require a very simple change to
adversarial training
so here rather than considering the
worst case perturbation
we can simply consider the worst case
semantic
perturbation and
through the developments of generative
modeling we can use
generator models to generate these
semantic perturbations
and using this methodology we can now
actually reduce
the gapping accuracy between two groups
based on skin tone
from 33 down to just 3.8
percent mitigating the bias that was
originally present in the data
so of course the things i have touched
on today
um definitely enhances specification
satisfaction to some extent
but there are still problems still a lot
of problems to be solved
for example there is just two
specifications i have mentioned today
there are many more specifications that
we would like our neural networks to
satisfy
and the more complex these
specifications become
the harder the problems become and even
on this
standard image classification example of
training our neural networks to be
robust
to these perturbations we still have yet
to find a single classifier
that is robust to these perturbations
completely
but if we do get this right with this we
can bring many more opportunities
we can enable safe reliable autonomous
driving systems
more robust ways of forecasting weather
we can help the speech impaired with
more robust audio synthesis
the possibilities are endless
so with this i hope i have motivated you
into thinking about these problems
and i'll conclude my talk thank you for
listening
thank you for the talk cheonbi i see
we've had
a number of questions submitted already
so let's kick off with the first one
so what what are the biggest factors
holding back the impact of
machine learning for the life sciences
do you think
so i think it definitely comes into
quality control so i think machine
learning has pushed
the boundaries in terms of a lot of the
metrics that we care about
but in terms of the metrics they are not
the only things that we care about
we definitely also care about do they
satisfy the right specifications for
example for image classification
are they robust to be used for
self-driving cars etc etc
if we're using it for say like a medical
application
we need to make sure that actually it
satisfies um
certain uncertainty principles so for
example if you give it
inputs that's out of distribution you
want to make sure that your neural
network reflects this correctly
um so i think yeah this might this is
definitely one of the biggest
factors holding your back great
thank you so much for that i think the
next question we have um
so what gives you confidence that
deepmind's machine learning technology
will be used by third parties according
to the safety and consistency
principles that you advocate for so
i'm very confident about this because
i'm part of a team
in deep mind where this is our sole folk
for focus
so basically our purpose is to ensure
all of the algorithms that deepmind
deploy or will deploy goes through
certain specification checks and this is
very important to us
so i'm very confident on this because
this is something i'm directly working
on
um even when it comes to the third party
use right
uh what do you mean by the third-party
use
so when they deploy um deepmind's
machine learning technology
that that is in accordance with the
principles that you set out
uh yes well it depends on the apple so
obviously this is still like humanly
designed
so um depends on the applications that
we are considering we first
need to think about what specifications
do we want it to satisfy
and then um it goes through some
rigorous tests to make sure that it
actually does
and yes okay
does that answer your question i think
so
i'm i'm i want to ask this uh this
person who asked this question what they
what they mean in the context of
third-party use but perhaps that can be
taken up over slack
uh so yeah for now we'll move on to next
question so
i think alexander asks how tractable do
you perceive
the technological aspects of machine
learning alignment
to be compared to the social aspects
in by this question do we mean like the
value alignment
the the ethics the ethical risks and
things like this
so um my talk was specifically based on
like the technological side of things
but i think like for the ethical side of
things for example if we're using
how are we making sure that machine
learning is being used for good for
example we don't want it to be used for
weapons
um things like this that has a
less technological aspect to it we
should think about this in terms of
deployment
in terms of how we design our algorithms
um but yes this is definitely something
that people should be thinking about
thank you jeromy next question
what can ai or machine learning
researchers learn
from the humanities for reducing the
discrimination
embedded in current models oh that's an
interesting question
what can we learn from the i think one
thing
um that i often think about
is for example echo chamber effects
so for example if you're going on
facebook
and facebook detects that you like
certain
kind of fees and it will feed you more
these sort of fees and
it has this echo chamber effect where
the things that you like will be
feeded to you whereas we actually want
to make sure that we have a diverse set
of opinions
that doesn't focus on a particular bias
so
for example something like this makes me
think about how can we design
our ranking algorithms to make sure that
this sort of effects
doesn't happen um yeah so
that's definitely something that's high
i think about
great so it's because you're like it's
kind of like applying
a sociological concept of echo chambers
to assess
you know the performance or so in that
in that sense i think definitely we
would be thinking about
like we would be learning from humanity
um how do humans react in terms of this
and
uh is it good is it bad and how can we
design our algorithms to make sure that
we
enhance the good and alleviate the bad
we have to take that from
social sciences great yeah i believe
that answers to questions very well
i mean i also think in terms of value
alignment i think that's quite important
for example if we're designing agents or
if we're designing
reinforcement learning algorithms we
want to make sure that it
satisfies certain values that humans
stand for or principles that humans
stand for
this is yes so yeah i would say social
science is very important
absolutely i think related to that point
um do you think regulation is needed to
ensure that
very powerful current machine learning
is aligned with
the beneficial societal values yes
regular regulation is very very
important
the reason why i say this is um i think
everyone has seen this in the talk
you can have a state of the arts
classifier
that beats um a lot of the metrics that
we
care about and still behaves in very
very unexpected ways
so given this sort of behavior we need
regulations we need to make sure that
certain specifications are satisfied
so yes i think this is highly important
thank you for that
so for the next question what uses
is machine learning already being
implemented for where these biases could
be
creating a huge undervalued issue
so i'm not so sure i have any like
all i can say is that um machine
learning is becoming more more prevalent
in society
and it's being used in more and more
applications there's not a single
application where
maybe machine learning hasn't had too
like at least a little impact on so i
think
um i think what the example i wish i
mentioned in the talk is something that
i think is quite important for example
in terms of language modeling
because language is quite a subtle
propagator of bias and we want to make
sure that
our languages that
through machine learned models doesn't
carry these sort of biases
so i think i can i can mention some
applications which i think
is important for us to be extremely
bias-free
and this is definitely something i think
is quite important
would you like to share any examples on
that so
oh i thought i already shared one which
is the language modeling right another
would be
for example in medical domains so i
think in medical domains if you're
suppose you're collecting data and it's
heavily from
maybe one population rather than other
we need to make sure that
our machine learning algorithms doesn't
reflect this kind of bias
because we want healthcare to be equal
for all um so that's another application
which i think this is quite important
great thank you for that
to the next question um what do you
think other key changes
needed to ensure aligned ai
if you consider that current engagement
optimization
might already be very bad for society
i want to be very specific about this so
i don't
think
i don't think metric design is bad for
society
i think the key here is knowing
that our metrics are will always be
flawed
and that's the key like we can always be
thinking about designing better metrics
so it's not a huge
breakthrough it's an evolution so you
train
you train your classifier for image for
images
and then you realize um oh this metric
doesn't capture the robustness property
so you add this back into the metric
and then you retrain and then you
suddenly find oh maybe it doesn't
satisfy
distribution shift and then we retrain
this is a progression
but the thing we need to realize is that
metrics doesn't capture
everything we wanted to have and we just
need to
keep that in the back of my our mind and
always make sure that we're rigorously
testing our systems and i think
that's a very very um paradigm shifting
idea where we need to think about
our metrics might be flawed and
discussing state of the art is not what
we
we're here to do we're here to deliver
safe algorithms
and i think that's that's the key idea
great thanks for reinforcing that
next question how much more important is
it to focus on
long-term ai risk versus near-term
or medium term risk in your opinion
um sorry my
my computer just uh
okay so now it's back home um
that's a really interesting question
because from my perspective i think
both have different advantages
uh what i mean by this is for the short
term i i know exactly what the problem
is the problem is concrete
and from having this problem being
concrete i can also tackle this more
concretely i can think about
the problem in terms of how what the
formulas look like
what what i can do to make sure that we
train our neural networks that
satisfy certain specifications but then
in terms of long term
it goes back to what you mentioned
before which is value alignments
or ethical risks or maybe there are some
things that we haven't even discovered
yet which can affect our
algorithms in a completely unexpected
way
um this goes into like a more
philosophical view of how we should be
thinking about this
and i think we can definitely take our
values from both
but because from my perspective i'm more
technologically driven in terms of
design i think more about the near term
so i can only answer in terms of i think
i see in the near term if we want
autonomous driving systems to happen
that we need to make sure that our
classifiers are robust that's a very
easy question to answer
but then at a much much longer time i
can't
my my imagination fails me and sometimes
i fail to think of like
what might happen to happen here what
might happen there and yeah but i think
that is not to say that is not important
that is also equally important but i
might have
less professional opinions on this
right that's very fair thank you yeah
on the next question out of all the
approaches to ai safety
by different groups and organizations
which is closest to deep minds in your
opinion
besides deep mind's own approach of
course
i'm not so sure that we even have a pro
and one approach
we're just a bunch of researchers i'm
sure in a lot of other organizations
uh they're also a bunch of researchers
looking at similar problems and thinking
about similar questions
um i mean i guess like the way
i can only talk about how i think my
group tackles this
so we're very mission driven so um
we really want to ensure that our
algorithms that we deliver
is safe reliable and trustworthy so from
our
from our perspective that is how we
think about
our research but i cannot make any
comments on
any other organizations because i don't
actually know how they
how how they work um
yeah great does that answer the question
okay so this next one is on phi
from five sorry so do you think there is
a possibility that
concerns and research on ai safety and
ethics
will eventually expand to the direct or
indirect impacts on animals
direct or indirect
impacts on animals
i mean i guess i'm not i don't know that
much about animal conservation
but i can imagine um
because machine learning algorithms are
so pervasive it's definitely going to
have an impact
and um and touching on everything i've
said before
um if you want to alleviate certain
biases in that
sort of area we need to also think about
designing our metrics carefully
so i don't know much about that area so
i don't know what it is that they
normally train for and how they design
their training
but all i can say is that we need to
think about it carefully you want to
think about
what it is that you want to avoid when
it comes to
animal conservation machine learning
algorithms
right that's yeah
thanks for that are there ways to
deliberately counter adversarial
training and other types of
perturbation mitigations
deliberately counter adversarial
training
yeah so what does that mean because
adversarial training is
a process um what what do what do they
mean by countering it
hmm how hard for me to pick this one
because i'm just reading off the slide
slider but
um
could you read the question again yeah
are there ways to
deliberately counter adversarial
training
and other types of perturbation uh
mitigation let me i'm just gonna answer
what i think this question
is asking um which is uh suppose if i
train your network with adversarial
training um
can i still attack this system knowing
that this is trained
adversarially maybe um and
i think one of the things that i didn't
touch on quite
for this presentation because i i wasn't
sure how much detail i want to go into
this
is that um specification during machine
learning
evaluation is extremely important so we
need to make sure that
um suppose we we do adversarial training
while we're evaluating we're going to be
evaluating more than just a
like a simple um adversary in the mix
we're going to be looking at all sorts
of different other properties about the
neural networks to ensure
that whatever it is we deliver maybe a
stronger attacker will come through and
it will still
we will still be robust to this so i
think the answer to that question is
that
we need to design our systems to be
extremely rigorously tested
um more so than our training procedure
does that i hope that answers
whoever asked this question i think that
touches some
aspects of it please what other aspects
what other aspects do you think
i think yeah we should we should you
know clarify the context of this
question on
or this scenario that's being imagined
perhaps inside
yeah yeah yeah yeah let's do that yep
let me find the next question
here's from puja what do you use
for interpretability and fairness
um there's not a sing one single
algorithm that we use
these are still like being developed
so like i said i'm not a ethical
scientist
i think in terms of fairness there is a
lot of different
metrics regarding fairness fairness is
not something that's
easily actually very easily defined
so when it comes to training algorithms
to be fair we assume that we have a
fairness definition
from someone who who knows more about
this stuff
and then we try to satisfy this
specification but i think designing
metrics that
are more aligned with fairness that's
where the difficult challenges come in
um what was the other one dimension
there was fairness and there was
something else
interpretability interpretability
so i could you repeat the question again
what what do you use for
interpretability and fairness
so again i think interpretability is not
there's not a single algorithm that we
use for interpretability
and actually depends on the applications
that we probably would eventually use it
for
how we can design our neural networks to
be interpretable
um is completely dependent on that um
yes so yeah i i hope that answers the
question
yeah um i think so it's rather broad i
guess
yeah for this next one um so assuming
only the current state of ai capability
what is the most malicious outcome
a motivated individual group or
organization could achieve
so i think what is the most malicious
again i'm not so sure there is
a single one so something which i i
think is quite important right now
is differentially privates
neural networks because suppose if we're
training on quite sensitive data
and we want to make sure that the
people's data is protected and is still
anonymized
and we don't want any um uh malicious
attackers who's interested in knowing
more about these people to come in and
just
look at these neural networks and say oh
we know more about someone someone's
and so and so so um i would say
that's possibly a very important um area
that
people should be looking at and from in
my
at least in my opinion is it would be
it's a
very malicious um yeah so
i think that's just off the top of my
head but there's obviously a lot
a lot more um yeah
great
so again a brother brought question but
do you have a field in mind
where you would like to see machine
learning being applied more
do i have a feeling actually
nigel we have talked about this earlier
which is
in terms of resource allocations for
charities i feel like
that's a lot of data and i feel like
machine learning definitely can
contribute a lot more in that area um
yeah so that's something i would really
like to see machine learning make a
larger contribution in um or making sure
that the charities data is formatted in
a way that's easily trainable that's
allows more interesting research
questions to be to be asked and answered
um sorry my computer keeps on blacking
out okay that's fine
um yeah so definitely i think analyzing
data
from charities to make sure that we're
allocating the right resources
more effectively because if people are
donating money they want to make sure
that their money is donated effectively
and i think this
is something definitely machine learning
can make a lot more impact in
great thank you so
the next question is it fair to say that
biases targeted
in neural networks are ones that humans
are aware of
is there possibility of machine aware
bias recognition
it would be hard to make machines aware
unless you drive it into the metric
um
could you just repeat that question
again yeah
so is it fair to say that biases
targeted in
neural networks are ones that humans are
aware of
is there possibility of machine learning
i'm not so sure that's
so i'm not so sure that's fair because
mike is from my opinion most um machine
learning researchers just handle
a data set and um they don't know the
properties about this data set and they
just want to see oh
we want the accuracy to be higher or we
want this to be higher
but the biases that's present in the
data set
maybe from some kind of data collection
scheme will be transferred in the model
until
you specify otherwise but would you say
the human
is aware of this bias i i don't think so
um
right that is not to say that maybe some
humans
are not like maybe they are but i would
say if in the majority of cases
um yeah it's we just need to be a
from that question i think the real
answer should be looking at
um if we're looking at the data we
should first
inspect what might be the undesirable
things that's present
in this data and how can we alleviate
this
great
the next question um how much model
testing should be considered sufficient
before deployment
given possible unexpected behavior even
well studied models
and unknown unknowns
i feel like there are several testing
stages so the first testing stages
is the necessary condition conditions
that we already know
um for example the image classifiers for
autonomous driving systems we know
it needs to be robust and then the
second stage comes in
maybe we can put these out in a small
scale deployment
and then they will report us back oh we
discovered all of these kinds of
problems and then
from these set of problems we can design
a new set of specifications and this
is an iterative process rather than like
just
one set of specifications go sort of
process this definitely requires
heavy testing um both at a conception
stage
and also small deployment stage so in
terms of deployment design
um definitely i think that's very
important
do you think there's a way to well i
think just building on that question is
there a way to
decide that testing is sufficient before
deployment
what would you say are the key
indicators of that so
i think one of the things i just
mentioned was we would never know before
deployment
so what we can do is we can deploy on a
smaller scale
yeah and to ensure that the risks
are being minimized and if things work
out or there's a certain specifications
that we realize still need to be
satisfied then we go through
second stage and then maybe we can go
for slightly larger scale deployment
and and so on so basically the key to
that question is
we can never truly know before
deployment which is why we need a small
scale deployment to make sure that we
understand what are the problems that
can exist
but before deployment we can only know
the things that we we can
envision um which is our robustness to
perturbations we want it to be
differentially private etc etc
great thank you for that
next question how can we discuss
um ai or machine learning concerns with
people
desperate for quick solutions for
example farmers using machine learning
and
agriculture because of their anxiety
about climate change
pharmacies because of that and
i think it depends on what you might
mean by quick um
even when it comes to climate change i
think
we're probably looking at solutions that
will take maybe
one year or two years to fully
understand and deploy
i believe if we do anything in too much
of a hurry
will might go go wrong
or it might not have the intended
effects that you will have
so even if this farmer is anxious
i think it is still really important to
make sure that these systems are
rigorously tested
so in my opinion we should not
like time is off the essence but we
should not rush it
um great
i think we have time for just a few more
questions um
well this one maybe second to the last
how do you distinguish between semantic
differences
and content differences in photos is
this possible to
do automatically for a large data set
uh content differences and semantic
differences
i think given a relatively good
generative model it depends on what you
mean by um can you distinguish because
i'm sure like to some
level to some extent um if we're using a
good generative model it can distinguish
some of the times maybe not other times
but to fully distinguish like using a
generative model i would say
maybe we're not quite there yet um
i think that answers semantic but in
terms of content do they mean
can they identify this image to be a
banana or something like this
yeah perhaps you know kind of um
could you repeat the question again
sorry yeah how do you distinguish
between semantic differences
and content differences so in terms of
photos in the context of
oh i see i think i see what they mean so
here when i say semantic differences i
really mean
uh features which should not affect your
prediction
so this is um exactly as
maybe this question is alluding to this
is actually a quite a nuanced point it's
very
it's very difficult to say um what
should or should not affect our
prediction um
but we can actually start from toy
examples that we know of
for example when it comes to if we want
to have
digit that's a data set in machine
learning called amnest digits
and um or colored amnest which is
basically an image of a digits
and you can have like colors inside this
and
for this simple task we know for sure
that the color of your mnist of your
digit should not affect its prediction
so we say this is a semantic
perturbation if you change the
color but of course if you move to more
complex data sets this becomes
more more difficult to to settle down
to actually understand so we can use
actually generative models
to um approximate but it should never we
would never
know for sure but this needs more areas
of research of course
thank you for that so maybe we have time
to produce one last question
to round out the discussion outside of
deepmind
where do you think the most promising
research in ai safety
is being done
oh that's a that's a difficult question
because i feel like that has
there's a lot of um great research
that's happening out there and i
i would not know all of them like just
in
my uh limited view i can see that
there are some very very good research
happening at uh
how to google open ai at
stanford at berkeley there's a lot of
i mean there's not a single one
everyone's contributing to the same
cause and i think
i i also don't think it's fair to
measure research and because i think
we're all trying to achieve the same
things
um so yeah i think anyone who's touching
on ai safety
should be commended and they're all
doing good work i think
that's a great note to end the q a
session on thank you
thank you but don't go away just yet um
for the audience um
discussing new ideas with other people
can be a really good way to understand
them
so we're gonna use the last 20 minutes
of this session for a couple of short
speed meetings with other attendees
if you check the session description
below this video you'll find a link
to an icebreaker session where we're
going together for those meetings
so please click on that link now and a
new host will meet you on the other side
thanks for watching thank you
you


How social science research can inform AI governance | Baobao Zhang | EAGxVirtual 2020

hello and welcome to the session on how
social science research can inform AI
governance with Baba Jeong my name is
more in the wing and I'll be an emcee
for the session thanks for tuning in
guys so we're first start with a
ten-minute pre-recorded talk by bubble
which will be followed by a live Q&A
session and you guys are probably
already familiar with this by now but
you can submit your questions on the
chat box on the right right-hand side of
this video and you can also upload a
favorite questions to push them a higher
up the queue and we'll get you will try
to get through as many questions as we
can and then after 30 minutes of
questions
I'll bring the Q&A to an end but please
make an effort to stay for the mini
isolated session afterwards I think that
this would be a really valuable
opportunity for you to solidify the
ideas you've learned and also to discuss
new ideas that you may have generated
during the session with other members of
this community you'll have two
one-on-one sessions for total 20 minutes
and I'll explain what to do when we get
there but now I'd like to introduce you
to the speaker to the for this session
Bob all
at the Brooklyn Client Center for
Internet and Society at Harvard
University and a research affiliate with
the center for the governance of AI at
the University of Oxford her current
research is focused on the governance of
artificial intelligence in particular
she studies public and elite opinion
toward AI and how the American welfare
state could adapt to the increasing
automation of labor without further ado
here's bubble
hello welcome to my virtual presentation
I hope you are safe and well during this
difficult time
my name is baubau Chun and I'm a
political scientist focusing on
technology policy I'm a research
affiliate with the center for the
governance of AI at the future of
humanity Institute and Oxford I'm also a
fellow with the Berkman climb Center for
Internet and Society at Harvard today I
will talk about how social science
research can inform AI governance
advances in AI research particularly in
machine learning have grown rapidly in
recent years machines can outperform the
best human players and strategy games
like go and poker they can even generate
synthetic videos and use articles that
easily for humans
looking ahead ml researchers believe
that there's a 50% chance of AI
outperforming humans in all tasks by
2061
this is based on a survey that my team
and I conducted in 2016 the EI community
has recognized the potential risks even
existential risks that unaligned AI
systems pose to humans
tech companies governments and civil
society have started to take notice as
well many organizations have published
AI epics principles to guide the
development and deployment of the
technology a recent report by the
Berkman crime Center accounted some 36
prominent sets of AI principles
now we're entering a phase where tech
companies and governments are starting
to translate these principles into
policy and practice at the Center for
the governance of AI org of AI we think
that social science research whether
that's in political science
international relations law economics
and psychology can inform
decision-making around AI governance for
more information about the center's
research agenda please see this report
it's also a good starting place if
you're curious about the topic and are
new to it now here's a roadmap for my
talk first I will present my research on
public opinion toward AI next I will
highlight some EA align social science
research on AI governance I'll then
present some research questions that
I've been thinking about a lot lately
finally I'll conclude by discussing how
one can be impactful as a social
scientist in this space why should we
study public opinion toward AI from a
normative perspective given that AI
could impact much of society we need to
consider the voices of those who will be
impacted by it
at the same time public opinion has
shaped policy in many other domains
including climate change in immigration
studying public opinion could therefore
help us anticipate how electoral
politics will impact AI governance the
research I'm about to present comes from
this report it's based on a nationally
representative survey of 2000 Americans
that Alan Day foe and I conducted in the
summer of 2018
here are the main takeaways from the
survey first we find that an
overwhelming majority of Americans think
that AI should be carefully managed
while they considered all thirteen
governance challenges that we presented
to them to be important
they have only low to moderate levels of
trust in the actors who are developing
and managing AI
now on to some results here's a graph of
Americans view of AI governance
challenges each respondent was randomly
assigned to consider five
randomly selected from thirteen the
x-axis shows the respondents perceived
likelihood that the governance challenge
would impact large numbers of people
around the world the y-axis shows the
perceived issue importance the ones that
were perceived to be high in both
dimensions include protecting data
privacy preventing AI enhanced
cyberattacks preventing mass
surveillance and preventing digital
manipulation all of which are highly
salient topics in the news I like to
point out that the respondents consider
all of these AI governance challenges to
be important for tech companies and
governments to manage but we do see some
variations between respondents when we
break them down by subgroups here we've
broken it down by age gender race level
of education partisanship etc and we're
looking at the perceived issue
importance in these graphs purple means
greater perceived issue importance green
means lesser perceived issue importance
here I'll highlight some differences
that really pop out in this slide you
see that older Americans compared with
younger Americans perceive the
governance challenges present it to them
to be more important interestingly those
with CSR engineering degrees compared
with those who don't perceive all the
governance challenges to be less
important we also observe this techno
optimism among those with CS or
engineering degrees and other questions
in the survey
despite Americans perceiving that these
AI governance challenges are important
they have low to moderate levels of
trust in the actors who can do something
about it
who are in a position to shape the
development and deployment of AI systems
a few interesting observations in these
slides first the American public places
relatively greater trust in the US
military this is in contrast to the ml
community who would rather not work with
the US military I think we get this
seemingly strange result because I think
the public is relying on heuristics when
answering this question while trust in
institutions has declined across the
board the US public seems to still have
relatively high levels of trust in the
military another interesting observation
is that the American public seems to
have great distrust of Facebook part of
it could be the fallout from the
Cambridge analytic
scandal but when we ran a previous
survey before the scandal broke we
observed similarly low levels of trust
in Facebook I'm sharing with you just
some of the results for our report I
encourage you to read it we're currently
working on a new iteration of the survey
and we're hoping to launch it currently
in the US EU and China to get some
interesting cross-country comparisons
now I would like to highlight two works
by my colleagues in the EA community who
are also working in AI governance first
off we have toward trustworthy AI
development that came out recently
it's a massive collaboration between
different sectors and fields to figure
out how we can verify claims made by AI
developers who say that their algorithms
are safe robust and fair this is not
merely a technical question a lot of the
suggestions in this report include
creating new institutional mechanisms
like bounties for detecting bias or
safety issues in AI systems we're
creating an AI instance incidents report
second we have the windfall clause for
my colleagues from dove AI here the team
considers this idea of a windfall clause
as a way to redistribute the benefits
from transformative AI this is an
agreement and xnt agreement where tech
companies would donate a massive portion
of their profits were they to make a
large amount of profit from their AI
systems and this report combines a lot
of research from economic history and
from legal analysis to come up with a
really inventive policy proposal there
are a lot of new and interesting
research questions that keep me up at
night
and I just want to share some of them
with you let's definitely have a
discussion during the Q&A so one of the
questions that keep me up is how do we
build incentives for developing safe
robust and fair AI systems and how do we
avoid a race to the bottom I think a lot
of us are rather concerned about the
rhetoric of an AI arms race but it's
also true that even a place like the EU
is trying to push for competitiveness
and AI research and development I think
that toward trustworthy AI development
piece gives some good recommendations on
the R&D front but in terms of what are
the market incentives and policy
incentives for businesses and the public
sectors to choose the safer AI products
that's still a question that I'm
interested in and many of my colleagues
are interested in another question that
I think about a lot is how can we
transition to a economic system where AI
can perform many of the tasks currently
done by humans and I've been studying
perceptions towards automation
unfortunately a lot of workers
underestimate the likelihood that their
jobs will be automated they actually
have an optimism bias even correcting
workers beliefs about the future of work
in my studies have failed to make them
more supportive of redistribution and it
doesn't seem to decrease their hostility
towards global
so certainly there's a lot more work to
be done in this space in terms of the
political economy around the future of
work and finally um I think a lot about
other geopolitical risks that make AI
governance more difficult in tobi
towards book he talks about the risk
factors that could increase the
probability of existential risk and one
of these risks that he mentions is great
power war we're not at a great power war
but there's certainly a rise in
aggressive nationalism from some of
these great powers instead of coming
together to combat the COBIT pandemic
many countries are pointing fingers at
each other and I think these trends
don't bode well for international
governance so thinking about these
trends and how they can shape
international cooperation around AI
governance is definitely a topic that my
colleagues and I have been working on
now I conclude this presentation by
talking about how one can be impactful
as a social scientist
I have the great luxury of working in
academia where I have plenty of time to
think and carry out long-term research
projects at the same time I have to
constantly remind myself to engage with
the world outside of academia to engage
with the tech and policy world by
writing op-eds doing consulting and
communicating with the media
fortunately social scientists with
expertise and aí and aí policy are also
in demand in other settings increasingly
tech companies have sought to hire those
who can do research on how individual
humans interact with AI systems or the
impact of AI systems on society
Jeffrey Ervin and Amanda a Scholl have
published this paper called AI safety
needs social scientists and I encourage
you to read it if you're interested in
this topic to give you a more concrete
example so my colleagues have worked
with open AI to test whether their GPT 2
language model can generate news
articles that fool human readers
governments are also looking for social
scientists with expertise in AI
policymakers in both the civilian
government and in the military have this
AI literacy gap they don't really have a
clear understanding of the limits and
the potentials of the technology but
advising policymakers does not
necessarily mean that you have to work
in government many of my colleagues have
joined think tanks in DC where they
apply their research skills to generate
policy reports briefs and expert
testimony I really recommend checking
out the Center for the security and
emerging technology or cset based at
Georgetown University they were founded
about one year ago but they have already
put out a vast collection of research on
AI and US international relations thank
you for listening to my presentation I
look forward to your questions during
the Q&A session
Thank You van I talked my mom so I seen
we've already had a number of questions
submitted so we're gonna get started
with the first one um so in general
strokes what concrete advice I mean you
give to a fresh college graduate with a
degree in a social science discipline
that's a very good question thank you
for coming to my talk in terms of
general advice one of the sort of
unexpected piece of ice that I would
give is that you should have strong
writing skills at the end of the day any
research you do you need to translate it
to different audiences whether that's
for an academic journal or if you're
talking to policymakers or tech
companies policy reports besides that I
think if you want to specialize in a
particular type of research for me
learning data science and statistics is
really important for the type of
research that I'm doing for other folks
it might be game theory or for folks
doing qualitative research it might be
how to do ethnographies how to do elite
interviews but overall I think having
strong writing skills is quite critical
great um Ryan asked what are the current
talent gaps in the AIA safety right now
that's a good question I must confess
I'm not a ice safety expert although I
did talk about this piece that folks at
open a I have written called AI safety
needs social scientists and I definitely
agree with the sentiment given that the
folks who are working on AI safety they
want to run experiments you can think of
them as psychic experiments and a lot of
computer scientists are not necessarily
trained on how to do that so if you have
skills and running surveys or running
psych experiments I think this is a
skill set that hopefully other tech
companies will acknowledge and wreck
is that wow this is really important
what's really interesting um would you
consider uh psychology to be within the
realm of social sciences people will
generally receive it um or more like
STEM fields
I think psychology is quite interesting
it probably depends on people who are
more on the neuroscience side might be
more stem but there's I I work with some
experimental psychologists particularly
social psychologists and they read a lot
of the literature in economics or
political science and communication
studies so I do think that there's a bit
of overlap so suppose like going off
this question up how important do you
consider interdisciplinary studies to be
whether that's like constrained within
the realm of social science or like
social science would stem on etc I think
it's important to work both with other
social scientists and with computer
scientists one of the realizations that
I made this is more sort of career
advice not related to a governance but
recently I've been working on a bunch of
kovat projects and having the expertise
so that you're not just an armchair
public health person is really important
so I try to get my team to talk with
those who are either in vaccine
development or epidemiologists or those
who work in public health and I'd like
to see more of that type of
collaboration in the AI Governance space
and we do that quite well I think at gov
AI where we do have in-house people at
FHI who are computer scientists that we
can talk to mm-hmm so um that's a really
interesting point it relates to one of
the questions that was just asked about
how can we more effectively promote
these international cooperation I mean
like concrete strategies that you may
advise yeah so international
corporations that that's a really
good questions so I do work a bit with
folks who are in the European Union
yeah governance space and bringing my
expertise to the table my the team that
I'm working with we just submitted today
a consultation to the your the EU
Commission so that type of work is
definitely necessary I also think that
collaboration with folks who work on AI
policy in China is also a really
fruitful area of collaboration in the
u.s. I do worry about the decoupling
between US and China so there's a bit of
a there's a bit of tension but if you're
in Europe and you do want to collaborate
with Chinese researchers I encourage it
I think this is a area that more folks
should look into mm-hmm yeah that's
really good point I guess in relation to
the last talk regarding biosecurity I
guess a topic of info hazards was was
very much staked upon so I guess within
the realm of AI governance can you think
of any particular info hazards that's
that's a good question
we do think a lot about all of our
publications at gov AI we talked about
being careful in our writing so that we
don't necessarily escalate tensions
between countries I think that's
definitely think something that we think
about at the same time there is this
sort of open science movement in the
social sciences so that's a tricky
balance that we play but we definitely
don't want to certainly we want to make
sure our work is accurate and
speaks to our overall mission at gov AI
of promoting you know beneficial AI and
not doing harm mm-hmm yeah I guess a
more specific question someone asked was
um do you think this in this concept of
info hazards if it's a big problem in
our governance would prohibit one from
spreading ideas but in the realm for
example that's a good question
I think I think the EI community is
quite careful about not spreading info
hazard we're quite deliberate in our
communication but I do worry about a lot
of the rhetoric that other folks who are
in this a governance pace are saying
right so there are people who want to
drum up this potential AI arms race it's
people who say you know competition is
the only thing that matters and I think
that's sort of the dangerous rhetoric
that we want to avoid because we don't
want a race to the bottom where whether
it's US China or the you they only care
about competition without thinking about
the potential risks of deploying AI
systems that are not safe great so we're
gonna shift gears a little bit into more
of the nitty-gritty of the talk so one
question that was asked is how do you
expect public attitudes to an AI to
defer by nationality yeah that's a
really good question so in the talk I
mentioned that gabay I were hoping to do
this big survey later on where we're
running it currently in different
countries um judging from what I've seen
of the literature I Eurobarometer has a
lot of good surveys in the EU I do think
that sort of
from what you would expect folks living
in Europe tend to be more concerned
about privacy
they have tougher privacy laws and so
that's to be expected but it's not
necessarily so that Chinese respondents
are totally okay with lack of privacy so
we're hoping to do this survey by asking
them same questions
around the same time because I think
that's really important to make these
cross national comparisons a lot of
questions you get different responses
because of question framing or question
wording so doing the rigorous social
science can really give a better answer
to this question yes I guess um I'd like
you to unpack that a little more so
extrapolate extrapolating from the last
question and I guess in what concrete
ways can we be more culturally sensitive
in stratifying these AI risks when it
comes to international collaboration I
think speaking the right language is
really important just when for instance
this is a guess a concrete example so
when I'm like when my team is working on
this u consultation we talked to folks
who work at the European Commission so
that we can understand what are the
particular concerns and speak their
language so to speak and and one of the
things that they're really concerned
about yes they're concerned about the AI
competition and potentially arm race but
they don't want to use that language
they care a lot about human rights and
privacy and so when we're making
recommendations those are the two things
that we try to balance because we read
their reports and we've spoken to people
who are involved in the decision-making
and in terms of the survey research that
we're hoping to do
we're getting a lot of consultation from
folks on the ground so that our
translation work is localized so that
it's that has a cultural nuances mm-hmm
yeah that's really good point about um I
guess using the right language which
also makes me think that you know how
people in the social science realm often
think differently you may use different
language than people from the centrum
example and the clash of those two
cultures this could sometimes result in
similar conflicts as like clash of
cultural conflicts I'm wondering whether
you have any advice on how to mitigate
those conflicts yeah that's a good
question so recently gabay I published a
guide to making impact statements that
are required for urops
the conference and one of the
suggestions I think that came out of
that is if you're working with if
computer scientists are sort of trying
to figure out what the societal impact
of their research is maybe it's good to
talk with social scientists who are who
can help them do this translational work
so I think again the interdisciplinary
research team strategy is quite
important mm-hmm do you think there is I
think you alluded to this you know talk
as well but do you think there was a
general literacy gap between these
different domains and it's so how should
these gaps be filled yeah that's a
that's a really good question so my
colleagues Mike Horowitz and Sarah Kahn
at UPenn have written a piece about the
I literacy gap in government and they
acknowledge that it's a real problem and
it would be certainly training course
with crash courses to train policymakers
or to train social scientists who are
interested in advising policymakers
that's one way to go about it
but I do think that I if you want to
work in the space doing the deep dive
not just doing the crash course can be
really valuable and then you can be the
person who can offer the instruction you
could be the ones writing the guide
material so yeah I do think there's a
need to increase the average level of
the I literacy but also important for
the social science master's programs or
PhD programs to be able to train people
who are more expert in this field mm-hmm
yeah that's a really interesting point
on which brings up a really interesting
question by chase so it has there been
more research into the source of Technol
optimism from CS / engineering grass as
an overconfidence in their own education
or in fellow Debs
/ engineers yeah that's a that's a good
question so we're we're surveying
wishing learning researchers at cafe ice
and this is future research that we're
hoping to share later this year and I
can't speak you know directly to that
research but I do think that there is a
perhaps a u-shape curve where on the
x-axis is your level of expertise and AI
and you're on the y-axis is your level
of concern about AI safety your risks
from the I so those who don't have a lot
of expertise they're kind of concerned
but those who have CS or engineering
degrees are perhaps not so worried but
then if you talk to machine learning
researchers themselves and a lot of them
are concerned I think they're waking up
to recognizing that what they work on
can have huge societal impacts you have
folks who
where can a I safety who are you know
very concerned about this but in general
I think that AI machine learning fields
is also waking up to these recognizing
these potential risks given as I
mentioned my talk this proliferation of
AI ethics principles coming left and
right from all these different
organizations mm-hmm yeah that's really
interesting how the public is for once
aligned with the experts in ml um I
suppose in this instance the public is
pretty enlightened um I'm not sure if
this question was asked ready I don't
believe it was but our organizations
working AI governance more funding
constraint or Talent constrain uh that's
that's a good question I think at this
point I can't speak for all
organizations but I do think that there
is uh can I just make a plug so I do
think that the kovai we're looking to
hire folks in the upcoming months we're
looking for someone to help us on survey
projects and we're also looking for a
project manager and other researchers so
in some sense you know we have the
funding and that's great and now we're
just looking for folks who can do the
research so that's my plug I can't speak
for all organizations but I do think
that there is a sort of a gap in terms
of training people to do this type of
work I'm going to be a faculty member at
a public policy school and they're just
beginning to offer AI governance as a
course so there is a bit of a gap and
for people who are interested certainly
a lot of self-study is helpful but
hopefully we'll be able to get these
courses into the classrooms at law
schools at public policy schools at
different PhD programs or master's
programs
mm-hmm yes so I think um perhaps the
motivation behind that question was more
along the lines of fun I guess yeah
governance is not specific to a per se
but a lot of there's a general a general
current trend that a lot of
organizations in yeah are extremely
competitive you get into especially you
know a specific job posts and I'm
wondering whether you can provide like a
realistic figure for how talent
constrain um hey governance is as
opposed to being funding constrain yeah
I can't speak to the funding side but in
terms of the research human resource
side of things I think we're beginning
to finally have some concrete research
questions but at the same time we're
building out new research questions and
it's hard to predict what type of skill
set that you will need necessarily as I
mentioned in my talk gov AI has realized
oh all these different social science
skills are really important whether
that's doing Survey Research whether
that's doing legal analysis or whether
it's doing you know elite interviews
more sorta on the sociology side or
doing translational work so I don't have
a good answer to that question but I
think getting some sort of expertise in
one of the Social Sciences and having
expertise and AI policy or AI safety
that's the type of us those are the type
of skills that we're looking for mm-hmm
okay so given that we have six minutes
left I'm gonna shift gears a little bit
and just try to get through as many
questions as possible um let's see what
insights from nuclear weapons governance
can inform AI governance
um that's a I think that's a really good
question um and i think you've caught me
here i think some insights that I have
is the dual use nature of nuclear
weapons people talk about AI as a as a
general purpose technology and it could
be you used for both benefits and harms
and sort of speaking from my own
expertise in public opinion research one
of the interesting findings in this
space is people tend to reject nuclear
energy because of its association with
nuclear weapons so there's a wasted
opportunity cost because nuclear energy
is quite cheap and not as harmful to as
say burning fossil fuels but because of
this negative association we sort of
rejected nuclear energy and thinking
about AI what I think about in terms of
trust in the AI systems we don't want it
to have a situation where people's
association of AI systems is so negative
that they reject applications that could
be beneficial to society what that make
sense but I do recommend some my
colleagues who work in international
relations space at organizations like
cset who have written about this I'm
sorry it's not my area of expertise okay
um the next question is is there a way
we can compare public's trust and differ
institutions specifically regarding AI
compared to a general baseline trust in
that institution so suppose in this case
on the US military may have a greater
trust across you know all domains in
general from the public yeah that's
that's a really good question I think in
terms of the public at this point
because a is owned you people are
relying on their heuristics rather than
their what they know so they're just
gonna rely on what they think about
what's a trustworthy institution and one
thing that I've sort of noticed that I
wrote about and my Brookings report is
some of the areas of political
polarization around AI governance sort
of maps on to what you would expect to
find in other domains and that's a
little bit concerning if we can't agree
on what's the right policy solution if
we're just going to map the partisan
rhetoric on to AI governance in the u.s.
context at least to give you a few
examples acceptance of facial
recognition software really maps on to
race and your partisanship so African
Americans really distrust facial
recognition Democrats tend to distrust
facial recognition
whereas Republicans tend to have a
greater level of acceptance so you see
this attitudes towards policing being
mapped on to facial recognition you also
see this in terms of regulation of
algorithmically curated social media
content where there seems to be a
bipartisan tech clash but when you dig
deep into it as we've seen recently
Republicans tend to think about content
moderation as censorship against
whereas Democrats tend to see it as
combating this information so
unfortunately I do think that
partisanship will creep in to the AI
governance space and it that's something
that I'm actively studying mm-hmm
so we all in one minute left I'm going
to go through that last question really
fast so with regards to the specific
data that was presented sorry you
mentioned a public considered all those
governance challenges important did you
consider including a made up governance
challenge to check response values oh
that's a really good suggestion we
haven't but we certainly can do that in
the next round that's a good suggestion
great okay um wonderful and that
concludes a Q&A part of this session
again please stay for the mini
icebreaker session we think this can be
a really good way to understand and
solidify what you've learned and
exchange new ideas if you check the
session description below this video
you'll find a link to the iceberg a
session and another moderator or meae on
the other side thanks for watching the
ice
you