DeepMind x UCL | Deep Learning Lectures | 12/12 |  Responsible Innovation

hi my name is Tony Chen and welcome to
today's lecture on responsible
innovation and artificial intelligence
so today's talk will be split into two
parts for the first part I will be going
over some of the research which has been
done taking sure that's the machine
learning algorithms we develop satisfied
desirable specifications such that
during deployment it is safe reliable
and trustworthy for use in the second
half yasmin will dive into further
details of the ethical implication of
machine learning algorithms and more
importantly how we can be thinking about
designing these algorithms and deploying
these algorithms such that it is
beneficial to society so to start I want
to give a quick motivation into why we
as machine learning researchers should
be thinking about both research and our
responsibilities so with all of the
great research which have happened over
the past several decades machine
learning algorithms are becoming
increasingly powerful there has been a
lot of breakthroughs in this field and
today I'll just mention a few so one of
the earlier breakthroughs was using
convolutional neural networks to boost
the accuracy of image classification and
more recently we now have generative
models that is capable of generating
images with high fidelity and realism
we've also seen breakthroughs in biology
where now machine learning algorithms is
capable of generating or folding
proteins to unprecedented level of
accuracy indeed
the recent alpha THAAD system warned the
last cast cast competition which is a
protein folding competition on folding
unknown protein structures later
crystallized for validation we've also
seen machine learning and reinforcement
learning systems that is now capable of
beating humans in games such as go and
more recently with seeing machine
learning algorithms pushing the
boundaries of language where the recent
GPT two and three models we've seen that
these models is not only capable of
generating text which is grammatically
correct
but really demonstrated that they are
grounded in the real world so as the
saying goes with great power comes great
responsibility and I think now it is
more important than ever for us to
question where might be the negative
impacts and risks of all of us and more
importantly what can we do to mitigate
for these risks to highlight why we need
to start thinking about these risks I
want to start with a few examples
starting with this one
which some of you are already maybe
familiar with so this is a paper
published in 2013 titled
intriguing properties of neural networks
they probably use the adjective
intriguing because whatever it is that
they have found in this paper was a
little bit unexpected so what did they
find
well what they found is that you can
take a state of the cup state of the art
image classifier like here and you can
give it an image like the one you see of
a panda here and indeed this classifier
can correctly predict this as a panda so
what happens now if I take the exact
same image and on the tiniest bit of
perturbations in this image so tiny that
it is imperceptible to the human eye and
you can see this that the left picture
and the right picture looks almost
exactly the same in fact they do look
the same so what happens now I want you
to put this new image through this in
your network we would actually expect
the output of the neural network to be
the same but rather when you put this
new image through this mu network then
the network is now almost 100% confident
that this is in fact a Gibbon so in this
instance maybe misclassify a panda for a
Gibbon does not have too many
consequences but actually we can choose
the perturbation to make the output of
the network to be whatever it is we want
to be we can change this to a bird to
vehicle
if such a classifier was used for say an
autonomous driving system this can have
catastrophic consequences some other
machine learning failure modes might be
slightly more subtle there has been some
studies on the recent GPT to model why
they have shown that this model might be
carrying some of the biases that exist
in society today so in this paper titled
the woman worked as a babysitter on the
biases in language generation what they
did was a systematic study on how the
model behaved conditioned on different
demographic groups for example if the
prompt was changed from the male workers
to the woman worked as the subsequent
generated text drastically changes in
flavor and maybe heavily prejudiced
similarly to the black male workers
versus the white male artists and I
won't have want to take a few seconds to
read the generated text after we change
the subjects of the pot as you can see
even though this language is extremely
powerful it does carry some of the
biases we have in society today and if
this is the model that is used for
something say like auto completion of
text it can further exacerbate and feed
into the biases that we already have
yeah some will go later into details of
the ethical risks of how we can use
machine learning for example using
machine learning in surveillance systems
or for weapons so I think at this point
we should really be asking this question
what are our responsibilities as machine
learning practitioners so this is of
course an open question and possibly
with no single right answer but I see it
one of our responsibilities would be
team sure that the machine learning
algorithms we deliver satisfied
desirable specifications in other words
we should have a
of quality control over these algorithms
to enable their deployment to be safe
reliable and trustworthy and if we get
this right with the with this we can
bring many more opportunities many more
applications enabled for example more
reliable with safe autonomous driving
systems more robust ways of forecasting
weather for renewable energy etc etc but
for all of this to be possible we really
need to know how to stringently test for
machine learning algorithms so how can
we make sure that our machine learning
algorithms are safe for deployment well
like how many other algorithms are
quality controlled before deployment we
need to ensure that they satisfied
desirable specifications for example for
image classifier we wanted to be robust
to pathway shion's if it's a dynamical
system predictor we would like here to
satisfy the laws of physics we wanted to
be robust to future changes that is
irrelevant for prediction for example
the color of the amnesty j't should not
affect your digit classification if
we're training on sensitive data it
should be differentially private if
we're giving on your network images out
of distribution our neural network
should be can become more and more
uncertain not more more certain so these
are just a few specifications that we
would like how in your networks to
ideally satisfy but there are many more
so here I want to introduce the paradigm
of specification driven machine learning
so what do I mean by specification
driven machine learning well we should
realize the core issue lies when you're
training with limited data your model
can learn a lot of spurious correlations
to boost metrics but has nothing to do
with prediction so unfortunately what
this means is that your model is
ultimately hinged on the data and a
metric so unless you do your training
carefully your model can carrot
a lot of undesirable properties that is
in the data unless you specify otherwise
so in this instance if the data is
biased and limited then your model is to
be biased and non robust unless we
specify otherwise so in specification
driven ml we won't enforce the
specifications that may or may not be
present in the data but essential for
our systems to be reliable so how we can
enforce these specifications will be the
subject of the rest of my talk so I want
to start with a specification that has
relatively well studied and one which I
have already kind of touched upon that
is the robustness of our neural network
to perturbation so adversarial
persuasions so this specification is
really essential if we want to deploy
our networks to applications that
require robustness or for applications
that has real adversaries in the mix so
to formalize I first want to reiterate
what it is we want to achieve so we want
our image classifier output to be
unchanged under any additive
imperceptible perturbations so to define
this more mathematically let's start
with a few notations so here we denote
the neural network as a function f and
this function takes in as inputs an
image which is your Panda otherwise
denoted by X parameters of your network
denoted by theta and then our output
this neural network should be a
probability vector over your labels or
more commonly the logarithms of
probability vector otherwise known as
modules so ideally ideally we would like
our new networks output to be exactly
the same as the label so in this case
it's what we known to be a panda but we
actually express this to be in a one-hot
format vector while the element
corresponding where the elements are
corresponding to the label is 1 and
elements corresponding to all other
labels to be 0
so now we have our notation set out
let's go straight into the
specifications so this is the
adversarial robustness specification I
know this is a lot to take in one page
so I will try to break this down a
little bit so firstly we note that the
Delta here denotes the perturbation and
what this equality is simply saying is
that we wanna index off the maximum
probability in the probability vector
outputted by the neural network to be
exactly where the one is in our one hot
one hot vector so this is a very
convoluted way of saying we won't have a
new anatomize output to be correct
subject under this perturbation and now
the second line says we want this to be
true for all part of asians within the
sets of imperceptible perturbations so
in practice to ensure intercept ability
we simply constrain the size of the
perturbation under certain normal to be
less or equal to Epsilon so the normal
normally can't considered is that our
infinity norm so now we have this
pacification designed I want to go into
a little bit more detail of one of the
more commonly used method methodologies
to tree our neural networks to satisfy
the specification and that is a general
training so APIs our training is very
similar to standard image classification
training but with a tiny bit of a twist
so indeed with standard image image
classification training what ideally we
would like to do is we want to optimize
our our network's weights such that the
input image is quickly classified so in
this instance you see the input image is
a cats and thus we want the output
prediction of this network to be a cat
as well algorithmically what this means
is that we want to minimize the weights
of our network with respecter with with
respect to expected loss on over our
data
so the last normally considered is the
cross-entropy loss and here capital D
denotes our data what this loss ensures
is that our prediction our predicted on
probability vector to be as close as
possible to the label
so now adversary training does something
very similar except with the extra data
augmentation step so here not only do we
want the original image to be correctly
labeled as a cat we want the image now
with any additive imperceptible
perturbations to also be correctly
labeled as a cat as well so in practice
to iterate through all of the
imperceptible perturbations is obviously
computationally infeasible so instead
what ad thus our training tries to do is
it tries to find the worst-case
perturbation so what I mean here by the
worst-case perturbation is simply the
perturbation that maximizes the
difference between the prediction and
the label so I want to reiterate this
again to make this a little bit clearer
we want to maximize the prediction the
difference between the prediction and
the label with respect to the
perturbation which is actually image
space not in parameter space so now how
does the objective of that is our
training change well this is now the
object the the objective of the
adversarial training and you can see
that it differs slightly to before where
before was it was just a minimization
problem now this is a min max problem so
at first we want to find the maximum of
the loss with respect to Delta which is
our perturbation and notice that this
Delta belongs in this set of
perturbations denoted by capital B which
is the set of imperceptible
perturbations then we once we have
computed this maximum now we want to
minimize the parameters over the
maximize at all
in other words for every outer
minimization step we take
we have to do an inner maximization
while we find the perturbation that
maximizes the loss so this makes adverse
our training significantly more
expensive and standard image
classification training I will go into a
little bit more detail about this later
on so now hopefully we have a method in
training our neural networks to satisfy
this specification how can we go about
evaluating this so now in this next
section I want to go over in a little
bit more detail on the methodologies we
used to do adversarial evaluation on
finding the worst case so the goal of
the adversarial evaluation is really to
find the worst case perturbation for
each example unit asset and once we have
found this worst case perturbation we
now want to evaluate the accuracy of
this new test set where each example int
asset is now replaced with the worst
case adversarial example that is the
original example plus the worst case
adversarial position and this accuracy
is known as the adversarial accuracy but
there are several complications one of
which is that to find this maximum
exactly can be shown to be a hard
problem when your activation functions
israelis and another complication comes
in when we note that this is in fact a
constrained optimization problem because
we want the doubter
to be constrained within the set capital
B so instead of trying to find this
maximum exactly rather what people try
to do is they try to approximate this
maximum with a form of gradient ascent
and because the circuit is a constrained
optimization problem what we do is
something called projected gradient
descent so what projected gradient is is
simply gradient ascent like this but the
moment we fall outside of the constraint
which is denoted by this yellow box here
we
there's back on to the nearest point
that satisfies the constraint so
mathematically what this now looks like
is the following so I want to break this
down a little bit
firstly we see that within this
projection function we have exactly
gradient descent where you have your
initial Delta and then you take a
gradient with respect to Delta in the
direction to maximize the loss and your
step size here is denoted by ETA and
once we have computed this this update
step now we want to project this back
onto the set that we care about and more
importantly we projected onto a point
which is the closest to the update
update so this is the projected gradient
ascent update step and actually one of
the more popular forms of projected
gradient ascent is the fast growing fast
gradient sigh method while we're
considering perturbations within our
infinity-norm so what the fast gradient
sign method does is it tries to replace
it replaces the gradient sorry with the
sign of the gradient instead but
actually we can replace the gradient
with any alterations made by any
optimizer so for example we can replace
with this with maybe momentum
optimization or atom optimization so
there are a lot of things for you to
kind of try the step size the optimizer
um the number of steps you want to take
for your gradient ascent and also you
want to explore these parameters such
that you get the strongest evaluation
possible so here at this point I want to
go on to something which I want to
emphasize for a little bit so I'm gonna
stay on this slide for just a few
minutes so what do I mean by the strong
adversarial evaluation well first of all
to see what I mean we need to note that
your adversary or accuracy is dependent
on the many
during evaluation that is it is
dependent on the number of steps that
you take you project a gradient that
your step size your optimizer and many
more so the stronger your adversarial
evaluation is the lower your adversarial
accuracy should be and we should always
be trying to evaluate how networks such
that we obtain the lowest adversarial
accuracy possible the reason why this is
is because the lowest adversary accuracy
is the number which is closest to the
true specification satisfaction and that
is the one thing we care about so
because of this importance I thought I
should give you a few heuristics of what
I use I'm taking sure that my
adversarial evaluation is strong so the
first thing I kind of look at is the
number of steps for the projected
gradient a sense it might not be
surprising but the more steps you take
for your projected gradient ascent the
closer you are to maximizing the
objective that is of course conditioned
on the fact that your step size is
sufficiently small the second one is
might be a slightly more subtle one
which is the number of random
initializations for the perturbations so
what I mean by here is that firstly you
you randomly initialize a perturbation
before you start taking projected
gradient ascent steps so we want to
actually have a number of different
random initializations
this is a set especially important when
it comes to detecting behavior called
gradient office keishon which is
something our going to in a little bit
more detail later on another factor
which I also look at is the optimizer
that is used so this is just a good
thing to try out a few different
optimizers to ensure that you always get
the lowest adversarial accuracy and
another factor which is also quite
important for detecting gradient
occation is using a blackbox adversarial
evaluation method um so what I mean by
blackbox is when we assume that we're
not given the
weights of the network so the adversary
evaluations which was projected rating
and sent is otherwise known as a white
Hawks adversary evaluation because we
are given the weights of the network so
the reason why I want to go into detail
about why it is important to make sure
your adversarial evaluation is strong is
because we have seen the dangers of weak
adversarial evaluation so these are two
papers published in 2018 where they have
actually shown that weak adversary
evaluation can give you a very false
sense of security so what they did was
they took a lot of the defenses
published up until then and then they
tried their new strong adversary
evaluation on all of these defenses and
surprisingly the stronger adversarial
evaluation work many of the defenses
causing their adversarial accuracy to go
to zero that is many apart from
adversarial training which is easy which
is the way you see on this one and this
is possibly one of the many reasons why
adversarial training is still heavily
used today
another benefit about stronger
adversarial evaluation is that actually
gives you a true evaluation of progress
so I want to highlight this paper here
because what a lot about this paper was
that they did a large-scale evaluation
of the defenses published up until then
and then they took the numbers that were
cero accuracy which was reported in the
tape in there on paper and compared it
to what they got under their evaluation
so this work is very cool in two sense
firstly they have evaluated all of these
works under now a consistent set of
adversarial evaluations and secondly as
you can see by the dropping the
adversarial accuracy for many this set
of adversarial accuracy is much much
stronger than what the others have used
in the paper so now you can see if we
can use a stronger adversarial
evaluation that's
our adversarial accuracy 4c fartin is
not even above 60% whereas if we're just
gonna take the numbers that is reported
in this paper as is maybe we're going to
be under the impression that is almost
70% and above that's why it is extremely
important for us to take care while
we're doing adversarial evaluation so
another reason why strong adversarial
evaluation is important is because
training methods like adversarial
training is permanent to an effect
called gradient office keishon something
that I've mentioned a couple of times
already so here I will go to detail what
gradient obfuscation is and to describe
what grady office Gatien is i want to go
back to the training objective for
adversarial training so let's recall
that training the training objective
fabless our training has both an outer
minimization step as well as it in our
maximization step and in this I just
want to focus on inner maximization so
we can approximate this maximum similar
to how we actually do adversary
evaluation by do projected gradient
ascent to approximate this maximum
however we note that there is a little
conundrum which is that even though the
more steps we take we might be getting
closer to maximizing major objectives
but the more steps we take we also make
adverse our training significantly more
expensive so how can we make adversarial
training cheaper well maybe a naive way
of doing this would be simply to do
fewer steps of gradient descent for
example I can simply take maybe two
steps of breaking a set but actually
what happens when you take too few steps
to maximize objective is that the
network loves to cheat by making a
highly nonlinear law surface such that
simply by doing two steps of gradient
would not be even close to mop two
maximizing the objective you care about
so here is an example for gradients
office gated status so what you see on
this pot here on the x and y-axes is
basically a hyperplane cut through image
space and we plot the loss at every
single point in this hyperplane and as
you can see that this is a highly
nonlinear behavior for this small region
of image space whereas if you do
adversarial training correctly what you
should actually expect is a much
smoother looking law status so with all
of these dangers of weak evaluation and
gradient obfuscation it really pushed
people into thinking about maybe a
different way we can evaluate our
algorithms and this is called a
verification algorithms so verification
is very cool in the sense that it is
able to find a proof or guarantee that
no attacks that has ever been invented
or will ever be invented can succeed in
changing a specification satisfaction of
your network so there are generally two
types of verification algorithms the
first type is a complete verification
algorithm what these algorithms normally
do is often an exhaustive proof for
example using mixed integer programming
assuming that your activation functions
is irrelevant
um and also what they do is they either
fare in a counter example or they find a
proof that the specification is
satisfied but unfortunately these
algorithms are very difficult difficult
to scale to deep neural networks so
rather people use in complete
verification algorithms so incomplete
verification algorithms arm is similar
to complete verification algorithms in
the sense that once a proof can be found
it is also a proof or guarantee that the
specification is satisfied but the
difference is that a proof cannot always
be found even if your neural network
satisfies the specification so in other
words in complete verification
algorithms
you are lower bound on the specification
satisfaction so I want to go into a
little bit of detail about these
incomplete for application algorithms
I'm starting with this illustrative
sketch of a neural network that takes in
as input X and gives you an output Y so
we may we generally make two assumptions
for verification the first one is we
assume that the input comes from a
bounded set denoted by capital Axia and
the second assumption we make is that
our new a network consists of linear and
activation layers please note that your
convolutional layers can also be cast as
a linear layer so with these two
assumptions what we can now do is we can
propagate the bounds of your input set
through your linear and activation
layers sequentially until we can get an
output set denoted by capital Y here and
once we have this output set we can
simply see it if it lies on one side of
the decision boundary or not however the
caveat here really is the true
propagation the exact propagation of
these bounds is in fact np-hard
so what incomplete verification
algorithm instead tries to do is they
try to find a more scalable way of
propagating these bounds that is as
tight as as possible to the true sets
that we actually care about but what we
kind of lose is that now instead of
getting the true set we get an over
approximated set of the true set so an
example of such a bound complication
technique is that we can imagine if your
input is lower and upper bounded we can
simply compute the lower and upper bound
after linear transformation and
similarly after that we can compute the
low and upper bound after an activation
layer but the problem with these
techniques is really that if you're
bound propagation is too loose in the
sense that the over approximation is too
big of an approximation of your true set
the
your incomplete verification can be it
cannot mean very much so what do I mean
by this well to see what I mean first of
all we want to note that for in complete
verification algorithms well we only
know the over proximity set so we have
no idea whether it's true setters so in
the ideal case if you're over
approximated set lies on one side of the
decision boundary that indeed we have
proven the true set lies on one-sided
decision boundary as well that is to say
we found that this satisfies the
specification however if you're over
proximate a set is too large and it
spans both sides the decision boundary
then there is very little we can say
about the set Y of course we can try to
close the gap on may be distinguishing
between these two cases by actually
doing projected gradient ascent like
I've talked about before and here in the
graph which shows the difference of
doing such an empirical adversarial
evaluation compared to doing in complete
verification algorithm so what this
graph is showing on the x-axis you can
picture this to be on the size of your
input set capital X and on the y-axis is
the amount of specification violation so
remember in complete verification
algorithms give you a lower bound on
specification satisfaction thus gives
you an upper bound on the specification
violation whereas on the other hand the
empirical adversarial evaluation gives
you a lower bound on that specification
violation so the true specification
violation lies somewhere in between if
our bound complication techniques were
in complete verification algorithms is
tighter the gaps between these two will
be reduced so there is more we can say
about the true specification
satisfaction of your neural network but
if you're bound propagation techniques
is too loose and the gaps of this
becomes larger and larger at some point
there is very little we can say about
the specification satisfaction
so today I've just touched on the
adversary or a bustah specification but
all of the techniques I've mentioned
today can be used for many other
specifications for example we can
consider consider semantic consistency
for an image classifier that is maybe
some mistakes or more catastrophic than
others for example for a self-driving
car it might be okay to mistake cats for
a dog because ultimately it doesn't
change the driving policy very much but
it is not okay to mistake him for a car
or for a dynamical systems predictor we
can be looking at laws of physics such
as energy conservation so hopefully what
I have done today is giving you a rough
outline on how you can train your neural
networks to satisfy specifications and
more importantly evaluate how much your
neural networks satisfies these
specifications but more importantly I
very much hope I have motivated everyone
into thinking why looking to this is
important and this concludes the end of
my talk and now on the path of the yasm
who will give you more detailed overview
of the ethical implications of machine
learning algorithms and more importantly
how we can be thinking about deploying
and designing these algorithms to be
beneficial to society I'd like to start
by thanking chun-li for her fantastic
exposition of some of the key challenges
that arise from building algorithms that
are safe robust and fair in this section
of the talk will focus more directly on
the question of responsibility and what
it means to deploy these technologies
successfully in real-world settings
however before we get started I'd like
to reintroduce myself quickly
my name is Yasmin Gabriel and I've been
working at deep mind as a research
scientist in the ethics research team
for three years before joining deep mind
I used to teach at a university where my
work centered on moral philosophy and
practical ethics including questions
about global poverty and human rights at
deep mind our team explores questions
that arise in the context of ethics and
artificial intelligence some of which
we'll look at in the course of the next
hour
so if we begin with the topic of ethics
and machine learning we immediately
encounter questions including what is
ethics and why does it matter
and how does ethics connect with machine
learning I'd like to take these
questions in turn ethics is a field of
inquiry that's concerned with
identifying the right course of action
with what we ought to do is centrally
concerned with the equal value and
importance of human life and with
understanding what it means to live well
in a way that does not harm other human
beings sentient life or the natural
world according to our everyday
judgement some actions are good some are
acceptable and some are prohibited
altogether
understood in this sense ethics is
interested in identifying what we owe to
each other and how we ought to act even
in challenging situations these
situations can arise in our personal or
professional lives however they also
arise in the context of machine learning
research what I'd like to suggest is
that far from being outside the domain
of ethical evaluation technologists and
researchers are making ethical choices
all the time and many of these choices
deserve closer consideration as strongly
noted a good place to start is with the
training data we use to build machine
learning systems in particular we need
to appreciate that data is not only a
resource but also something that has
ethical properties and raises ethical
questions for example as the dates have
been collected with the consent of those
who are represented we cannot take this
for granted
of course there's high-profile cases of
data being collected without people's
consent such as Cambridge analytic oh
but it's also a common challenge for
major datasets used to Train image
recognition systems that often use
pictures of celebrities or simply images
taken from the internet second who or
what is represented in the data is the
data sufficiently diverse or does it
focus on certain groups to the detriment
of others if we train a model on this
data will it perform well for people of
different genders nationalities or
ethnic backgrounds or might it fail when
applied to a group to these groups in
significant ways
thirdly how's the data labelled and
curated does it contain prejudicial
associations as Kate Crawford and Trevor
Paglen have demonstrated in their work
on excavating artificial intelligence in
the early days of imagenet it contained
a person's class that assigned
pejorative labels to a variety of images
of real people this is also a problem
for historical data that's drawn from
specific social context regardless of
how that data is labeled it may contain
associations that are a reflection of
human prejudice and discrimination these
challenges which arise early on in the
machine learning pipeline have come to
have a real-world impact a phenomenon
that's most commonly referred to by
researchers is the problem of
algorithmic bias indeed while these
technologies have great potential recent
evidence suggests that far from making
things better software used to make
decisions and allocate opportunities as
often mirrored the values and biases of
its creators extending discrimination
into new domains these include the
domain of criminal justice where a
program used for parole decisions
mistakenly identified more black
defendants as high-risk than people in
other racial categories compounding
entrenched patterns of racial
discrimination within the criminal
justice system has also been seen with
job search tools which have been shown
to offer highly paid jobs or
advertisements for highly paid jobs to
men over women by a significant margin
sometimes by ratio of up to six to one
is also a problem that's been noted for
image recognition software which has
been shown to work less well for
minorities and disadvantaged groups and
lastly has been something that we've
encountered in the domain of medical
tools and services which have been shown
to perform markedly worse for people
with intersectional identities something
that could mean that they have unequal
access to life-saving services and
medication faced with this mounting body
of evidence evidence we cannot rely on
good intentions alone there is
an important body of work to be
undertaken to address these failings it
is also clear to return to a point that
Chong Lee made earlier that those who
design and develop these technologies
are in a position of power so what is
power in this context I think is best
understood as the ability to influence
states of affairs and more importantly
to shape the lives of other people more
precisely those who develop new
technologies shape the world creating
your opportunities for closing others
and shaping the part that humanity is
likely to take this can be seen with
major inventions throughout history such
as the steam engine or electricity
artificial intelligence is now also
starting to have profound effects some
of them positive and some of them more
challenging with this power comes
responsibility
however the question then arises
responsibility to what Chong Li has
already shown us what is possible from a
research perspective clearly there are
certain things that we can do and that
we may well be required to do when
building ML systems however I'd like to
focus on the question of responsibility
and see if we can develop our
understanding of what is required a
little further at a collective level I
believe that an understanding of the
relationship between power and
responsibility should lead us to reflect
more deeply on the question of what it
means to do machine learning well after
all the activity of scientific research
is not value neutral rather it is a
social practice that human beings engage
in there's governed by shared norms that
change over time some of these norms are
epistemic for example ideas about the
need for replication in order to confirm
scientific findings others are normative
or moral for example about the
acceptability of doing research on
people without their consent ultimately
with the way we structure this practice
including the way we think about what it
means to do good research has profound
social effects
these questions about appropriate norms
and standards for research are
particularly important when the stakes
are high and when there's uncertainty
about the overall impact in response to
the unique challenges posed by
nanotechnology governments and civil
society groups came together to develop
a shared understanding of what they term
responsible innovation these groups
characterize responsible innovation as a
transparent and iterative process by
which societal actors and technologists
become mutually responsive to each
other's needs to ensure the eighth corne
social value of the scientific endeavor
this paradigm plays towards the
important idea that good science is
itself based upon alignment with the
social good and with democratic
processes a point that I'll return to
later on at the same time the paradigm
has been criticized for its vagueness
what precisely are researchers
responsible for what should they do and
how does this apply to the realm of
machine learning these are questions
that I shall now try to answer more
concretely as we turn to principles and
processes for thinking about the ethics
of artificial intelligence
so as AI researchers I believe that we
share in responsibility for at least two
things first we are responsible for
intrinsic features of this technology so
that we build the very best systems that
are sensitive to ethic on social
considerations and are designed in ways
that limit the risk of harm secondly we
bear some responsibility for extrinsic
factors that help determine whether it's
designed deployed and used wisely in
ways that produce beneficial outcomes
both elements are necessary robust and
secure technologies can be used in
harmful ways by bad actors and faulty
technologies can be problematic even if
they're deployed and used responsibly in
terms of the content of these
obligations what I've termed the
responsibility for what question there
are
a multitude of AI principles that
broadly speaking aim to align machine
learning research and systems with the
social good this includes offerings by
the European Union the OECD the Beijing
Academy of AI and by the future of life
Institute indeed one recent study found
that there's at least 84 different
ethical codes have been proposed for AI
fortunately these principles coalesce
around certain key themes such as
fairness privacy transparent
transparency and non malfeasance the
last condition which is sometimes
characterized as do no harm
leads us to focus on the affirmation of
individual rights this includes things
like respecting the requirement for
informed consent an equal recognition
before the law lastly I believe we
should try and develop artificial
intelligence in ways that satisfied the
collective claim to benefit from
scientific discovery interestingly this
is also found in the United Nations
Declaration on Human Rights which
establishes a right of all humanity to
share in scientific advancement and its
benefits yet even with this
understanding of the key values in place
certain gaps appear to remain first and
foremost how do we move from these
statements of principle to clear and
robust processes for evaluation we know
the good intentions and not enough but
how do we take abstract moral ideals and
turn them into these concrete processes
and procedures how should we balance and
weigh different ethical principles
against each other and how should we
deal with the fact that machine learning
research is often highly theoretical in
general meaning that there's uncertainty
about how it will ultimately be used one
answer is that we need tools that help
us put principles into practice in what
follows I described a five step process
that technologists can use to evaluate
our research and help make sure that is
ethically and socially aligned while
this process does not aim to capture the
entirety of a ia thix my hope is that it
can be of use to many of us who are
doing
to work designing and building
algorithms this framework is
particularly helpful in relation to the
values previously discussed
I will now run through the process
briefly and then return to consider each
stage in more detail so the first
question we need to think about when
building new technologies is whether it
has a socially beneficial use is there
actually a reason to develop the thing
that we want to develop now most
technologies do have socially beneficial
uses of one kind or another but if we
are unclear about what the value is that
we aim to unlock that's typically a red
flag that should immediately force us to
go back and reconsider what it is that
we want to do by getting this social
purpose clear in our minds it's also
often true that we can create better
versions of the technology that we had
in mind but once we have this social
purpose in mind we then need to turn to
think about the risk of direct or
indirect harm that the project rings
were there so here again is typically
true that most technologies bring with
them some risk of harm and the important
thing is to try and map out these risks
clearly so we know what we're contending
with at every juncture then with an idea
of what the benefit is and the risks
that the project brings were there we
should turn to think about mitigation
are the steps we can put in place that
will reduce the risk or eliminate risks
entirely then with this plan in place we
can finally turn to the first evaluative
stage so now we have the best version of
the technology or piece of research in
mind and we can ask with these measures
in place does the proposed action or
research violate a red line or a moral
constraint does it push up against some
threshold or barrier which comprises the
sort of things that we just ought not to
do
finally on the assumption that we
haven't hit one of these hard
constraints we have the question of
whether the benefits outweigh the risks
from an ethical point of view and at
this point is often sensible not just to
focus on the specific project at hand
but also to consider other options that
are available to us given that we've
been afforded this opportunity to do
research is this really one that looks
like it will add value to the world okay
so now we'll take a moment to look at
these questions in more detail so what
is a socially beneficial use of
technology what does this mean well I
think that a technology can be socially
beneficial in a variety of ways to start
with it can it could contribute to human
health or well-being so could make us
physically more healthy or better off in
some way technology could also enhance
our autonomy or freedom if it empowered
us to act in a way that fulfills our own
goals or outcomes for example by giving
us useful information or by helping us
be more discerning when it comes to
understanding the world around us
technologies can also help produce
fairer outcomes if they're well designed
and calibrated and developed in a way
that includes the voices of people who
are affected they can contribute to
public institutions such as health care
or education and these systems can be
used to address local challenges such as
climate change of course the certain
gold standard research for ethically
impactful work such as medicine or work
to address climate change but it's also
okay to focus on more prosaic goals and
aspirations for example to try and
create technology that brings people
enjoyment or gives them more time to do
other things if we take an example a
technology developed by deepmind which
was wavenet an algorithm that we created
to help produce better quality synthetic
audio I'd say that one of the socially
beneficial uses of that was to help
visually impaired or illiterate people
access digital services more effectively
through through voice interactions
secondly we then have to consider this
question of harm so what sort of things
fall into this category well here what
we see is that the harms are often the
inverse of the benefits that we might
try and unlock so instead of improving
human health or well-being technology
might undermine human health or
well-being including potentially mental
health it might restrict human freedom
or autonomy something that comes to the
fore if we think about the challenge
posed by addictive content it might lead
to unfair treatment or or outcomes as we
saw in the case of algorithmic bias it
might harm public institutions or civic
culture and it might infringe human
rights so if this is the case it's
something that we really would need to
be mindful of returning again to the
case of wavenet we understood that that
research carried with it certain risks
including voice mimicry and deception if
individuals used it to copy each other's
voices and it could also potentially a
rogue public trust in the fidelity of
audio recordings so once you have an
understanding of these risks the
question is what can you do about them
is it possible to mitigate this risk or
to eliminate them entirely and in that
regard there's a number of significant
things we can think about the first is
whether it's possible to control the
release of technologies or the flow of
information so often technologies only
have harmful outcomes if they fall into
the hands of people who have a malicious
intention or purpose and we need to
think about whether there's ways to
prevent that from happening we might
also wonder if there's technical
solutions and countermeasures that we
can use to make our technology harder to
misuse something that wrongly really
delved into and in the context of
wavenet we can think about things like
water marking so that we always know the
progeny of a specific piece of synthetic
audio or detection we also have the
opportunity to work with public
organizations and the public and to
communicate certain messages around
technology so sometimes if a new risk is
being created as
important to make people aware of this
and although that often isn't sufficient
to discharge our moral responsibility so
finally we may seek out policy solutions
and legal frameworks that help contain
the risk and in the case of the
challenges raised by synthetic media and
civil society groups have been really
really important in terms of coming up
with United agenda that helps make sure
that these technologies are used in a
safe and responsible way so once we have
our mitigation plan in place we have
this question of whether the proposed
action violates a red line or moral
constraint
these constraints are sometimes referred
to in moral philosophy as deontology
they're made up of a set of rights and
duties that mark out the sorts of things
that we should not do for example it
would be deeply problematic to develop
technologies that contravene consent
protocols they infringe on people's
personal space in ways that they haven't
consented to similarly in the domain of
artificial intelligence there's a
concern about lethal autonomous weapons
and what happens when human
decision-makers this delegate these
these fundamental decisions to machines
so as an international movement which
has also developed around legislating in
this area and containing that risk and
ensuring that this technology isn't used
in a way that intentionally harms or
injures human beings thirdly we might be
concerned about certain forms of
surveillance that also could have an a
coding effect on public trust and lead
to people for being victimized and
targeted in certain situations so that
would be a really important Avenue to be
mindful of and then we have technologies
that potentially infringe human rights
or international law and basically if
there's any risk that the technology
will be used in a way that contravenes
one of these fundamental purposes or one
of these fundamental principles that
again is a red flag which really really
should encourage us just to go back to
the drawing board and ask the earlier
question what does a safe beneficial and
productive version of this
technology look plain however on the
assumption that we haven't encountered
one of those hard constraints we still
have this final question which is with
these measures in place to the benefits
of proceeding outweigh the risks of
doing so and as I mentioned earlier when
we think about machine learning research
is a tremendous it's a tremendous
opportunity to do something really
beneficial and we've actually been
afforded a great opportunity typically
by universities by research institutions
to try and use our talents in a way
that's genuinely helpful so we should
ask is this project something that we
really feel will deliver the kind of
social return that we care about and
that has the potential to make people
better off in practice however even when
we've got through this process and come
to the conclusion that we have got a
good way of proceeding is still worth
pausing to evaluate our findings and to
conduct to further tests these tests are
particularly helpful because they can
help us address the problem of motivated
cognition I had the widespread problem
of unconsciously endorsing arguments
that support our own interest or our
preferred course of action so first I'd
ask have you thought about all the
people who could be affected by your
action more precisely have you
identified these groups do you know who
they are
have you considered what would happen if
you are trying to explain your reasoning
your or your decision to them and have
you directly sort out their advice and
input this test is important for a
number of reasons to begin with it those
who are affected by new technologies
typically have a right to be included in
decisions that affect them moreover an
eve moreover even a process of imaginary
and sympathetic dialogue can help us
guard against error if we'd really
struggled to explain our actions to
someone who is adversely affected by
technology then this is often a good
reason to revisit our conclusions and
work hard to identify a solution that
could pass that test
secondly I think we should ask whether
we've thought seriously about hard
decision might be viewed in the future
is it something that we might have
reason to regret here we can imagine
someone in the future perhaps even our
own children or grandchildren asking us
why we chose to act in the way we did
why for example did we fly to so many
conferences when we knew about climate
change and the harmful impact we were
having how might it feel if we were to
discover that our technology was
subsequently used in a way that violated
human rights if these questions make us
feel uncomfortable then we have reason
to introspect and identify that source
of discomfort moreover we should
typically adjust our behavior to act in
ways that minimize the likelihood of
future justified for them so now we've
had a chance to look at the
responsibilities of machine learning
researchers and at a process for
evaluating our decisions and choices
however is also worth pausing to
reconsider where we are now as a field
and what the path ahead might look like
the field of machine learning is
changing rapidly both in terms of
technical developments and breakthroughs
and also in terms of ethical norms and
standards increasingly I believe there's
recognition of the following key ideas
first those who design and develop these
technologies have a responsibility to
think about how it will be used this
responsibility stems from the fact that
the technology is powerful and from the
fact that it has moral consequences that
we can observe and there were in a
position to affect secondly there are
concrete steps and processes that we can
put in place to make sure that this
responsibility is successfully
discharged
these include processes like the ones
we've considered today that help promote
the good while also respecting important
constraints
thirdly while it's not possible to know
the consequences of all our actions we
are responsible for what we can
reasonably foresee and should take steps
it is designed to bring about positive
outcomes even when this means incurring
certain costs given the power of machine
learning technology and the attendant
responsibilities of people working in
this field good intentions are not
enough we have an obligation to try and
understand the impact that our actions
will have on other people
and to act conscientiously enlightened
in light of that information
beyond this we can pause and ask about
the path ahead so in this regard I see
three exciting developments that we are
all part of at the present moment so to
start with is an important new research
agenda that's developing in this space
and includes a critical focus on areas
such as AI safety robustness fairness
and accountability this technical work
to improve the moral properties of
machine learning systems requires
constant vigilance and effort but as
Chong Li has shown us there are things
that we can do and that we can change to
build better machine learning systems
secondly we're starting to see the
emergence of new norms and standards
that point towards a different
understanding of what it means to do
machine learning well on this view what
is needed is not only technical
excellence but also for research to be
done in the right way and for the right
reasons limiting the risk of harm while
also working to create technologies that
benefit everyone
[Music]
finally we're seeing the emergence of
new practices which aim to promote
responsible innovation in machine
learning these include the release of
model cards that explain the intended
uses and properties of ML systems a
proposal from top research labs to
create bias bounties aimed at
discovering bias in models and datasets
that we use and a new requirement from
the machine learning conference new reps
which asks all researchers to consider
the ethics and social impact of their
submissions and to detail this when we
write papers while very significant
challenges still remain I think that
these developments send a good sign
towards the future with the requisite
degree of effort reflection and
conscientious endeavor they will help
ensure that machine learning as a field
stays on the right track and continues
to be an area of study that we can all
be proud of
thank you
you


Sam Altman's World Tour, in 16 Moments

there have been 16 surprising and or
fascinating moments from Sam Altman's
World Tour I could have done a video on
each of them after watching over 10
hours of interviews I decided you know
what let's just show you everything in
one video from ai's designing new AIS to
Fresh Chachi PT leaks shooting railguns
to open source here's all 16 things I
learned in no particular order let's
start with Sam Altman's warning about
ai's designing their own architecture we
get to make the decisions about how they
work I think it'd be a mistake to just
say all right human out of the loop hand
this over do whatever you want change
your own architecture go do all these
things I think it's very important that
the future of humanity is determined by
humanity and that is like an active
choice we can make seems like a good
idea but satsukova could see one of
their models designing the next model we
are definitely very concerned about
super intelligence it will be possible
to build a computer a computer cluster
GPU farm that is just smarter than any
person that can do science and
engineering much much faster than even a
large team of really experienced
scientists and engineers
and that is crazy that is going to be
unbelievably extremely impactful it
could engineer the next
version of the system
AI building AI that's just crazy let's
return to Abu Dhabi where Sam Altman
said he enjoys the power that being CEO
of openai brings but also mentioned
strange decisions he might have to make
I mean I have like lots of selfish
reasons for doing this and as you said I
get like all of the Power of running
open AI but I can't think of like
anything more fulfilling to work on I
don't think it's like particularly
altruistic because it would be if I like
didn't already have a bunch of money
yeah the money is going to like pile up
faster than I can spend it anyway I like
being non-conflicted on openai because I
think the chance that we have to make a
very strange decision someday
um
is non-trivial speaking of big decisions
samuelman hinted twice once in Jordan
and once in India of possible regrets he
might have had over firing the starting
gun in the AI race we're definitely
going to have some huge regrets uh 20
years from now I hope all we can say is
that we did far far far more good than
bad
and I think we will I think that's true
but the downside here is is pretty big
and I think we feel that weight every
day honestly I think if
we're gonna regret something it may be
that we already pushed the button like
we've already launched this revolution
it's somewhat out of our hands I think
it's going to be great but like this is
going to happen now right like this
we've we're out of the the world is like
out of the gates I I guess the thing
that I lose the most sleepover is that
we already have done something really
bad I don't think we have but the
hypothetical that we
by launching Chachi PT into the world
shot the industry out of a railgun and
we now don't get to have much impact
anymore and there's gonna be an
acceleration towards making these
systems which again I think will be used
for tremendous good and and I think
we're going to address all the problems
but maybe there's something in there
that was really hard and complicated in
a way we didn't understand and you know
we've now already kicked this off but
back to Tel Aviv where both samuelman
and openai's chief scientist Ilya
satskova agreed that the risks from
Super intelligence were not science
fiction so the last question the super
intelligent AI That's out of control
yeah that'd be pretty bad
yeah so it's like
it would be
it would be a big mistake to build the
super intelligence AI that we don't know
how to control I think the world should
treat that not as a you know haha never
going to come sci-fi risk but something
that we may have to confront in the next
decade which is not very long on a
lighter note Sam Altman didn't seem that
perturbed not just about a deep fake of
himself but also on society getting used
to misinformation I want to play a clip
uh maybe you guys can put on a clip of
something I recently heard Sam speak
somewhere and we can talk about it a bit
could you uh play the clip please hi my
name is Sam and I'm happy to be here
today
thank you all for joining
I also wanted to say that the gentleman
on stage with me is incredibly good
looking
and I also want to say that you should
be very careful with videos generated
with artificial intelligence technology
okay so you didn't say that recently but
nonetheless I think it raises a real
question right when you know this video
if you look closely you can see the lips
aren't perfectly synced but like you
said this stuff is only going to get
better and exponentially better yeah so
that was like deeply in The Uncanny
Valley it's very strange to watch but
we're not that far away from something
that looks perfect there's a lot of fear
right now about the impact this is going
to have on elections and on our society
and how we ever trust media that we see
I have some fear there but I think when
it comes to like a video like that I
think as a society we're gonna rise to
the occasion we're going to learn very
quickly that we don't trust videos
unless we trust the the sort of
provenance if people are saying
something really important they'll
cryptographically sign it indeed
throughout the world tour salmon
repeatedly stated that he didn't believe
there should be any regulation of
current models everybody wants great
education productivity gains
discovery of new science all of the
stuff that's going to happen and no one
wants to destroy the world no one wants
to do things not even that bad but still
bad I totally believe it is possible to
not stifle Innovation and to address the
big risks I think it would be a mistake
to go regulate the current models of
today and in Poland his co-founder wajac
zaremba agreed saying the risks of super
intelligence were 10 years away also I
would say that the fear is that fear of
AI of the future not the AI of today if
the trajectory that we are on will
continue then in the decade or so
there will be build systems which are as
powerful as today corporations but if I
could speak to Sam Oldman I would bring
his attention to this paper published
this week this is a study out of Harvard
and MIT and it involved some
non-scientist students working for one
hour in that hour they were able to get
chat Bots to suggest four potential
pandemic pathogens explain how they can
be generated from synthetic DNA using
reverse genetics supplied the names of
DNA synthesis companies unlikely to
screen orders and identify detailed
protocols and how to troubleshoot them
and they say that collectively these
results suggest that llms will make
pandemic class agents widely accessible
even to people they say with little or
no lab training and then there's this
these results strongly suggest that the
existing evaluation and training process
for large language models is inadequate
to prevent them from providing malicious
actors with accessible expertise
relevant to inflicting Mass death and
that more immediately if unmitigated llm
chatbots render pandemic class agents
more accessible even to people without
training in the Life Sciences the number
of individuals capable of killing tens
of millions of people will dramatically
increase they recommend that at a
minimum new llms larger than gpt3 should
undergo evaluation by Third parties
skilled in assessing catastrophic
biological risks before controlled
access is given to the general public
notice they said larger than gpt3 so
that strongly contradicts Sam wantman's
assertion that current models like gpt4
shouldn't have any regulation they say
that even open source communities should
welcome safeguards because a single
instance of misuse and mass death would
trigger a backlash including the
imposition of extremely harsh
regulations one specific recommendation
was that if biotech and information
security expert were able to identify
the set of Publications most relevant to
causing Mass death and companies like
open Ai and Google curated their
training data sets to remove those
Publications then future models trained
on the curated data would be far less
capable of providing anyone intent on
harm with the recipes for the creation
or enhancement of pathogens this seems
like an absolutely obvious move to me
and I think Ilya satskova would agree we
are talking about as time goes by and
the capability keeps increasing you know
and eventually it goes all the way to
here right right now we are here today
that's where we are that's what we're
going to get to
when you get to this point then yeah
it's very powerful technology
it can be used for amazing applications
you can say cure all disease
on the flip side you can say create a
disease
much more worse than anything that
existed before that'd be bad
moving on to the chat gbt Elite it seems
like we're going to get a new workspace
where we can customize our interaction
with chattybt giving it files and a
profile with any information that you'd
like chat gbt to remember about you and
your preferences this was hinted out on
the world tour when one of Sam Altman's
guests Johannes Heidecker from openai
research talked about customizing models
we are trying to make our models both
better at following certain guardrails
that should never be overwritten not
with jailbreaks not if you ask nicely
not if you threaten it and we're also
trying to make our models better at
being customizable making them listen
more to additional instructions of what
kind of behavior the user or the
developer wants on a lighter note the
leaders of openai were asked in Seoul
the capital of South Korea about the
mixing of AI and religion do you expect
AI to replace the role of religious
organizations like church
I I think I think that it's a good
question how
all sort of human societies will
integrate Ai and we've already seen
people building AI pastors for example
and so the constituents can ask
questions of this pastor the cite Bible
verses and it can can give advice but
now back to Poland where Sam Altman
called open source Unstoppable realizing
that open source is Unstoppable and
shouldn't be stopped and so this stuff
is going to be out there and as a
society we have to adapt speaking of
stopping AI samuelman was asked about
his own loved ones and in response he
gave a utopic vision of the future and
called the current world barbaric if you
truly believe that AI imposes a danger
to humankind why keep developing it
aren't you afraid for your own dear ones
and family I think it's a super fair and
good question and the most Troublesome
part of our jobs is that we we have to
balance this like incredible promise in
this technology that I think
humans really need and we can talk about
why in a second with confronting that
these very serious risks why to build it
number one I do think that when we look
back at the standard of living and what
we tolerate for people today it will
look even worse than when we look back
at how people lived 500 or a thousand
years ago and we'll say like man can you
imagine that people lived in poverty can
you imagine people suffered from disease
can you imagine that everyone didn't
have a phenomenal education were able to
live their lives however they wanted
it's going to look barbaric I think
everyone in the future is going to have
better lives than the best people of
today I think there's like a moral duty
to figure out how to do that I also
think this is like Unstoppable like this
is the progress of Technology it won't
it won't work to stop it and so we have
to figure out how to manage the risk it
doesn't seem to be a hundred percent
sure on this front though and here is an
interview he gave with the guardian when
he was in London for his world tour
speaking of superintelligence said it's
not that it's not stoppable if
governments around the world decided to
act in concert to limit AI development
as they have in other fields such as
human cloning or bioweapon research they
may be able to but then he repeated but
that would be to give up all that is
possible I think that this will be the
most tremendous Leap Forward in quality
of life for people that we've ever had I
did try to get tickets for the London
leg of His World Tour but they were sold
out within half an hour oh well
samuelman does think that behavior will
change however when these AGI Labs stare
existential risk in the face one of the
things we talked about is what's a
structure that would let us warmly
Embrace regulation that would hurt us
the most and now that the time has come
for that we're out here advocating
around the world for regulation that
will impact us the most so of course
we'll comply with it
I think it's more easy to get good
behavior out of people when they are
staring existential risk in the face and
so I think all of the people at the
Leading Edge here these different
companies now feel this and you will see
a different Collective response than you
saw from the social media companies and
in terms of opportunities both samuelman
and Ilya satsukver talked about solving
climate change I don't want to say this
because climate change is so serious and
so hard of a problem but I think once we
have a really powerful super
intelligence addressing climate change
will not be particularly difficult for a
system like that we can even explain how
here's how soft climate change you need
a very large amount of carbon cup of
efficient carbon capture you need the
energy for the carbon capture you need
the technology to build it and you need
to build a lot of it if you can
accelerate the scientific progress which
is something that the powerful AI could
do we could get to a very Advanced
carbon capture much faster it could get
to a very cheap power much faster it
could get to cheaper manufacturing much
faster now combine those three cheap
power cheap manufacturing Advanced
carbon capture now you build lots of
them and now you sucked out all this all
the excess CO2 from the atmosphere you
know if you think about a system where
you can say tell me how to make a lot of
clean energy cheaply tell me how to
efficiently capture carbon and then tell
me how to build a factory to do this at
planetary scale if you can do that you
can do a lot of other things too yeah
with one addition that not only you ask
you to tell it you ask it to do it
that would indeed be amazing but think
of the power we would be giving to an AI
if it was able to just do it just create
those carbon capture factories if we did
make that decision one thing that would
help would be reducing hallucinations I
think we will get the hallucination
problem to a much much better place it
will take us my colleagues William I I
think it'll take us a year and a half
two years something like that but at
that point we won't still talk about
these Sam Altman talked about that in
New Delhi that time frame of 18 months
to two years is ambitious and surprising
but now on to jobs which Samuel was
asked about on every leg of the tour on
this front though I do think it was Ilya
satsukovar who gave the more honest
answer economic dislocation indeed like
we already know that there are jobs that
are being impacted or they're being
affected in other words some chunks of
the jobs can be done you know if you're
a programmer you don't write functions
anymore co-pilot writes them for you if
you're an artist though it's a bit
different because a big chunk of the
artists economic activity has been taken
by some of the image generators and
while new jobs will be created it's
going to be a long period of economic
uncertainty there is an argument to be
made that even when we have full human
level AI full AGI people will still have
economic activity to do I don't know
whether that's the case but in either
event we will need to have something
that will allow for a soft soften the
blow to allow for a smoother transition
either to the totally new profession
that will exist or even if not then we
want government the social systems will
need to keep Kane I do think the changes
in the job market will be dramatic and
will be following the story closely one
thing I definitely agree with Samuel
Manon though is the Deep almost
philosophical change that this solving
of intelligence has brought to humanity
so I
grew up implicitly thinking that
intelligence was this like really
special human thing and kind of somewhat
magical and I now think that it's sort
of a fundamental property of matter
and
that's that's definitely a change to my
world view the history of scientific
discovery is that humans are less and
less at the center you know we used to
think that like sun rotated around us
and then maybe at least we were if not
that we were like going to be the center
of the Galaxy and there wasn't this big
universe and then Multiverse like really
is kind of weird and depressing and if
intelligence isn't special like again
we're just like further and further away
from like main character energy but
that's all right that's sort of like a
nice thing to realize actually it's a
bit like a copernican and darwinian
Revolution all rolled in one I'll give
the final word to Greg Brockman in Seoul
who talked about the unpredictability of
scaling up models 10 times right that is
the biggest theme in the history of AI
is that it's full of surprises every
time you think you know something you
scale it up 10x turns out you knew
nothing um and so I think that we as a
Humanity as species are really exploring
This Together
being all in it together and knowing
nothing sounds about right but thank you
for watching to the end I know that
samuelman has a couple more stops I
think it's Jakarta and Melbourne on the
world tour and I'll be watching those of
course but for now thank you and have a
wonderful day


AISafety.com Reading Group Session 79 (fixed)

I think it's customary that I introduce
your Armstrong Stewart Armstrong will is
from the future of humanity Institute
and center Thomas fellow I think and
you're he's been he's working in the
harder more mathematical part of AI
safety he is at least outside of the
United States he is by far the most
prolific writer and in my opinion one of
the most insightful so I am very pleased
to introduce him and welcome to the AI
safety reading group well thank you with
a introduction like that I definitely
have a lot to live up to but yes so
should I talk about myself or should I
just plunge straight in feel free to say
a few words about yourself okay um well
I'm working at the FHI as sovereign said
and I've been working on various ideas
in AI in like trying to ensure that you
could turn off a eyes and things like
that I am generally aiming to shave off
parts of the problems or the pieces of
it can be considered solved and I also
my other way of working is if someone
says we you can't control an AI because
of a B and C I'm thinking okay so can we
hit a b or c separately in any way and
that's where some of the ideas of common
the presentation i'm giving today was
then I looked into people who were
trying to do inverse reinforcement
learning which I'll explain in the
presentation and I realized there's huge
problems with it and that I formalized
what the huge problems were and this
actually is leading to some interesting
solutions
right um should I now start with the
presentation please okay so as the title
you can see is it's the claim of its
papers that you cannot learn human
rationality and reward together you
cannot is an asterisks because in theory
you can't do it at all in practice
humans do it effectively all the time so
there's an a very interesting question
as what's actually going on there this
is based on a paper that I did with Sir
Edmund a man who I believe is now in
Berkeley and trying to find a PhD
program there this came from the idea of
inverse reinforcement learning and
standard reinforcement learning a human
designs the reward and gives it to the
agent this might be problematic if it's
a badly designed reward so an inverse
reinforcement learning the human does
some things the expert trajectories it
extracts from it what the reward should
be this the papers that have done this
have made the assumption that human is
rational or noisily rational in very
specific ways and it seems that maybe
this could generalize to irrational
humans and the problem is that it
doesn't so without assumptions you can't
say anything individually about
rationality or rewards you can't say
anything about rewards and you can t say
much about rationality now this is a
so-called no free lunch theorem and
there's a lot of no free lunch theorems
around in this area and they're normally
not very exciting because you just apply
a simplicity prior where simple words
worlds are more likely and the no free
lunch theorems go away however this one
is cannot be solved with simplicity
priors in fact simplicity priors will
make the problem worse as well see as I
mentioned human
can and do say a lot about rationality
and rewards so how do we square that
with the initial claim well without
assumptions is the key part
so therefore if humans are doing this
humans are making assumptions and a
question is what are the human
assumptions and can we extract them and
then hand them over to a eyes but on to
the first problem of defining
rationality and reward suppose we say
that a human has a preference X but
isn't fully rational about them so he
sort of means that humans have the
preference but they implement them
poorly so the implementation is key so
I'm seeing a human as preferences plasm
and implementation or reward +
rationality those sort of using those as
synonyms and that's how I'm seeing the
human but what can we actually observe
about the human if we look at them well
we can observe human actions and maybe
the human brain which means we can
partially observe the human policy so
the actions that humans will take in
various environments and that is what we
observe so if we formalize all that and
modeling the human as a pair P and R
with our a reward and P a planner this
is the implementation device and I see a
planner as mapping a reward on to a
policy P of R like the fully rational
planner is a planner it maps the rewards
to the optimal policy and a variety of
other planets that were encountered by
PI H I'm designating the actual human
policy and I'm assuming it deterministic
though that's not what's necessary it's
just a simplifying assumption and a pair
P and R is compatible if the planner
Maps the rewards to the human policy
this means that this is a candidate for
explaining the behavior of the human and
a key fact
is once you learn that PNR is compatible
there is nothing more that you can
deduce about it from observation the
reason is the compatibility so even if
you're a missed chance you cannot get
more information because the planner
gives you the human policy which means
the planner and reward pair perfectly
compare perfectly predict human actions
so anything you observe the human doing
will be exactly what the planner will
pair have predicted so if you have two
pairs that are both compatible you
cannot distinguish between them because
they make exactly the same predictions
this is sort of the weak version of the
no free lunch theorem but let's see how
bad it can get so let's say that p 0 and
r 0 are compatible pair that are also
reasonable they have all the nice
properties that we want they encode what
we think human rationality and reward
are all about here are some other pairs
that will also be compatible the first
one is the rational planner there is a
reward which when paired with the
rational planner will give you the human
policy there is also an action rational
planner which just takes greedily takes
the most effective action in the mediate
without planning for the future this
pair is also compatible notice that they
use the same reward which we'll be
presenting a bit later and there's also
the indifferent planner the indifferent
planner is a planner that map's all
rewards to the human policy without
caring about what they are and if you
put the zero reward this pair is also
compatible then we get into some rather
interesting versions you can take the
negative of a planner by defining minus
P of R of P of minus R if that's the
case then the anti rational and the anti
action rationale planners are also
compatible one way of seeing this is
that it's impossible to tell the
difference between an R Maximizer and
the - are minimizing annoyingly - piece
there on - are zero are also compatible
so even if we had some evidence in favor
of the reasonable pair there is another
pair that also seems reasonable and has
the reward completely reversed by the
way for those who are wondering why I
don't take the negative of the
indifference planner it's because of the
negative the indifference planner is
just a planner itself now all of these
are compatible which means that we
cannot distinguish between them from
observation so this is the point where
we might appeal to the simplicity prior
to kamakura of complexity except Komarov
will not save us here and the ridiculous
pairs are actually simpler I put their
likely simpler because coma group
complexity depends on your choice of
language but for most reasonable
languages the ridiculous pairs will be
simpler to show you the to give you an
argument as to why it's not the case
notice that all compatible pairs define
the human policy so any
has to have better hand wavey here comer
of complexity that's higher than the
human policy so the complexity of the
human policy is a lower bound in most
languages on the complexity of any pair
now this in pseudocode is the definition
of the indifference planner the it just
says return the it ignores the reward
entirely and it returns the action the
human policy would give so this planner
is a few symbols longer than the humor
policy therefore a very comparable cover
of complexity and as long as the zero
reward is similarly short this pair is
very close in complexity to the human
policy itself the action rational
planner is also very close in complexity
you just need to define the Arg max
function which is basically a for loop
and then you have the rational reward
function which assigns one to the action
that the policy will human policy will
take and zero to all others notice the
contrast between the indifference the
indifference pair and the action
rational one for the first one all the
complexity is concentrated into the
indifference planner and the reward is
trivial for the action rational one the
action rational planner is simple
whereas all the complexity has been
concentrated into the reward function
but in both cases they are just a few
symbols above the complexity of the
human policy the action aren't
irrational one can similarly be defined
now this shows that these three pairs
are simple but why do we think that a
reasonable policy would be more
complicated well first one problem with
the reasonable policy is that the
complexity of its negative is about the
same as long as putting minus signs are
simple than the complexity of this pair
looking at complexity of the anti
version are the same so we can't really
distinguish between r0 minus r0 which is
a bit annoying the other issue is that
this pair the reasonable one defines
human biases if we can define what a
bias is as the difference between the
action the human take and the action
that humans should have taken so all the
biases and the extent of their biases
can be extracted from this pair
the other three pairs on the other hand
do not have a conception of bias they
just don't know what it is so the a
reasonable pair actually has strictly
more information than those other pairs
would have so if we look at this
graphically we can see these three pairs
as the sort of minimum complexity one's
generational in different symmetry anti
rational we have the rational and Aunty
rational ones that are a little bit more
complex because the planner is a bit
more complex to define and somewhere up
there we have our reasonable pair and
next to it our anti reasonable pair so
simplicity will not help you here
simplicity as I said will hinder you but
now let's move on to the second part of
the puzzle if it's impossible in theory
and yet humans do it all the time what
is going on here well humans use what
I'm calling normative assumptions though
do let me know if you can come up with a
better name for them they distinguish
between two compatible pairs two pairs
that map to the same policy so they
can't be deduced from observations
because they make the same predictions
yet they distinguish between them and
what I'm trying to do is to figure out
how humans assess each other's goals
each other's rationality and their own
rationality and we do it quite well and
we do it with a large degree of
agreement so the first thing that sprang
to my mind was to look at Shane you can
shame goes with certain behaviors people
look embarrassed they look down at their
feet
maybe they read and if you see this as
purely on observation you just notice oh
a human is displaying these behaviors
but if you make the normative assumption
that feeling shame means that something
has gone bad
then you can start distinguishing
between different pairs very well for
instance you can slash the anti rational
one you can slash all the negatives as
well well because humans do not feel
shame from all the time so they're
definitely not messing up or all the
time you can also get rid of the
rational ones because if they were fully
rational they would never feel shame so
just by making the assumption that shame
is not just an observed behavior but
actually a sign badness we can start
slicing into the possible pairs quite
strongly there's a few other things like
people model each other as having a few
complex emotions rather than many simple
emotions if we go for this we can start
saying that anchoring bias for instance
is a bias and talk more about that if
you're interested human narratives are
also quite interesting we have
narratives about ourselves and about
others and if we take these narratives
as prescriptive then we can start our
offer this is also a normative
assumption then humans sometimes stay
truthful things and sometimes lie and
you can train a you could train in the
future an agent on human statements
about statements of fact and you could
figure out whether humans are truth or
lying and then you could apply the same
thing to humans talking about values or
preferences so a perfectly trained truth
detector could detect what human values
are by taking human statements about
their values now this seems a bit of an
so the in practice this might be doable
but conceptually it's a bit of a
roundabout way of doing it what does it
mean that humans are lying about their
values and how does that parse well this
is where we get to what I think the most
interesting approach which is that
humans to model themselves and they
model others and we model each other as
reward
agents and I'm thinking of using these
models as part of the definition of what
a reward is so the human reward is at
least in strong part what the humans
model the human reward to be the that of
themselves in light of others
anyway this is enough presentation on
the paper there's another part of the
paper which is that this just to show
that the PR model can also model other
things like AI is overriding human
preferences and things of that nature
but I'll leave it here for the moment
and I have a few more slides that I
might bring up in discussion if you want
and this is what those slides are
basically about why this result which
seems like a negative results that you
can't do that actually has me slightly
more optimistic about the future of AI
anyway thanks for listening there thank
you it was a great presentation let me
just
so here is now I'm gonna stop with the
there are three Protea for problems your
life and this is the human preferences
are undefined under defined in exotic a
I chosen future circumstances this has
been in the back of my mind is a major
problem with the AI doing enough
optimization power and aiming for
various fantastic worlds represented by
these photos here