Aligning ML objectives with human values

welcome to the afternoon their session
first talk key by Paul Christiana
about AI line hey so this talk will
cover some slightly unusual topics so
you should feel unusually free to stop
for questions or clarifications or
heckling my voice is not doing super
well so if I start whispering it's just
because trying to conserve it feel free
if I'm unclear to ask me to speak a
little more clearly so what I mean by AI
lineman is the problem of trying to
build AI or the problem of building AI
which is trying to do what we want it to
do so that is the idea is that even if
you build smarter and smarter systems
you'd like them to be applying that
intelligence in the service of the tasks
which we construct to themself and the
reason I think this may be a problem or
it's sort of an theoretically
interesting problem is that ml works
particularly well in the setting where
we have a clear objective where we can
say like here's your score in a game you
want to optimize your score or here's
some data that you want to predict but
we often want to use a melon cases where
we don't have some simple short-term
objective and maybe this is particularly
true as we look forward to speculative
applications so if you imagine say in
the AI system which is trying to advise
me on how I should spend my week or an
AI system which is trying to advise us
on some policy decision the actual
metric we care about is something that
we can't measure easily the actual
metric is something like how good are
the consequences over the next ten years
or 20 years where like first we can't
wait to see those consequences and
second even if we could we don't really
have a precise definition of what is
good we don't represent of what it is we
want and we might be concerned that if
we construct some simple proxy that we
can optimize with ml we'll end up sort
of optimizing the proxy at the expense
of breaking the connection between it
and what we care about
so I'm going to be talking about
alignment sort of as a reduction that is
I'm going to assume that were handed
some powerful and melt techniques that
is were handed some algorithm which is
able to optimize a well specified
objective and we want to use that
capability sort of as a black box to
perform some hard to specified tasks and
we'd like to do that reduction in a way
which is both efficient so it doesn't it
reduce much overhead it's competitive so
the policies we get out are in some
sense about as smart as the policies
which we could have gotten out of the
sort of black box optimizer we were
given I'm in a scalable so we don't want
to have to make some assumption of the
form this works as long as the we're not
optimizing over to largest base or we're
not optimizing too hard over that space
we saw I'm good I'll be a little bit
more precise about what I mean by this
in the next few slides there's some
clarity about the problem feel free just
so first to talk a little bit about what
I'm going to assume that I have what we
have to work with what we're given and
then it work in sort of an online
learning setting so I'm gonna imagine
there's some fixed class of policies so
you might think of this as being like
some space of neural nets with a billion
parameters there's some space of inputs
and outputs maybe to make life simple
these are like sort of questions and
answers which are both just expressed in
natural language and the inputs might
also have a bunch of other context as
part of them though now the algorithm
expects to see some sequence of
decisions where a time step T it
receives a context or X provides an
output Y and we provide a loss we might
be either in sort of a bandit setting or
we just provide the loss for the
particular output that it gave or might
be in more like a supervised setting
where we provide sort of function which
it can use there was give it the loss
for every value of y this algorithm
brings some combination of optimization
a clever choice of model class clever
regularization clever exploitation
strategies that allow it to achieve a
low total loss and by low total loss I
mean by comparison to the best model in
this class will say this learner is good
if that's what a loss that receives over
some arbitrary sequence of decisions is
close to the total loss of the best
policy in this class so it might imagine
like if the space of neural Nets was
convex then something like SGD will give
you some guarantee of this form if we
make the regularization or the
exploration or a model class or
optimizers better then we end up with
bigger class to see that we can compete
with and lower bounds on the regret this
is kind of what I'm imagining that we're
given in the reduction this is where we
start we have some algorithm which has a
property of this form so I'm regret
bound some model class
to say a little bit about what we want
I'm gonna try and present it in a way
that makes it as structurally similar as
possible to what we have to really
emphasize the distinctions again I'm
going to imagine that there's some class
of policies we'd like to compete with
they're all nuts with the billion
parameters say at each time T we want to
be able to come to the system we've
built make some query again we wanted
maybe ask it some question receive some
response and now we have some
preferences over the different responses
that the system could give so at time T
based on the actual context in the world
not merely based on the context we
provide we have some utility function of
the different answers they could give
how good would it actually be if this is
to produce this answer we don't directly
observe that utility function and as
before we sort of want to build a system
which delivers a lot of utility compared
to like the best model in this class
like best-case some would have come down
and said here are the parameters that
you actually want to use in your
question answering system and we would
like to achieve a high utility compared
to that benchmark when I say that this
is competitive when the reduction say
the reduction is competitive I mean that
I wanted to compete with a similar Class
C so maybe we're going to end up
competing with the class that's smaller
than what our algorithm could have done
I mean I don't want to increase the
regret by that much and they say it's
scalable I say this reduction should
work for arbitrary large classes then
you have to make some assumptions on the
Class C but hopefully they won't be of
the form it's not too big
I presented these in a way that makes
them yeah so I think they're basically
two gaps so one is that in order to get
our bound we have to provide the loss at
every step but we can't provide the
utility you that's the first gap and the
main one that I'm going to focus on in
this talk and there's another large gap
where the regret is going to depend on
sort of the maximum difference in losses
that's going to give us a vacuous bound
if we try applying it to like an actual
practical deployment of ml in the world
but it's gonna give us some bounds like
you know at there will be at most one
day on which every self-driving car in
the world crashes which is going to be
really the actual amount of utility the
results is really weak we need something
stronger which is where we get into sort
of ideas like verification or
adversarial training or so on I see
these are members which I think about
these are the two main gaps between what
I'm happy to help myself to and what I
want to achieve yeah so we're going to
have to in order to get any traction on
this goal we're going to have to make
some kind of normative assumption that
relates some observations to the utility
function and we're gonna have any hope
of having our own without such an
assumption I think of that as part of
the statement of the problem and part of
what makes that sort of a fuzzy problem
is like what kind of normative
assumption do we want to make I'll talk
a little bit about that over the course
this time yeah
so we need some normal assumption really
something we see to the utility function
one option would be to say the humans
just know what is good so for example we
can ask humans which is better this
outcome or this outcome or this output
or this output and say that if the
humans prefer one to the other then in
fact the utility of that is higher if
you imagine that you that have Oracle
access to a human and you make this
assumption relating to that human to the
utility now we're in the regime where we
can at least hope to have algorithms
today in some sense this is the most
natural it was like the simplest or most
natural assumption they sort of run into
two problems one is the humans are
expensive and so if we make this
assumption the complexity of our
algorithm is going to depend we have to
make some large number of calls to this
human and a second is that I don't
actually want to take humans as my
source of ground truth because I might
think that in the long run
myah systems are making decisions that
are more complex than what a human could
understand what we'd like them to in
fact help us make better decisions and
we could have otherwise made I think
this is a good starting point for the
kind of assumption we might make and the
first part the talk and we'll discuss a
little bit problem one and then the bulk
of the talk I'm going to talk about
problem two
our first problem was that humans are
expensive I'm relatively optimistic
about being able to overcome this
problem in practice although it's
certainly a fairly like it's a real
problem it's a serious problem I'm
optimistic about being able to do
something so examples of like how we can
get traction or why we might hope to be
competitive and part of competitiveness
means if the algorithm we were starting
with didn't necessarily need to make
these calls to humans we want to not
have to make that many calls to humans
we want to not increase the total cost
of rather than that much the reasons why
I'm optimistic about being competitive
one is that what we're trying to do or
sort of the definition of the goal is
often simpler than a policy which in
fact behaves well so it might hope that
we can exploit that gap to reduce our
dependence on humans maybe from humans
if to interact with humans in order to
learn what you should be trying to do
you have to interact with the real world
in order to learn how to achieve it if
the statement of what you're trying to
do is much simpler than how to achieve
it then most of your sample complexity
is just going into these interactions
with the world which we're not making
any worse by this reliance on humans
their idea is that we can just query the
humans if we have this assumption
vitamins we can imagine querying them on
points where we can learn a lot about
humans so we don't just have to be
asking about random decisions that come
up we can imagine learning from cheep
correlates so we could say maybe humans
have some complex thing they want if
they have a system acting in the world
on their behalf but most of the time
they can just learn from some simple
proxy like how do I make a lot of money
or like how do I sort of control the
environment in some simple way it's like
useful for practicing how to become
intelligent and we can imagine
pre-training on other tasks with cheap
feedback in the world of treatment do
you know would say don't covered a lot
of these in his earlier talk and it's
one of these I'm not going to dwell too
much on this challenge I'm optimistic
about being able to find something I
think right now is a hard problem
yeah so it's worth noting the like if I
actually start with this assumption this
assumption is not enough to actually get
off to the races because like there's
going to be trade off this algorithm
needs to make where there's uncertainty
so we actually have to be able to say
something and at least some probability
like need to ask them about some
probabilistic mixtures so I could pick
some reference to reference outcomes and
ask them about mixtures between those
reference outcomes and Y and y Prime and
so I need to be able to query them about
that kind of information to actually
help I think there's really a lot this
is a reasonable example this like if
humans prefer Y to y prime them to
Tilly's their practice like we're not
gonna want to make an assumption in this
simple you're gonna want to be a little
bit more sophisticated they talk a
little bit more about the sense in which
humans are approximately optimal or kind
of know what's up because something more
mild though these two problems will
still both apply I mean everything was
forming and sending those humans and you
want to minimize the number of crystal
there's only certain type of queries
that they won't cook an answer
all they want at the end we greatly
respect yeah so this is unfortunately
just a feature of the talk but the level
of formality is going to continuously
drop starting from a high point and now
it's lower it's going to become lower
still you couldn't set up this problem
at this point formally by saying of
access to some Oracle H we're going to
make this assumption that like hy Y
prime is 1 if and only if u of Y is
greater than Y Prime and then we can
talk about query complexity of that
algorithm unfortunately like so as we
moved to this worlds where we started
asking like what are the actual
assumptions under which they like active
learning works well reward learning will
work well then things are going to get
dice here to try and formalize as we
talked about going beyond humans these
normative assumptions are going to get
even diced here is I think part of the
game is trying to make those assumptions
to something more precise and part of
the game is going to be coping with like
formalizing what we can and coping with
what we can't I'm not super happy with
situations or any way so what do you
feel about humans do you sometimes see
TV until well so certainly if I make
this assumption I made this assumption
then certainly that implies transitivity
which is a good way to just debunk this
assumption outright like you can tell
this assumption isn't true because we're
in fact intransitive so I think in the
long run you're going to want to make
something weaker which is like example
of assumption you could make is that the
probability with which Eamonn prefers
one outcome to another in some sense
depends on the utility gap that's also
obviously going to be false since then
you would say that ensemble of humans
fish large ensembles transitive and
again this gets into where the game is
not like it seems quite hard this is
like the problem we care about in some
sense but seems very hard to get like a
real formal statement which is still
both possibly true and plausibly soluble
as a problem so I expect I'll just get
compromise on the two sides there formal
statements which are obviously not true
like assumptions you can make a humans
that are obviously too strong under
which you can solve the problem and then
you have the actual thing you care about
which is not formalize do you hope the
progress on these dim assumptions of
issues strong are getting you towards an
algorithm which actually works in the
world
like I said this is just gonna get worse
once we ask about how you go
distributional become so in this setting
y is just actions proposed by the system
so you could imagine relaxing which is
something we'll talk about in a second
to humans observing outcomes of
decisions and having preferences over
outcomes so you here I'm saying we have
preferences over actions yeah so this
this object to you reflects not only
like a human's preferences of our
outcomes it also reflects like a human's
expectations or subjective beliefs or
which outcome which actions will lead to
which outcomes which is a sense in which
wanted to do better than humans
interferes with this so one way in which
you can do better than humans as you can
say instead we get to observe the
outcomes of decisions and now we think
humans preferences are correct about
outcomes that game is still going to be
hard are still going to be intractable
to work with because the outcomes humans
care about are a long distance in the
future sort of in some sense can't get
around this problem of needing to fold
in both human preferences and some like
somehow you have to get these
expectations about comes there ten years
hence actually if we just have an
assumption that human preferences over
outcomes are right then we're going to
run into a wall once we say well now we
care about like the effects of this
policy in ten years and we don't get to
observe that or we can't have enough
observations to possibly have a regret
bound using only that feature we like
need to learn something more about the
structure of the world I want to make
some further assumptions about the
structure of the world motor of
Attraction we can't just rely on
feedback if the time horizons are long
yeah so just just just to make sure I'm
thinking in the right spaces so putting
a concrete example if I would think
about the self-driving example you were
using before then the reference
judgement would be in fact over all the
various controllers and actuators and
all the other things that are yes well I
did some of the complexity yeah Val I
did some of the complexity by just
saying you have a context then you have
an action and in reality you may care
about sequences of actions but setting
that aside your context would be like
the video so far of what has happened in
action would be like some set of motor
outputs and their preferences would be
from a human's perspective what do they
think is the utility marginalizing over
like all the places where that video
would appear in the whole world
what's this utility from taking this
action
that was kind of asking for it by
beginning with a more formal statement
he was continuously getting less formal
yeah this is the kind of thing I don't
feel great about it's the kind of thing
I work with I don't know how to I don't
have a better way to go sort of imagine
working from to end so like gradually
formalizing better and better and then
also trying to have algorithms that seem
like they possibly
and we won't isolate this challenge now
we can't be in the regime where we can
make it completely normal in various
ways they can I think all the
formalizations would probably they're
not going to be quite right and so as
normal there's going to be some
sequences that are better formalizations
to capture more
um I'm gonna talk a little bit about
this first feature that might make it
possible to be more efficient where
maybe what you want done is simpler than
the policy right so back here I just
assume that the humans directly had
preferences over actions and then
learning these preferences over actions
is quite hard but I could assume instead
that humans are preferences over
outcomes with more of the complexity
being in the mapping between actions and
outcomes I can get further so we can
imagine again removing the sequential
setting and just talking about the
single step setting it imagine there's
some transition function where the
outcome depends on the situation and
which action was taken the humans now
have preferences over triples like
there's some function R or maybe you can
query our by talking to a human and this
tells us like if you're in situation X
take action y produce how come Z how
happy are you with that
if we have to consult a human if you
imagine like a human spending an hour
you may be in the regime where querying
R is very very expensive compared to
trading t control the sample complexity
in terms of R as well as the complexity
in front
so if we can have much lower sample
complexity in our than an T then we are
sort of ok if R is a lot more expensive
than T and the bound on like how
expensive it's ok for our to be just
depends on how low we can make the
sample complexity I think I here would
make some similar assumption like that's
like saying that you has this form sort
of amounts to saying that's like saying
because this form amounts to assuming
something about the relationship between
whatever the definition of R is and you
AG humans a spread our there correctly
report their preferences or otherwise
so in this setting is a very simple
approach although analyzing even this
very simple approach in theory is
already fairly difficult where I just
say in parallel we're going to fit some
model R hat using supervised learning
and then minimize sort of that the
learned reward function this is some
assumption about the learn ability of
what the human is doing sort of some
 about how that is the complexity of
learning that is simpler than the
complexity of learning a good policy but
actually even further assumptions will
be needed to get this going
so that gives us if we imagine the
normal RL setting where we have sort of
roll out workers interacting with an
optimizer that makes our policy good or
trajectories go from the will that
workers to the optimizer and updated
parameters go to the role of workers
that picture just gets expanded by we
take the result in trajectories and send
them to some human Leibler again perhaps
at this point I want to use sort of some
clever active learning scheme to decide
which directories to query at maybe we
want to use pre training to be able to
more quickly fit this predictor which
says what is a human going to do how is
a human going to label those if ever
predictor of human behavior we can hand
that to the optimizer instead of
directly having the human involved in
sort of a trajectory
guess again there might be a label back
from the reward predictor to the human
label or if you want to do procedure by
a switch
what's a label this is an example of a
very simple strategy for hopefully
helping with the sample complexity or
the dependent something that's expensive
it doesn't really deal at all with the
second challenge of wanting to go beyond
humans and it's also certainly not a
whole story for how you would reduce the
you can be okay with having a really
expensive
yeah let's have a question ongoing the
audience okay when you say that do you
mean going beyond any individual human
there's an intrinsic noise and then the
individual human or going beyond
aggregate of an average of a large
number yeah I think there's a lot of
assumptions we could make that sort of
imposed weaker and weaker restrictions
on a human maybe the weakest one you
could imagine is like if you assemble
the largest group you can feasibly
assemble then they have some statistical
edge in every case where they're more
likely to choose the better outcome
that's like sort of maybe not literally
the weakest but one of the weakest
things you can imagine I'm sort of not
happy even with that I'm not happy even
with that but even getting to that you
sort of will have some problem or sample
complexity there becomes primitive you
want to bet that you throw something we
want to do better than that oh that's
right so I think I problem sort of from
this perspective is if you're fixing
that ensemble you're not going to be
scalable in this sense that if your
model costs is substantially better than
humans will definitely start doing the
wrong things are there group of humans
so far why I talks a lot about this
despite not in the long run being
interested in sort of doing what a human
would do or doing what a human thinks is
good is that I think this building block
of like coping with the very expensive
sources of feedback about what is good
is an important building block even if
they want to go beyond humans I mean I
think it is one of that in this regime
we can actually do things that are
theoretically well specified and so
that's nice when everyone can plop a
problem that at least makes sense I'm
willing to assume that the human is
actually telling me which things are
better which outcomes are better now I
can actually start to prove theorems or
at least break down algorithms that way
we can characterize what they're
supposed to do relatively well
so the other thing I spend a lot of time
working on over the last few years
cooking with humans I still don't think
any a lot of people have worked on this
problem it's like a important problem
and lots of domains as part of why I'm
relatively optimistic that there's
something we can take on happening in
the long run
as many wants to ask about this part of
the talk maybe now is a good point to
pause otherwise I'm gonna jump right
into like desire to outperform humans or
how we can even think about what might
work
so our basic problem here now is the
humans are not actually ground truth
that is I can look over the course of
say 10 minutes or some length of
feedback which I can actually get over
the course of a training process but if
I want to evaluate how good is the
situation after 10 minutes and I'm
asking a human a human isn't actually
giving me the ground truth about that
what I actually care about is like you
know after 10 minutes the world has
entered some state I care not just about
the intrinsic value that state but like
what is the value with respect to so if
I want the value function for this like
human evaluation I want to say are we in
a state where like things are going to
go well going forward so I'm going to
lump all that into this utility function
U and say this utility functions
unobserved even if we assume that humans
can tell how intrinsically viable estate
is like they can look at everything and
be like there's good stuff happening in
the world they don't know like how much
good stuff will be happening five years
down the line and we would like a reward
function R such that optimizing our
reward function is equivalent to
optimizing the Sun observe utility
function
I'm going to talk about a possible
approach sort of a waxer assumption we
can make on the humans so that they know
the right answer and then some of the
difficulties involved in that approach
so rather than assuming that a human can
give me correct answers to questions of
the forum which of these two outcomes is
better I can assume that a human could
decompose that heart evaluation task
into slightly easier subtasks and then
likewise feature the subtasks that the
human could decompose evaluation of that
sub task into slightly easier still sub
tasks sort of like saying instead of
making an assumption that a human can
directly answer our questions assume
that they know like consistency checks
amongst these questions if I have that
assumption I could then try and do
something sort of like alpha zero where
I take my hard problem I have my system
try and reform that hard problem and
then to tell how well was doing I asked
a human to decompose it into easier
subtasks use the current model to
perform those subtasks and thereby
defined some really expensive or work
function it's gonna be expensive because
it involves not only consulting a human
it involves using the current model to
perform a whole bunch of subtasks so
it's necessarily going to be at least
some like large constant factor more
expensive than it executing the model
grounded and like an example it's maybe
not the most grounded example in the
sense that it's like a very speculative
long-term example but hopefully gives a
flavor of the kind of decomposition I'm
thinking about I'm happy to talk and
some other examples will conflate her
there a little bit closer to so if I
imagine like some proto Matic tasks that
the human is not very good at like
design some regulation in this industry
which we might in the long run know as
the world gets more complex we like ml
to help us solve this task better we say
like a human can't just look at some
piece of legislation and say was this
good but we hope that a human could say
if I had assistants who are able to
solve a bunch of two subtasks very well
the human can orchestrate that effort in
order to give like slightly better
answers than one of the assistants could
have given directly so I could say like
in order to evaluate how good this
policy is there's a bunch of sub
questions like what are the effects
going to be on various industries what
are other consequences should be paying
attention to for these possible outcomes
how good are these level enforcement
costs be etc like I could hope that each
of these tasks individually is easier
than solving the top level task or
evaluating the top level task that's
that if I have a model with a given
level of sophistication and I apply it
to these subtasks
I get our award signal which is smart
enough to guide the model to improve at
the high level task
I hope that we can continue applying
this down the road so if I like take one
of these subtasks like creatine effects
of regulation industry I can break it
down further and say like what are
activities that are affected what people
would be affected how might they change
their behavior which of those changes
are they most likely to engage in and I
could hopefully just continue drilling
down until I get to questions where I'm
at least happy just accepting human
level behavior on those questions so if
I ask like once I get to very concrete
queries about the world that human can
just go check now I say great I'm happy
just training an m/l system to do it a
human would do but you
this is the kind of thing I'm hoping for
like the kind of setting in which this
assumption about decomposition might be
both plausible and substantially weaker
than an assumption about directly being
able to say which piece of legislation
it seems like this requires a couple
capabilities that we like and I wonder
which one you think is like the harder
one like one is a couple seems like an
understatement
yeah composition with everything you've
decompose it into it's like the kind of
counterfactual question that we don't
like great tools for answering in
general is that the challenge the
decomposition earth the challenge that
these questions are all the nature a
very different problem like boy yes I
think there's it's one part of my
responses I definitely think the hardest
part of building a system which answers
questions like this is probably not
producing an objective that induces good
answers like it's probably like these
tasks are all absurdly hard tasks like
if you throw like some big language
mullet these tasks is going to give you
terrible answers okay that's like the
first first thing is that's like the
hardest part definitely like the biggest
difference between the world now in the
world we can solve these problems so I'm
just used as a motivating example in
that way then more directly your
question like it is the case that all of
these questions are the kinds of things
that are not it's not easy to get a big
data set with answers to these questions
to train a model so you could maybe say
there's two possible reasons existing
models can't solve questions that
involve like this kind of counterfactual
reasoning one would be the models sort
of architectural II there's no way of
setting the weights of the neural
network they'll give good answers to
these questions that's like one
possibility and a second possibility is
like there's no training signal which
actually would induce too little I was
to optimize somebody just cover that so
I'm sort of setting aside or Brackley in
the first question and say it's looking
forward to some world where in fact
there's a setting of the parameters
which we're allowed to answer these
questions which i think is a little bit
of an open question right now it's we
don't really know exactly neural nets
were capable of doing if you have an
objective that incentivizes the right
behavior we're getting quite good at
this next word prediction tasks but that
doesn't imply that were very good at
these other tasks I tried to bracket the
first issue and then the second issue of
like do we can we train roll nuts to
these things can we have objectives I'm
just hoping to recursively continue
applying this approach so I sort of just
learning how to reason about such
counterfactuals by looking at what a
human would do or looking at what
answers the human order card is good to
these questions and so you ground out a
counterfactual question so simple you
can hope to learn them from just
it's more like human saying you know is
this answer good or demonstrating here's
the answer I would have given to that
kind of question those are the two
states think it's reasonable to give
some sort of estimate as to how many
like each time you recursively do this
you're gonna have more subtasks verbs
tests and so you're growing in the
number of possible tasks and then that's
right you'll eventually get to a test
that's straightforward that you could
use the ml model to solve but at this
point we have like two purple a back out
of that recursion we have so many sub
tests that were aggregating together but
actually running this decomposition
would be very impractical if you imagine
you have a branching factor of like 10
and you think you have to go to a depth
of like 20 in order to solve a task
that's a very big number of tasks so if
this I never actually performed this
kind of decomposition more than one
level deep or like it sort of imagine
analogous to a tree search and alpha
zero or you might say we could run this
tree search actually exploring the
entire game tree in this way is deeply
impractical but we hope that a model
which is able to solve this task at the
top would also have a reasonable level
of performance on all of these subtasks
and all their subtasks and so on such
that if we create a model not only to
solve the top task but also to solve all
of the subtasks at the same time the
total capacity that model needs is not
much higher than the capacity a model
would have needed to solve just the top
level tasks and then we can help in
parallel with solving the top tasks were
actually gonna train all of these
subtasks and during training we're going
to use the model to solve the subtasks
that appear rather than actually
expanding out the recursion so I'll next
slide is gonna be like a little bit of a
schematic of the training process which
maybe that I've been before going on if
there any other questions about the last
slide I'm happy to yeah
yeah so again I want to make this
distinction between like first I want to
hope that if there's like upsetting a
parameters in your neural net they could
solve the top level tasks there's a
setting that can also solve the subtasks
to a high enough level of quality to
certify that solution as the top level
tasks and then this is a question of how
you train those tasks but like 16 to
game theoretic question comes up I'm
just asking is there a way that a neural
net can solve that kind of game
theoretic question and then do we have
an objective and incentivizes it the
objective the incentivizes is not going
to be like a traditional game theoretic
objective it's going to be more like ask
a human to evaluate that response or to
generate the possible answer to that
question using the same kind of
decomposition so if you have a game
theory question you're going to ask
ultimately you'd ask like what does the
equilibria thus game you submit that
question to your neural network it
produces an answer a human looks like
that answer now I want to say is that a
good answer so the human is going to ask
some sub questions which might be like
are there better responses for this
other player they would like ask that
sub question in that case we're sort of
implicitly doing tree search in the same
way you currently would so in that case
you're gonna have a more game theoretic
algorithm but it's not a de special case
I mean just sort of the same procedures
used to evaluate all of these answers
namely ask the human how would you
evaluate whether that answer is good and
one kind of thing a human can do was ask
is that in equilibrium
like what are possible deviations if
that makes sense
yes we're not hoping to learn about
these things by any of these things by
feedback in the world we're hoping to
learn sort of humans have some reasoning
process and like part of the game here
is how do we do these things we don't
want to wait 10 years to it happens so
we're hoping to learn about those pairs
of strategies of asking human like is
this a better response that makes you
skeptical of the original equilibrium
and we're not necessarily getting impact
input from the world and we can talk
about that a little bit later how that
comes in you're learning to reason sort
of in the way implicit in what humans
currently believe but I've taken to the
extreme where implicitly assembling this
very large network of tasks
I talked a little bit concretely about
this training process would look like
we're trading some system to take his
input like a question X and produce an
answer Y or whatever just Iggy's input a
video and produce them work for your
robot the way we're going to produce
through a work function we're gonna
construct this expensive reward function
that we then optimized in the same way
we talked about optimizing a human
reward function earlier now instead of
just going to human that pair X Y goes
to human who was allowed to interact
with many copies of the current model
that human interacts with many copies of
the current model to produce a reward
function and then we provide that to the
top model as its signal call to us and
you can either imagine sort of
constantly doing this using the current
model in the sub pieces or you can
imagine doing a training a sequence of
models which we hope are increasingly
competent but the first one is just
trained imitating human subsequent
random to be human interaction with the
first one not to imitate but optimize
this world
yes just like I said this talk started
off much more formal at this point where
to a thing that has pseudocode if you
want to analyze this and say like
whether it works
things are getting quite hairy and the
thing I've thought about a lot they're
not going to get into and as much stuff
as I would like over the next 10 minutes
so I think they're a bunch of big
questions if you want to take this kind
of strategy so one big question is can
we actually use this kind of
decomposition to train a machine
learning model so is this competitive it
does introduce a lot of overhead does it
actually like discover the best model in
the class that answering these questions
we're like how much longer does it take
and if I just had these targets directly
there are a ton of theoretical questions
about whether decomposition is a
plausible strategy at all it's like how
do we articulate the kinds of normative
assumptions we are making for which
tasks should we expect it to be possible
at all how do errors compound if you
imagine a machine that makes errors on
some class of statements with some
probability how do those compounds you
iterate training when can we stop at
saying something like this works in
practice like the strategy makes good
prediction in practice or happy and then
there's a question about reasoning which
maybe in some sense could be a
theoretical question but for now is just
an empirical question which is does
human reasoning in fact to have the
structure where a human can decompose a
heart evaluation task into easier parts
and you so there's some of the big
questions about the overall feasibility
of the strategy you may open the very
long run you end up with like some sort
of working examples and some theory that
explains why they work right now I think
I'm more in a state where like here are
some big problems maybe we see ways to
attack each problem and it's a ways off
do you have like a system which which a
good argument that it's going to work
big entries is multi-man step back and
take questions there's anything about
like what is basically going on what is
the hope here how would this training
strategy work at all
yeah we're stepping back
humans are hard do you have any hope for
proxy of some non-human thing that can
be used yes I guess I would say for me
right now the kinds of tests I'm
thinking about divided into a few
categories so some are like empirical
tests with ml systems today and those
are mostly going to have to use simpler
systems which may be arranged from like
a toy systems to a little bit less toy
what do you say we have some kind of
decomposition and we want to understand
like what properties that decomposition
need to have in order for this training
process to work well so that's one
category of tests on a second category
of tests is like in fact to take humans
and we can even without ml just probe
whether human decompositions have the
kind of structure we want like if you
replace an RL system with a human who's
actually just trying to receive a high
score on some game to find in terms of
other humans like what did the
equilibria that games seem to be we can
sort of have this more ivory sail game
where we probe like are the equilibria
actually good it's kind someone started
to show that it was like bad equilibria
strategies that are yeah so there's like
two kinds of empirical tests we can do
now and then maybe the third kind of
thing is just theory understanding such
decomposition so like especially what
properties would human reason you have
to have there's any question of like on
what time skill do those come together
such the ML systems are actually able to
imitate the kinds of human reasoning or
answer the kinds of questions that
plausibly have the structure there
possibly admit this kind of
decomposition and I think I just like I
would like to be in the place where you
can test that we can test that as
quickly or as frequently as reasonable
so we see whether that's true and when
it's true a little bit all
there are simple domains in which you
can maybe test this even with humans but
I said there are domains like eg if you
imagine some simple estimation problem
where human has been asked like
estimated how many cars would fit like
if you pay of doll of Chicago or
something that you can ask that human if
they're going to solve that would like
to break it into pieces break them into
smaller pieces eventually go to Google
to look something up and you can say can
we use that kind of decomposition
already currently to train machine
learning systems to answer that kind of
estimation of problem and that's the
kind of thing we can test now and sort
of on that spectrum between toy and real
application where the structure that are
made is simple enough you may not need
to use humans in this way to solve it if
you do get to learn some non-trivial
decomposition from humans or to see more
I'm just wondering in engineering yes
there's two kinds of things in that
space that seem to be a one is like
there are lessons from the architecture
of South Korea which is just like
largely about this game of like how do I
break some complex tasks into pieces
which gonna be composed in some way that
is easier to specify then the whole
program and so I do think that's like
part of my optimism about the project or
like there's a lot a lot of what we know
about decomposability comes from people
needing pointy decomposed things for one
reason or another I do think there's
like a challenge there where like in
some sense the workability of the
proposal depends on like can we apply
this in domains that haven't been a
meaningful to automation historically
like are there hard domains where this
breaks down has pasting software is it
partly because you have to be really
creative breaking a system that doesn't
exist here it's an interesting
intellectual
yeah I definitely we can find these hard
demands or it's interesting where
they're like close enough that it's
possible you can apply the same ideas
but not obviously you can just automate
it and then we can ask in those domains
I think there's a lot to learn from the
cases in which people in fact work on
decomposition the sort of similar set of
lessons in the context of like actual
organizations which have to I'm in other
places they can actually hope to get
sort of easier versions of this problem
that don't involve humans from domains
that have this kind of structure already
in software engineering is a potential
source of life on that spectrum from toy
problems to thing we actually care about
software engineering they move a little
bit past Toit what's well it's tough
they actually care about long run I
think about what those
okay I'm gonna talk a little bit for the
rest of the talk I don't know if I
should wrap up at 3 or 45 minutes after
starting and I talked a little bit about
sort of STIs Twitter demands asking in
some sense the simplest part of this
question was if we have this kind of
decomposition structure can we use it to
train models so the easiest to
investigate empirically as I mentioned
before this is sort of a similar
situation to alpha zero where we're just
going to invoke the current model a
whole lot of times in order to generate
training data and alpha zero we use tree
search here we're going to do some
different kind of decomposition it might
wonder like does that strategy work only
because tree search is kind of
optimizing the reward or like quality in
the game as a mana variant or does this
just work in general for similar reasons
yes I would be reasonably optimistic
coming in and I think Cavalier provides
good evidence that it's going likely to
work it is I mean doing RL is a little
bit different than other tasks and it is
nice when you have an invariant that's
going up over the course of training
which is not the case in this
decomposition
so in part just to get experience
implementing this kind of thing and to
see what difficulties came up last year
I worked on simple prototypes systems
trained in this way so if we take some
algorithmic tasks which just very
naturally have this decomposition
structure can we create a system to
solve them using this kind of approach
so the idea is we fixed some simple
demands with natural decomposition for
example like given a permutation to
compute large powers give it a series of
assignments evaluate variables given
some function sum it up over all inputs
matching a certain wildcard pattern and
these are all tasks which the neural net
could be trained to do if I had
input-output examples so if I give you a
permutation at a power and I say what
collectors out a set of 100 million of
those then the transformer can learn to
do any of these tasks but I could ask if
I don't have input-output examples and
instead I'm just handed this
decomposition if you have some Oracle
that implements this decomposition but
like can I use that to train a neural
net roughly as efficiently to solve this
kind of task we're not going to assume
we have any efficient algorithm for
solving these problems and in some cases
like iterating this algorithm doesn't
lead to a sufficiently efficient
algorithm like it array in the given
decomposition and constructing the whole
tree doesn't well and this problem you
want to use dynamic programming if you
want to get an efficient algorithm you
don't just want to actually construct
the little tree can we learn about as
well it's a campus
again this is like a really simple proxy
for these cases we ultimately care about
where we think maybe humans are able to
decompose the task and easier parts but
can't solve it directly just gets like
one tiny aspect of the difficulty here
but the system ends up looking like is
we have some humans or in this case some
Oracle which we're going to count how
many times we call which implements our
decomposition we hand them some problem
that were interested in solving like
given this function what's like the sum
over some large space of inputs the
human that interacts with the current
model and effort to solve that problem
for example the human might say if I
want to compute this sum I would first
cut the domain in half and send each
half to one of my models one copy of my
model so we don't questions into the
human we ask them to try and solve the
problem by interacting with the current
model we then take transcripts of those
interactions and send them to another
model which is going to try and predict
how a human would try and decompose the
problem take that predictor of human we
send it to workers these workers now get
handed questions they try and copy what
the human would do in order to decompose
the problem and interact with the
current model in order to produce the
big data set of questions and answers we
send that to some optimizer
that tries to train the model to imitate
now we're not doing RL we're just to him
supervised learning train the model to
imitate what would have been produced by
this kind of decomposition and then we
send that model both back to the humans
so that they can interact with the
updated model and to the workers that
they can produce better answers
like roughly what the flow of data and
parameters looks like this
again the tasks we worked on were very
toy on these tasks so I guess I should
also say the distribution of questions
which the human answers depends on the
sub questions that occur in the course
of this decomposition there's not a
fixed distribution in to be able to
answer so in the same way they like an
order to play go you do know which
states are going to occur in this game
tree in order to answer these questions
you need to know which questions are
going to occur in these decompositions
so I'm admittedly very Toit asks you
don't have like big ml difficulties
introduced by this recursive process you
trained like roughly as fast to fuse
these decompositions if you have browser
things that's really just isolating one
small part of the way things could have
gone wrong
why did you gain experience with any
the next steps there I think in two
directions one is making the toy demands
more complex and stressing more of the
ways in which this kind of decomposition
might fail especially working with much
larger sets of possible questions so the
non stationarity of that distribution
becomes a real problem and then another
direction is just to actually start
exhibiting decompositions and
interesting domains so a lot of we're
working on right now is working with
language models that can kind of start
to produce the kind of reasoning a human
would engage in when the decompose a
task and trying to move from these
domains to look more toy to domains that
look a little bit more like actual
interesting question answering that's a
large engineering project cool I'm going
to stop there real quick overview so we
can stare at that but we want some
normative assumption there relates
utilities to observations
I don't want use humans as a gold
standard coconut the expensive reward
function seems like a powerful building
block since it lets us start ignoring
the complexity of their word form so we
constructed just focusing on what's a
positive and ordered assumption and then
decomposition is one plausible strategy
for relaxing this assumption of human
optimality but there are lots of open
questions both about does this kind of
thing ever work does human reasoning in
fact have this kind of structure which
we'd need and is this suitable as the
training paradigm internal systems
thanks great we have time for one or two
questions
yeah
I do think you're going to get
non-unique answers in general probably
are like maybe there's a single optimum
in this reward function but like it'll
be something we're like stochasticity in
the sub answers you get will affect the
answers you get at the top level it's my
main take on this is that it's an
empirical there's like an empirical
question of whether humans can in fact
so you take like a complex question it's
much much more efficient for human to
solve that question holistically often
the new gauge in this kind of
decomposition there's an open empirical
question of like is that merely a matter
of efficiency or is it actually like an
essential part of how human cambria is a
good answer
or is there a way that a human can like
to recognize good answers with this kind
of decomposition closely I think at this
point that's an empirical question about
human reasoning on these sort of complex
or open-ended tasks and the main thing
I'm excited about for making headway on
that is just running a bunch of
experiments with humans in order to sort
of identify like kinds of questions for
which the sort of decomposition yields
for answers or about evaluations
no questions geillis Thank You speaker
[Applause]


Stuart Russell - Clarifying AI Alignment

Oh
well it's certainly related I think
often when people hear the phrase value
alignment they think that the the the
goal is to build an AI system whose
values are aligned with those of humans
and I think that leads to two
misconceptions one is that it's the AI
the AI system is kind of adopting the
values right so if you're a vegetarian
family and you buy this domestic robot
that's going to cook food for you you
want it to be a vegetarian even though
it doesn't eat you wanted to have
vegetarian values so that only cooks
vegetarian food that's not the right way
of thinking about it the right way
thinking about it is you want it to
understand what your values are but you
know if your friend next-door borrows it
to do a barbeque you know with lots of
ribs and steaks one weekend when you're
away that's fine right it's not going to
have a you know a real crisis of
conscience about cooking ribs for the
next-door neighbor because it's not
adopting the values it's simply learning
to predict the Preferences of ultimately
you know all seven billion of us right
and we can all be different and that's
fine we it can maintain seven billion
preference models I mean Facebook
already does so that's that's fine the
other thing is that we absolutely do not
expect the machines to have complete and
correct models of the Preferences of the
people that it's on whose behalf it's
working
you're always going to be dealing with
fundamental uncertainty about the true
preferences of the individual and yet
you still need to be useful and one
important point is that if you're
uncertain about the individuals
preferences then it turns out that
you're necessarily deferential to them
because you know that they know more
about their preferences than you do
which means that if they want to switch
you off that's because you're about to
do something that they don't like even
if you don't know that they
like it right and so you're quite happy
to be switched off if you believe you
know the objective perfectly then any
attempt to switch you off would just be
a mistake and therefore it should be
prevented and so there's this absolutely
necessary mathematical connection
between uncertainty about preferences
and the deferential behavior of the
machine


Stuart Russell - What role can psychologists play in AI Alignment?

Oh
so if we think about this connection
between underlying human preferences if
they exist and I think that's also
something that psychologists can help
with and behavior you know that that's
sort of really in one conception that is
our whole cognitive architecture right
our preference is the kind of the
wellspring of our behavior but then
there's all of the learning perception
decision-making and so on that in the
end produces the behavior so if you had
to say well you know we can't really
make progress until we have that
complete model and then we can kind of
invert it so that when we see a behavior
we can infer what are the underlying
preferences of this person that's a
really ambitious long-term project and
we have to wait a long time before we
had finished it if that's what we have
to wait for so I think you have to look
at what are the major deviations from
rational behavior and here I think there
are things we can do so for example we
know that human beings are myopic right
so you know when when Lisa doll lost the
game of go to alphago right he made a
move that lost the game at some point
right he made a move after which there
was no possibility of winning but did he
do that on purpose right did you know if
you thought he was rational then you'd
have to assume that he wanted to lose
but of course we don't think that way we
think he's trying to win but his
computational limitations mean that he
can't always do that perfectly so it's
perfectly obvious to us that people
don't have to be perfectly rational but
you can still figure out what they want
the the biggest deviation I think is
that at any given moment when when a
human being is
behaving they're behaving within a very
restricted context right so here I'm in
the restricted context of I'm doing an
interview so you know there are lots of
other things that people can do but none
of them are even in my decision frame
right now right I mean I could I could
set fire to the building and claim the
insurance money or who knows what right
I mean I could take by I could take my
phone out and start trading stocks in
the middle of the interview just but I'm
in an interview so there's only certain
things that one does in interview and
then well and that interview itself is
part of a larger commitment to this
particular you know Center and the
research project and and so on so in any
given moment we're embedded in actually
several concurrent hierarchies of
commitments and activities and if you
want to understand what someone does in
terms of their preferences you have to
understand what that hierarchy is right
what a you know that there are short
term goals within that particular
decision frame but those short-term
goals exist because of a larger frame
and larger frame and somewhere up there
the the real underlying preferences
about our our long-term future and we
may not even be at all consciously aware
of the connection between those so for
AI system to understand what people are
doing you know even a system you know
the relatively restricted system that
you know helps you with your calendar or
something like that then it needs to
understand what are the activities
you're engaged in you know and where are
you right now in that subroutine so to
speak
and so I think this is something that
psychologists can can really help to
study right what is the what is the
nature of this hierarchy of commitments
how does it vary across individuals how
do we pop in and out you know when do we
decide to abort one commitment and
replace it with another what's the
control structure and so on so these are
these are questions that can be really
helpful II explored with real human
beings


Sino-Western cooperation in AI safety | Brian Tse | EA Global: San Francisco 2019

our next speaker is Brian say he is a
policy affiliate with the Center for the
governance of AI at the University of
Oxford he's also a senior advisor at the
partnership on AI and the author of the
Chinese translation of the open AI
charter his research focuses on
International Cooperation for safe and
beneficial development of advanced AI
and he's been invited to present at both
deepmind and the Asilomar conference on
beneficial AI previously
Brian worked at JPMorgan and a
multibillion-dollar global private
equity firm which specialized in on East
Asia he studied at Harvard University
Tsing hua university and the university
of hong kong excuse me
Brian was a member of the World Economic
Forum's Global shapers community and
today he primarily works on AI strategy
as it relates to global coordination at
the Center for long-term priorities if
you have questions for Brian you can
submit them via the busy bow app and for
now please join me in welcoming to the
stage to speak about improving
coordination with China to reduce AI
risks Brian say
it has been seven decades since a
nuclear weapon has been detonated for
almost four decades parents everywhere
have not needed to worry about their
children time for smallpox the ozone
layer
far from being depleted to the extent
once is feared is expected to recover in
three decades these defense on
non-defense amount greatest achievements
by humanity this achievement would not
have occurred without international
cooperation from a multitude of
different countries the Dutch serve as a
reminder that international cooperation
can benefit every country and every
person living in every country together
we can achieve even more in the next few
decades AI is poised to be one of the
most transformative technologies in the
Chinese language there is a word wage
here which is composed of two characters
one meaning danger and other opportunity
with both of them being presented at a
time of critical juncture with the eye
we must cease to minimize dangers and
captured upsides ensuring that there is
robust coordination between stakeholders
around the world especially those in
China and the West it's critical in
achieving this endeavor so far the idea
for national competition for
technological and military authority has
dominated the public narrative when
people talk about China and AI the
country's ambition to become the leader
in the iPad 2030 is always invoked in
contrast there is very little attention
paid to the call from the country on
international collaboration in security
ethics in confidence of AI which are
areas of major interest I believe it is
a mistake to think that we can either
have international cooperation or
international competition today some
belief that china-us relations can be
described as Selectric adversaries I
believe we need new strategic concepts
to capture the incentive to compete
as well as the need to cooperate Joseph
Knight well known for coining the phrase
soft power has suggested that we use
cooperative bribery as idea to describe
the relationship
Graham Ellison the author of destined
for war has proposed the idea of
coopertition allowing for the
coexistence of competition and
cooperation at the same time so a
deliberate effort to move the world
towards greater coordination or at least
co-op petition is urgently needed in the
rest of my talk I'm going to talk about
three promising areas of common ground
for coordination these are risk from
accidents misuse and raising dynamics of
AIT filament for each of this risk I'm
going to talk about their importance
visibility for coordination and leave
you with some recommendations and last
that was risk from accidents as
deployment of air systems have become
more commonplace the number of accidents
related to AI have also increased for
example on May 6 2010 the Dow Jones
Industrial Average experienced a sudden
crash of trillion dollars and that was
known as flash crash it was partly
caused by a use of a high-frequency
trading algorithms and the impact was
immediately spread to other financial
markets around the world now as the
world becomes increasingly
interdependent as is the case with
financial market local events have
global consequences which demand global
solutions the participation of Baidu in
the Panaji Nai provides a encouraging
history of global collaboration in the
present released last year Baidu
said that the safety and reliability of
air systems it's critical to their
mission and was a major motivation for
them to join the consortium the
companies think of autonomous vehicle
safety as an issue of particular
importance looking at other to use
technologies for inspiration there seems
to be coordination between China and the
US on the crew
security one example is the Chinese
nuclear technology Center in Beijing
which is by far the most extensive
nuclear program receiving tariff funding
from both the US and Chinese government
it focuses on building a robust nuclear
security architecture for the common
security a phyto feature of this
partnership is the intense focus on
technical exchange as well as the
reduction of risk from accidents it is
noteworthy that all of the Chinese air
at the co-principal so far have
emphasized the need to ensure the safety
and reliability of air systems in
particular the Beijing high principles
and the one from 10 cent Research
Institute have highlighted the risk of
AGI systems calling for preconditioning
concerns with this shared understanding
of the risk from accidents I believe
Chinese and international securities can
collaborate through the following ways
first researchers can collaborate at the
increasingly popular AI city workshops
as some of the major machine learning
conferences second there are efforts
such as the AI safety group wolves
coming out from deep mines which allow
labs and researchers to measure and
benchmark the safety properties of
reinforcement learning agents third
international bodies such as ISO
encouraged to continue the effort in
technical standard setting especially
around the reliability of machine
learning systems
lastly Murray Sakura alliances such as
the Polish in AI can facilitate some of
these discussions on best practices
especially through the safety critical
air working group now even if we can
mitigate an intended consequences of AI
systems there is still a possibility
that you'll be misused for example
earlier this year openly I decided not
to release the trained model of GPD -
which is a record-setting language
learning model openly I cited concerns
that you might be misused to impersonate
humans creates misleading news article
or trick victims into revealing their
personal information this motivates a
case for global coordination because
malicious actors from anywhere
have gained access to the technology and
deploy them in other parts of the world
in the future cybersecurity there was a
well authenticity of a global response
to security incidents in 1989 one of the
first computer worms
known as Swank attacked a major American
company the incident prompted the
creation of an international body called
first to facilitate information sharing
and enable more effective response to
future security incidents the origin
first has been one of the major
institutions in the field and currently
lists 10 American and ate Chinese
members including companies and public
institutions another source of optimism
is Korean research view of effects are
examples these are small input samples
that have been moderated slightly in a
way that causes the machine learning
classifier to miss classify it now these
efforts are examples poster critic
concerns because they could be used to
attack a machine learning system without
the attacker having access to the
underlying model fortunately many of the
leading AI labs around the world are
already working hard on this problem for
example in nips 2000m 17 google bring
organized a competition on this research
topic and the team from Chinua
University won the first place in both
the attack and defense tracks of the
competition similar to the risk from
accidents many of the Chinese air
ethical principles have included concern
of the misused risk of AI systems one
promising selling point of coordination
between Chinese and foreign scholars
especially the AI Labs is publication
norms so after the release of opening
eyes ability to model and the policy the
panel Shanaya organized a seminar to
discuss the topic of research openness
there was no immediate concern that to
whether the AI committee should restrict
research openness due to such concerns
however they did agree that if the AI
community moves to that direction then
their review parameters in the norms
should be standardized across the
community and presumably this should be
done across the global air community for
it to be most effective the third type
of risk that I'm going to talk about is
the risk from raising dynamics in AI
development under competitive pressure
they are labs might put aside safety
concerns
in order to stay ahead in the
competition to illustrate this point
there was a fatal self during car crash
in 2018 by uber when the crash happened
Combinator's initially thought that the
increase incredibly brittle vision
systems was a culprit however later
investigations showed that the victim
was detected early enough for the
emergency braking system to work and
could have prevented crash so what
happened eternal that the emergency
braking system was turned off
intentionally by the engineers because
the engineers were afraid that the
ophélie sensitive protein system would
make them look bad as compared to their
competitors so this type of trade-off
between safety and other considerations
to seem to be very concerning especially
if you believe that AI system would be
increasingly powerful this problem is
going to be even higher stick in the
International Security context and we
should seek to draw lessons from
historical analogues for example the
report technology will led by Richard
10sec discusses the norm of no first use
in its contribution to the strategic
stability during the nuclear era notably
China was the first nuclear weapon state
to adopt such a policy back in 1964 with
varying degrees of success other nations
have also used the norm to moderate
their proliferation and use of various
military technologies including binding
lasers their mouths and weapons in outer
space
now with AI as a general purpose
technology there is a further challenge
of verification how do you ensure that
the certain AR technologies that one is
committed in not using can be specified
and verified and we literally the
Chinese Nuclear Posture has been
described as a defense oriented one
now the question with a is is it
technically feasible for parties to
differentially improve defensive
capabilities rather than offensive
capabilities thereby stabilizing the
competitive dynamics I believe these are
still open questions ultimately
constructive coordination depends on the
common knowledge that there is this
shared risk of a race to the bottom on
AI safety and I'm encouraged to see that
there is increasing attention paid to a
problem on both sides of the Pacific
so from China for example madam fooiing
who is a five year person of the Foreign
Affairs Committee and the influential
diplomat has said that Chinese
technologists and policy makers agree
that there is a threat of AR to
humankind at the world peace forum she
further emphasized that the Chinese
believe we should cooperate to prevent
such a threat preemptively there is also
the Beijing high principles which in my
view is the most significant one coming
out from China so far also highlight the
need to avoid a malicious AI race and
this principle has gained support from
some of the major academic and industry
institutions in the country from my
understanding the discussions around
I sold my air principles the book super
intelligence by Nick Bostrom and
warnings from Stephen Hawking and other
thinkers have made a meaningful
influence on the thinking in China now
building common knowledge between
parties is possible as illustrated by
the Sioux City trap con by the scholar
Graham Allison through CDs trap
describes the idea that the rivalry
between established power and a rising
power often result in conflict and this
thesis has captured the attention of
leaders in both Washington DC and
Beijing in 2013 presidencies being to a
group of Western visitors that we should
cooperate to escape from such fu City
trap in parallel I think it is important
for leaders in Silicon Valley if not
Washington DC and Beijing to recognize
that there is this collective action
problem of a a I raced to the
precipice or what I might call the
bostrom trap with this chair
understanding I believe the world can
move into several directions first there
are great initiatives such as the ice
warmer air principles which commit many
of the signatories to the principle of
arms race avoidance expanding the
breadth and depth of set of dialogue
especially between Chinese and Western
supporters will be critical in
stabilizing the expectations of each
other's belief and fostering mutual
trust second AAI safety research
collaborations can be initiated between
labs across the border for example labs
could collaborate on some other topics
lay out in the seminal paper concrete
problems of AAS safety which was itself
a joint effort of multiple intrusions
lastly which is also the most ambitious
one is that leading air labs could
consider adopting policy in open air
Charter the Charter described that if
there is a safety concerns fellow
Alliance a two-year project that comes
close to the following such technology
then open AI then open air I would stop
competing and start assisting with this
project
now this policy is the incredible public
commitment as well as a concrete
mechanism in trying to reduce this
undesirable racing dynamics now through
the talk I have not addressed many of
the complications involved in such a
endeavor there are considerations such
as industrial espionage civil military
fusion and impact on civil liberties I
believe each of those topics deserve a
nuanced balanced and probably separate
discussions given that I will not be
able to do a proper justice to these
topics in a short presentation like this
one that's said on the broader challenge
of overcoming public attention I would
like to share with you a story the Cuban
Missile Crisis was believed by some to
have a one-in-three chance of resulting
in a nuclear war between the US and
Soviet Union after the crisis president
Jeff Kennedy was in a desperate search
for a better way for it
before he was assassinated in one of his
most significant speeches about the
international order he proposed the
strategy a concept of a world safe for
diversity in that world the US and
Soviet Union can compete rigorously but
only peacefully to demonstrate whose
value and system of governance can best
serve the needs of their citizens
this eventually evolved to what became
didn't thought trend which contributed
to the easing tension during the Cold
War in the history of Chinese thought
there is a similar doctrine which is
harmony in diversity or her Putin in
Mandarin so the war must learn to
cooperate to tackle our common
challenges while accepting our
differences if we're able to achieve
this during the Cold War I believe we
should be more hopeful about our
collective future in eternity first
century thank you all right
again questions for Brian you can submit
them through the busy beau app we've got
a few that are coming in so far so I
think last time I saw you was just under
a year ago right and I yeah we were in
London if I remember correctly how do
you think things have gone over the last
year are we on a I think if you were to
take a call back from Philip telex talk
this morning an attentive reader of the
New York Times right you would probably
think things are going very badly in
us-china relations do you think it's as
bad as all that or maybe the news is
hyping it up to be worse than it is
what's your perspective it is endearing
I think the perspective that I will add
to the discussions are two one we're not
only thinking about coordination between
governments so in my talk I was not
focusing on state to state cooperation I
mentioned a lot of potential areas of
collaboration between AI Labs
researchers academia and civil society
and I believe that the incentive and
they willingness to cooperate between
those decoders are still there
that's one thing the second thing is
that my presentation is meant to be for
looking and aspirational which means
that I'm not not looking at the current
news I'm thinking in five to ten years
or even twenty years if AI systems
become increasingly advanced and
powerful which means that there are
potential tremendous upside for everyone
to share as well as potential downside
that everyone should be worried about
the incentive to cooperate or at least
what Co Co opposition call compete I
don't I don't know how to use that for
should be there so I guess for people
who are interested you'll be interesting
to think about game theory such as the
hunt game rather than prisoner die
Manama I wouldn't go into the technical
detail there but the basic idea is that
if there are tremendous upside and also
cheer downside for parties then it is
more likely that parties would be
willing to cooperate instead of just
competing a question from the audience I
don't know if you'll have an opinion on
this but do you think that there's any
way to tell right now whether the US or
the West however you prefer to think
about that or China have an edge in
developing AI and do you think that
there are political or cultural
differences that contribute to that if
you think such a difference exists just
in terms of the potential for developing
capable systems we're not talking about
safety and ethics right well you can
interpret the question I will focus on
capabilities so I think currently it is
quite clear to me that china is nowhere
near the US in terms of the overall air
capabilities and people like drafting
and others have have AG UITs at length I
guess I would add a few things there one
is that if you look at some of the
leadership structure and some of the
recent deployment at Chinese air
companies
for example $0.10 it seems like the
incentive to develop really advanced and
interesting theoretical research are not
really there their air companies are
much more focused on products and like
near-term profit and one example that I
would give is that there was this $0.10
AI lab director Zhang Tom who was quite
interested in like AGI we live well ofin
ideas and he work at $0.10 for like two
years or so and he decided to leave the
AI lab early this year and he is now
going back to academia he's joining Hong
Kong University of Science in Ala G as a
faculty member and even though he didn't
mention the reason explicitly but you
know there are a lot of discussions
around this event and people basically
just think that the incentive Duty for
really long-term interesting research
it's not their attention and honestly at
many of the AI companies now another
point I would raise it that if you look
at some of the u.s. air labs for example
fair or likely will bring the typical
structure is that you have like two
research scientists and then one
research engineer as a team obviously
the ratio could check the ratio could be
the same but the number could be greater
but the ratio of research scientists and
research engineers is the opposite for
Chinese air companies so you have like
one research scientist and like to
research engineers which kind of implies
that there are much more focus on you
know putting their research ideas into
practice into applications so I'm
addressing this question mostly in the
perspective of like long-term a fonzie a
system which is probably the concern for
this audience so that's a kind of
surprising answer to me because I think
the at least the naive you know again
like attentive New York Times reader
point of view would be at least at the
government level Chinese government is
way better than the US government in
of long term planning and priority
setting so if you agree with that how do
you think that translates into a
scenario where the Chinese mega
companies are maybe not doing as much of
that as the American companies sure I
think the Chinese model is too
interesting from a kind of long term
mega project perspective but there are
variances in terms of what type of mega
project you are talking about if you're
talking about railways if you're talking
about bridges infrastructure in general
the Chinese government is incredibly
good at that like you can just build
tons of buildings and just in like days
and I think that it takes us or you can
many other hot governments like maybe
years but but there they are engineering
project right we're not talking about
Nobel prize-winning type of projects and
I think that's really a difference there
are some analysis on like where the top
AI machine learning researchers are
working and all of them are in the US
but if you look at you know pretty good
researchers but you're not top like
potentially Alan Turing prize-winning
researchers then yeah China has a lot of
them so I think we have to be very new
ones in terms of looking at what types
of you know scientific project we're
talking about whether it's mostly about
scientific breakthroughs or like
engineering challenges fascinating a
bunch of questions are coming in via the
app we probably won't get to them all
are you gonna be available for office
hours yes
OB yeah perfect so I'm gonna do my best
to get through as many as I can here one
question is about kind of the general
fracturing of the world that seems to be
happening or bifurcation of the world
into sort of Chinese sphere of influence
which might just be China maybe it's a
few you know surrounding countries and
then kind of the rest of the world
obviously we're seeing Chinese
technology companies getting banned from
American networks so on and so forth
yeah so do you think that that is going
to become a huge problem isn't already a
huge problem or is it not really that
big of a problem after all well it's
definitely concerning and the lens that
I am concerned
is the impact on the international
research community so what I was
alluding to was this pretty
international and like interconnected
Committee of research labs and machine
learning researchers and I believe that
would still be a good mechanism for
coordinating on like different a
applause issues you can think that they
would be great at you know raising
concerns through open letter initiative
they can collaborate through workshops
and so on but this larger political
dynamic might affect them in terms of
well Chinese scientists not being able
to travel in the u.s. they just can't
get visa right and maybe in the future
US scientists might also be worried
about getting associated with with
Chinese individuals so the thing that
I'm worried about is really this channel
of communication between the research
communities yeah lead that will change
yeah you're sort of anticipating the
next question which is the idea that
individuals are maybe starting to become
concerned of if they appear kind of on
either side potentially of the of the
divide that if they appear to kind of
friendly Chinese scientists in America
or write American in China whatever that
they'll then be viewed very suspiciously
and might suffer consequences that so
doing that is already a problem and if
so then what can individuals do to try
to you know bridge this divide without
or at least with while minimizing the
consequences that they might suffer for
trying to do so I feel like it's hard to
give a general answer to this it
probably depends a lot on the
trajectories of individuals and what
other constraints so I guess I could
leave that to office hours all right
a question about the Communist Party so
the questioner sort of assumes that the
Communist Party has kind of final say on
everything that's going on in China I
wonder if you think that's true
and then if that is true how do we work
within that constraint in terms of
international collaboration and what
might be possible yeah well I mean is
there a is there any way to make
progress without the buying of the of
the Communist Party or do you need it
and if you need it how do you get it
yeah I think one assumption is there is
that it is bad to have involvement from
the government that's why we need to
navigate a space and try to avoid that I
think I think that's the assumption of
the question and I can just smell the
assumption when people ask this type of
questions I think it is not necessarily
true
like I think there are ways that the
Chinese government can be involved
meaningfully we just need to be thinking
about what are those spaces again one
promising channel would be basically a
ICT conferences through academia
if general university is interested in
organizing a AI safety conference with
potential buy-in from the government I
think that's fine and I think it's still
a venue for research collaboration and
the world just need to think about what
are the mutual interests and what are
the honestly the magnitude of stakes
that we are worth dealing with yeah at a
minimum the Communist Party has at least
demonstrated awareness of these issues
and it seems to be thinking about them I
think we're a little bit over time
already so maybe just one last question
do you see that this competition
cooperation dynamic and potentially kind
of race to the bottom or race to the
precipice dynamics do those get repeated
across a lot of things in your view
Thursday I obviously there you know in
an earlier era there was nuclear mmm-hmm
rivalry that yeah hasn't necessarily
gone away obviously either but then we
also saw this news item of the first
CRISPR edited right babies being born
and that was a source of a lot of
concern for people that thought my sure
kind of losing control of this
technology so how many of those what's
the portfolio of these sorts of
potential race dynamic problem yeah yeah
I think these are wellif in history and
unlocks but what makes AI a little bit
different is that AI is really a general
purpose technology or like Omni use
technology as people say it and it's
used across the economy is kind of a
question of political economy and not
just international security it's not
just a nuclear weapon space weapon that
could be used in civilian uses or
military purposes but like it's really
everywhere right it's more like
electricity in the industrial revolution
so one thing that I want to add there
which is related to the previous
question is the Chinese response to the
gene-editing
incident from the scientists in Shenzhen
so many people condemned the behaviors
of the scientists because it was a kind
of unilateralis behavior right like he
didn't get enough regulatory compliance
and like he was just doing it at a small
lab in the city but what you can see
there is that there is this uniformity
of an international response on the
incident and the responses from you know
US scientist and UK scientists and
Chinese scientists or basically the same
you know there there was a open letter
on the paper on the journal Nature with
hundreds of hundreds of Chinese
scientists saying that this behavior is
unacceptable and what followed was that
the Chinese government wanted to have a
better regulation of gene editing and
one of the ethics so I think this
illustrate that we can actually have a
much more global dialogue of ethics and
safety around science and technology and
in some cases the Chinese government but
it is interested in joining the school
put a log and takes action with
don't make the policy well that's a very
hopeful and I think appropriate note to
end on so how about a round of applause
for Bryant a thank you very much find
them at office hours for more
you