Governing Transformative AI

hello
my name is mauricio and i'm excited to
welcome you to the talk
governing transformative ai i'm thrilled
to introduce you to marcus andreyan
andrew young is head of operations and
policy engagement
focusing on growing the centre for the
governance of ai at the future of
humanity institute which is based at
oxford university
andrew young focuses on making the
centre's research relevant to important
stakeholders
acting as an enabler for research and
contributing to fhi research
he has a background in history and
philosophy of science with a focus on
the philosophy of economics and
evidence-based policy
today he'll be discussing the field of
ai governance with an emphasis on
long-term risks and opportunities
we invite you as always to use the
software platform during the session
on the right side panel feel free to
share your thoughts on the live
discussion board
and leave an upvote to any questions for
a speaker to answer at the end of the
presentation
after the session we invite you to use
swapcart to set up meetings with fellow
conference attendees
without further ado welcome marcus hello
everyone
good to see you all or i hope rather
it's good for you to see me
i'm going to be talking today about the
field of the governance of ai and
particularly with a focus on more
transformative possibilities
and giving you sort of a showcase of
some of the work going on at the center
for the governance of ai at the future
of madness institute
where i work um
firstly you might start by sort of
thinking a little bit about what vi
governance is
um ensure it sort of refers to how
humanity
can best sort of navigate the transition
to a world with advanced ai systems
um it sort of focuses in particular on
sort of the ethical
economic political military decisions on
ai development and deployment that's
going to sort of
shape the impacts of this this
technology around the world
but it's not just about sort of this
kind of descriptive set of questions is
also about the normative so it's trying
to
understand how these decisions will be
made how this technology will play out
with an eye towards figuring out what
actions we should be taking now
um to sort of make sure that its impacts
are as good as possible
another thing that sort of another kind
of definition i quite like
is more sort of a negative definition so
this is trying to
basically just saying something like um
even if we solve the challenges of
technology
of sort of aligning ai systems to the
intentions of their users for example
we have a whole bunch of problems still
still left to go
and ai governance is trying to
understand what those problems are and
trying to see what we could
could be doing about them
i think one thing that's particularly
useful when thinking about this uh
this field is sort of dividing up risk
into a few different clusters
um one is sort of accident risks um
where
something happens uh despite the sort of
best intentions of their
of their user or the of the creator and
causes some kind of accident that causes
harm
another one is sort of malicious use so
the system does what it's supposed to do
but it's actor or sort of user tries to
have it to do something
something bad something that's harmful
to the world i think one other sort of
category risk that i think is
particularly important to pay attention
to especially if you're sort of
coming out this problem from a a
governance perspective
is sort of structural risk and so these
are risks that sort of
either might basically risk that might
change the incentives of actors
such that bad outcomes happen one
example of this might be sort of
um you uh another sort of important set
of risks that people often think about
is sort of
how strategic or sort of competitive
dynamics
might lead actors to for example under
invest in things that would make sure
that they
their systems are safe and that might
then then cause
to sort of increase the risk of
potentially bad accidents happening
another sort of cluster of things might
be sort of
changing the incentive structure for for
actors in a more more sort of concrete
way and so one one sort of
interesting example of this i think is
thinking about the invention of machine
gun
which was sort of the early ones were
invented by a man called gatling
who actually when he invented it he was
sort of thought this would be a good
thing for the world
um because he reasoned something like um
well
we have a certain amount of bullets
going out on the field
and if i sort of figure out a machine
that could sort of shoot
more bullets per person of soldier that
i have on the field then
surely we could just have fewer soldiers
on the field and then
but then sort of lo and behold this
turned out to be some some fairly poor
reasoning because he he forgot that
uh this sort of efficiency gain might
have just been taken out by having more
bullets in the field
uh which is uh what happened and which
also led to this
sort of uh quite uh defensive dominant
sort of technological suite that we
ended up having
in that in the world war one um in that
legislative trench warfare
uh which led to a whole a whole bunch of
deaths um which i think interestingly
was also partly because
um the actors didn't quite understand
the structure that they were in
they didn't quite understand how
defensive dominant this this technology
suite that they had of a sort of barbed
wire of um
machine gun etc was and so they sort of
kept sending troops and kept hoping that
they'd be able to to break through when
that was going to be very very difficult
i think another thing that's sort of
important to to think about in this
space
is to sort of realize or reflect on all
the ways in which
ai governance is going to be a very very
difficult problem
i think one way to to get that is to
think about ai technology
as a general purpose technology i.e a
technology that's going to have impacts
across
a whole bunch of economic social
military processes
similar to those of electricity for
example and
if that's the case then this is going to
be a very very difficult technology to
to govern more exclusive control
primarily because this sort of harms are
going to be very very diffuse
and also there are going to be benefits
and it's going to be very difficult to
separate these benefits
from the potential harms and it's gonna
be difficult to sort of
see which things are harmful which
things are beneficial and it's also
gonna be difficult to
make sure that actors sort of only do
the beneficial thing
uh rather than the harmful thing as well
um i think this
means um this it's sort of unclear to me
what this means in terms of whether this
is
an important problem to work on on the
one hand it might make it
more even more important to work on if
this is sort of a um
a problem that sort of we end up needing
having to solve uh to be able to get to
uh safe and beneficial outcomes from ai
um or it might be able to uh might just
mean that sort of this we should be
focusing on
other problems instead we should be
focused on sort of
complementary goods to air governments
such as better safety for example
uh yeah i'll go through some examples
that come from
the research group at the center for the
governance of ai
um there are also a bunch of other uh
important and interesting research
groups in this in this space i'll focus
on our
research because i that's the one that i
know best um i'll just sort of
hop through sort of a romp of like
different kinds of uh research parts
that we've been doing but there's a
whole bunch more out there a whole bunch
more that's being done
and a whole bunch more that has been
done in the past that you might find
find really interesting i'm mainly
highlighting things that are sort of
more recent that you might not have
heard of and that i find i find sort of
more interesting at the moment the way
that we sort of try to divide up the
space of research uh mainly comes from
this
research agenda that helen defoe our
director wrote in 2018
which tries to divide up the space in
sort of these four categories so first
is sort of the technical landscape
trying to understand what are the kinds
of capabilities that we're going to have
when so trying to do forecasting and
also trying to understand what actors
are going to have access to
to what capabilities then there's sort
of politics so this is trying to
understand how political actors how
politicians governments uh maybe even
sort of companies
and other actors with a significant
amount of power how they're going to
interact with us with this technology
and what kinds of decisions we should be
expecting them to make um
uh in this and this domain then there's
sort of this other cluster of sort of
ideal governance
so this is trying to understand um what
are sort of the the
kinds of uh places that we want to go so
what are the kinds of norms that we want
to be put in place
what's the kind of institutional design
that we want companies to put in place
so for example
um what are sort of um yeah if
if you were to design an ethics board
for example what should that look like
uh to be able to sort of make sure that
this technology is used for
uh use for good purposes and then so the
fourth question is trying to take all
these
uh these these three different clusters
these three previous clusters
trying to translate them into what do we
do today to try to lead to
some of these sort of more long-term
beneficial outcomes that we're
that we're looking for and
starting with the technical landscape
two people that i find very interesting
uh one is sort of this
um piece or this this work on this
concept of structure transparency
uh which has come out of um a bunch of
different
directions for example a bunch of work
done at fhi over the past few years a
lot of the thinking done by
uh drexler for example and the thing
that you're trying to do here is you're
trying to understand
what are ways in which we might be able
to what are the sort of potential
benefits of the potential effects of
being able to
unbundle information uh more from each
other
um and so one way to sort of exemplify
what we're talking about here is to
think about
bomb sniffer dogs that sort of when they
walk around the airport
they have this really really nice
feature where they they'll be able to
tell a guard
whether a bomb is in a particular bag
and so then we only need to search
that particular bag that the dog thinks
that the mom is in and so this means
that the privacy infringement
for the benefit that we get i.e fewer
bombs on airplanes etc
is is really is really quite small
if we had a different technological
suite for example if there we
didn't know how to train dogs to be able
to sniff
sniff for for bombs then we might end up
in a very different system where maybe
the way that we need to be able to
guarantee that there weren't any
any bombs in the airplane might be to
look inside of everyone's bags
for example um and
this is the kind of thing that we're
thinking about in this space
it's trying to understand um how can we
use new technologies or maybe even just
use
um sort of technologies and sort of uh
and sort of processes that we that we
have today
and that we can we can sort of unbundle
um sort of privacy infringing
information being spread um from the
sort of potential beneficial information
to be spread so you can imagine for
example
um you might set up a system where a
police officer might only be able to
access
a certain kind of um footage
certain kind of studio footage um by
maybe putting in the face
of someone who has been approved as
being a
likely suspect of a crime and in that
case maybe they they
the only thing that they get access to
might be then get the get the videos
where that face
shows up or sort of the time around that
that time and then the thought is that
that might be a lot less privacy
infringing
and so therefore it might come with a
bunch of benefits i think there's a lot
of
interesting uh interesting work here and
in particular uh work that i'm excited
about is sort of thinking about
what the potential effects of having
more technologies
like this might be in the world for
example you might imagine
um sort of one question that
particularly interesting is sort of how
these
potential benefits will be taken out so
will we take this
sort of efficiency gain out in terms of
uh
simply having the sort of roughly the
same amount of privacy infringement
happening in the world
um but we just get more benefit out of
it or is this
these are these kinds of technologies
that on net will be privacy preserving
or not
and what might this kind of technology
do to a democracy versus an
authoritarian state for example
seems like very very important exciting
questions to me
another set of questions looks at this
uh concept of cooperative ai
which is something that alan d for our
director has been working on a bunch
i think one way to motivate it is just
to think about um
the distinction between horizontal and
vertical coordination
and so vertical coordination here is
sort of um
the an ai system doing what what its uh
principle or what its user wants it to
do so this is
at least historically a lot of the work
uh that we've been talking about in the
sort of
technically a safety space focuses on on
this particular question which seems
very important if we don't solve it it
seems like things are going to be bad
but the thought here is that even if we
solve that problem
we're going to have additional problems
which consist of how do we make sure
that there is
cooperation or coordination between
between these actors or between sort of
maybe human ai system pairs
and and other human ai system pairs
and one one sort of way to pull out that
instinct might be that
today you can imagine that sort of uh my
actions are at least
to a pretty large extent um aligned with
with my will
or sort of what i what i want want to
happen or what i want to do
and even if that's the case even though
it's case with with most of humanity we
end up with a lot of conflict
and we end up with wars we end up with
all kinds of kinds of bad stuff
and so if we don't solve this sort of
cooperation problems between ai systems
as sort of
their ability to get things done in the
world increases and then we might be
leaving a bunch of value on the table or
we might be creating sort of unexpected
risks that we weren't uh thinking about
before
and so this is sort of a sort of line of
work that's pushing in the direction of
trying to
trying to build up a field that tries to
solve these kinds of problems so it's
trying to understand how
ai systems can be cooperating with each
other how ais and humans can cooperate
with each other
but also trying to understand how we can
build ai systems that might be able to
help
human to human cooperation so for
example are there ways in which we could
bootstrap or enhance democracy if we
find ways to be able to
extract people's preferences in a more
efficient way in a more
sort of high fidelity way than sort of
someone voting
once every four years for example
i think there's a lot of future
interesting work going on here um
one in particular a bit of work that i'm
excited about is
trying to understand questions that are
important to what the impact of doing
this this work
uh might be in particular if you're
trying to sort of shape the world in
particular russian and so one question
that seems particularly important is
this question of
uh whether we should expect these
capabilities of a sort of ais and be
able to cooperate
uh to be developed sort of by default uh
or whether they're something that we
sort of should put extra energy into
making happen in the water
in the policy in security space um
one sort of set of work that i'm
particularly excited about looks like
sort of similar to this this report that
we recently published called
ai policy levers which looks at what the
u.s government
what their tools might be to try to
shape ai research
development and deployment and this is
basically just a mapping of what are
they what are
all are the different tools what are
those which ones are they most likely to
use
given given what they currently look
like and i think that seems like a very
very important set of questions
primarily because the us government
might be one of the most sort of
important actors
in this in this whole ai governance
problem um so the future work that i'm
very excited about
sort of relates to this is one is just
looking at other actors
um and uh looking at what kinds of tools
they might be
be using in the future so looking at the
eu looking at china seems important
um and sort of having writing these
pieces that people can use that sort of
reference
uh reference work to sort of get a brief
overview of what's what's going on
i think another bit of work that seems
very important to me is trying to
understand
um how the tools that these
actors have access to might change uh if
we're gonna
if we end up in a world where ai systems
end up being very uh very transformative
and so you could imagine that if air
systems end up being very transformative
we'd end up in a situation where um the
sort of
looking at so the past president of how
for example anti-trust legislation has
been used
might not be all that informative um
because
um the legal or sort of the
the policy makers might just simply
change the laws um to sort of suit
suit the current situation or they might
just sort of creatively uh reinterpret
laws that currently have in place so
trying to understand that
i think will be very important in
particular i'm interested in that in the
in the area of antitrust and competition
law
space another piece of work that we've
been uh focusing a lot on and that i
would
recommend you have a look at is looking
at um what we can learn
from other historical analogies that we
might be able to draw
to ai technology for example and so
one example here is a piece of work
looking at the period right after the
second world war
where arguably we have this quite unique
opportunity
where um where the world was sort of
uniquely situated to be able to put
international controls on a very very
powerful technology in nuclear weapons
so we had recently had a world war the
world was
suffering war fatigue we recently
dropped these two bombs that sort of
were had a big impact around the world
and sort of shaped people's
imagination in all kinds of ways and on
top of that we also had
the us which for about six to seven
years had this
this monopoly on nuclear weapons and so
you'd imagine that this would be a
uniquely
useful uniquely important uniquely good
window
uh to put in place international
controls of of nuclear weapons
uh it turned out that this didn't really
work out um and
to to learn more about why that is and
what kind of lessons what might one
might be able to draw from that case
study i would really recommend you
have a look at this report um
looking at the ideal governance space um
we sort of the sort of main way that i
think of this work
is basically looking at
what are the beliefs of different kinds
of actors in the space so for example
looking at
what our machine learning researchers uh
what are their beliefs about
uh what direction they want ai
technology to go what are the public's
beliefs about where this technology
might be able to go or
where where it ought to go and then that
seems like important
inputs into this question of what are
the kinds of
systems the kinds of uh institutional
design the the kinds of uh policies we
should be putting in place over time and
so this is sort of
a piece of a sort of set of work that we
put a bunch of energy into uh for
example uh by baba zhang and
and no immediately thinking about what
kinds of uh what kinds of surveys we
need to be running here we're also more
and more trying to look at the space of
uh
ai policy uh and so one sort of example
of a report here that i'm particularly
excited about is looking at
the extent to which the eu it's for
coming sort of ai legislation which
compared to the world looks like it's
going to be quite stringent
and what global effects that might might
have um there's this thing called the
brussels effect
where in some situations we actually see
something that's kind of the opposite of
sort of a
regulatory race to the bottom uh i.e
that um the eu
putting in place more stringent
legislations end up having a global
effect
primarily because a company that needs
to comply with eu rules
uh ends up not it ends up being cheaper
for that company to then just comply
with those rules
across the world um and so um the sort
of core question that we're interested
in here is just trying to understand
should we expect something similar to
happen in the ai space and if that's the
case
that would be very interesting for a
number of reasons for example it might
just mean that
the eu as an actor becomes a lot more
important than we
than we might have otherwise thought
another piece of work that we've um we
worked on a whole bunch
is this question of uh whether and how
if so a company should be put in place
something called a windfall clause
which is basically trying to have
companies um promise that if they sort
of win the ai jackpot
if they end up seizing and producing a
whole bunch of wealth
by their ai systems um what should we
then in that case they promise that
they'll give back
some of this benefit some of this wealth
to the rest of the world
and the thought here is that this might
be able to sort of reduce incentives
from
from companies but also from government
etc
to sort of race or try to really really
push to be
among the sort of the first or the most
dominant actors in the space
and i think in general uh this is sort
of one mechanism in in this space
of sort of how we might be able to
redistribute um wealth if it's the case
that ai ends up producing
a whole bunch of it that seems important
to explore i think there's a whole bunch
of other other ones we ought to be
exploring as well like for example how
sort of taxation
might be might be used in this space as
well
um yeah so those are some examples of
the kind of work that we're doing and
that one might be able to do in
in this space hopefully those that's
given you giving you some ideas uh if
you want more i'd recommend you
uh have a look at you could have a look
at our website governance.ai
you could also have a look at some of
these uh some of these pieces here and
you could also
just shoot me an email if you're
interested as well thank you all very
much
thank you marcus for laying out this
landscape of ai governance
we've got lots of interesting questions
from the chat so let's dive
in one thing people are wondering about
is generally for undergraduates or early
career professionals
interested in making a positive impact
on ai governance
what advice do you have yeah
thanks uh yeah and thank you all for for
listening to me
to me yamaron about these things um yeah
so
i think there's in general like giving
curious advice to
a huge number of people at once i find
very tricky
um i think the sort of the most useful
things i can say is stuff like
if you want to have an important impact
in especially in this space but i think
in a lot of spaces
in sort of this general area of trying
to do something about essential risk and
whatnot
is uh try to make sure that you keep
engaged with the research coming out
um and that you like as much as possible
try to
um do your do the work required to start
forming inside views
on as many of the sort of crucial
questions that you care about
that's not to say that your inside views
are the ones that are going to matter
the most
uh it's more that like the work that
like
um the benefit that you get out of
trying to build an inside view is like
very it's very very helpful
uh in particular because it yeah because
it means that you'll you'll sort of um
discover things that might be wrong
discover ways that you might be able to
contribute etc so i think in general the
the kinds of advice i would give would
be to try to learn as much as possible
about space
keep engaging with the research go to
conferences like these talk to people
um and then sort of once you've done
that and you're still still sort of
excited about the field then
do do what you can to try to get your
foot in the door somewhere
uh and sort of try out the work um you
could do your dissertation about
something well then those kind of things
great thanks then uh christina from the
audience asks
is there a global initiative or what
kind of work is happening
uh toward creating institutions for
international governance
yeah so there's a there's a decent
amount um so
first there's like a number of groups or
sort of like igos
setting up ai specific bodies um
and so yeah for example we have
something called dpi
uh that was set up fairly recently uh
this the oecd has set up something
called the oecd observatory
um those things are happening uh in my
mind
some of the most important sort of ai
governance
like in my mind the most important
governance that will happen in the world
won't happen at
these kinds of institutions it will
happen at the institutions that just are
doing the global governance
uh so it will happen at institutions
like g8
and you know the general assembly at the
un and these kinds of things
and so i think that's the kind of stuff
that you should be thinking about when
you think about how
yeah how ai will be governing the global
scale
uh-huh great and then another question
we you discuss how a lot of the current
work in ai governance tries to forecast
the the long-term future dynamics of
technological and the political dynamics
of ai
it seems like like a really hard problem
potentially hard enough that people will
have to make
a big governance decisions well before
we figure out these things
uh so is there some point and if so
at what point should the field shift
from big picture forecasting
to more concrete policy work yeah i mean
so um yeah to some extent i think that's
already happening
like uh i think there are you know there
are like important decisions are being
made in the world that presumably will
have
long-term consequences on this field so
the eu
is yeah rolling out a bunch of
legislation on
on how ai will be used and that's likely
to have uh sort of
trajectory changing effects um and
those those effects are likely going to
be important and if one could
sort of uh yeah have an impact on those
that seems important so i think in in
sort of
overall my answer is just that like um
there is
there is productive work to do in sort
of the
sort of more concrete sort of what
should policymakers do today
um there's unproductive work to do there
today and i guess in general just
sort of ask this space um yeah there's a
bunch of different variables that will
change
what kind of work is most valuable when
um and one of those variables is just a
thing of like how much just
strategic insight do we have um and that
will hopefully just sort of go up over
go up over time and that will mean that
sort of as as that goes up then the like
value of the policy work goes up
um but it's also it might also be the
case that sort of you have particularly
important
opportunities to have an impact now it
might also be the case that you want to
sort of
um it might be one thing i'm unsure
about is like the extent which it's
like cheaper to buy influence now or
like if you want to buy
sort of if you want to become a person
who will like end up being important in
future decisions
uh be made in the space um then maybe
you want to invest in that now and maybe
that's
a better investment to do now than to do
in the future and that seems kind of
plausible to me
um and so yeah in short i think there's
a bunch of uh that kind of work that
should be done
should be done now um keeping in mind
that like yeah maybe this
picture isn't just quite as as clear as
you'd like
great thanks let's do one more quick
question on this note
are there actions policymakers can take
now that seem robustly good for
the future of ai um yes i think
i think there are um i think overall my
view is that i would like there to be
more
i would like to have like more thorough
answers to these questions um and so one
thing that i like i'm very excited about
they're happening more and i think is
starting to happen more is like people
doing what i kind of call like policy
development
um research in the sort of long-term ai
governance space
um and yeah i think there's a bunch of
potential actions or there's a bunch of
exciting like pretty
powerful levers that are sort of
starting to be pulled by
by the world so for example in the us
it's likely that we're going to have
some kind of some new stuff going on
without trust um
and that will sort of affect tech
companies in a large way presumably that
will affect the future of ai
um it would seems like a great thing to
have good answers to
which way is is good to go or like what
kinds of antitrust uh like legislation
we want in place
uh for for the future of ai governance
for example um
and so i think that there is yeah a
bunch of that kind of work that would be
helpful some examples of actions that i
think
can be taken today like i think a really
big cluster is
like goes under this umbrella what you
might call like differential
technological development
uh so you try to differentially sort of
boost a certain
direction of the technology so for
example you could imagine that like oh
we want the nsf and whatnot
to invest more in ai safety things or we
might want them to invest in
ways in which you can verify what a
system or what a user is doing with a
certain system
where those that kind of research will
be likely be in in a lot of different
worlds will be very very helpful
to various kinds of governance solutions
that you you're going to want to put in
place
um i also think that there's sort of one
if you're interested in sort of
um my in my case in my mind sort of the
best example of
uh work that does that goes sort of the
full way
from like um tricky high-level strategic
considerations to some like concrete
actions that policymakers can
take today um that's like been done
quite thoroughly it comes out of cset
um which is this this whole bit of work
on
whether or not and to what extent the
u.s should try to keep
uh semiconductor manufacturing equipment
uh and other other processes that might
be needed to create high-end
chips um from uh from like china having
access to this
um and yeah i think i would love to have
more
work that is done as thoroughly on on
other questions um
um that's not to say that like i'm even
though that's the case i'm still not
confident what the what the right call
is for for policymakers
uh but it seems likely that at least
engaging with that work will will
hopefully
improve your uh world picture and
hopefully have you make some better
decisions
yeah thank you that's all we have time
for today so thank you so much for a
wonderful session and for sharing your
time
thank you so much for coming thanks guys


AI Research Considerations for Existential Safety

hello
uh my name is jack and welcome to ai
research considerations for existential
safety
i am excited to introduce you to andrew
kritsch
andrew is a research scientist at the
center for human compatible ai
at uc berkeley and is also the founder
and president of the berkeley
existential risk initiative
his current research interests include
logical uncertainty
open source game theory and mitigating
race dynamics between companies and
nations and ai development
prior to his work at chai andrew worked
in technical ai safety
and financial trading he completed a phd
in mathematics under
barron sturmfields at uc berkeley today
andrew will be discussing some arguments
for technical research areas that he
thinks will be relevant to existential
risk
but which a don't currently fall easily
under the header safety or alignment
and b could benefit from more
existential risk focused technical
attention
we invite you to use the swap card
platform during the session
on the right side panel feel free to
share your thoughts on the live
discussion board
and leave and upload any questions for
our speaker to answer at the end of the
presentation
after the session we invite you to use
swap card to set up meetings with fellow
conference attendees
without further ado welcome andrew
kritsch
yeah thanks so much i'm excited to be
here um everybody
thanks for coming this is a great
conference so far i've really enjoyed
the talks and really enjoying the
context that i get to come into
uh it's so much nicer not to have to
define existential risk
at the beginning of a talk when
everybody already knows what it is
so thank you for that um
in this talk i would like to invite
everybody to take note of the slide
numbers or take screenshots of slides
that you want to ask questions about
because i'm going to keep the slides up
during q a
uh and i'm going to go pretty fast
through the slides uh the conference i'm
told is a little bit behind schedule so
i don't want to push that
and uh so we're going to go fast
uh here goes and
i guess while we're going fast i'd like
you to keep keep track of what my main
goal
for you is as a listener is that if you
are if you're interested in computer
science if you're interested in in
either doing computer science research
or supporting computer science research
that relates to existential risk from ai
in any way i'm hoping to expose you to a
diversity of different
tasks that human society needs to
perform
in order to be existentially safe with
ai technology
and that's going to go beyond it's going
to include alignment but
in my opinion it goes a little bit
beyond alignment and you probably heard
some of that from
from marcus's talk already and i'm a big
fan of their work so you'll hear a
little bit about that too
so here's my very short case for aix
risk
if you've ever studied robotics you know
uh
jack can you tell me if people can see
my mouse while i'm presenting
my mouse visibility yes great so
um in robotics everybody knows you know
you've got your laboratory where you
train and test things to stay in some
kind of safe state while it operates but
then you deploy in the real world and it
gets
it gets outside of that comfort zone of
states in which it knows how to operate
and eventually ends up
crashing or having some kind of an
accident uh
so that lesson
is important for the state of the entire
world as well if you think about earth
as a machine that's not being operated
by a large number of plants animals and
humans and
increasingly ai technology there's
states of the earth in which humans can
exist and i'm going to denote that by
this
human uh this like fuzz here
that denotes humans um and then there's
other many states of the earth
uh that would not allow human beings to
exist like if you imagine if the earth
were suddenly like mars humans wouldn't
be on it
um or if the earth were a random
configuration of atoms
uh humans would not be in that
configuration it would just be
lots of dots and specks so
uh so it's important to realize that
most states of the earth could possibly
be in are states where human beings
simply do not exist and as a result
most ways we could set the world in
motion under the control of an automated
system
will result in the world ending up in
states
where humans don't exist so we have to
do this kind of alignment work
to sort of keep the global not just
individual machines aligned with human
values but the global state of the earth
aligned with this narrow pathway that we
need to take the earth through
in state space so that it remains in a
state where human beings exist
um so i'm going to start by talking
about uh the
the assistance relationship that we want
ai technology to have
to humans you can think of a human as
having
an intention uh this could be this could
be a human institution as well it
doesn't have to be a single human but
it's
something with a coherent intention and
what we what we want is for ai
technology to assist the human or human
stakeholder institution
in having the impact that they want so
the impact of the ai needs to match the
intentions of the human
and some people call this an assistance
game because there's two
players in the game some people call it
impact alignment because you have to
align
the impact of the ai with the intent of
the human
so lots of different names for it but
that's that's that's the situation
um and one approach to this problem is
to sort of break it down into a sub
problem where
let's think where we think about what's
the intention of the ai is the ai trying
to do
something can we design in a way that it
has an intention that matches
the human's intention uh and some people
call it intent alignment
and if you if you can think about the
machine's intentions you can also
wonder is the human able to really
understand what the what the machine is
thinking can the human interpret the
thoughts
of the machine or are its are its
actions legible
to the human so you can think uh you can
think intent alignment is about making
sure that
humans intentions are successfully
transferred into the machine whereas
legibility or interpretable
interpretability work
in machine learning is about making sure
the machine's intentions are understood
by
the human so that for example the human
can correct the machine if its
intentions are off course
or the human can decide whether or not
the machine is ready to be deployed
based on whether its intentions
uh seem competent so that's
interpretability
or legibility and then
you can also ask okay if you have a
machine whose intentions are well
aligned with those of a human and maybe
they're well understood by the human as
well
is the machine competent to safely and
robustly execute
on those intentions so when you hear
people talk about ai safety
most work in the field of artificial
intelligence that's carried out under
the heading safety
is about making sure that the machine
can really safely operate
and not crash in in new in new
situations
so you can imagine decomposing this
impact alignment problem
into an intent alignment problem
followed by
a safety or robustness problem uh with
an interpretability legibility check in
the middle to help
uh to help uh understand what's going on
inside
and uh each of those perspectives gives
rise to a whole different
um set of technical problems and
research questions that people
interested in existential risk
end up working on so i'm not going to go
over all these papers
my slides jack i think it has the
ability to send out the link to my
slides to everybody
so you can just open these up and click
on the links or search
google scholar for any of these paper
titles screenshot them if you want
i'm going to zoom through them but here
are some papers
and books about
what i would consider assistance uh the
assistance problem
uh where we define success for an ai
system in terms of whether
it achieves what humans want uh
and here's some more work coming out of
the center for
human compatible ai and interact at
berkeley these are both berkeley
research groups so we work closely
together
um and we've also got people at deepmind
and open ai
working on what i would consider impact
alignment
uh solutions and uh i mentioned this
last one by paul
you'll see paul's presenting right after
me so don't miss his talk um
and this is a this is a really
interesting work because it's
uh it's very much uh you can view it as
an
intersection between impact alignment
and intent alignment which is something
paul's written about extensively
um you can read these blog posts that
he's written
um and uh the
the previous i'll flip back a slide this
this work here works by
figuring out the human's attention and
then trying to get a a little sprite
here to perform a backflip
um after a lot of uh positive and
negative feedback inputs from human
observers watching it
trying to do a backflip um
here's a couple more pieces of work uh
that you could consider
under the intent alignment header um we
have rohan is also presenting
um today uh talking about how you can
infer
the preferences of humans by just
looking at the present state of the
world as something that humans have
worked towards
um and uh anka dragon at berkeley has a
lot of work on on
intent inference or preference
difference in robotics so instead of a
bunch of papers here's a video
i would highly recommend anyone to watch
uh her work's really fascinating
and now moving on to interpretability i
direct you to
the work of chris ola and cynthia rudin
uh they
they both got really fascinating
approaches to
uh to how to make machine learning
systems more understandable to humans so
we can look inside and say that's a bad
thought don't think that thought or
that doesn't seem like it's going to
work in the wild we're not ready to
deploy
and their their approaches are
counterposed um
chrysol's approach is to look at
something very complex like a neural net
and try to understand
how it works cynthia's approach is to
try and make something understandable
from the get-go and uh you can see from
the title of her paper
uh that these are complementary uh
approaches
um and now moving on to safety and
robustness i would direct you to
uh the work of dario amade at google
brain open ai
um he has this paper with a few other
co-authors including chris and paul
um on concrete problems and ai safety
that was really popular
uh in 2016 as you can see it's got a lot
of citations and it really helped open
up the conversation about
risks from ai in any context even as
something as simple as a car crash
became easier to talk about
after this paper came out um
and i'd also directed to jaime axe work
he's now professor of princeton
uh via berkeley and uh he's got he's got
what i think is going to be pretty
um pretty transformative approach to how
robotic systems and systems operate in
high-dimensional state spaces can remain
safe
while they learn so in you know when we
deploy ai systems in the real world
they're going to have to adapt to the
real world while being safe during the
adaptation process
and jaime's work really hones in on that
issue
also david kruger um is a long time
uh x-risk enthusiast slash pessimist
depending on how you
how you want to label um and
he's got work on uh similar problems
involving
the change of environment that a system
faces from from
training to testing or from testing to
deployment so
so there's all these different areas um
that people are thinking about
and uh the first thing i want you to
take from this is hey there's there's
people to work with there's things to do
uh if you're interested in x-risk if any
of these topics
uh interests you reach out to these
people and their co-authors and try to
get involved in working with them
whether it's on computer science front
or if you have questions about policy
that might
but that might need input on a on a
technical side
um and as these as these areas progress
um what's next is that i think we're
going to start looking at as marcus
mentioned in his talk
as well we're going to look at
interactions between hybrid
human ai teams or institutions uh where
at first you know
like with calculators machines are just
helping us and we're still interacting
with each other a lot but
as you can see with social media uh
technology is becoming interposed in our
interactions such that um
some of the interaction you could say is
being carried out by the machines
and eventually uh the more
um the more our economy allows us to
automate
uh tests the more economic activities ai
capabilities allow us to automate
we could start feeling like we're really
delegating responsibilities
to ai systems almost like there are
personal assistants
and as that happens you know we might
get to a point where you can delegate
the operation of an entire company uh to
ai systems
so we'd have something like ai ceos or
ai government agencies and when we're
delegating
that much responsibility to ai systems
you can imagine situations
where uh where humans uh just get left
out of the picture eventually
and if you if you ask how could that
happen the answer is many many many many
ways
i don't want to limit your imagination
to one story for how humans could not
exist
there are just so many states the world
could be in that don't have humans any
of them except for the small tiny number
of states that you can imagine that do
have humans
are states in which humans are extinct
that said people like stories so i'm
going to try and categorize some of
these states into different story
categories involving conflict and
cooperation
starting with conflict obviously if if
you've watched movies where ais go to
war with each other
we can have a destructive outcome where
um
where a lot of uh humans die as
collateral damage in a war between as
systems
um or or ai systems that have been
designated to
to kill humans um thankfully we have
people working to
prohibit the design and construction of
lethal autonomous weapons
um that is weapons that are allowed to
autonomously decide to kill humans
uh and i think more work on that front
is definitely helpful to
um to reducing existential risk um
and another thing you can do is is try
to complement ai's destructive
capabilities with cooperative
capabilities
so um uh if ai
is if ai technology is enabling
competing states to cooperate better
they might have less reason to go to war
and thankfully we have people from fajin
oxford working with deepmind
to try and pioneer this field of
cooperative ai
it's it's really building on types of
thinking that's
already been done in economics
multi-agent systems game theory social
choice
key machine interaction etc but as an
objective it's something we can all work
together
and make an important we can make it an
important vision
for ai as a field to just help make the
whole world more cooperative
um and one way we can make things more
cooperative
is we can make systems um and
institutions more mutually transparent
if you look at the work of joe halpern
um he's got pretty fundamental
findings in game theory about how um
systems
or agents that are somewhat or
completely mutually transparent are
able to achieve more socially
cooperative outcomes
and so i focus a lot on that in my
research sometimes i use logic
to study what happens when systems
reason about each other
and whether or not they can use that
reasoning to achieve cooperation
um and moving further up the causal
chain from cooperation to mutual
transparency i just want to make a
second plug for these same researchers
chrysol and cynthia rudin because
interpretability doesn't just help you
interpret your ai system it can help you
interpret
a competitor's ai system if they open it
up to you and that can
that can make teams more mutually
transparent to each other if my eyes
my ace my ai system is inspectable to
you and you're as inspectable to me
if they're interpretable during that
inspection then suddenly we can gain a
lot of trust in each other
another area that i think is is upstream
of of x-risk is fairness and
unfortunately
i don't have super x-risk oriented
researchers that i can point to
who are working on fairness right now so
this is something if you want to show up
and be really x-risk oriented
and working in the field of ml fairness
you'll definitely get an applause for me
because i think
there's a lot of reasons why fairness um
it's kind of unfair right for someone to
build an ai system that destroys the
world
while you weren't there right so uh it's
it's a tremendous power imbalance
between
powerful people who can build ai systems
and technologies powerful enough to kill
all humans
and then the rest of the humans who get
killed and i think
work to make technology
more fair in how it's deployed and used
is definitely progress on limiting the
limiting the acceptability of taking an
existential risk with the
rest of the world just because you and
your friends that you work with
think that you're doing a good thing um
so i would definitely encourage more
work in fairness
uh and if we if we are really successful
in things like fairness mutual
transparency and legibility to the point
of really
getting a cooperative um dynamic going
in the way the world uses ai technology
uh if we achieve that it's possible that
the ai technology cooperates so well um
that the entire economy can be automated
and humans get left in the dust i i've
written a blog post about this you can
read more about it but basically here's
a
set of industries that could form a
totally self-contained production loop
uh if we managed to use ai to automate
all companies in those sectors
um so cooperation itself has some
ways of getting out of hand if we
delegate a lot of cooperative behavior
to ai systems and then they don't end up
retaining that safe state for the earth
in which humans can exist
and then they're more complex processes
that are neither cooperation nor
conflict and i think it's an open cat
research
area to identify and categorize those so
i'm not going to try and
taxonomize them here instead i just want
to say
that i think if if you think of the
field of all ai researchers
some ai researchers are going to turn
out to be pivotal they're going to be at
companies
or project government projects or
wherever you can imagine
really transformative ai technology
being developed
these researchers are going to have a
say over how it's used
um and i think one of the most robustly
valuable things we can do to reduce
existential risk
is is to think about you know
which and how many ai researchers are
what i'd call ex-conscious or
existentially conscious you have
environmentally conscious
as a as an important um problem
uh important perspective for humans to
have i'd like to expand that to just
being existentially conscious
and think you know how many and and
which ai researchers are extensively
conscious
and uh and work on expanding that set
because the more
ai researchers are existentially
conscious the more the higher fraction
of those pivotal researchers
um will be existentially conscious and
here's if you can screenshot this i'm
not going to go into it but here's my
super
super rough back of the envelope
calculation for how expanding the number
of existentially conscious ai
researchers
is linearly helpful to reducing
existential risk
i don't know how helpful it is i don't
know if it's
if you get massive increasing returns to
just getting a few existentially
conscious
researchers or or if actually you need
all of the ai researchers to be
existentially conscious in order to in
order to have full existential safety
i really don't know i just put these
plots here to signal how much i don't
know
but it does seem right to me that
whatever way you draw it
more existentially conscious ai
researchers is good and it goes from
zero to one
in terms of risk um to safety
as the percentage of of
conscious existentially conscious
researchers expands
so if you are interested in existential
if you're existentially conscious
um and you're interested in ai research
you know that could be pertinent
to existential safety i would say
don't stress too much about which of
these areas you work on
if any of these topics excite you or you
feel like you have insight
that you can contribute to these topics
just just dive in
and you know get into a good phd program
or if you're already a faculty
member just reach out to colleagues and
start collaborating with them
um and i don't think i don't think it's
it seems
way more important to me to get involved
in ai research
uh and and you know maintain your
your interest in existential safety uh
as it relates to your research than it
is to
to find right away the most high you
know the highest impact
ai research that you can do you you'll
you'll get there with
time once you have uh enough
experience under your belt um so
um let's just work to grow this set of
people
um for some reason uh
these cartoons just appeared backwards
but i don't know
who's going to be pivotal uh i just i
just
i just want to say i really like all
these people here and i'd like to see
more of them so thank you for for your
time and
i'll go to questions now
thank you so much for that presentation
um
cool yeah we have um there's a pretty
popular question
um what would x-risk oriented fairness
research
look like got it and so
let's go back to my fairness slide um
here's
a bunch of uh here's a bunch of
papers that i think are important to
existential safety even though
the authors weren't necessarily
targeting existential safety as their
purpose
i think um successes in defining
fairness
uh successes in in uh
in clarifying the fact that there are
many different conflicting
notions of fairness uh or is
is important because eventually there's
going to be a lot of debate over what
powerful i mean you can already see in
ai policy there's a lot of debate over
what powerful ai systems should do
including debates over what what kinds
of fairness we want to prioritize and so
you can imagine building ai systems that
listen to that debate
and take into account all the
perspectives
and make and comes back with its own
proposal hey everybody here's a proposal
uh it takes into account a lot of
different beliefs
i call this negotiable reinforcement
learning if it's a reinforcement
learning system i have a paper on it
um and uh if you do take into account
everybody's beliefs bi system could
could make a proposal that everybody
likes um and
uh so that's an example uh area that i
think is
is very underexplored um and and these
are all
these are all successes i think in just
the co developing the concept of
fairness
in a way that i think will benefit the
structure and integrity of society which
in turn
then helps us work together minimize
extras
cool um all right that's actually all uh
the time that we have for this session
um thank you so much for the for the
presentation and for your time with us
today thank you all for coming
thank you jay