What are the Key AGI Safety Research Priorities?

I would like to start with the question
like what is your frame on a on a GI
safety that what is the frame on a GI
safety that bribes how you come up with
research priorities and I think I'll
just go this way um sure yeah I think
there is yeah I guess there's a lot of
different levels to answer this question
at I think one question is sort of what
is the you know what's the goal we
should be aiming for um so I'm
definitely in the camp where we want to
be aiming for something like porridge
ability you know like I've grown did a
did a pretty good job explaining that
like we want to make systems which in
some sense are trying to do the best
thing and to help us rather than you
know learning the perfect values to
optimize for um I guess another let me
just say one way of saying what what
drives your research priorities is that
you thought from the goal right and then
you're like okay what are the problems
that we need solving or just get that
goal and your goal is a very tough level
goal of like we want a GI to go really
well the future yeah so I think I think
that like I'm a pretty big fan of the
sort of top-down approach where we're
sort of thing what's what's the what's
the big goal and then like and then
let's let's try to write down as much of
a description of how we could train safe
AGI and save powerfully I system and
then and then Steve what's missing right
which which piece is there are we are we
missing what what constraints that
top-level goal what's on the strategy
strategy could be like write a safe AGI
and then see where you gets done yeah do
you have a version of what drives your
research priorities um yeah so I guess
first the the kind of question that I'm
that I feel like I'm trying to answer
like the way that I define AGI safety is
similar to the way that I would define
like car safety when you talk about car
safety you have roughly these like two
camps of these two clusters
of possible outcomes there's like the
car crashes and people die and there's
like you get where you're going and the
car doesn't crash and we don't focus on
things like how comfortable the car is
stuff like that so it's this this top
level like oh that there's there's two
different clusters and outcomes and the
main thing that like the word safety
means to me is something like how do we
like affect the probability of which of
these clusters we end up in and this
might not be the same as like actually
what is good because there might be
important things that are where we end
up with in each Buster but like where I
think of safety is like which cluster
doing enough and how do we affect the
probability of which cluster we end up
in so what I hear there's this an
important there's like a catastrophic
outcomes like avoiding catastrophic
outcomes is one of the things describing
you and we when we try to predict the
future in like what is gonna happen with
AGI we can we don't know what's going to
happen there's like the future is very
uncertain but we can discern some
structure and the structure tells us
like there may be many catastrophic
outcomes over here and they may be like
ways to avoid catastrophic happens
altogether over here and we want to try
to keep the probability of getting over
there yeah yeah I'm yeah yeah and then
like after like when I'm like trying to
answer the question like in my day to
day a lot of the way in which I'm
approaching how to do this involves
trying to steer into the parts of the
things that I'm thinking about parts of
the problem that are that feel least the
least crystallized to me and feel the
most confusing and kind of try to like
always kind of push in the direction of
noticing when I'm confused
and try to like open that up so much I
think much more forward Jamie as opposed
to backward cainy right yeah Daniel was
I think it's about backward chaining
from the goal and I want to hear from
you is curiosity driven like where am I
most confused by the way of that Eric my
background is in engineering and and
here's to work by backward training
goals that's very much my orientation
and ICAI safety as a problem of finding
safe ads from where we are to leave the
kind of future that we're looking for
and that means paying attention to how
AI systems are being developed today
which is from research and development
processes what they do is bounded asked
and bound in time with bounded resources
what I think of as providing services
noticing that those services can include
the service of developing new services
as we move forward the higher level AI
and that that is a form of general
intelligence if you have a general
ability to develop new services that's
the general ability to instruct systems
that perform new tasks and further this
of what this this frame the AI services
model if you look at the world through
that lens it's possible to see how we
can move orde asymptotically recursive
improvement in a way that is compatible
with the problems the limitations that
Yahshua is emphasized this place is the
intelligence of flexible ability to
generate new kinds of systems in these
episodes Tim level not inside a box but
inside a increasingly automated research
and development system and this just
provides a different context for
thinking about a wide range of
approaches day eye safety I didn't say
all of the above AI services about
services who are has emphasized that
you'd like systems to perform paths for
people that sort of baked into the cake
of the description of AI services Seward
is examining how do we get distance to
do that that's part of an AI services
research agenda and one can frame pretty
much everything that people are doing as
something it fits into the spandrel
picture something that I think can be
understood better at priorities better
for his new affordances new problems if
you through this - say that what you're
doing is having noticed that the ADI
safety or ai safety community has as a
particular frame on how they approach
the whole problem and how that does not
match what you see happening in the
world then that's kind of given you the
research priority
of articulate and explain a different
Braemore like the discrepancy between
the two youth the agent focused you and
the services book yes and the agents or
his product and recognize that human
level intelligence gives us super
intelligent level performance
intelligent systems that can get us the
moon but their systems of people who are
relatively specialized and are in effect
before providing services to
organizations in society so it puts the
focus on this isn't and what thank you
especially parity I think I'm someone
Erica thing um I have to confess that I
work on e time safety as a bit of a
side-effect of working on getting things
like robots and maybe more and more
capable eyes to do the things that we
want them to do and I get to energy I
safety only because as capable I imagine
as capability improve you know you had
these problems that we're hitting just
more and more and more so but I tend to
work on cars and truth it turns out then
you try it when you try to write an
objective function for a car um
eventually it when it optimizes that you
hit some sort of edge case where it
doesn't do what you want and then you
say oh man I didn't think about this
other thing let me add it into the
objective and then you rinse and repeat
and I work on robots that move around
people so I hear what you're saying is
like start with applications or like
what is the the next interesting thing
that we might want a AI based systems to
do and with a view to life we might want
to scale these off in capability by far
in the future what kinds of things are
going wrong now that are relevant to
that and how do I fix them a little bit
maybe I'd rephrase it as I mainly do I
mean II try to get robots to do stuff
right they're not doing it very well and
part of the problem is very similar to
the kind of problem I would anticipate
that more and more capable I systems
would also have okay thank you I'd like
to open it to the audience to now like
just give your thoughts or comments or
questions for the panel I'm not a Sarah
like and I don't feel free to jump in if
you think you're interested in the
question but no obligation
for the good instructions my question is
mostly for Scott I think I'm curious
what you have thought of if you can
describe kind of qualitatively what are
some patterns of things that underlie
catastrophic risks or existential risks
or low-level um I think I'm gonna take
the no pressure right yeah so I think
who's asking about like what other what
are the details of things that you're
confused about like we can get to that a
bit later
anyone else have questions or comments
on what people is dead I have a question
about these two paradigms where I'm the
agent-based paradigm there might be
something in between on something
different which is extended AI or the
extended human or a time where we have
very couple services so they are not
decoupled from us yes so coupled with us
and maybe humans we might become a
danger I mean another variant of the
agent-based situation beyond that which
is like I'm very interested in kind of
extending the power of all of the humans
by kind of building better and better
tools I see this is compatible with
Earth Eric's model which is like where
as a civilization going towards by
building better and better automated
tools and this can be seen as a GI but I
think the point that Eric is making and
make feels really great you know I'm
wrong is that when you think about the
safety problems if you take a frame of
like we're going to make a single agent
safe that's very different from thinking
about we're going to be dealing with a
world in which there are many tools that
we can call on and something unsafe
might happen as a result those interact
as if you could make an agent they if
you haven't made the world safe the
world isn't necessarily adopting all of
your safety procedures and they are
necessarily pursuing that goals that are
lined with yours one of these services a
service for someone else might be to
kill you
yeah that's a service service of the
system might be to give you what you
want but really didn't want service yeah
can we get another I think another
variant of what you were suggesting is
what Dylan was saying in his stock which
is come up with a definition of
rationality what do we mean by an agent
that's really not just the agent itself
in a box around it but really a human
robot system because ultimately that's
what we're trying to build yeah I would
like to jump like one level of meta and
ask you a question that I struggle
myself quite a bit is how do you scale
the AI safety ecosystem in terms of
number of insights that it produces so
like to the degree that you have thought
about it like what are your thoughts I
can I get David okay so my question is
sort of I understood the the prompt of
this panel to be like what is the most
important thing to work on it is sort of
like this having question of like you
know what is the low-hanging fruit
what's most important and neglected and
then you can ask the follow-up of like
why aren't you working on it
so I'm curious to hear people's thoughts
about that they're like what is your
having question what is your having
trouble with regards to HIV yeah so in
particularly what is what is most
neglected and what is like most
important to work on right now given all
of the context and circumstance that
we're in I want to point out something
about the the picture of like how do we
scale up is actually very complicated
related to like incentives and
researchers such there's the thing that
that I spend a lot of my time doing that
might from a view of like how do we
scale up be viewed as like affecting
against this like scaling up system
where I'm like very focused on like
getting my own understanding better and
getting the understanding of like my
local team better and like not like
focusing on like writing papers that
would like to make
other people to be able to get a bunch
of stuff too
I think I'm making the right choice
there but like I don't know and I think
there's like a lot of benefits that are
coming out of that not just in how much
it affects my time but also in whether
or not I'm creating an incentive in
myself to be able to like create the
kind of output that other people would
approve of and like when you're thinking
about scaling up there's there's this
counter this counter thing which is like
there's actually some benefit the silos
in how they allow you to be able to
think I think young question was as I
understood your question is like what
are the generators for Moorea AG I say
to be progress on the planet and Scott
is saying like well there's kind of two
views on that one of the views is that
we need more AGI safety research as we
need ways to like train them up and make
them do stuff and we need them you
negate their ideas and get better at
life building a common common shared
knowledge of what the problem is and
what the solutions are and there's
another view which is that Adi safety
it's a really difficult problem and we
need like the solutions and the problem
to be understood really well in at least
a few heads in order to make like
progress of the form of knowledge
building and that's what I hear starts
things like I'm ignoring all of this
part in order to concentrate all my
efforts on like making an incremental
step in fundamental knowledge maybe I
can give a different kind of answer to
that question which is I think we you
know might be at an exciting point where
we're able to leverage or kinds of
talent and a wider rind range of
people's abilities for or answering at
least certain kinds of AG ask a few
questions like you know on the opening
safety team we're starting to do a lot
more concrete experiments with like sort
of trying to prank to write down roughly
what good what good a system look like
that that could through say Fiji I if we
scaled it up and then just doing
experiments with that and that's one
time where we where we can use ml
engineers and people that aren't
necessarily you know off board doing
this sort of really high level
conceptual progress but instead you know
just just want to write some code I
think we can use
use social scientists or answering some
of the questions about how do you know
how do things like to be play out in
practice like if we have this this
zero-sum game set up between between two
agents and a human asset judge like what
norms can we set up so that this debate
is like maximally truth seeking and
helpful um so I think there's a lot of
room for including more and more people
like there's a portfolio approach to AGI
safeties like we don't understand the
problem but we have like many different
frames and hooks on it and some of them
are more amenable to bringing
researchers in right now orally even
bringing engineers in right now and just
like give us some empirical evidence or
these kinds of questions and like this
will help us by the research and third I
agree with that and I think like it's
good to have some silos it's some people
working very hard and very deep
technical problems and then it's also
good to have like a bunch of exploration
of what the different possibilities are
like that okay I think may be related or
building on everything I was sad I can't
help myself was saying this but we're
maybe not doing a great job tapping into
all of the folks doing AI research and
robotics research who largely think that
we're here talking about the Terminator
and robot apocalypse as opposed to
talking about clearly stated technical
problems I think in a sense if we
communicated better about what those
technical career technical problems or
I'm why they're important we might do a
better job getting more people excited
about it and and therefore make more
progress so just unlike David
Kruger's question earlier unlike what is
your having problem what do you think is
the most important thing to work on and
you like okay in the in the position if
like I'm gonna communicate what the
technical problems are would you like
which is the most important problem you
would work on it like how are you yes
let me give a little bit of context on
on that which relates to also how do you
then communicate these things right what
are the technical problems and I think
part of my approach so far has been to
say people are working on building
technology X let me play this this game
where I assume that they've done a good
job and that's done
what is the next problem that I see so
so that's kind of my generator for
technical problems in AI safety right
the first one was you know some
optimization or RL works enough what's
the problem okay well the objective we
have no idea how to write it down and
then we know what it does things like
searle and so on what Stewart and Dylan
and and I think if we assume that we can
do a good job at that then my next thing
which is the thing that I'm very excited
about right now is how do you define a
good space of objective functions or of
constraints or whatever to begin with I
mean I think when Rohan was talking
about that you could either say I want
to do what's good or I want to satisfy
these constraints that avoid catastrophe
I think it's a bit of a red herring
herring to say that those are actually
different things than one is easier than
the other they're both really hard to
specify and I think you know with with
China that you get two challenges one is
what are good human models that help us
link what people do and all the
information we get from people to these
things that are in their heads yeah but
maybe number two is what is the
hypothesis space and that's something
that I tend to struggle with what you're
talking about there I think is something
close to what I'm very interested in
which is a specification problems like
how do we even write down what we want
and what are the different ways of doing
that related to stay on the topic I
would like to actually like to do tie
together what Dave was saying and and
John a little bit Dave was asking you
know what's most important I would ask
which is more important your heart your
lungs or your liver we need all of these
in a car which is more important this
year and we'll the the engine the brakes
and in a you look through the lens of
the services model you see many kinds of
systems many kinds of problems many
different ways affordances for
addressing problems
you were saying assuming that various
problems are solved what are other
problems I think we should assume in
many context that we someone out there
is able to build a good predictive model
of human concerns that envy is not an
agent predictive model but can serve as
an advisory Oracle to systems that are
acting in the world so just very
generally there are many different items
of
problems humans ills and interests are
not unabled there's a lot of surface
area on this problem and I think what's
most important in this immediate context
is to look at the broader context and
for people to think about the fit
between this broad problem spaced and
their interest with the right weighting
of contact yeah so I'd like to get it
okay let's go the response I like pretty
like I pretty broadly agree with that
and like I think it does affect like
things that I work on I just wanted like
broad a warning that I I think I see a
lot of people like replaced the notion
of comparative advantage with like what
am i bestop and that's and I think
that's a very common mistake and I want
to just like black that there's a common
mistake to say compared to advantage
means like the thing that I'm best at as
opposed to a combination between the
thing that I'm best at and the thing of
the world actually needs and like I just
want to like flag that possible failure
mode yeah that's a good point about
having Baron we could the comment from
Almaty and already answers but you think
the we don't have to I just made the
safety of the AG AI and AI is the almost
same way and we don't have to dismiss it
okay so I your point I'll write down
your point is about like is there
difference between AI safety and AGI
safety you want you wanna think if
anyone this is the question of being
switching sides or on for the past 20
knots how much safety is actually
capacity so for example we discussed
robustness robustness to poisoning
however recent results will show that
you have to trade all trade off for
example or bust nurse to poisoning if
you want to have savings opportunity or
you have to trade off safe intro teacher
or what's not to poisoning and discuss
this with Yan and some of you we ended
up staying a maybe robustness to things
like poisoning and adversarial example
are just like capacity capacity which
convinced me so how much of these is
actually capacity
that could take you a question - meaning
like how much and progress in AI
actually helped us with AG I say thank
you so much as so my question is about
non-technical AGI safety and the
relationship it has to the research that
you're doing so I'm curious whether you
believe there's a gap between the
theoretical research that's being
conducted and the kind of institutional
economic political project of building
AGI and yeah just hear your thoughts on
that topic
thank you any thoughts on what I just
feel work interactive nah I think a lot
about the context of governance and
again matter of reframing it's one frame
to say we're waiting for breakthroughs
or I don't know if you're looking toward
recursive improvement a self improving
agent which looks very difficult Joshua
says we don't know how to do that if one
is looking toward through the lens again
if the AI services model one sees a path
to accelerating incremental automation
of AI are indeed that looks like
something that moves toward recursive
improvement without breakthroughs it
looks like a more distributed process
that increases the urgency of problems
if the problems and puts more of the
questions in the domain of systemic
interactions which looks more like a
governance problem yes I know that your
job title is AI macro strategy so yeah
governance in the whole you're a sea
grant AI a macro strategy yeah so on the
question of like AGI versus AI safety
does anyone find that if you're
restricted to the focus on AGI how does
that if like what you find most exciting
or confusing or interesting work on yeah
I mean I think well so this like a
question between like AI and agin like
ow
general general a range of tasks you can
solve but I think in my view maybe the
more interesting question is now what
level of capability are we operating at
and I think there are some problems that
go up
when you go well beyond human
capabilities or like when you're trying
to solve asked that that need to take
account you know you amount of
consideration that humans been no longer
meaningfully overspeed and I think at
that point some things get a lot harder
right like it's a lot harder to provide
good human feedback and so a lot of our
approaches are about having the ice
itself hope the human you have better
feedback um so I think that's I think
it's absolutely worth thinking about
those kinds of problems that's that I
think a lot of a lot of work on or or
you know some human mobile capabilities
or is helpful for not
brightest to do that as a force of
service yeah I would like to interject a
little little remark of that for they're
extremely brief I just want to pass on
that because I think it's not about AGI
it's just the it can be narrow but very
capable I I this particular way and then
you get problems like you know I have
people push on my robots to correct them
and learn what what they want from that
and it's not the kind of feedback that I
can anticipate you make a very different
task with a or where the robots ought to
say is now much more capable you need
elicit different kinds of feedback and
interpret them differently I think
there's certainly a view that there's a
there's a view that there's a spectrum
from like you know near-term or AI or
narrow or whatever word we used for
problems that occur in systems of this
kind all the way along and including
versions of the problems that are also
applicable to be like far future or AGI
or highly capable systems this is a view
that I've just heard to be like thing I
would like to do research like all along
that spectrum and I'm motivated by the
ones in the near term I'd like to know
if there's anyone that finally like
strongly disagrees with that that
spectrum or thinks that they're like
important problems that they're
particularly motivated by that only
appear when you consider highly capable
or highly general AI have a comment that
I think very much builds on that which
is that the services model says that
complex high-level very general
capabilities are composed of of pasts
and there's a kind of divide and conquer
strategy built into that into kind of
offense for learning in alignment if I
have a system which is is asked with
getting people from place placed very
rapidly you could say well what about
its goals doing things very rapidly and
patent you need to know about urban
environments so on but if what is
calling on is the service of driving
people she was in what what mode and
choosing driving and we have systems
that have learned through drive to in
accordance with with you know human
values and traffic laws and so on now we
have a compositionality that they a that
many other tasks can be exploited by
high-level systems
high-level systems don't yak to know
about all the details of alignment all
the way down they're intrinsically more
safe because they're using in effect
thief operators seems to me that the
services model is very much related to
what we've seen in life factored of
mission and in iterative amplification
it's like those diagrams intuition that
like breaking problems down into parts
that I see across all those I'd like to
get back to that question of like yeah
given those models does that mean that
we can like work on the general problem
of breaking problems down into
subproblems and and then not have to
worry about anything extra that comes
with like I like equal wage guy I think
this is not a direct answer it's the
crime in the space for the last three
question which is that like I I think
that I I don't see a strong connection
between the kind of solutions that would
it would attack like current current
narrow a narrow AI and a GI problem but
I do see a lot of value in using AI as
like a source of intuitions using narrow
AI as a source of intuitions and data
that we can use to like make our model
better about what AGI is gonna be like
and so bringing back to the other
question without the human touch I think
also we can like look at humans as a
source of what things are gonna be like
and we can also look at mathematical
models all sorts of things gonna be like
and I think that we're very kind of
starved of good data because we're
trying to like solve all this problem
ahead of time and and we kind of have to
like take a we can get right I'd like to
make a comment that I think may a
corrective potential misfit
misconception or misperception if you
have a system of services that is in
fact operating at a high level has high
capabilities there are affordances for
control their affordances for better
understanding but there is build of
potential for emergent behaviors of
various kinds any of which look like
classic HDI problems I very much want to
understand from that perspective better
I think it will help inform our
understanding of systems of business I
think you were I noticed you had your
hand up earlier that's the other
question to ask I think this cuts across
everyone's viewpoint and particularly
yawns question about how do we sort of
scale up the volume and rate of
production of interesting ideas about
safety it seems to me that the way to do
this is to encourage a mindset in the
broader area ai community that an unsafe
AI system is a bad AI system and one way
to do that is just to think about things
like liability this comes back to
governance and regulation and liability
so when you think about car
manufacturers you know the Pinto lawsuit
that said no you can't make a completely
fragile gas tank that blows up when
someone rear-ends you at five miles an
hour
that meant that gas tank design got a
lot safer
you know brake design in cars has been
pretty safe a complete break pay layer
leading to a serious accident is quite
rare I would say on the other hand tire
blowouts probably a more frequent than
they should be and I'm sure there are
other kinds of failures so if we think
about canonical kinds of applications
where people are starting to put out AI
systems in the real world and probably
given the way the software industry
works completely denying any liability
for anything the system software does
though for example if you you have
Alexa
Alexa buys a hundred and fifty thousand
dollar Ferrari because the television
tells it to rather than the owner tells
it to you know do you think I guess
Amazon right they produce Alexa do you
think Amazon is then liable for the
Ferrari probably they deny all liability
they just say well you shouldn't put it
next to the television or you shouldn't
have had it tuned to that channel or you
know and so on so forth no yes I think
if we look at these applications they
eat our you know eat your own dog food
build applications and then actually use
them in the real world give them your
credit card you know put your kids in
them you know put your senior executives
in the self-driving car and have them
drive having be driven around with no
but no human safety driver yeah and I
think that would encourage the much
larger number of Engineers to just like
bridge builders all think about safety
of the bridge it's not like there's two
bridge builders who think about safety
and another 50,000 bridge builders who
couldn't care less
they all do I think the thrust of your
comment is about changing minds best and
there's changing mindsets of be like
whoever is developing AI at the moment
so that AI sake the NHS I think you're
both within what they consider part of
the problem and also like changing the
mindset of the idea would be that we
you're using the example of liability or
the knowledge of the liability and of
bringing the safety minds better more
widely
can you comment on trying to
characterize the transition points where
various types of safety concerns will
actually become relevant and perhaps
trying to makes them more formalized
a way to communicate that to the ml
community at large No so we yeah I mean
I think we've got a couple of questions
now and the panel feel free to answer
whichever you want
I'd like to end with if you want to any
thoughts on who would be like your ideal
student or ideal hire or like the one
thing that you if you could press a
button make it
you wouldn't make happen to improve AG I
think anyone has any bad don't pick on L
Matty's question as I understood it and
I think the way I think about it is a
trade off among three different things
on the one hand you have you want to
have any kind of guarantees on safety or
you want to be robust on the other hand
you want to do well with respect to the
actual tasks in a sense the way I think
about is the actual ground truth utility
and then at the same time you want to do
it with doubt bothering the person too
much with sort of with low sample
complexity and I think whatever you you
can fix one and then you can choose a
trade-off between the other two but
whatever you do impacts where you are on
this on this triangle so there's
obviously something that we have to
decide on right like what level of
safety is good enough for instance what
I what is our chance to strain that we
want to impose I think I can speak
Western I would like to see a process
that develops AI safety guideline
initially ones that align very closely
with existing practice that they're not
at all on risk visiting practice is they
don't not leaving catastrophe in any
direct way and then with that engagement
asking how principles be expended and
adapted to higher level III going
forward that the institutional process
that I think is relevant your question
okay thanks thanks everyone for all your
wonderful
you
[Music]