251  A Generalist Agent 2

hello and welcome to session 251 in the
ai safety.com reading group tonight
we'll be discussing the second part of
the article a generalist agent by scott
reed and others
this is a work done by deepmind when i
say it's done by scott reed it's
actually with 21 other co-authors
and tonight we're talking about the
second half of the paper
so we previously saw looked at uh
gaeto uh and and saw some of its
capabilities and in this case we're
trying to do uh in the in the second
half of the paper we are looking at what
can um what what can we learn about how
data performs and um
with ablation studies and scaling in
different parts so we'll start with a
scaling laws analysis
so here on the x-axis you have the
number of tokens processed in hundreds
of millions and on the
y-axis you have like what is the score
how good is um
is this model actually performing on
these 600 different tasks
and as you uh probably expect um
the uh the models
it looks like there's like a factor
three four in between these two and it
looks like there is like a different uh
roughly the same order of magnitude
improvement and
there seems to be a reasonably steady
improvement
according to a number of tokens
processed
it looks a bit or it looks almost too
regular for me when i look at this like
i would have expected um
maybe some bigger models to uh
to uh learn slower like
we expected smaller models are less
efficient but on the other hand of
course it's an easier problem getting to
40
to 60 and in this graph it's kind of
seems like they're canceling each other
out
it's possible to look at this and say oh
it looks like this graph is probably in
yeah and this the difference between
these are logarithmic you you could say
something like that i'm
it looks like there's not enough data
here that we can really conclude
anything strongly um but um
yeah
also like my catchphrase um this scaling
laws is something that we have seen so
incredibly many times and it just looks
like we are on a straight path towards
um having an ai that's competent enough
to kill us and so even though it looks
very boring to again and again and again
have the same kind of uh grass i do
think that they're perhaps the most
important thing
so one of the things that the deepmind
cares a lot about is different kinds of
uh generalization out of order out of
distribution tasks
transfer learning this kind of thing
and
this is of course a question that they
um what you know not just can solve them
but can do so efficiently and they
hold out four tasks so after 604 tasks
they have four tasks that they choose
not to train on
and then they
won't wonder what is the best way to
actually use london uh
sorry london we just talked about number
in the introduction this has nothing to
do with the lambda model this is the
gator model um and so the obvious way to
think of how to do this would be to
just have the prompt uh being some kind
of um
a demonstration that would be really
nice but remember the the prompt has
only uh
1024 uh
tokens so that's not simply enough for
the agent to um
to invert anything
they use the word hint over i couldn't
quite figure out what that precisely
means it probably means it doesn't like
heaven in short-term attention or
something like that um english is not my
first language
um
i was a bit confused here in that um
the um the context uh
the obviously you prefer to have as long
consciousness as possible but the
execution time i've read somewhere and i
couldn't quite find it that the
execution time grows quadratic in the
context length so you get much worse and
that's why they need to shorten it and
they can't just put it inside the prompt
and gato is
has to run within a
20 millisecond
cycle uh due to constraints from the
robotics so that means that the running
time must be really constrained so the
context has to be really small so they
have to work around this however it
should be noted if you're
that
the way it's actually described is in
the robot that they are constrained on
memory rather than on um than on
execution time so
it looks like both of these are actually
hard constraints
so having having to run on a robot is a
substantial problem
so the obvious thing you can do if you
can't just edit and prompt and hope for
good meter learning
is to um
fine tune on some demonstration and
that's obviously what they're doing um
i think that's a
very reasonable thing to to do and i
think uh for all uh
things that we care about
fine tuning
is um almost by definition going to be
good enough um but i think still think
figuring out what it can do in meter
learning is important
like deepmind probably thinks fine
tuning is just fine but i care a lot
about how it um what kind of things it's
capable of doing out of distribution um
because
those good positions should be unsafe
there was a comment on twitter by
someone called jennifer saying that
if gator had achieved robust transfer
learning that's a technical success
another safety failure um because um uh
the the the
failure transfer learning was not a
safety plan because there is no safety
plan
and so that's a very um
negative view on uh deepline's work
and i think i
probably endorse that
let's have a look uh
uh at these four tasks
where
generalization was uh attempted where
transparent was intended
was done in a smaller model and before
called uh cardboard swing up meet well
assembly dm lab one of apple's forage
and atari boxing
and again you'll see on the
y-axis we have performance and the
x-axis we have the number of fine-tuning
episodes
they do this with four different ones
they have a
an expert run that is done by a um a
reinforcement learning algorithm um
which is like a
align with something that is like
they're trying to imitate this agent um
to steal this agent in some way and so
obviously we can't expect this uh in
general to have better uh uh results
than than the expert and they have in an
untrained model called scrap scratch and
one that is trained with um
with uh um like language understanding
and images but not with control data and
we remember that control data was like
85
of the of the training and then they
have some that has only been trained by
the control but not with language and uh
and images and finally one that's been
trained with everything
and if we look at cardboard swing up
first
as you probably would expect
doing
using just a standard language model
only starts to really get make sense
when you have
like a thousand uh fine-tuning episodes
and even then it could be kind of like
an artifact um
teaching someone about language and
images doesn't really help them to do
card poll swing up
you would want to do other control tasks
that does indeed seem to help and in
particular
if you
have both of them then
well eyeballing this it looks like it
has
better performance earlier to have both
uh
both modalities of training to have all
modalities of training
so this is kind of
the expected view
and there's another
here where we can again see that the um
the model train from scratch takes a
long while but eventually actually ramps
up but the one that is trained by
language it can't do this as simply this
control task and there seems to be like
negative
transfer
in this case it doesn't help but again
having both
kind of maybe like i feel perhaps i'm
over interpreting here but it looks kind
of like the yellow line is above the
green
so having more data seems to help
there's another one here
um
order of apples
forage
in this case it looks much more like
this
obviously if you have control data then
you can easily get uh 19 of the apples
but even here just having the language
seem to somewhat help uh i it's it seems
kind of clear that just that this helps
compared to just having the stretch
algorithm and finally for atari
we see
mixed results i i'd say right it's um
you could definitely argue that the red
one
just from scratch is just better in that
atari is just so different from the
others
that uh
yeah fine tuning on atari health and
pre-training language and control kind
of doesn't really matter as far as i can
tell like these lines are
changing positions it's not really
is that obvious what kind of what you
can influence this
so these are four very different cases
and i'm a bit sad that we don't have
more than four
like we can't really make any strong
conclusions from this like um if i have
to put on my conspiracy head i would say
that they wanted to to test it on 600
different tasks and they had a set of
tasks that was 604 and that meant they
could only leave out four um so i think
definitely think this work would have
been much better if they had held out
more than four tasks so we if like if we
had seen 20
of these tasks then it would have been
interesting if they were all like atari
boxing or all like cardboards would
swing up
a criticism of this
work was made on uh unless wrong uh well
not on uh that's from an interview with
uh
blake richards a um assistant professor
at the university of
uh montreal i think um claiming
in this case that um they're getting
better transfer effects if you train
domain than you trade across all
possible tasks
and
like this can be interpreted in two ways
they can be interpreted like if you it's
better to train in domain than in other
tasks that's the obvious thing right if
you want to be good at control tasks
then training on other control tasks is
better than training on language tasks i
mean in that sense that's probably
really obvious the thing the other
argument however is that you get better
transfer effects if you only train on
control and not train on language so
that would be uh the argument about what
is in general from this the relative
position of the yellow line and the
green line
and i would disagree with blake richards
here in that like
i i i really don't have a
like it's hard to look at these images
and see say that the green line is above
the yellow that does not seem to be the
case it seems like
there is some kind of transformation
happening
possibly not in this atari boxing but
again remember with atari if you go back
with something like three four years
that was the ale um which was very very
famous for not having negative transfer
back then there were we always had
negative transfer and now we are
seeing something that does not look like
negative transfer and um and that is
indeed very surprising
well that it's very surprising like if
you if you went back for five years that
would be very surprising uh we've seen
this kind of thing
many times recently and it's no longer
surprising but we are certainly seeing
some kind of generalization transfer
learning happening um
i think that is reasonably clear
okay so let's have another look at a
real-life
skill
that's something that uh
uh
of course that's one of the key things
that this paper is about having also uh
the robots actually do things in the
real world and see how that works in
this case they have a robot that picks
up some cubes and stacks them and have
been trained on stacking the red cubes
on top of the blue cubes
and that's the training task and then
the realization task is then can the
robot figure out to put the blue cubes
on top of the
the green tubes
like can they generalize from being
really good at stacking red on blue to
stacking blue on green
and they of course
they can't do this in impromptu so they
have to fine tune on that
and so they try that in simulation and
uh here the the colors are different
there's a behavioral cloning algorithm
and an expert reinforcement
algorithm and
three models you can see the uh
uh
the the blue one the smallest model
picks it up relatively slowly and the
green one uh much faster and the uh the
red one the biggest model at 1.2 billion
parameters
is much faster and even starts out
better than the behavioral cloning
algorithm
they do describe that there was one
problem one episode where overfitting
was a problem and
that's the kind of thing that i always
want from these um
these papers to actually dig into just
one precise one interesting example it
would have been really interesting to
see like what is actually going on in
this particular episode that is
problematic um but unfortunately like
papers don't
like to give very very concrete examples
and so they are also this was the
simulation and then they want to like
how does this work in reality and in
this case uh when they transfer this to
reality then gato obtains a 60
success rate 60 is this line so that is
to be expected it seems like what gatew
is doing transfers directly to reality
um
and
then they try the behavioral cloning
algorithm and that totally fails to
correspond to reality
it's this sim to real problem that a lot
of algorithms have a surprising amount
of difficulty uh going from simulations
to reality um and in this case it looks
really marginally like 0.5 percent
success is very very bad but they note
qualitatively when the robot needs to
stack up within the
behavioral cloning
it seems like it's almost always doing
the right thing but that then in the end
it's almost there but then it the the
tower collapses um so uh
like this result here like
60 compared to 0.5 percent looks really
really wonderful but um the this
qualitatively you can't really say that
much
because the the bad algorithm gets
almost right every time
okay so now we look at how good can the
robot actually get at stacking these uh
these boxes and uh in this case
the thing we were trying before it just
joins into the the training set and
compares different algorithms
and they um
yeah uh people who write these kind of
people always like to go into a huge
amount of details about how they beat
the state of the art and you can look at
it here and they say this is in an
earlier version of gator that didn't use
from fine tuning uh i think i would have
liked to have some kind of idea about
how difficult is this task actually they
say that it took 20 seconds for a human
to stack these with a 3d mouse and
a failure rate for humans is also like
unknown this is the kind of thing i
would like to know like is are we
talking about something that is above
the human level or below the human level
i would like to know
the authors of gator do more
they try to train some specialist single
domain multi-task agents so
within this control domain they try to
train it on all these things and nothing
else and see if they can like distill
this into a general controller that is
capable of doing all these tasks and in
meter while they're using one with 79
million parameters and they managed to
train on all these and distilled into a
single agent that has success rate of
96.6 which is state of the art um
at first when i saw this i was kind of
surprised like 79
million parents is not a lot these days
like if you go back uh three years sure
it was a lot but but no longer right
this is this is a really small thing and
um
and i think it's surprising that can you
learn that much uh with such a small one
i think uh
i would almost argue that the uh the
transfer of learning results we have
seen have been really meager like the
difference between the uh the blue line
and the green line we saw before were
not very convincing to me but this seems
like um some really neat distillation uh
and um i think that's kind of surprising
and and
but but i also noticed that i am in fact
a bit confused
because
it stays here that it beats the state of
the art with uh taking a small model and
just fine tuning it
uh in the most naive way and then you
get something that is wonderfully good
and beats the state of the art and i'm
wondering like what was the state of the
art before right why
it's not clear what they're doing that
is so smart that they can take a small
model and just distill in the most
obvious way and be state-of-the-art
normally you need to do something smart
really smart to be state of the art
right
then they also have the atari
problems
where
they use the full model and the
total demonstrations had some kind of
problem it was not in childhood
but in the rest of them the 44 out of 51
um they had uh
performed above the human uh
level and even without any kind of fine
tune
23 of the 51 atari games will have
superhuman performance and they expected
just scaling would would improve this
um
this uh
the problems with training can be
summarized as
the specialist the terry agent achieved
better than human performance for all
games where data contains superhuman
episodes
again because they're not directly doing
reinforcement learning they are
distilling other reinforcement learning
algorithms
then there's a section called related
work i won't call it related work
because what they're actually doing is
trying to make some kind of argument
about why uh why this will scale um and
so i just as an illustration of scaling
i took this classic graph from kaplan's
uh scale notes
um so why do we expect this will scale
well obviously uh we have substantial
experience with
the models that are larger than 1.2
billion like we have qt3 most obviously
but also many others and uh like more uh
more models are arriving uh or every
week i would honestly um
[Music]
yeah so there's a good argument for why
this will scale and we've also seen like
good results in general with these kind
of ultra-aggressive models we are seeing
this uh
uh
perform very well in many many other
circumstances and that's an also an
argument why we should expect this to uh
to transfer and this is uh the first
single generalist uh trained on hundreds
of vision language and control tasks
using modern uh format works at scale
i'm somewhat confused that um yeah i
mean sure it's the first one that's that
has done this um
but it seems like a
i should be careful about calling it
naive and obvious and all these kind of
things because i can't do it myself
right um they have some more arguments a
bit more fluffy like
the brain seems to have like a uniform
structure and that's kind of an argument
why you don't need different things for
uh like motor control and processing
visual because if you break some of the
motion control then the motor control
can just move into the digital area
right um so uh and we're also seeing
this with with neural network that
everything can be processed everywhere
and that's like an argument for some
kind of transformation
um finally they talk a bit about why
this uh atari uh might be uh more
different than
more difficult than many other tasks
and i mean conceptually a lot of control
tasks and a lot of robots are working in
reality
whatever that may mean and so reality
like having a robot that is big or small
or doing different kind of things they
have like the physical underlying
structure and um atari games doesn't
share any kind of online structure in
the same way as reality does
finally they talk about the implications
for aicc and so i um i tried to look up
deep mind on and safety and i got this
wonderful uh result from google with
deepmind's twitter and then missing
safety and google suggested i could
repeat my query with the word safety
because the word safety is not one that
deepmind is using at all
um
and they
say that yeah this
is safety seems to be an important thing
and someone ought to to look into this
and i think i've read enough papers to
to realize that this kind of call for
action is just you know empty words
they're saying that yeah it would be
nice if someone did this we will take no
actions or
doing anything
further and
as problematic is the fact that when
they talk about safety the the examples
that they give about safety are things
that are not actually related to safety
as we conceptualize it but
lack of capabilities in the obvious
sense that a
an agent can be unsafe because it is
trying to do the right thing is just not
capable of doing the right thing
compared to an agent trying to
competently optimize for something we
don't want
it's not completely fair they do in fact
mention stuart russell
so it's not 100 that they only look at
capability but you know it's a it's a
weak chapter this and they also have
seem convinced that this safety isn't
actually any kind of
meaningful problem and they're uh
very optimistic about solving that and i
think
in my in my estimate this if you
conceptualize safety as a race between
capability and safety alignment work
then this work seems clearly to be
advancing the state of agi and building
high towards ati it seems uh obviously
on
that as a strong negative
zooming out a bit is deep mind in fact
on track to destroy the world i think
definitely this paper seems uh
like
it is not trying very hard to engage
with this or trying to
[Music]
look into this whether we
we
humans end up with a better strategic
situation or a more strategic situation
and
in my analysis it kind of depends it's
not obvious to me that this is a big
step towards agi um
like
multimodal transformers how important
are they actually going to be that's an
interesting question and it's not
obvious to me at all that they will turn
out to be very interesting that it's
it's possible that just plain
learning on text is sufficient and uh
images and
soon will probably have video maybe that
might not matter very much how about
transformer agents how important are
they going to be compared to uh just
decoder only um
transformers
again it's a it's an open question i
don't know it's not obvious to me that
just former agents is going to have a
much larger impact than just uh
gt3 or something like that um
but it's not uh it's also not obvious
that it won't have so this is the kind
of thing that i would focus on trying to
determine whether this work is in fact
having a substantial uh positive or
negative or well whether it has a small
impact or a substantial negative impact
another thing that
would have been nice and would have been
possible to do with this work was it
would be interpretability work like in
particular some of the models are really
small 39 million neurons seemed like it
would be easier to do injury ability
work but
apparently as far as i could tell from
this no such work was done at
all uh and finally there is a uh a quick
reference to uh
to the book super intelligence saying
that many important vitamins make
technical agi safety more difficult
and that's decided as nick ostrom
superintelligence do not 2017 and as i
have the book and i've read several
times and it does in fact not say this
and also like the donut is the french
book distribution i think which i i
don't think that's a great citation but
that is like the smallest to pick
that is all for today thank you and see
you next week


Working in AI | Jan Leike, Helen Toner, Malo Bourgon, and Miles Brundage

the theme of the this series of
lightning talks is working in AI and the
possible careers you could have in that
space I'm going to announce each speaker
and they're going to give a short talk
they're going to go sit back down and
then everyone will come back up at the
end and I'll ask questions okay
so our first speaker is a yan Lika he is
a PhD in reinforcement learning and
research scientist at deep mine working
on technical AI safety so please welcome
yan Thanks
all right how do you build a career in
technical Asst I'm going to start with a
big spoiler and my main point is
basically you should just it's the same
thing as building a Korean AI so if you
want to do a safety you should learn any
I so but before I get into that what
does research on technical AI safety
mean roughly you can categorize a s
safety into three may like categories of
research first is alignment research
which basically about how do we build
agents that we can teach arbitrary goals
and that end up being end up doing what
we wanted to do the second so another
like includes things like
semi-supervised reinforcement learning
reward hacking the safe exploration and
so on and the second part is about
robustness how we make sure that machine
learning algorithms don't brain
unexpectedly and there's include things
like machine learning security
interrupts ability and robustness to
distributional shift and and other
things and finally the third category is
about trust how do you make sure how can
we establish trust that we build the
right systems and that entry includes
things like interpretability how do I
look into the black boxes and see what
what is going on there and formal
verification and I should emphasize that
this is kind of the machine learning
perspective
and there's been other kind of research
agendas around safety that have been
raised but most notably the agent
foundation's agenda by advocated by Mary
so but in this talk I'm going to take
the machine learning perspective
so what could technically a safety
research look like this is an example of
a recent collaboration between open air
and deep mind and this is a new
technique to teach goals arbitrary goals
to agents and what we do is so here this
is kind of a schematic of what the
research what what the setup looks like
you usually have NRL give'em and acting
with environment so this is just normal
reinforcement learning set up and what
we did is we put as a new module here
that is called the reward predictor
which learns reward function it learns
to predict the reward for what the human
wants so and be kind of important part
of this is that it is two people who
work in machine learning this looks like
normal machine learning research and
fast and it's motivated by like
long-term considerations of AI safety so
if you want to do this kind of research
it's amazing we need a lot more people
doing this stuff and there's currently a
huge talent gap and so you might be
asking yourself how do they get there so
let's start with your under get to
undergraduate degree what should you be
focusing on and like ideally your
undergraduate degree should be in
computer science or mathematics should
learn all the regular basics like
programming and algorithms and algebra
calculus and so on and then later on you
might focus more on machine learning and
deep learning and enforcement learning
and those are kind of like the major
core skill sets that at the moment are
mostly in demand and there's some some
kind of important things I think to keep
in mind you should try to prioritize
harder courses or easier ones if you
want to
like their math courses that you find
interesting but they don't seem as we
related like phew they still feel free
to take them you should but the end of
say your master's degree you should get
you should try to aim to have some
research experience by them so you have
in a better position to apply for PhD
positions or similar and so you want to
get you want to start working with
researchers early on you want to get a
paper published and for that it's
usually advisable to find someone a
supervisor who is good at supervising
and not necessarily the most famous
supervisor because the most famous
people usually don't have time to really
easy supervise students and the idea is
that you should really try to find out
early whether you're good you're good
fit for this line of work so that you
can you know that this is worth pursuing
this career path for you because if you
go into a PhD that's usually a many year
commitment so that's the next page how
do you get a PhD and I I should
emphasize that getting a PhD is usually
the prerequisite for someone to hire you
as a researcher but it's not strictly
required so you could get away with
having experience equivalent experience
but getting a PhD is usually like a
one-stop-shop package for all the
relevant research skills that you need
that includes like coming up with ideas
knowing what knowing the literature very
well knowing all the basics and being
able to write and present your research
results I think an important point to
make you is that you shouldn't you
should not be afraid to work on regular
machine learning research there's been
there's like a mistake that I see every
once in a while where people are really
ambitious about doing a safety research
and they want to dive into a a safety
researcher by the way but right now the
field is on a stage where is often not
quite clear what good projects are and
so you can you can get very lost very
easily so usually
I recommend to people to work on a
project that you supervise the finds
interesting so you can get a lot of
feedback from them and and ideally like
if you if you doing your PhD you just do
a normal machine learning research but
you're still you're interested in moving
into a safety after that I think that's
still a very good idea there's also if
you have a master's degree then you
could do a PhD in Europe which usually
takes only three or four years with this
in the US which can take over five years
and then you should for PhDs in general
like there's lots of advice out there
like to do a PhD where should you do h
PhD which you focus on should look at
that I think like right now the ideal
background for doing a sh t research is
a PhD in machine learning and there's
some alternatives to that is the brain
residency which is very competitive and
there's you can do internships in
various spaces is research engineering
where you which is like a lot of people
at open air and deep mind are doing and
this is basically working together with
researchers on code but more from from
engineering perspectives and for that
kind of work you usually don't need a
PhD but yeah so at the moment PhDs and
machine learning are extreme competitive
but if you can get into a program I
think it's very bold with it and you're
also like your date usual you will
probably have very good exit strategies
because the people want to do one of
your higher machine learning people
right now that's it if you have
questions there is two excellent
resources on the ad self now is page the
career review on machine and PhDs and BA
a safety syllabus and if you're
interested in the source stuff come talk
to me afterwards and once you get your
PhD and machine learning you should come
work it did mine door opening I or other
places thank you
[Applause]
thank you yawn next we have miles
Brundage speaking to us about careers in
AI policy miles is a research fellow at
Oxford future of humanity Institute his
research focuses on AI policy and
strategy and recently he's been working
on the relationship between AI and
security as well as thinking through how
international agreements on AI might
work
welcome miles so I'm just going to say a
few brief words about what AI policy is
and what sorts of topics you would be
working on and worrying about if you
were to pursue a field at AI policy and
how on toner is going to say a bit more
about the institutional landscape and
you know the more practical side so to
begin with I think there are a lot of
misconceptions about what AI policy
means that sometimes conflated with you
know sort of strict top-down regulatory
you know bans on AI or something to that
effect but I I define policy as just
authoritative social decision-making
about a certain topic or about a certain
technology or issue so AI policy is
really just about what we as a society
should do and we doesn't just mean
governments but also corporations have
policies towards AI in terms of what
they publish and what the protocols are
for privacy and so forth so working on
AI policy doesn't necessarily mean that
you think we need you know restrictive
government regulations anytime soon or
even ever in the same way that someone
who is say a left or a right leading
healthcare policy analyst might conclude
that you know the government should have
a hands-off approach on health care
policy rather it's about asking the
question of how we can ensure that AI
has positive social consequences there's
a common distinction made in AI policy
between short and long term AI policy or
AI policy issues and I'll say something
briefly about that because I think it's
important to wrap your head around so
the way that I define long term AI
policy is worrying about the issues and
trying to make progress on the issues
related to AI that won't materialize for
some
I'm or will have very long-lasting
consequences and I think that maps on
somewhat to the the differences in focus
between say organizations like the
future of humanity Institute which see
themselves as doing long term AI policy
because we're concerned with AIS that
have transformative economic and social
consequences whereas others who you know
one might call doing short term AI
policy or focus on issues that we have
right now so accountability for
algorithm based decision making and
privacy issues related to the use of
training data for AI but I think this is
also a somewhat misleading distinction
because there's a lot of uncertainty
about how quickly AI will develop so
some of the issues that we call long
term might actually arrive sooner than
we think
so you might all instead think of it as
policies focused on current AI
technologies and policies that are
specifically and policy analysis that's
specifically about more transformative
scenarios so there are already a lot of
government policies that pertain
indirectly to AI even though there isn't
you know a department of AI policy or
anything like that in any government and
there are things like drone
registrations and no-fly zones for
drones that sometimes use AI their
intellectual property laws that pertain
to technologies in general and therefore
AI and there's a lot of government
funding of AI research but at the same
time they're still an awareness that
something more more serious and more
sort of wide-ranging might be useful in
the future so for example as was
mentioned in my bio one thing I'm
interested in right now is whether
International Cooperation might be
useful for dealing with some of the you
know purported arms race situations that
might might already be arising or might
arise in the long term as countries
strive to use AI to pursue their
military advantage so I think it's not
clear yet what the appropriate level of
policy implementation or analysis is for
AI but I think there are a lot of
potentially very thorny issues to think
about so just give a quick overview of
some of the issues in this landscape on
the short term front there are issues of
economic impacts of AI those still you
know smaller than one might see over the
long term they're issues around
liability
bias security including you know
automated defenses and attacks involving
AI as well as privacy and I think these
are all potentially very important
issues in their own right and I do think
that from an ei perspective you might
want to be particularly focused on
issues that will have very long-term
impacts such as those associated with
general AI or super intelligence but
it's also the case that you can gain a
lot of career capital by working on the
short term issues and it might be that a
lot of the same tools and institutions
will be relevant to some of the short
term issues as well as a long term
issues so and I'm happy to talk more
about that offline but I think it's you
know important to note that there might
be synergies between you know different
timeframes and different types of work
over the long term we might see scaled
up versions of the sorts of issues that
I just mentioned so even more severe
concerns around privacy even more severe
concerns about the economic impacts of
AI but there also might be some
fundamentally new or at least you know
qualitatively differently large changes
such as you know big concerns around
wealth distribution created by huge
economic productivity gains from AI or
the risks of catastrophic misuse or
catastrophic safety accidents there
could be concerns about you know even
more extreme curtailment of privacy or
freedom in a society where you could
have nearly perfect surveillance as a
result of automated you know
infrastructure and drones and so forth
so I think there are very serious issues
to think about in this area it's not
clear yet what the right solutions are
and I think that's why it would be good
to have more people working in these
areas and I think you know we we need to
move from a general awareness of that
there all these problems and start
towards start thinking about what
exactly can be done about them if
anything and what the right level of
governance should be whether it's
corporations or international bodies or
national governments there are lots of
open questions such as you know how
should we evaluate AI policies so
there's a paper for example by some of
my colleagues at the future of humanity
Institute called a policy to sit errata
for the development of machine super
intelligence which raises lots of issues
around how we should think about justice
and and speed of transitions towards a
more powerful AI there are questions
around
logistics of cooperating internationally
if we if nations were to agree that they
don't want to just race to build the
most powerful AI but instead they want
to cooperate in some fashion how would
you actually implement that so I think
there are interdisciplinary questions
that involve the science of AI and you
know thinking about the global
distribution of software and hardware
that that that also require a policy
analysis in a various discipline so I
think a policy is an exciting area and
one that requires a lot of different
perspectives in order to make progress
on so with that I will wrap up and look
forward to any questions later on Thanks
next we have Helen toner speaking about
policy funder and strategy her research
careers in AI Helen Connor is a senior
research analyst at the open
philanthropy project where she focuses
on policy governance and strategy issues
related to the progress in artificial
intelligence
before joining open philanthropy she led
EA Melbourne while studying chemical
engineering and Arabic at the University
of Melbourne
please welcome Helen great so as Martha
said there's a huge range of societal
strategic policy political issues
relating to AI that are going to need a
huge amount of work over the coming
years my guess is that navigating the
effects of progress in AI machine
learning robotics automation is going to
be the defining policy challenge of the
coming decades so now that miles has
laid out some of the topics that could
be valuable to work on and are likely to
need work over the coming years I'm
going to briefly touch on what kinds of
institutions are out there where you
might be able to do some of this type of
work and obviously there's a wide range
of options here so I'm going to kind of
zoom over them at a high level rather
than going in in detail early on in
particular options and actually one side
note I'd make is that although there are
a large number of kind of related fields
where you might be able to do work on
these kinds of issues there really isn't
an obvious home for work on AI policy
and strategy in the same way that
machine learning is a pretty obvious
home for work on at least the machine
learning angles of technical AI safety
until what this means and this is
particularly true I think for the kinds
of more transformative and longer-term
issues that ei is tend to be more
interested in and so what this means is
that if you want to work on these topics
on AI policy and strategy topics you're
going to need to be a bit more
self-directed and a bit more
entrepreneurial then you might need to
be in other areas so there aren't
necessarily going to be existing courses
of study or you know existing
fellowships or journals or or that kind
of thing there for you and you might
need to be a bit more sort of
self-directed and tread your own path in
terms of choosing what topics you want
to work on persuading supervisors and
superiors that what you're doing is it's
interesting great so with that intro to
break down the space into two main types
of work you might do which I'm going to
call research and practice and then
split each of those into the kinds of
institutions that might be good good
homes for each kind of work so the first
one research this is like going deep and
and doing original thinking on the kinds
of questions that miles is talking about
I think three key types of institutions
where you might do this would be
academia think tanks and ei
organizations so I'll go through one by
one
so if academia is likely looks like you
know going to grad school maybe doing a
postdoc or a fellowship maybe eventually
going into more senior roles in a
university and academia is really well
suited to choosing one specialized topic
and going into a lot of depth on that
topic there's a lot of room for you know
fundamental and theoretical research and
is there's an ever-increasing number of
interdisciplinary centers springing up
at universities that could be good homes
for this kind of work so I'm thinking of
places like the Center for International
Security and Cooperation at Stanford the
Center for long-term cybersecurity at
Berkeley pulse at UCLA there's a bunch
of other centers like this or you could
also go into a more traditional
department like International Relations
or economics and you know the best home
for the work you want to do is just
going to depend on what topics are
interested in and also which people
you're working with and what they're
interested in as well
cool so number two for research think
tanks I think think tanks are fairly
similar to academia in some ways and
that they're relatively mainstream they
tend to have established areas that they
focus on establish ways of doing things
and the key difference would just be
that they are often more focused on
concrete policy proposals and specific
ways of implementing ideas rather than
the kind of theoretical or fundamental
research so there's definitely some
overlap between between think tanks in
academia there are already some some
newer think tanks that are specifically
focused on AI issues and particularly on
the the kind of neuro term issues that
that Myles was talking about so AI now
is one and data and society is another
and you could also aim for a more
established and kind of general purpose
think-tank as well like you know the
RAND Corporation the Center for
Strategic and International Studies
Brookings the Center for a New American
Security just to name a few there are
lots subsisting tanks the third type of
place you might want to do research
would be a TA organizations like the
future of humanity Institute at Oxford
or the Center for the Study of
existential risk at Cambridge and these
places are going to be you know really
best if you want to be focusing on types
of issues that pas tend to care about
disproportionately so you know again
these kind of longer-term more
transformative scenarios and I guess the
the trade-off that comes along with
having the flexibility to work on these
issues is going to be essentially that
you know these organizations don't yet
seem to have as much connection to or
recognition by political decision-makers
so you know I don't want to sell them
short there's certainly been some
contact between FHI and UK government
for example and that's that's likely to
continue but I do think there is a bit
of a trade-off there in terms of
flexibility versus influenced okay so
that is the research side of things what
I'm calling practice is is roles that is
that are more about you know
understanding what research is out there
and putting it into practice you know
making decisions based on it for some
given body and I think I think there are
two big categories of this type of role
probably you know government and
government ish organizations and
industries sort of start with government
it seems very likely that the role of
governments and government like bodies
is is only going to increase in you know
in the their role in the development and
use of AI and because of this it would
be really great
I have more people from the effective
altruism community working in government
I think particularly in the US
government given how influential the
u.s. is on the international stage but
certainly also in other country
governments and also in you know
multilateral organizations like the UN
and others which are starting to show
some interest in AI issues and I think
that's that's likely to continue so who
should work in government I think
there's a unfortunately low number of
effective altruists who seem to have you
know an appropriate profile for roles in
government so I think if the description
that I'm about to give sounds like you I
would I would really seriously consider
it so I think a good profile for someone
to do well in a government role I guess
I'm thinking particularly bout the
federal government but I think this
applies to too many types of roles it's
someone who is you know well-rounded and
generally capable I think if you're a
genius in any one given thing you'll do
better in that thing outside of
government most likely but if you can do
you know a wide range that might be good
you'll want to have decent interpersonal
skills I think particularly the kinds of
interpersonal skills that will let you
go into a meeting you know with
different people you don't know very
well from different agencies with
different goals and you know sound
reasonable to them get them on board
behind the kinds of things that you're
interested in I think a couple of other
important traits are to be really
patient with bureaucracy unfortunately I
think that's going to be a key part of
working in government and then the last
one it's just going to be to be you know
a citizen of whatever country you're you
want to work in so if you want to work
in US government be a US citizen in my
case that's an area for improvement
I think the second major major category
of you know practitioner type work would
be working in industry so this could
mean working within a old AI
organizations like deepmind or open AI
could also mean other industry
organizations like the I Triple E or the
partnership on AI I don't think there
are many positions like this available
right now deepmind certainly has some
policy staff but I do think that the
number of these types of positions is
likely to increase and this is likely to
be you know a less sort of stuffy and
bureaucratic option where you're still
close to to the relevant decision makers
and I think um you know which of these
whether you're more interested in
and type roles or industry roles is also
going to depend of course on how you
expect the the development of AI to play
out and how influential you expect these
these different sectors to be okay so to
sum up and again noting that this talk
is simplifying a lot a naming to you
know skim over the landscape rather than
cover anything in great detail if you
want to do research work you might
consider academia for in-depth or
theoretical work think tanks for more
policy relevance or ei organizations if
you want to really stick to the topics
that ears tend to care about more than
others if you want to work as a
practitioner you could consider
governance if you have the patience for
the bureaucracy or otherwise in there
may also be roles in industry
organizations that work for you
including in AI research groups that's
just about all that I have to say if
you're interested you can check out the
80,000 hours careers guide which miles
wrote on these topics I think if you
google 80,000 hours AI strategy that
should be the first result and just
before closing I want to throw in one
bonus angle which is that China has been
showing more and more interest and
sophistication in AI and machine
learning technologies so I don't think
this is for everyone but in the spirit
of taking a portfolio approach if
spending time in China and getting to
know the scene there seems appealing to
you I would definitely consider that as
well thank you alright thank you so much
Helen next we have mala Borg on and
Andrew Snyder Bedi tag-teaming it to
talk about careers in operations and
management in AI malo is the chief
operating officer of the machine
intelligence Research Institute mala
oversees all day-to-day operations and
program activities at Miri he also
co-chairs the Committee on the safety
and beneficence of artificial general
intelligence and artificial
superintelligence of the I Triple E
Global Initiative for ethical
considerations and artificial
intelligence and autonomous systems malo
joined Miri in 2012 shortly after
competing a map completing a master's
degree in engineering at the University
of Guelph Andrew Schneider BD is the
research director at the future of
humanity Institute University of Oxford
before that he worked as a project
manager at the future of humanity
Institute and as a researcher at Mehta
his projects @fh I cover existential
risk research fundraising recruitment
and outreach
while at Shi Andrew obtained over 4.2
million pounds from grant writing and
wrote popular articles that received
over 500,000 readers his current
interests include observation flexion
effects technological forecasting and
longer-term biosecurity issues
please welcome Malo and Andrew so I
think if I was to summarize the point in
one sentence it's just that operations
is an incredibly incredibly important
set of skills and career paths for
people that want to have an impact at an
EA organization so that would be like
the overall summary I'll give a little
bit background as as kind of like my
experience as a project manager at FHI
um so I I went into FHI as a project
manager there are a number of other
project managers who have worked at FHI
and I think it's a tremendously a high
impact career option for someone who
cares a lot about the research is really
interested in the research but also
doesn't want to entirely specialize on
the research and wants to also kind of
integrate with the real world and like
translate research and impact and do
fundraising and and interact with the
world in these ways so so three-three
like kind of main areas as a project
manager that i found super useful
so one obviously is fundraising so if if
you're familiar with the research and
you're good writer and you enjoy writing
grants which is like relatively rare you
can have a tremendous impact so a
typical project manager FHI raises
something like 1 million in grant
writing every year from non ei sources
so these are like Research Council
funding or various trusts and whatnot
and so even even just from a purely
fundraising standpoint I would argue
that project management within an EI
organization could potentially be a
superior option to earning to give if
you're interested in these kind of
issues so that's like one kind of
easily measurable a way in which these
careers have an impact another is just
kind of outreach and ensuring that the
the research is having an impact in the
real world and this requires a very wide
range of skills so I think my favorite
example of this was the Asilomar
conference um you know this was after
the publication of super intelligence
and this was really kind of the first
conference to got a lot of the
stakeholders together to kind of create
some common knowledge around Yai safety
would be important in the future and
this took a tremendous amount of effort
this wasn't done at FHI this have done
it fli but but basically all the effort
that went into that was an operation
skill set effort and I think that that
event really kind of got the ball
rolling for the AI safety and AI
strategy a trajectory that we're on now
and was a tremendously tremendously
important piece of work potentially more
important than all the research that
like any any single piece of research
that that's happened recently
other examples kind of within FHI
include things like writing press
releases and making sure that the right
stakeholders have access to the the
research that one is producing or
interfacing with governments and kind of
setting up collaboration so recently we
had a collaboration with the finnish
foreign ministry who is kind of
interested in some of these issues and
and setting that up so that takes a lot
of effort as well that isn't necessarily
just on the research end one thing that
I'll jump in there is I think on the
communication side there's also a very
underappreciated skill of being able to
both be interested and kind of deeply
understand the work that people are
doing but also have the skill of being
able to like very accurately model the
different stakeholders and people that
you're communicating with because a lot
of the topics that we're talking about
have different audiences that has maybe
different like pain points or concerns
and if you're kind of delivering this
message that sometimes is a little
complicated or hard to buy or whatever
there's like a very valuable skill of
being able to kind of like create these
models and speak to different audiences
and so if you feel like you have this
writing skill or this communication
skill and you feel like you're good at
kind of knowing your audience and
different people like this is very high
value I think you could wander
into basically any AI or who's doing
like AI safety type of stuff and even if
they don't list a position if you're the
type of person and you hang out around
there you'll very quickly be someone who
they want to kind of work with more so
yeah please get in touch with any of us
that's the thing you have yeah
absolutely
and then I guess the the final thought
is just kind of on the basic operations
and management style roles that need to
be filled within this space so this is
this is kind of like a really
fundamental role basically being the
foundation on which things get done and
on which an organization can rely on you
to ensure that there's a platform on
which the research and outreach and all
the impact can occur and I think I think
there's relatively few people with an EI
community that have considered this as a
longer-term career option but I think a
lot of people really ought to um so I
think well I don't know malla maybe you
can say more about the impact of just
yeah yeah sure so one thing I'll note
kind of right out of the gate is
oftentimes people who are good at this
don't realize how like few people have
kind of this skill and so it seems
obvious to them that this is like a very
replaceable position or oh you know I
can't do a you know direct policy work
or AI technical safety work and so I
guess I can go earn to give or use my
skills elsewhere and see this kind of
like ops role is something that's very
replaceable and kind of a little bit
second-tier and I think that's like very
wrong and that is even from my own
experience basically everyone I know
who's good at this role kind of doesn't
understand why other people think it's
special and so if you think you have
this thing like you are special and or
just like like FHI and Miri and other
places need your kind of help and the
other thing I kind of want to emphasize
is that there are a lot of like
interesting opportunities for
advancement in these types of roles so
even if you like the initial role is
something where it's very much like
you're the person who's making things
happen and you're not making decisions
and managing people if you have an
interest in these subjects and as you're
working at these organizations you
developing more of a specializations or
interacting with everybody and you have
the ability of getting things done and
moving projects forward there will be
lots of opportunities especially in the
smaller organizations for someone who's
a good generalist in that way to take on
more responsibilities you know go into
management that sort of thing I mean it
in kind of Mary's particular case we
have this thing where we're always
trying to eat our superiors job and so
I'm always trying to steal all the
things I came from Nate our office
managers trying to always take all of my
job and so I would get encouraged people
who think they have the skill to jump in
on kind of intro roles and kind of build
their their capital that way and there
are actually a lot of opportunities
right now in this space and maybe Andrew
can talk a little bit about some of the
the openings that FHI is going to have
or has already yeah so FHI is going to
be hiring for two maybe three Operations
roles in the very near future the open
fill is also I gather hiring for a
Director of Operations so there are a
number of positions already within this
space that that you know we're searching
for really top-notch people like kind of
elite operators and so I will be doing
office hours at 2 o'clock
along with Carrick and Neil and some
other FHI people so if if you're
interested in a role like this please
come get in touch I'd love to chat more
yeah and i'll say generally as well if
this is the type of thing you're
interested in and you want more tips and
thoughts on how to kind of make your way
in feel free to catch me around the
conference
the speakers could come up and sit for a
few questions thank you all so let's
start with a question for Yann for those
of us who have already completed an
undergraduate and perhaps graduate
degree in something other than CS or
mathematics how do we gain the technical
expertise to do AI technical safety
research yeah that's a good question
generally so there's there's things you
can so the first step will just be
reading up on literature right there's
the ASAP syllabus which is again really
just an AI syllabus and has like lots of
textbook textbooks that you can look at
usually in order for someone to hire
your researcher in machine learning if
you have a PhD in related field is that
you demonstrate that you can do research
in the space and like one idea would be
that you may do an internship in a place
like Mila and then try to get a paper
published in machine learning and that
then you can point to you the fact that
you can do this kind of research
and there's other avenues as well that
you can take to to gain this kind of
skills like for example the band
residency and other things so we have a
question for the policy people Helen and
Myles what are the best places to get a
graduate degree at which universities
for people who want to influence AI
policy yes so I think in general any top
university is like any university good
reputation is going to be a good place
to start the area that I can speak to
best on this question is national
security or security studies and I think
they're sice so the Johns Hopkins School
of Advanced International Studies and
Georgetown which are
in DC have really strong reputations
there yeah my impression is that any top
school is going to put you in good
standing and maybe some of them have you
know slightly more or less established
programs for different areas so that
some right to you yeah I agree with that
and I would add you know in addition to
schools that are top you know across the
board like Stanford and Harvard and
others so like at Harvard's Kennedy
School would be an example but also uh
places where there's you know an
especially strong program in policy so
like Tufts Fletcher's school or
something like that and I would also add
that you know to some extent the the the
school is not the not the only question
it might be the important most important
question but might also not be the most
important question it also matters that
you're you know in involve with a
productive research group and have a
good advisor who is interested and the
sorts of issues that you're interested
in and can provide good guidance so I
think that's that's another factor to
consider just adding that FHI might um
so so we recently have Alan Defoe who's
come to FHI and he's also concerning
building out the team of PhD students at
Oxford so that would be a good
opportunity like if you're interested in
kind of advancing doing a PhD like
within kind of an A org
this is something that we're trying to
get a pipeline started for um but uh
yeah I mean yeah so that would be kind
of an exciting opportunity for some
people
question number the next question is is
there a place for philosophers of mind
in technical safety research like deep
mind or are they better off doing policy
type stuff I guess it depends on like
your exact background right so usually
in technical is safety we do technical
research so you'd have to have a
technical skill set to do this kind of
research whether I guess if you have
like a few background is like solely in
philosophy
I would guess you better placed in like
a policy or strategy position but again
and I would just add a general comment
on like the value of philosophy I think
it's a very important skill set and a
lot of people who have done pioneering
work on you know AI policy and strategy
like Nick Bostrom and Toby Ord have
training and philosophy so but I think
there's a there's a need for those with
that skill set and who you know are in a
program in that area to sort of focus on
the sorts of questions that are policy
relevant so for example I could imagine
a lot of false philosophy of mind or you
know ethics to not be super relevant to
say you know the design of AI systems
but I could imagine very fruitful
research that looks at the specific
questions of you know machine you know
the moral patientsí and and moral agency
of like the sorts of AI systems that are
being developed or will soon be
developed as opposed sort of abstract
analysis of you know the space of
possible Minds it Thanks next question
what are the AI related career
opportunities for people who are unable
to get a PhD for whatever reason I mean
I think most of the positions that
Andrew and I spoke about lend themselves
well to people who don't have PhDs
there's you know a wide spectrum of kind
of just making things happen at an
organization we're kind of very little
specialization and anything is
necessarily required if you have the
right abilities up to writing grants
we're kind of a lot more domain
knowledge is important but again it's
the type of thing that you can kind of
pick up through interaction with the
space and working at an org and
developing those skills to the point
where you can kind of take on those
kinds of responsibilities I guess
another thing I'll say is at me REE
credentials for researchers aren't as
important if you can do the type of you
know very big picture kind of mix of
math and philosophy research that we're
doing and you kind of have the skill set
for that whether you have an
undergraduate degree or a PhD that's not
particularly interesting to us it's
whether you have the skills and so if
you're in that direction
you know you should feel free to get in
touch yeah and I got that in the in the
policy and strategy space I think a PhD
is not super necessary they're a bunch
of graduates and master's programs and
things like international relations or
MPP Master of Public Policy
that are great options and will set you
up pretty well even for the more
mainstream you know sort of think-tank
government type things especially if you
can get internships while you're doing
them and again you know echoing Marlowe
I think if you want if you're aiming to
work in EI organizations there is even
less important and if you you know
having I think having some good work
sample and showing that you can write
about these issues is going to be more
important than credentials for those
organizations so a question for the
policy people can you talk about the
benefits and costs of pursuing a law
degree in order to work on AI policy
issues versus the other sorts of things
Helen was mentioning um so I think the
benefits are you know somewhat analogous
to philosophy it's sort of a disciplined
way of thinking and and you know
analyzing you know institutions and laws
and I think there's also a lot of
subject knowledge that that could be
useful in terms of understanding you
know what what sorts of you know legal
apparatus are relevant to the
development of AI or what as you know a
fruitful legal policy would look like
and actually analyzing particular
legislative proposals in terms of costs
I mean you know there always opportunity
costs with any discipline and and I
agree with what Helen said earlier that
there's not any one disciplinary home of
of AI policy so you know you like the
the opportunity costs would just be like
learning more you know developing other
skill sets like you know game theory in
an economics program or whatever so I
think you know there are lots of ways
there are lots of disciplinary
contributions to be made and and you
know different degrees that could be
useful yeah I agree with that and I'd
also just add that if you have a
particular incident going into to
specifically government I think another
benefit of law degrees is that that
generally pretty well well looked upon
in instead of a wide range of government
roles any recommendations for AI related
work for those with an interest in
reducing far future suffering in
particular as opposed to existential
risk I mean I don't know if they have
any open positions but Fri is definitely
kind of like the main place that's
thinking about those sorts of things
oftentimes with small ei orgs
many people who get positions there
aren't people who applied for a general
posting but if an opportunity presents
itself for someone with that there's
certain skill set you know is in contact
with the organization that might be
enough so if you know that's a subject
that you're interested in I would
definitely recommend getting in touch
with those folks and seeing whether
there's there's some fit there I also
think there's a good argument we made
that working in kind of any ei
organization is a good fit for someone
with those interests I mean you know
unless you want to be specifically and
only working on those things where you
know something like Fri might be a
better fit I just briefly add that in
the same way that some of the same
concepts and tools and issues are
relevant over the short and long term it
might also be the case that you know
some of the same you know policy
concepts are relevant to addressing
particular risks over the long term so
if you're able to you know make
international coordination work to
prevent you know automated you know
hacking or whatever then you might also
be in a better position to prevent
future suffering so I think both you
know Fri and also potentially other
organizations it could be good for that
okay so we have five minutes left so
we're going to do two more questions the
first one for yan what's the difference
in the work that research engineers and
research scientists do a deep mind you
say a bit more about the qualifications
there - um yeah so usually the research
engineers are more focused on janu
engineering part of projects so a lot of
most of the work that goes on at the
mind is very engineering focused where
we build new machine learning models we
implement new things we try out most new
stuff and that involves a lot of
implementation necessarily and research
engineers are more focused on the
implementation set of things so usually
you have teams of research in years and
researchers working together on a
particular project on particular
research project and usage scientists
will focus more on like the conceptual
side and like on figuring out like the
high-level goals of the project and
research engineers will work more on the
implementation side but that doesn't
mean that research engineers are just
coders and so research engineer
expected to know machine learning pretty
well and like be able to read and write
machine learning papers so but the
qualifications for uses scientists are
usually more selective so for a research
engineer you're like you wouldn't be
expected to have a PhD in machine
learning and usually a master's degree
in physics can be the science or maybe
it's efficient we are engineers from
like other disciplines you have like
done physics undergraduate juries or
other cognitive fields and finally for
the policy people could you talk a bit
more about if someone is trying to
decide between academia versus advancing
policy research directly in AI
organizations or ei organizations excuse
me what sort of considerations might tip
them one way or the other I I think it
really depends on what sorts of issues
you're concerned about and what sort of
work you want to do so I think you know
there tends to be a greater orientation
towards like temporary or contemporary
issues that you know a research
organizations because they have fires to
put out and yeah you know current issues
like privacy and so forth to worry about
and they have to worry about you know
the particular concerns of the
organization if you want to you know
take a more sort of you know a global
scale perspective then it might be
better to you know not be tied to a
particular organization but I think you
know they're their pros and cons of both
and there isn't there there isn't a
clear you know choice between them like
even if you just want to do the research
so like there are people who focus on
research at AI research at AI
organizations and I think it also can be
very helpful to you know be close to the
research and to know what the actual
technology is capable of but you know it
also to some extent is sort of a false
choice because one can you know you know
be in close collaboration with AI
research organizations from the outside
and vice-versa I'm Adam spoke I meant
effective alterus organizations such as
Shi or open philanthropy project or
whatever as opposed to academia I'm
supposed to academia
okay sorry about that yeah so I think
again I think it's it's somewhat of a
false choice in that like there are
people at FHI who are you know in the
process of getting their PhDs or D
filled and you know it's housed at you
know a university but generally speaking
academics tend you know like non you
know ei organization academics tend to
be focused more on their disciplinary
skill set and like getting you know
really high competence in a particular
you know in a particular skill whereas
at you know at an EI organization will
be more interdisciplinary and drawing
out a bunch of different perspectives
you might you know might lend itself to
more Jeanette general analysis at you
know they're drawing up to good
discipline so if you want to get like a
really strong foundation in a particular
area then you might want to go to like
you know the best Poli Sci program or
the best computer science program yeah I
think I think I just add that probably
the best thing to do is to figure out
which which specific topics you're
interested in and then figure out who
you think is doing the best work on that
and so that could be you know it could
be an academic institution or could be
an EI organizational could be somewhere
like FHI that's opposed thank you so
much everybody
you