Fireside chat: AI governance | Markus Anderljung & Ben Garfinkel | EA Global: Virtual 2020

hello everyone yeah I'm Marcus I'm Ben
and we are here FHI and we both work at
the Center for the governance of AI yeah
the future of humanity Institute and
we'll be chatting a little bit about
like research coming out of our Center
and sort of giving some tips about what
you might want to do if you want to go
into a career in governance basically so
it's like to take us away yeah Ben like
how did you get into the field it's a
governance
yeah so I guess I got involved a few
years ago so I was just graduating from
college I was majoring in physics and
philosophy and was considering actually
going into the philosophy of physics and
then had some doubts on the basis of
perhaps and not being either the most
useful or employable field of all of the
fields just around the same time as well
I was getting interested in ideas
associated with effective altruism and
then professor of mine University Island
a folk who heads the center which was
himself transitioning into focusing on
AI governance from a sort of long-term
perspective and so he could have a call
for a research assistance looking for
people who might want to get involved
and then yeah I've just happened to me a
good time where I was ready to
transition into something else so it
seemed important thing it seemed like at
the time not many people were thinking
about it so that's how I got involved
cool cool what happened next
yeah so I basically worked as a
part-time research assistant for about a
year or so at the same time I actually
got a job at the Center for effective
altruism and I was sort of like doing
that part-time thing with Alan while I
was working here right and then after
about a year or so of doing that I
decided I wanted to sort of transition
sort of into the area full-time and
there having the opportunity to do that
right right okay so it sounds like a
kind of sort of random or something you
just this had opportunity happened to
pop up and you you sort of were position
such that you could yeah I don't want it
so there's definitely a high degree of
randomness like it wasn't it wasn't just
like oh this is a random interesting
interesting thing entirely so I
definitely at the time had gotten more
interested in ideas to search with
associated long-term ism and effective
altruism so I guess the topic of
governance was a bit on my radar
roughly almost exactly the same time
they became interested in topic this
opportunity came up so I was definitely
cool yeah
Elementor brand new mr. cool cool yeah
I'm sort of yeah how is sort of your
your work in the field like changed I
think quite I say quite a lot so I think
I I still you know have a fairly broad
focus in the area but I think when we
first started there was this real sense
that you know AI was gonna be a thing
that was really important
everyone had like you know started to
believe that like just around then I was
like just around the time like alphago
ready like 2016 yeah exactly
and then super intelligence had been
been written and that sort of specific
variation of like long term was
concerned around AI it was broad science
you know oh man yeah seems like could be
this really transform technology there
might be these risks that you know
people who only understand very well yet
almost no one almost always working on
for example a safety at the time but
there was also this sort of thing people
were saying were like almost literally
no one is thinking about like long-term
sort of like governance challenges
assert you to say I um like there was
even at the time not that much just like
you know a governance work of any kind
going on and so I think a lot of early
stuff was just like a little bit like
what even it's going on here what
exactly are we trying to do and you know
I think a lot of it also was probably
like a lot more you know naive then
research then because you that people
really just I think a lot of people were
just transitioning and you know when
you're a that much about the area right
yeah okay cool and you're now doing a
default here at here at Oxford so like a
that's Oxford ease for like a PhD
basically yeah so just started yeah
Oxford PhD essentially an internal
relations just this like just past term
just because yeah so I guess you know my
only credential in the area so far
continues to be an undergraduate degree
and completely unrelated fields insert
seems you know somewhat useful to maybe
get a proper degree in education and in
a field that's that's somewhat more
relevant to governance then Rick then
physics great right yeah cool yeah so
it's to turn your question around around
on you Marcus
yeah yeah tell me about how did you get
involved I realize actually I don't have
a really great sense of how you first
sort of right yeah yeah yes I guess I
it's sort of slightly convoluted story
so I got like sort of involved in a
syringe that incident raphson style
questions while at university so that
was like maybe 2011 2013 or something
like this and then for a while I sort of
I sort of got involved with gimmick and
these kinds of organizations and then
sort of had a few years of sort of
transitioning from a belief of like oh
okay like long term something like long
term ISM I guess we didn't have a word
for that back then but something like
the long term ISM seems true but I'm
like not convinced that like these sort
of like focusing on like emerging tech
style questions we're like reasonable
because it just didn't seem to me like
it was difficult to see like sensible
concrete things so you could do in the
area but then over like a few years time
I think I was like I became like
increasingly convinced that like there
seem to be things to do and so then I
just like when I graduated uni and 2016
I yeah basically I was I was moving back
to Sweden at that point and so like
looking at like what to do in Sweden I
was like not really I couldn't really
find a lot of like super exciting
options and so I decided that I would
like like initially like try to like
build up career capital and so I did the
classic thing of going into management
consulting and did that for a few years
that was like really really fun really
interesting I felt like I learned a lot
and then after awhile I just like felt
like sort of the development trajectory
kind of like leveled off and I just like
wasn't learning as much anymore and it
was like less interesting I also felt
like I could probably like do more by
like working like more directly on like
the cause areas that I cared about and
then I transitioned into working for
like effective office in Sweden
basically with like a belief that like
it seemed like yeah building sort of the
EI community seemed like a good idea I
wanted to stay in Sweden at that point
as well and so it looked like one of the
best opportunities that could have would
be to like head up that organisation
so I got funding to do that for like and
did that like for about a year and then
during that time I basically got
convinced that like if I if it was like
willing to like live elsewhere then
Sweden I would like could do like more
good with my career elsewhere and so
then I like started looking to like
other options and basically my algorithm
was like I'll look for things where it
was something like I my my description
of like my comparative advantage was
something like I'm like a imagining
consultant person who kind of like
understands the research or something or
like the the simpler version is like I
was a magical consulting who like gets
it
and so then I like basically like looked
for roles that looked like basically
working for like organizations doing
research in areas that I cared about
we're sort of my like broader skill sets
like being able to like help out with
operations help out with like
recruitment and those kind of things
could come in handy and then I just like
applied to a bunch of things that fit
that bill we're like God I was one of
them so it wasn't specifically that was
like oh yeah governance seems like the
very most important thing and then I
like just like looked for ways to help
out there it was a little more like a
broad class of things that seemed useful
yeah something like that yeah cool yeah
maybe next let's chat a little bit about
like research that you've been up to
recently so in 2018 for eajy London you
did this talk on title income how sure
are we about this AI stuff right okay
and so I guess like I'm inclined to ask
like how sure are we about this AI stuff
um so I think still still not well I
guess there's two ways of being sure so
there's like you know there's I guess
the robust like how sure we that if we
had sort of all the facts and we had all
the considerations we'd think that this
is one of the top let's say you know
like two or three priorities for the a
movement and there's a separate sense
which is like given the Lumina mobile
information we have at the moment and
given a limited number of considerations
we have how sure are we that even on
that basis just at least an expectation
it makes sense for us to be putting a
lot of resources in this area
and I think the first one I think we're
you know I think we're really not very
sure but it's just very hard to be sure
because there's so much we don't know we
don't know um like we don't know what
future AI systems will look like we
don't know what the pace of progress
will look like we have left trouble even
imagining the world with like you know a
systems are a lot more advancement we
have today we don't have a great sense
of you know exactly what institutions
will matter when you know interesting
stuff happens we don't exactly have a
great sense of just so much of it that's
that's to some extent the nature of yeah
you know any technology was gonna be
massive changes his wrist we don't have
a great picture it might not be
different for like AI versus like other
cause areas um something more for AI so
if you think about you know let's say
climate changes or something there's
definitely a lot of uncertainty we have
surf example you know climate models
yeah we don't exactly know the miles
your feedback loops and we know exactly
know you know like how to think about
low likelihood events like how likely it
was at like one in a thousand or like 1
in 100 for like extreme rain back loops
making things much worse than my mom's
predict yeah we still serve a basic
sense of you know the rough parameters
of things like the mainframe it was just
like how hot well things get to some
extent and I'm there there's some stuff
you know that is hot
yeah how bad is hot you know exactly
where hot where we're less hot yeah
that's right of thing
where is Raya is just you know if we're
talking about let's say in a long run
world we're just you know human labor
doesn't really you know isn't any
necessary for most tasks just AI systems
are essentially replacing it it seems
like there's so many dimensions along
which we just don't know what that world
you know looks like we don't know what
there's a s systems are like we don't
know just how that's organized or it's
just very very difficult to picture it's
a bit like as like an analogy like it's
a bit like being in you know 1500 and
someone tells you describes in very
rough terms like the internet like they
say oh you know communication is gonna
be much much faster and more efficient
and information retrieval it's gonna be
better there's a bit you can do the
reason you know about about that sort of
world like oh I guess you know maybe you
could have larger businesses because
they can communicate you know
differently
but you definitely if you try to
visualize it you're not going to be
visualizing the internet you can be
visualizing a very fast carrier pigeon
yeah or something you know it's just
you're gonna be way off on it and it's
it's hard to even identify a single
dimension it went along which like
you're answering and so I guess broadly
I feel to some extent that's a little
bit we're at with AI right which a long
way of saying I think it's really hard
to be to be sure I think just in terms
of the the sort of weaker standard of
like how I'm sure we like how sure we on
the basis of glue we understand the few
considerations we have that the very
least what make laugh sense for you know
decent chunk of the game there's meant
to be focusing on this yeah I do
actually feel I think pretty good about
that like I I think ideally I would
maybe have a a slightly smaller portion
ei portfolio focus on AI but it seems to
be clear to me that you should at least
have you know you know a few dozen
people like really actively thinking
about the long-term implications of AI
and it seems like we're not that far
from out number sadly at the moment hmm
okay wait that sounds smaller than I
expected like so so when you say like
you know like a smaller like a
proportion of the portfolio yeah be
focused on AI what's your current view
unlike what that percentage is um yeah I
think that's I think that's very
difficult um so but like ballpark yeah
so it's a not unreasonable if I'm I
think it's yeah it's really hard to
think of like exactly how I'm defining
the display between things yeah I didn't
maybe I want something like one in five
people who are like you know sort of
fully engaged like long-term oriented
yet people to be thinking about AI
primarily right okay whereas now you'd
like you'd feel like the number is like
if a in five four and five yeah it feels
to me like it might be more than half
I'm not sure if that's correct right um
yeah yeah yeah okay and so and so what
would be the like intervention that you
would like to have her like what would
you like these people to be focused on
instead yeah um so I think just broadly
it's it's I think it's love uncertainty
about you know something a lot of this
is just base
the skill sets that people have and
never worked at people naturally
inclined to do so it's possible that
just the basis of there being lots of
people with you have computer science
reads and things like that
I mean to me a that balance I'm like
justify the the really strong AI focus
but just imagining completely fungible
people it's sort of like work equally
want anything yeah I really still think
that sort of fundamental like sort of
causation research is like still pretty
neglected yeah so I think similar like
there's a lot of really could work I
think being done at the global priority
is since a few but I think there's just
there's some broad topic so just seem
you know pre-development along to my
something just not that many people are
thinking about the team kind of crucial
so these questions like for example
thing I felt Rama was working on um
should we just be you know trying to
basically should we assume that we don't
have really great opportunities to
influence the future now relative to
what future people might have if we sort
of save our money and now it's a rate of
chemo light over time should we be
trying to sort of pass resources on the
people in the future yeah to sort Vinci
generalize it seems like very crucial
for the for the like long-term is
community yeah
broadly even with any eye as well I
think there's strangely not that many
people thinking about just for someone
that a level like what exactly the
pathways for influence and a safety
ninety governance what exactly is the
nature of the risk yeah I think there's
a handful of people doing this this row
of work to some extent like part-time on
the inside you know and I have other
things so except like Rowan Shaw has
someone identifies like someone who's
doing I think a lot of grade like what
exactly is a case for high-risk but
there's not that many people on that
list compared to that the set people
working on AI safety I think there is
just it seems like some abstract
argument that just before you put you
know a ton of resources into doing
object level work it's you know quite
useful early on to try and put a lot of
resources into yeah yeah part is asian
between different kinds of object level
work or try to figure out what exactly
is the thing that's motivating this
object level work yeah yeah so one of
your one of your like complaints in that
talk was like basically there somewhere
oh it seems like we're putting a lot of
research into this but I'm not seeing a
lot of like proper sort of like
write-ups or people like actually laying
out the arguments that are like
motivating them to sort of make these
make these career transitions or like
moving a lot of people's choices do you
think we've
seemed like an improvement there I guess
like there are a few things I've been
like yeah sense that yeah so actually I
think there's definitely been there's
definitely a degree of improvement since
I gave a talk so the inkling I gave the
talk was maybe this was the low point in
terms of so I think there's this initial
period of time where you know super
intelligence had been written and for a
lot of people that was like roughly
corresponded to you have the picture
that they had of like the nature of a
risk and the motivation for a government
and things like that and then over time
there's some sort of transition the
people often having like fairly
different visions of what the nature of
the risk is or what the motivation is
for a governance and you're a lot of
different dimensions about to some well
you know one dimension is
superintelligence focus a lot on sort of
this like very discrete transition to
world with you know like advanced AI
it's it's sort of it sort of presents a
picture where there's not that much it
interesting stuff happening and then
there's maybe you know a day or a week
where you transition into having you
know quite advanced systems and a lot of
people moved away from that and also a
lot of people I think you know myself
included had started thinking more about
risks which weren't necessarily safety
oriented which although those are also
discussed debating superintelligence
definitely like the main focus what I'm
but no safety oriented oh so so it seems
like there's lots of concerns you might
have about you know the future of AI not
necessarily being great so just um you
know I mean someone see like major
category it's just um it may be the case
that you know if you transition to world
where human labor is like you know
mostly worthless yeah and you know the
functions of the government can be
automated you know maybe that's not a
great great you know world in terms of
like democracy or like the can the
preferences or values and those people
being taken into account right
or you might be concerned for example
about you know if in the future there
might ultimately need ethical questions
around like the moral status of you know
AI systems it also seems like maybe
those decisions are made made wrongly
one way or the other
and I think there's lots of ways you can
imagine or just you know maybe there's
some just best case in there on you you
have for how a US government how it's
used and maybe even if you know stuff is
okay maybe the difference between the
best possible world you can achieve and
you know like a mother we're always
actually you know quite substantial
right right okay so these are like
wrists aren't like accident risks from
like very powerful systems or something
like this yeah basically where I guess
one way you know from payment is you
know we've had like major technical
technological transitions in history it
seems like you're not uniformly good so
you know classic one is Neolithic
Revolution
you know agriculture introduced that has
a few you know later I'm not gonna facts
like the rise of the state and things
like that it's a bit difficult to do an
overall assessment but um you know
that's the example of technology change
world really radically and a lot of the
the results were just really very not
you know very not positive so things
like slavery becomes a massive
institution and people seem to come more
nourished and disease you know becomes a
thing and you know you know hierarchies
or establish as opposed like more
decentralized decision making right it
seems not that hard to imagine you know
the same way if you make if you do
another transition it's a human labor
becomes you know replaced by by capital
basically that that might have various
knock-on effects they're really not not
you know exactly what we want right yeah
yeah and then like so in these previous
transitions that sort of the bad
consequences that we've seen have like
primarily been these like structural
effects or something yeah exactly yeah
yeah like yeah slavery becomes like more
yeah more economically viable and these
kinds of things and so this was kind of
like the low point so like yeah like
November 8 2018 it's a low point yeah in
the sense of like in sense of yes in the
sense of um people have like seemingly
you know their justifications have
changed quite a bit
a lot of different areas but that's not
be reflected in almost any published
writing it's still you know roughly
superintelligence and then I suppose
also like some polka shown blog posts
and yeah that's roughly that's read the
state of state of affairs yeah instantly
I mean it's not been I think a massive
improvement but definitely there has
been a fair amount stuff that's come out
since then that you know I think has has
been useful so example I believe this
was off for example like like Paul Chris
Shauna rode up you know a series of blog
posts that we're describing an argument
for I guess safety risks even in the
context of continuous transition Richard
know did some work trying to talk
sodomize different different arguments
and sort of like lay out what the space
county was Tom seller did some similar
work
Rohan Shah also wrote um I wasn't really
trying to prevent the case for AI risk I
think a really good sequence called the
value learning sequence missed the
series of essays trying to lay out I
would say roughly like a picture of the
nature of the alignment problem and then
I believe this was also afterwards I
think reframing yeah I believe so I
think eric drexler his work yeah FHI i'm
also came out framing he's like quite
different picture of sort of future AI
progress with the nature of the risks
there are sir has been you know actually
a decent amount of this stuff I think um
star think quite a bit less than then I
would ideally and also me really put out
a paper on what they're planning based
optimization which corresponds one of
their main arguments for why they're
worried about you know a iris that
wasn't example in super intelligent I'm
starts gonna throw him on this stuff I
think it's still quite a bit less than
then I would ideally want right because
I think still yeah I think still there's
lots of viewpoints or I'm gonna capture
the existing writing and I think as well
like a lot of it just really is a fairly
short black post saying well you know
there's those are useful I just don't
really feel very comfortable I think
with a lot of sort of I guess
career time and a lot of resources being
put into an area if a lot of the main
justification is just fairly short blog
post and some of them also you know this
roof it's obviously really really
difficult to communicate clearly about
this stuff because we just don't me have
the right concepts or the right picture
of how things will go but definitely you
know I think it's not uncommon for
people to like have arguments about
let's say what a given post is actually
like sort of saying which seems like not
not necessarily great sign for us is
community really being on the same page
about what the landscape of arguments
and considerations is right right and
why do you think this is like do you
think that we like like why are we in
this and this particular situation are
people like individually making mistakes
here
like they should have spent time on this
like metal level question just I think
to to some extent yes I mean I think
there's a there's a few things that they
make it complicates one is I think this
sort of stuff is in fact just like
fairly difficult because I think to some
degree ever like I think to some degree
it requires both of these understanding
of like what all you know what's the
sort of current landscape of arguments
consider it especially in safety realm I
think it like it does require off and a
pretty good understanding of like just
what's going on and safety just a good
understanding of what's going on in
current machine learning good ability to
do things at conceptual analysis and
synthesis and things like that and
that's you know just maybe not that many
people in that bucket at the moment
another aspect which I think also helps
to explain it a bit is I think aloud to
people who are currently working this
area have just you know quite recently
coming to it and I do think that there's
a sort of unfortunate dynamic where a
lot of people have the sense that the
you know arguments or problem framings
or currently like more worked out than
they are and they it's sometimes just
having been published or they're in
someone's head or you know all the
people in the know you know have a good
understanding and then especially
becoming to an area you know just sort
of begin with you're not necessarily in
great position to to sort of maybe do
this like high level framing work
especially cuz you don't really know
what's out there what what exists in
terms of like unpublished Google Docs
right and so I think it's quite easy to
just like and maybe sensible if you're
just you know entering the area - I'm
not trying to do as high-level stuff but
sort of pick an object level topic and
I'm trying to work on that right and
then I think a lot of people just you
know and then once you're doing that
it's like it's somehow it somehow feels
maybe a bit unnatural to sort of drop
the object level research program you
now embarked on - to this rift stepping
back thing like yeah I think there's a
lot of people who have felt inclined to
start doing let's say high level problem
investigation in their sort of spare
time in a sense while doing you know
their their object level thing is the
main thing but for that reason it does
seem like a somewhat difficult
transition to just drop an object level
project you have to write to this
somewhat more loosey-goosey like like
what are we even yeah
do anything yeah yeah are there like
particular view points that you feel
like haven't been accurately represented
or like ought to like be written up like
more thoroughly yeah
sir I think maybe I guess so I think um
even the ones that have been written up
to some extent I think could really
stand to be enough like further and and
more clearly so one example is something
the Apopka Shawn does for example have a
few good blog posts describing I think
one was called what failure looks like
where it's meant to be sort of saying
like you know here is what like a bad
outcome might look like even in the
context of like a slow transition where
stuff is fairly gradual like here's what
like a bad safety outcome life might
look like
I think the blog posts are quite good
but also at the same time I do at least
personally feel like there you know
there's drove a limit to how thoroughly
or clear you can communicate and I strip
block those and you think there's still
a lot of ambiguity like about what
exactly is you know what exactly is the
scenario being scrab like what does this
look like like but we're talking about
something that's like you know an act of
disaster or just like lost opportunity
like what exactly is your argument this
being likely because to some extent it's
it's trying to just present the picture
what it might look like as opposed to
making the argument that this is you
know this is necessarily plausible I
think yeah just I think basically like a
lot you know more kits or be done there
I think similarly for this idea of like
Mesa optimization which um is now I
guess one of the primary justifications
that Mamie has for you know signing a
high probability to a sort of safety
risk I still think is lava immidiately
about what this concept exactly
exactly is um like it seems like
different people tend to characterize it
differently or perhaps misunderstand the
paper or the paper is may be ambiguous
or things like that but definitely
doesn't seem like everyone's on the same
page about what exactly this thing is
right and it also doesn't really present
the case for the paper doesn't know you
try to do the thing of um it it's River
argues that there might be this
phenomenon that it's calling Mesa
optimization but doesn't really you know
try to make the argument that because
this phenomenon
arise then we should beer as an
existential risk or like a plausible
existential risk so that I think that
works till like hasn't really been done
sure yeah so a lot of these so the
arguments that like are out the ire they
like ought to be like scrutinize more
etc are there like arguments or like
classes of views that you feel like
don't even have that like the initial
yeah you know written up thing that you
can actually just like starting yes I
think there's a couple so I think one is
um there's a bit of us for example Alan
has done has done some of this work and
is you know still thinking like quite a
lot about this stuff but in in terms of
them I think they're still not a lot
written on the idea of structural risks
we're just this this concern that maybe
stuff just is like gets really bad in a
Java sensor is just very disappointing
in terms of our current values and it's
a bit nebulous it's a bit like you know
the Neolithic Revolution where it's like
there's some concrete disaster it's just
there's some sort of structural forces
that just push things in direction that
you know but you really sort of you know
ideally wouldn't want I think similarly
a little bit of writing but really
definitely not a ton this question um
should we think that certain kinds of
you know future AI systems have moral
status and if so is it you know
plausible risk that just that won't
really be like respected and that will
be an important enough development to
you know for like long term is to really
focus on I think those are probably two
of the main things that stand out as in
my mind plausible justifications for a
lot of focus my eye that now it's or
struggle to point someone to you know
long ride up and making a case for these
these pathways justifying like eaf era
right right okay and then if I were to
like turn the thing around on you like
okay so like you're you're here you're
working on these issues like will be
you're like stab at the like the
justification yeah I mean I think some
of my stuff a lot of my stab at to
justification
it's you know very very not not details
basically there's not a thing of okay so
we're quite you know the starting point
is were like it seems very very likely
that AI will be really really transform
like it seems most people expect yeah we
will eventually it to the point we're
just you know human labor not necessary
for things or promotes things and that
world whatever that looks like it's like
obviously extremely different and then
we have this you know a set of however
you want to count them like maybe you
know half dozen ish you know big
sometimes vaguely sketchy arguments for
why there might be some you know II a
path like you know place like Yale
average to try and make this girl like
much better or much worse or like why
this could go much better and much worse
and so it seems like there's not that
many maybe sort of taking like a
long-term perspective and is thinking
like what topics might have really long
run its gonna fans that there might be
some chance to influence I think that
there's that sort of things is not that
large it's relatively hard to find them
I think and then would lease at the
stage at the moment where I think
there's a ton of at least value of
information and trying to get much
clearer about what's going on with AI
but it seems at least you know one of
the few places right now where I think
it's like plausible um long-term it's
like you know have an opportunity to do
useful future work right um and so it's
a bit of another one where I think so
the argument is like oh it seems like if
you're a long term rest you should like
think about the like the areas where
like that seem like they'll have like a
large lever leverage of the future this
seems like a like one of our best bets
of well yeah something yeah yeah I mean
I think yeah that's basically my
viewpoint we're just I think most things
just you know if you look throughout
like history most things that have I
been you know mattered at one cooker
point in time just um it just becomes
extremely ambiguous whether significance
was you know if you look at hundred
years in the future and especially if
you try to think like how applause what
it is it but like people could have you
know none with the impact would be it's
like let's say you know if you're in I
believe like thirteen hundreds yeah you
may be the big thing going on at the
time is like Genghis Khan yeah yeah and
it seems like you know now you eat like
the folk maybe your focus is a thirteen
others person it's like the Bartlett's
kangaskhan things seems like really bad
yeah exactly we should you know really
focus on you know I'm stopping this
Genghis Khan thing I think you know
probably
would be ethically justified but you
know at the fully long-term perspective
it's not clear if you know like you know
this historian I would say things like
oh well like you know get maybe Genghis
Khan was like good in the long run
because the trade networks or whatnot I
think it's probably we probably really
don't have a great sense and so you know
there may be like a you know from the
short-term perspective like a good
justification for focusing on getting
his cotton at the time yeah but if
you're you know long-term research
turnaround and thinking like what's the
most high leverage you know thing I can
do to affect the future I really don't
have a time face that like you know you
would have really been have to predict
like what's good and what's bad from a
lot long-term respectively right I
doesn't give you pick like any given
century more than like five centuries
ago you're gonna you know probably wind
up in the same position with the the
issue of the day I'm and so yeah I think
just we have a strong prior that just um
most topics we look at today we just
really can't like we may be justified
and working on like a you know like just
from their presidents by Freud like
doing long term prioritization I think
we should ever prior that most things we
really can't predict very well like yeah
you know what the difference will be
hundreds of years in the future and so
then there's not that many candidates I
think I think interpreters at least semi
plausible or eventually or seemingly if
you squint at them so many plausible
arguments for AI be in this bucket yeah
I think that alone is reason enough to
want to put you know non natural
resources interact one thing this
argument suggested that was that perhaps
it's some more useful to put in resource
put resources and try and figure out
just like what is going on here then
maybe object level resources at the
moment because lava there's a lot about
even for me value of information in
terms of trying to figure out like you
know what's going on hearing what's the
case for focusing on it yeah yeah cool
yeah okay so yeah Marcus so I am
although I work here I'm really not very
observant I'd be interested if you could
tell me sort of you know what discovery
I sort of currently up to there in space
mm-hmm cool yeah um so I guess like
broadly what I think that we should be
doing or sort of like what we're trying
to do as an organization is trying to be
the place that does sort of the the best
research in this
yeah governance space or like what you
might call like long-term is the a
governance space so working on a
governance questions from like a long
term perspective and like yeah what I
spend most of my time doing is like
doing what I can to make sure that we
like build that organization that does
like the best research in the space
basically and so like what that looks
like in practice is you know we have a
lot of different research projects going
on and I personally spend a lot of time
on like us like recruiting and like
growing in a serving station and so that
means for example where we're running
this like cubby I fellowship we're
bringing in people to do basically like
three months stints of doing research on
some topic that like relates to the
kinds of things that we're incident and
then sort of that's like past like a
path into this like a governance space
basically and that's something that
we'll probably yeah we'll continue doing
like I think for the foreseeable future
like maybe bringing in like roughly like
10 people to like every year I guess
we'll see we'll see this year
how like whether we'll be able to like
bring people in this summer but yeah I
expect I expect us that I continue to do
that going for it and I'm like so far
pretty excited about that as a way to
like get people into the field so like
currently like this like a our
governance field as sort of like I guess
like we conceived it I guess like you've
been there since like the start yeah
like since like some time in like 2016
roughly yeah basically and so given that
it's not like you know you you haven't
had time to build up like a lot of like
institutions that you need for like
building a field you haven't like built
up a lot of like very clear like career
pathways for people and I think this
like fellowship is like one one example
of that that we can like start building
for people and then the other thing that
I'm like really excited about us doing
it's like continuing to find like great
excellent researchers such as like
yourself
and bringing them into the team and then
well like I guess like a few years down
the line I guess my hope is we're a team
of like I don't know we're like about
like a dozen of like truly great
researchers
hanging out together here in Oxford
doing really good research
that's kind of my kind of my help and
then in terms of like the research that
we'll do I think it'll like currently
where we're sort of a funny organization
and that we're like the way that we're
like defining or like thinking about
this like yeah governance problems is
like very very broad so like one way
that I like sometimes describe what this
like a governance thing is it's like oh
it's like all the things that are
required to like make a I go well that
aren't technically I safety and that's
like a lot of stuff so that's like
several things that we do yeah and like
it like it'll span like you know feels
from like economics like IR to like law
to policy to like a yeah a tremendous
amount of different topics and so and I
think that will like probably continue
to be the case going forward but I'm
hoping that like over time will like
start building up more and more like
narrow expertise so like as we like
hopefully get clear on this like meta
picture that like you're among other you
among others are like working on we can
like get clear all like oh here like
some specific fields that we need people
to work on and we can like start like
properly like figuring out a lot of
questions so like I think a few years
down the line I would like really really
want us and like others in this space to
have like just like really solid at
least like somewhat solid answers to
questions like I don't know like a
company comes to us and they ask like
what kinds of publication norms should
we have or like how should we like like
what sort of internal mechanisms should
we have to make sure that we like are
held accountable to these like lovely
beautiful principles I've written about
about how will like benefit humanity
with our research or whatever I don't
think we we have that so far but yeah
that those are like examples of the
kinds of questions that I'm like hoping
and thinking that will like make some
good progress on in the next few years
yeah I guess once your last question
besides just you know apply for the
gabbai fellowship do you have any other
like recommendations for just
career-wise like
people who might be interested a should
do if they want to potentially enter it
or get a sense of if they want to enter
it yeah yeah I think like in general I'm
like I quite like this this sort of
recommendation of like if you if you're
trying to figure out whether you should
be doing next and like try doing a bit
of X so I think that's a pretty good
suggestion so find some bit of research
that you can do and try to do it I think
for most people like so currently in
this space yeah there aren't a lot of
opportunities that look like this
Gaby I fellowship at least like in the
like what you might call like the
long-term is yeah governance space but
like other things that look similar to
that might be like the Center for
security and merging technology so see
set based in Washington there's like
sometimes they're like some like junior
ish roles in places like deep mine and
open AI where you could do like this
kind of research and this kind of work
but I think like that alright that's not
like a lot of roles that's like probably
like I don't know like tops a dozen a
year probably less and so I would
encourage people to like think much more
broadly and so things that you could do
are like looking at like one thing I'm
very excited about people doing is like
looking at trying to work in sort of
policy teams etc in like a wider set of
technology companies so working out like
places like Microsoft like Facebook
these kinds of places as they start
building up like policy teams or the
Catholics teams I think would be awesome
to like have a lot of people like join
these kinds of roles I guess another
thing that a lot of people are doing
that I think seems like reasonably it's
like like a good idea for a lot of
people is like just like using your
studies to like dip your toe in the
water and so some people do this during
like a bachelor's or master's they'll do
like a dissertation on like hey
governance related topics and others
will like I think quite a lot of people
currently are like looking into doing
sort of like PhDs in this area as well
which I think Beach teas like one reason
that PhD might be a good option is just
that like if you're a field and you're
like trying to grow it's gonna be like
very difficult to it's very difficult to
open up new roles so in some cases it
can be like much easier as a field to
grow by like I guess
a poor way of putting it it's like
co-opting other roles or something and
so like you know they're all these PC
like positions out there in the world
and so we could probably like have some
people join those and like you know
swivel them around such that they are on
like topics that we think seem like long
term is like really really important in
this area I guess like the other like
main tip is just like engage a lot with
the research and so like yeah read all
the all the things like coming out of
the institutions such as us but also
like places like that like see set etc
and in doing so like try to like really
really engage with the research and like
when you read it like yeah like keep in
mind that like these people who are
writing these things they're like the
next Martin they're like good at what
they do but they're like by no means
sort of any Oracle's or like don't
actually like have that much knowledge
actually there's like a lot of
uncertainty in the space and so you
should like yeah go into like reading
these things with an attitude of like
you know
so if keeping an open mind to a lot of
these things being wrong and like being
very very like critical and like trying
to like actually form your own
interviews or at least like yeah I think
like another like similar thing to that
eyes like don't just like read things
and sort of take people's conclusions
instead of try to like build them up
yourself from the ground up or something
so like try to build up your like
internal models of like how you think
the world would go so it's good very
wise do you have any any tips or like
what would you have told your like past
self I'm I don't think anything really
beyond what you said I guess why we've
totally and past self is just check your
email in case Allen Defoe sends out an
email looking for research assistance in
this area right with like it with like a
fairly non-competitive process at that
time it's not that many people were
interested in yeah okay so like get real
lucky get real lucky is something
exactly the way through my past oh yeah
yeah yeah cool yeah well it's it's been
it's been fun chatting I'm great yeah
and enjoy the rest of the broadcast yeah
and my new sign off is stay safe stay
sane see you later


AI Safety Coordination : What Capacity & Projects are we Missing? | Jaan Tallinn

feel free to so from now on we are going
to be recording this session and so with
the option to publish this as I think
that many of the things that we'll be
discussing in the remaining minutes will
potentially be quite valuable for a lot
of people that are seeking to and that
are seeking to start and organizations
in this space all right
and so I've already dedicated the slack
channel and I've said a few things about
the note-taking I think we're gonna have
a short conversation which I'll be
kicking off with a Q&A with a young
before we open it up really to all of
you guys as questions and all right so
without further adieu and yeah maybe a
few words about you you really have a
really strong physics background you are
quite a devoted consequentialist and you
see you a comparative advantage and
existential risk and I think you said in
your introduction really as a as a
catalyst there also in addition you know
after being born behind the Iron Curtain
and Soviet occupied Estonia you then
became the computer industry really in
Estonia and before and becoming I want
after the Skype voice and and you know
since then you know since you were kind
of like really fundamental efforts and
in getting Skype going you have sense
used your ability and to really think
from first principles and your ability
to connect people and to catalyze kind
of project including your opportunity to
fund efforts in incredibly effective
ways to reduce existentialists so I
thank you all really and for joining
today and I think we're gonna be talking
a lot about a few of the projects that
you actually helped launch and
throughout the session but maybe you
know as a little bit of an introduction
at first
what made you initially interested in
the AI safety space at first and how has
your thinking on those priorities and
safety kind of shifted since then
right so I think I got interested in
this topic in 2008 when I stumbled upon
les wrong or packed animals it was kind
of overcoming bias and aleeah's arse
writings there and I I found like you
know it seemed that they were saying
something that is incredibly important
and when I look around like nobody was
paying attention to that so I I should
shut the LEAs or an email and we met up
in March 2009 11 years ago and and
basically went from there because I
really got convinced that this is
important now one thing that has changed
over time as the current community has
grown and there have been like more and
more new points is that I become like
much less certain about the future
initially I was like very strongly
bought into eleazar's like heartache of
towards ending in a few minutes after a
starts building itself improving itself
but now I'm I do think I give like
significant probability mass as in like
definite double digits more than 10
percent to more complex views by for
example Paul Cristiano or like AI
impacts and so it's I think it's yeah I
got a lot more humble in terms of
epistemic so I would say Wow okay yeah I
think it's it's interesting to see how
you know the community kind of like
updates together over the years and and
I think that that this one is quite it's
quite prominent for I think really
updating and updating quite quite
effectively so and so I think that um
you know we can talk a little bit about
your other worries in terms of
existentialists for example as they
pertain to biological risk and about
kovat in a second but I think you know
to focus on AI for for just a minute and
you know you in the the way in which you
kind of like propel projects and in the
space and the way in which you've
started
is that you know you don't have one and
like umbrella organization but you know
you have like an investment arm with
which you have fund a few M AI related
projects and you also have a nonprofit
that funds a variety of different
organizations so I'm kind of like
initially do you think differently about
your investments to your philanthropic
issue
efforts or do you think they're they're
basically tracking the same thing oh
yeah so I do indeed spend quite a lot of
time on just investments and I like
portfolio of like well over 100 startups
by now and and in in this investment
portfolio there are like a few
categories or few classes of investments
and definitely one category I call GaN
strategic investments where I don't
necessarily invest in order to in order
to make a good ROI on investment but I
do think like how the world would change
if that company would be successful for
example like in general I'm very cynical
and skeptical about health care
investments I think this is like corrupt
marketplace so it's kind of very
difficult to look at that look at
technology and make predictions about
where this thing is going to be
successful or not because it doesn't
really depend on technology so much but
like still there are a few things that
I've been messing healthcare because
like if they against all odds and end up
being successful the world would be just
much better place and also like I've
done a few investments in in AI
companies know that - yeah most
prominently deepmind in order to kind of
connect the eye community with AI safety
community yes and and also I do think
that some there are there's this concept
of
impact investing and I have enough
it's not obvious is it is like how does
this thing work because from it's like a
basic truth that if you're trying to
optimize for two separate things on a
very high level you're not going to get
like best from from either either think
so like if you are a for-profit
organization that that is trying to do
impact that's trying to maximize the
impact well like you sort of have to
choose like I'm going to go for profit
or you're going to go for impact at
least on a naive level on the other hand
like basic what I'm saying is that like
if you're not like not-for-profit then
you have like then you're less
constrained in what you can do you can
kind of direct the address traget of
Commons on the other hand like these
commercial constraints can be can be
usefully constraining because you will
end up in now in a nice short feedback
loop so yeah I do think that there is
like a definite overlap between my kind
of investment activities in my
philanthropic activities okay very nice
yeah I mean I think as a
consequentialist you know there's also
this concept of like earning to give and
but I think you know that even even if
you're earning to give you know that
even when you're earning and the two
aren't mutually exclusive as to just be
trying to monitoring or tracking at
least some kind of good but I think we
are we going to also talk a little bit
about investment in the recession was
was with Luke later as a pertains to AI
risk I think for now and you've kind of
co-founded and funded founded and and
have been kind of like at the at the
inception of a variety of really kind of
like quite crucial organizations in the
space and one of them you know is the
Center for the Study of existential risk
and then it's the future of life
Institute met I met and you also donate
to a variety of organizations in the
space for example future humanity
Institute global catastrophic risk
Institute Miri and so on and so forth so
the list is really long could you maybe
say in a few words about what was the
founding story about for example FLI and
the Center for the Study of existential
risk how you came about
those two and what space do you see the
two occupying in in the space of it is
sensuous so when I entered this
existential risk racket then I gonna
feel me that you thought that okay like
what what is it what am I gonna compare
the advantages what I can bring and like
one obvious thing was just funding but
the other one was also very obvious to
me was kind of brand so so when I have
like when I have functioned as like a
co-founder
I definitely basically just I would have
invested my brand into into this and and
this like a prime example of working as
a catalyst therefore like I kind of I
can tell the founding stories but I
don't think that are super super
relevant that every sort of like
coincidences like for example with
future of life Institute it was max who
already knew from before he said that he
had just finished his book and because
he was writing this book
his first book he had cleared up a lot
of crap from his calendar and now that
had finished the book he said like I
want to fill up like empty space with
something meaningful rather than like
bring back the crap so he thought okay I
can I like I would like to start this
Institute and kind of kind of be my
co-founder sure and with you Hugh price
he has actually written like a New York
Times article about how Center for
Caesar got started and I was just I got
me selling AI risk to him in in a taxi
cab in Copenhagen oh wow
well it's it's quite a romantic or do
you still have if people are thinking of
founding organizations and or in the
process thereof do you have any kind of
like personal tips of you know how they
may go about founding a new org or
what's important there to know from the
get-go
oh there are so many things and I don't
think there's they're kind of I don't
think I have a kind of top three the
things that just come to mind for
example
going back to the one of the earlier
questions for profit and not-for-profit
so like the nice thing about for profits
is that even if it's kind of like for
profit in order to have impact is that
you have kind of your work cut out for
you you know you know what the
short-term things are like an and you
know what is the what is the core of
this thing that recruiting people
whereas like if you are doing like
not-for-profit it's much harder to kind
of coordinate people and I've learned a
lot from Andrew Creech actually with
whom I worked for the last couple of
years and he says that like when you're
doing trying to do like a non-profit
people come like it's hard it's much
more like herding cats because people
expected their their voices heard much
more than they would if they kind of
join us like 10th employee of a
for-profit Institute so text one
important thing to keep in mind like if
you're doing not-for-profit expect to be
it would be like way more like latarian
for pretty good and for the bad it
surprises many people alright yeah
that's definitely um yeah there's
definitely like a feature in the back
you could say to some extent and alright
and I think if you if you just let's say
like survey the a a I safety landscape
for a moment and which areas do you
think are we doing well and already and
where do you think we could flex our
muscles a little bit more and so which
kind of like over the time over the
course of the years that you've been
kind of like catalyzing those
organizations where do you actually
think that we have a good grasp already
and at a problem and where do you think
Oh actually and we need to we need to we
need more capacity in this area so I
think strategically I have this model
that are now been calling drill and the
city so they have the idea is that
imagine like a drilling operation
weather is like a drill going into the
into the earth and that's like a small
city or like village around it that is
that is basically taking the products
and and or taking the stuff that the
drill pumps up and this product I think
those things so like the AI industries
like that so you have like drill growing
deeper into the technologies or
intelligence or algorithm algorithmic
improvements going from like I don't
know linear regression to connect
optimization to the later stuff is like
transformers and things like that and
then you get those technologies and
those get get packaged into products and
services so I think that's like a
booming industry of like AI ethics but
they only focus on what's happening
above ground so so it's like and I think
there's like a lot of expertise that can
be kind of reused from technology ethics
in general so that's why a lot of people
think correctly that they are experts
may I
ethics but I think is that they don't
even know that there's something that's
happening on the grameen technical they
know that yeah there are Terrell I kind
of seem to be some new papers coming out
or something but like the interesting
interesting question is the thing that
attracted me interesting question is can
you have an AI lab accident so if you if
the answer is yes and that's actually
that possibility which got me involved
in this like when I when it is I
basically said look we're going to have
an AI lab accident where AI starts
improving itself then like the world
will basically blew up before you get
the product ice the result so I do think
that in strategic picture there is
increasing amount of work being done in
the is aesthetic space which means
regular like constraining a
products and services and and there's
like a very little work I think which
has been counting the numbers in like
you know less than 20 PhDs are full
working full-time on what happens at the
tip of the drill like how can we kind of
constrain intelligence in general until
the work of either from first principles
or or or or kind of empirical a okay and
and do you think this this this type of
work should be done in existing
organizations do you think we need to
kind of like grow capacity more there or
do you think that there's actually like
something to be said of starting like a
new our effort that that focuses
specifically on this I would really like
to see there be more effort because
right now take after what happens at the
tip of the drill the work is only done
being in in the four cities in the world
San Francisco open AI London deepmind
Oxford FHI and Berkeley of course a
couple of couple organizations there
plus like a smattering of akkad
academicians etc but like in terms of
organizations yeah I perhaps incorrectly
but I'm not aware of people doing kind
of fundamentally a safety in the same
way that yeah
Miri is doing or or or did my safety
team is doing so I would really really
like to have like more in fact like one
thing on my once this covert thing kind
of lifts is on my roadmap is to kind of
take which on a road show the different
AI labs in the world to present his work
on wait he had for the last two years he
and and his colleagues have been working
on breaking down the AI safety research
and and safety building like a a safety
research agenda yeah yeah it's a quite
comprehensive agenda I think and that is
available as on proxy honest website
quick I don't think they have published
it yet although my information is a
couple of weeks old
the agenda is called arches at least
tentatively okay I'm not even aware of
it I think yeah it's really really
really awesome at least based on the
draft okay I was thinking of just of the
general principles and they don't impose
website but I think Andrew actually was
interested in doing a session at this
meeting on that map so that there is
right right right so I think he is going
to talk about that yep yep I highly
suggest it's so good
okay well I'm hoping that we'll get a
peek into into that map into that map
soon um okay so if you think about I
mean you've already talked about and
kind of like I guess the the spatial
spread of the different of the different
places in which AI and AI safety is
happening or not happening and do you
think and from your kind of like you
know kind of like um from your
experience and in input and in a few
geographical context where how do you
think the the kind of like existential
risk landscape differs and let's say
Europe versus u.s. are there any kind of
specifics that if people were hoping to
start organizations in one versus
another that they should be aware of
when tackling problems yeah I don't
think I had like a very good answer
there because right now there are just
sort of concrete people who some of them
happened to be in Europe some what would
happen to be in the US and there isn't
Jack enough enough people to enough
start averaging and looking at kind of
yeah cultural differences there so it's
just right now in terms of like the
fundamental idea safety research I'm not
kind of less knowledgeable about I mean
I could talk about the ethics things I
have like some window into that but I
don't think is very important to discuss
right now okay so we're not even there
yet to tackle that problem okay good so
I think one thing that you know you
shared with me and before the meeting
and were the notes that you recently did
on on the current crisis and and I've
had to look at it and and and they're
quite quite comprehensive and so I would
love I mean importantly like I just
started the document
yeah but like most of the content
actually has been a contributor to
playing by other people yes okay
well I'm okay so basically to bring the
others up to speed there is a living
document that Diane is is at least
curating and it has a lot of discussion
on how we can leverage code 19 in the
general context of existentialists and
you know there's a few kind of like
general points that you talk about in
there but perhaps it's interesting to
note father's here too and what what
what were the main findings that you
have them in that doc I mean this this
document is still gonna be commented on
etc and like they agreed idea is that
like post Kovac crisis I would expect
people to be enough
taking more serious of species white
problems are hopefully going to be taken
much more seriously post-commit as
people going to remember it especially
like an up like bio bio issues obviously
but the question is like can we gonna
generalize from it and and for example
like Singapore seems to be getting like
a lot of praise for for the current
ability to handle things so and I happen
to know like many people in Singapore
and administration and so I'm gonna I'm
gonna glad to see that that they're
there you know I knew this before that
they seem to be fairly saying government
I mean they had a quirk straight caning
at all etc but but they seem to be
fairly rational and glad to see that
that they they seem to be guns doing
well on this particular axis now the
question is can economy can we use that
kind of shift our mindset to to
generally promote kind of rationality I
once heard a podcast by the guest the
podcast was Eric Weinstein he made this
claim that like everything about war is
awesome except the horrible price of
course that you have to pay but the idea
is that like if you have war time it
kind of surfaces rush more rational
leadership and you know suppresses the
usual guns or some social games that
happen in in governance so like it's
plausible that the COI crisis will give
it's like in some ways like a vaccine
for Humanity that might kind of promote
like allergic reaction or like not
allergic reaction but immune research
reaction do that by making of the
governance of the species to what degree
we have it more rational for example UN
more started like post-world War two
right so like the recycle precedence
that that war were actually crates
crates this window during which we can
offer
promote things so yeah like the things
that have been discussed in that
document are things like just for
example just in order to elongate that
window of rationality just think about
the magnetic things like what is the
what is the narrative that we should be
promoting and the ways to promote things
are working with kind of popular content
creators for example or we can create
prices like for example future of future
of life Institute does have prices
already - you know that you're going to
promote a message that message that we
think are useful for the species then
yes and the other thing that we were
thinking about like like unique kind of
looking at which garments actually we're
better at dealing with this and trying
to kind of use them as examples as good
examples to to make the rest of the
planet more rational and also I think
things like prediction markets
I just gave bunch of money to meet
oculus because they have like 20
pandemic specific sub site and so they
can increase the prices etc to actually
I do think that having more decisions
based on on you know objective
predictions would be good for their
species really yeah I totally use
meticulous at vision weekend to do like
a year-long forecasting tournament and
we're going to give out a prize for that
as well later this year and I think to
your and to earlier points I think Tom
Kalil with the Schmitt futures they are
already doing some grant making at least
for for covert 19 that's not really a
prize and so it's lacking this kind of
like public aspect of it but at least at
least there's some grant might like
making happening they were going to be
talking about later and I think in this
with mocking Christine we're also gonna
be talking a little bit about and I
guess like which type of governance
systems may may be better or worse for
this type of crisis but I think what you
mention of like there's this vacuum
right now we know to claim the narrative
and to actually and like do something
substantial and I think is something I'm
important and important to consider so
I'm I don't know I think are you
limiting the number of folks on the
document to about a hundred by perhaps
if people are interested and to having
access and to actively contributing they
could shoot you a message or something
and and you can get from their instant
yeah I guess so can't deliver the trying
to kind of keep it keep it small for for
now to some people wouldn't be like kind
of like Chatham House Rules reason like
I don't want people to be kind of like
commenting as if they were in a public
discussion yeah all right okay that
points taken I think you know I would
love to hand it over to a few
participant questions and if you'd like
to either raise your hand here in this
meeting or if you'd like to see if you
have a question I I think just feel free
to to ask right away either here on
slack we can see Andrew Andrew Rogers
you have a question yes I do have a
question I like the analogy about the
drill and then the cities around it we
really can't see what's at the tip of
the drill I actually do a lot of work
for the government's doing surveillance
of advanced computer science that is not
public in designing programs around so
my question is to what extent do you
think we really have an insight into how
many drills there actually are out in
the world and and you know the kinds of
ways we can actually kind of keep tabs
on that my sense actually is from the
work I've done is that we don't have we
have a far less of a we see what we can
see from the academic that people who
publish a lot but what has concerned
governments is that they when we start
looking around and doing you know more
systematic surveillance we find other
things that really are not on the radars
uptick popular AGI called mean sphere
you know communication channels oh yeah
so it's it's totally possible that there
are sort of covert things happening
there is like there's at least one
counter-argument to there being like a
lot of covert progress that humanity is
making towards AGI and that is or
actually like a more specific argument
argument that governments are not
investing heavily to towards kind of
superhuman capabilities when it comes to
like calm general general spectrum
supreme capabilities and that is
somebody pointed out that it's actually
way easier to hire top-level AI talent
than it is to hire top-level
cryptography talent and we know that
governments are heavily investing in
cryptography and that's why it's kind of
very difficult to start a cryptography
company and get like top level top level
talent or even like computer security it
seems to be like really really tough
market to to get get people to join the
X risk area safety community but like
yeah just I mean deep mind was basically
the core of the deep mind was just like
mechanism to hire talent what what Timmy
Steele was just like play around the
world and talk to the top professors in
different universities and made sure
that they got options in deep mind so
they will say what they would Center top
students to deep mind so so that's like
a counter counter argument while they're
probably aren't like that many but it's
just it's not super strong control
agreement I would definitely be
interested in having a better
understanding of what's actually
happening on this planet all right
thank you great anyone else feel free to
raise your hand even in present yeah
move on yes so for others that may be
interested in
donating or investing do you have a list
of organizations that you involved in
and maybe any investment thesis for each
one yeah so actually like one thing that
again Andrew Grich has helped me
recently all spearheaded and also Oliver
haruka from les wrong
he has been has been contributing a lot
in terms of kind of code we've been
working on system that we call s process
and right now it's still like in a
better mode we going to do like a third
grant round but it's the idea is that it
should be like a scalable process for
Forgan of involving fairly painlessly
involving additional donors in in a way
that doesn't I mean I could very long
about this process is a wonderful thing
but okay let me just a quick quick
mental image is that the idea is that
that there is like imagine like a free
layer neural network at the bottom you
have this what we call opportunities
basically people or organizations who
apply for grants the second layer we
have recommenders and they are basically
people who investigate the opportunities
and then on top we have donors and that
the job of the recommenders is to rate
we have like this by marginal value
function like free parameters that you
give to each each of the links and then
the donors is to rate the recommenders
so it's it's like it's it's a really
untidy as you can basically build up a
tree from these like free free layer
structures so the idea is that that as
we kind of will be debugging this
mechanism on very
trying to get like more and more donors
involved there yeah hopefully it would
be like something that people could be
like fairly painlessly participate
without actually having to build their
own kind of nonprofit philanthropy
organization very interesting thank you
oh wow that's neat okay very cool and
then I thought I saw Georg having his
hand up yes thank you
Jung do you perceive any blind spots in
the institutional AGI environment or put
differently if you could implement a new
institution tomorrow which does
something different from the existential
existing institutions what would that be
yeah I don't have like a very very good
object level tall serving for the last
decades what I've been doing is just
like kind of like letting others do the
initial work and kind of like supporting
them a little and see like like yeah
which one of them actually grew and and
are sustainable so one thing definitely
what what is happening seems to be
happening is that the rest of the world
is kind of waking up
even without the co-ed crisis are
getting more and more interested in in
what will become of our species and it I
don't think that kind of the community
who has been working on this for the
last ten years and has their own
internal Shibboleth is like super
well-positioned to the Judas outreach
like one thing that that is really
valuable I do you think is sort of like
a buffer between the places like
alignment forum where like top level the
SAT research is being discussed and the
rest of the world there are people I
kind of like concerned but I don't know
like tech I mean very common mistake
that I see especially in the East by the
way I've been going to China and and
Japan quite a lot recently is that
people confuse competence and
consciousness so it's a very very simple
mistake I think people say that well
machines get more and more competent and
more and more smart than state and then
suddenly become conscious but we don't
know anything about consciousness so
like let's not worry about it
it's like yeah many mistakes in that
sentence but ok so this is an example of
how how others there might be like
valuable organisations in the middle
hope basic understand both they're both
the fundamentals but also understand the
mindset of the of the rest of the planet
ok any thanks anyone else raise your
hand if you'd like even in person yes I
Joseph had a question those of us who
are working on some of that sort of what
you talked about surface level AI
alignment work constraint approaches how
would you I guess what would you
recommend in getting to a point to work
at the transformer lever other push it
to the tip of the drill how do you
recommend like switching over to that
kind of space yeah so I I mean once
under creatures roadmap is is out it's
such a super comprehensive you can sleep
basic like yeah it's like a tree
basically you can cool and like I'm
interested in this thing this thing this
thing this thing okay and then you have
like a list of papers that that I've
been and I've already been published in
this and like a very thorough
explanation of what is this particular
leaf of this research tree that you're
that you're looking at so I think like
does it like my immediate suggestion
just wait until this thing is or like
yeah ten danger creatures session if
he's he was going to have a session and
yeah global this meeting but he yeah and
yeah Gruber and at this meeting but you
couldn't attend anymore that being said
I think it's fair to say that and he's
probably happy to talk to you about it
if you reach out to him and yeah thank
you
he has also written like if you go to I
think was choice
Chai's website where he has this like
annotated bibliography and so he already
has like some someday some published
pointers thank you very much
all righty any other questions feel free
to literally raise your hand in the
video yep done yes I I guess I had a
similar question to sigh I've been
thinking a lot about explained ability
and robustness and there's an enormous
number of people working in on those
topics
so it's wondering how can we tie all of
that work into longer term AIC issues or
do you think that you know they're
they're actually disconnected so my
model is that most of the like near-term
work on this like on the city level
about which is about like face
recognition and like pretty particular
narrow products and services isn't very
useful when it comes to kind of making
sure that we're not going to be killed
as species however the things that you
mentioned like the explained ability and
transparency they I think they are like
the best shot or actually being relevant
both in the short term and in the long
term because like explainable things are
commercially valuable so you get
immediate kind of gradient incentive
gradient to work against on the other
hand explainable and transparent things
are I think which make that makes that
argument are much less likely to defect
much less likely to do like treacherous
turn and which are kind of like firmly
concepts from this on the ground say you
are safe to research okay very cool
thank you yeah thank you um is there
anyone else with a question we have five
minutes we could squeeze in another
question if you'd like
guess or exam hi
sort of on that note for the things that
there aren't as much of a market
mechanism to implement are you worried
that there's going to be a gap in
translating especially the kind of
safety work that Miri does and maybe to
a lesser extent the kind of Holland and
are doing and like actually getting the
relevant groups to implement them is in
and if so do you think there's like a
space to have some sort of initiative to
transfer technical safety work into
hydrogen yeah yeah just yes like I mean
it's kind of also it is important to
think about applicability domain of this
like fundamentally I say to research
like it's not like move that the work
that happens noticed most of the
capabilities were that happens like
above-ground it's like taking existing
research perhaps even existing code and
then training the AI for this particular
domain for example I do you think like
face recognition most of the work
commercial work is just doesn't really
do like fundamental research some
westerns just actually training and
debugging the systems etc so like there
is like less work or less
it kind of less relevant like the
fundamental research summary for example
is doing is much less relevant in those
things but like they're definitely like
fundamentally a research labs out there
that to my knowledge don't really care
about the impacts like I mean previously
at least Google brain used to be one I'm
not sure how things have been developing
recently perhaps to have like become
like more responsible over time but like
I prom what I heard some X X Google
brain research as well that seems to be
like actively toxic environment used to
be importantly about talking and talking
about their safety so like is like
definitely something that I really
really would like to see like whenever a
fundamentally a research groups in
academia are well funded companies like
that they would be actually gonna know
what the problem is and wouldn't dismiss
it alright thank you and I think is
there another question and we have about
a minute left
[Music]
look it's about the quantum computing
and and its influence influence on hgi
or strong AI could you see the potential
of the development of quantum computing
can have a big impact on the development
of HCI yeah it totally can like the
basic are my my mother I start quantum
computing also together with with other
techniques like 3d 3d chips
what is this think for photonics stuff
basically techniques that are able to
massively accelerate the amount of
compute computing operations that you
can do per second this is like not like
my model one way I've been saying is
that remaining runway that humanity has
is not measured in in wall clock time
its measured in clocks
so like this like ammount of like
computer precious that we now have left
so whenever it somebody comes up with a
with something that that can do a lot of
computation in very short time it
effectively kind of squeezed this runway
and we have like less physical time left
to get our house in order so like yeah I
think quantum computing if they kind of
manage to solve it in a way that
provides a lot many more compute cycles
then we should expect or they'd be even
left even less the time to do our
homework all right okay and with that
being said we're now at 11 you're ending
right on time so I would all encourage
you now and to head on over into the
main zoom room again and we're gonna do
a little report out there and young and
so get ready for that thank you all so
much for bearing with this and I hope
you had a great session and yeah I can't
wait to see you in a second again in a
different room alrighty bye-bye thanks a
lot