123 - Robin Hanson on AI Skepticism

initiate conversation and if I could ask
people to mute the microphones please
I'm here I was just waiting until the
sign down dead Sunday died down sorry
anyway so I'm not sure how you'll manage
like who initiates a question and how
somebody raises their hand but I presume
you've worked that out before so I'll
just trust you to know what you're doing
now I'll follow your lead it's great to
talk to you all I figured I should just
have a very short introductory outline
you probably know what questions you
want to have ask and what you want to
talk about we can just get into that but
let me just set the largest framing I I
definitely believe that eventually there
will be powerful AI powerful software
that will be much more powerful than we
humans are today collectively and even
individually if you can divide it out
that way I definitely believe that
eventually growth rates will increase
although they eventually have to
eventually later have to slow down there
will be faster growth and faster growth
could come relatively Stud suddenly in
the sense of within less than a current
doubling time of 15 years we could be in
a whole new regime of much faster growth
and that a some sort of artificial
intelligence is is probably the most
plausible explanation for a next faster
growth mode that we might enter into
artificial intelligence definitely has
the potential to to be different not
only in terms of its ability but in
terms of its preferences you know the we
are somewhat flexible in our preferences
with respect to culture and culture has
changed just over time and we are now
different people than our ancestors were
in terms of our preferences and that our
preferences can be summarized in part
has ancient human preferences that were
evolved in humans long ago say million
years ago and then more recently
culturally imprinted preferences that
are somewhat the results of cultural
selection over the last few thousand
years
and recent events of course objective
created recent cultural values and we
should expect that our descendants will
also differ from us in many values ways
both because they will just be in a
different world and they will have
adapted to that because there's just
randomness and random change random
value drift if you will and it's an open
question just how much they will have in
common with us value wise and I
understand that many people very
concerned about that and would like to
not have our descendants have different
values from us I have this book on edge
of em all about very human-like
artificial intelligence and what that
world would be like that's not the only
scenario it's a scenario explored and
there are other scenarios of less
human-like software at the moment I've
been working on a project to try to
imagine what the future after m's would
be and where human-like and non-human
like software would win and I do think
that human-like software has a long
future and can win in many places and I
can talk about that if anybody wants and
I so I think and I think non human like
software will win in other places I
definitely think that eventually
there'll be a concern that any one piece
of AI could be out of control I mean we
we've always had to worry about any all
of our devices of staying in our control
there's more of a concern in the long
run for more powerful things being out
of control I I'm relatively skeptical
about the scenario where all the sudden
one thing gets out of way out of control
as opposed to a world where many many
things are slowly becoming harder to
control or you know and so I'm tend to
imagine a world where there are many
roughly equal equally powered things and
as an economist imagine world where no
one's in charge no one's setting
everything there's more competition that
produces the outcomes and I think that
it's just too early now to do much work
on how to keep future things in control
I think you'll have to see roughly what
there's
truckers are like and what their roles
are like and what the incentives are
like and what the key accountability
metrics are etc before you have much of
a hope to to think of useful ways to
keep things in control but then anyway
so that's roughly my overall set of
attitudes and interests and things like
that and that's probably enough to get
us started and I'll let whoever is the
moderator here decide who talks now okay
thank you for your presentation
so if people could write in the chat if
they have any interesting questions and
then I will start out with the first
question so my first question was back
when I first read the AI Foom debate
between you and Elliot Kowski there
seemed to be a people what you were
talked about beside each other in that
Iliad cast was claiming that that a firm
was dangerous because it happened so
quickly an intelligence explosion was
something that that was so quick that we
would not have much time to prepare and
you were arguing that was not so likely
because it was that that a very local
fool was was unlikely so that to me
leaves open the question of a global
Foom in that an intelligence explosion
that is rather broad but also very fast
do you think that's like oh so I was
trying to signal my beliefs on that when
my initial comments to say that I do
think that rapid change is possible and
even likely so that the next best guess
next doubling time for our successful
economy would be say monthly today we
doubled roughly every 15 years so that
would be a much faster growth and that
could happen you know any time in the
roughly the next century or two probably
not in the next two decades but possibly
and that that would be literally if
that's driven by artificial intelligence
that's faster growth driven by smarter
machines if that counts in your mind as
a global phoom fine it's even to me that
a lot of the concerns were focused on
the one machine that takes over the
world in a weekend sort of thing and
that if that's not your snare you you
have different concerns you want to do
something different about it
to clarify until someone else has
written another question oh my my worry
was about a Foom that took on the order
of months to half a year or something
like that
which is basically too fast for us to
for humans to do much other than
pre-planned precautions well I would
imagine whatever happened is the results
of many organizations the world having
access to powerful machines and then
using those machines as they intended
and you know if that took a year that
could still be you know a decentralized
scenario where no one of them has great
influence then the question is what what
is it about that snare that worries you
is the key question so in the Foom
scenario the simple the local Foom
scenario the scenario is there's one
machine that takes over the world in the
weekend and then every it controls the
world it makes the world do what it was
that was the presented scenario and that
it's values would be pretty hard to
predict from that context that and that
was the thing that would make you worry
there but in a world where many many
parts of the world have are growing
together fast the question is what
exactly is your concern in that snart I
think I will return to that in the
meantime Robert has a question that
would be hard yeah anyway there so so
there's this there's this mistake that I
think people make and I know that I used
to make of kind of imagine a GI
suddenly coming into existence in our
world
one that looks very much like our well
and I think your position if I'm not
mistaken is that by the time we have AGI
the world will look fairly different in
that we will have all of these other AI
systems which are almost AGI or you know
that are like not not not as powerful
but there will be a large number of like
existing powerful systems in place
already and so this like Foom where one
things suddenly gets a decisive
strategic advantage over the entire
world becomes unlikely and more likely
to be like a multipolar scenario where
the the new system has to like take into
account the the values of all these
other systems but like I can see two
different possibilities there one of
them is yeah that there's all of these
existing like capable systems which
could keep the new one in check the
other one is that like you might have
kind of an overhang where if you have a
lot of narrow but very powerful systems
those could be co-opted or taken over or
you know hacked or whatever by a new
system that just gained the that's only
really good at like hacking and and
psychology and strategy and that kind of
thing yeah yeah I'm wondering which what
seems more likely I would regard those
other scenarios as other variations on
local food maybe elaborating that
category makes you think the more likely
or more concern but the key difference
is does one thing grab control of
everything or are there many sources of
power and influence that are roughly
comparable so right you know the default
the multipolar scenario the default
scenario that I tend to think is far
more likely is that there are many
systems in the world that grow in
capability but no one of them is vastly
better than all the rest nor does any
one of them suddenly grab control of all
the rest right it doesn't grab the
financial markets and win all the money
it doesn't grab all the military and and
have all the bombs it doesn't grab all
the computers and and
therefore you know etc right that the
people are already people have always
long been protecting their things from
theft and that there's no sudden Foom of
theft to allow one thing to grab
everything through whatever channel
theft war clever persuasion interpretive
dance yeah okay yeah cuz I'm just kind
of thinking about it in the same layer
of overhang that people that people are
talking about like hardware overhanging
all around overhang would be a like a
when you would get one little thing that
allows you to also get a whole bunch of
other things in essence it's a
refreshable threshold effect so
threshold effects can certainly allow
more variance and ability but the
question is just how much variance are
we talking about here we certainly see a
lot of threshold effects where say any
one company gets a new innovation and
allows that company to beat its
competitors out for a while you know
we're talking about a threshold that
allows one thing to take over the world
so that's a vastly larger threshold than
we've ever seen right and so I thought
this will give me just a chance to make
my side comment I mean I decided to
instead of making longer speech at the
beginning to have some other speeches
ready when this topic came up so one
little speech that I have ready is just
the story that innovation isn't
continuous it is lumpy and therefore
there could be big lumps but we have a
lot of data on the distribution of
lumpiness of innovation and at least by
say academic citation of papers that
distribution is pretty constant across
different academic areas so it's
actually a pretty robust distribution
and in the standard distribution most of
the weight and innovation is in many
small lumps but every once in a while
there's a big lump and a key question is
what's the chance of an enormous lump in
this area obviously a priority the odds
of that are pretty low if you think it's
not different than all the other areas
we've ever seen so then the question is
well what are the arguments for thinking
that this area is unusually different in
making us expect there to be a really
big lump just at this point where the
phoom sort of thing would happen
I I guess should we go to other
questions or yes I think Ashwin had a
question following up on the global film
idea yeah so I guess pretty similarly in
this vein
I guess so it's so like it seems a
little bit unobvious to me that just
because the like level of resources is
like fairly similar across different
groups that therefore there's no
decisive strategic advantage like
there's this idea of like
offense/defense balance for example and
of offense/defense scaling which i don't
know if you've seen from alan Defoe I'll
link to the paper here which basically
argues that like it's possible depending
on the technology and depending on the
situation
for more resources coming in to either
allow for more decisive first strike or
to allow for sort of defenses to better
protect against a first strike and it's
not at all obvious to me that's going to
be the case in like every area or most
areas that resource investment do day I
will be defense scaling rather than
offense scaling if you want on a frame
it in those terms like allowing for
prevention of getting your you know
technology hacked or resources still
under whatever rather than the reverse
and so it seems possible they you
couldn't fact have this kind of
snowballing and doesn't seem that we've
hired any particular sort of large lump
necessarily so much is just like the
ability to have like a moderate
advantage that you happen to be able to
like leverage and really I would call
that a lump if you happen to stumble
into a regime where you get a scaling
advantage then you end up accumulating a
lump the question is how often do such
lumps appear I think often in these
discussions there are just two very
there's two different styles of analysis
one is to look at you know historical
data sets and ask what you know how
often is there something like this event
and another is to just look in the space
of abstract mathematical models and say
what fraction of the space of models
looks like this and usually the space of
models has much larger fraction of very
concerning scenarios than the space of
actual data so for example if you look
at economic growth models
you know abstractly mathematically it's
just as likely that more growth produces
excel it makes growth accelerate as it
makes it decelerate but it almost always
the way we see in the actual world
there's very rare to have accelerating
growth scenarios that at least last over
very large scopes and extremely common
to have decelerating growth scenarios
sure um and similarly for this
offense/defense thing I mean yes in
principle you could have an offensive
about instead of let's one anything take
over the world but we've had a lot of
history of offense defense and that's
almost never been true on the large
scale I'm sure but I guess I call points
in that when it one is like this is a
slightly different point but related
that like it seems like resources are
like pretty lumpy like if you look
especially if you imagine like I don't
know like it seems fairly obvious that
if you have this kind of economic growth
type effective AI it's gonna be like
governments are very interested and
applying it to like military type
technology and there in particular it
seems like maybe like nuclear weapons
are a big concern like there's some
evidence now that like improved like
just like data analysis type stuff is
providing a greater chance of like a
decisive nuclear for a strike and it
seems like that's like a particularly
like you know wait it's gonna happen
then you could have like a sort of
ability of military dominance coming
from like a relatively small advantage
you know obviously offensive advantages
sometimes happen and they happen
sometimes happen in history and you
could look at our history to find the
things closest to that and the most
concerning and you know first strike
millet nuclear war might be the closest
to stay analog into that but you know to
leap from that to say that therefore an
AI will kill the world in a nuclear
strike is you know a big jump right it's
just it's it's itemizing the extreme
most extreme possibilities you can
identify but you should admit that you
have search for the most extreme example
and if you look at the rest of the
distribution it's much less extreme than
that I mean I guess but like it's also
true that like you are going to have
people like trying to sample from like
the extremes of like the most powerful
they can get
like but that's always been true that's
that's been true for thousands views so
the distribution we see is the result of
people sampling as best they can for the
most extreme things and rarely is
offensive win on the scale
sure yeah I don't want to close too much
so a lot of the people jump in okay then
this question is from Chris oh yes hi
I think basically it's perhaps been
answered already so the picture I get
from you is that you regard an AI where
accelerated growth
sorry Foom happens globally or in lots
of different areas as being
intrinsically potentially it could be
intrinsically safer because they're
likely to be checks and balances between
different centers of of AI basically is
is that it in just the way that
companies and cities and nations I want
to distinguish the kinds of concern you
could have so if your focus is on one
machine taking over the world min a week
and I'm much less concerned about that
but now I have another one of my
standard speeches at this point which is
to say there's another concern that I
have to get say is overwhelmingly likely
to happen and there's probably not much
you can do about it which is just the
general fact that so far in history each
generation has had values of different
from its immediate ancestors and that on
the long run looks like a random walk
and that's that you can't predict it
very well and that it seems very hard to
prevent that so that I think your
default expectation for the future
should be that when values can change
they will to some degree and that's
roughly a random walk and if you don't
like where that random walk tends to go
which is everywhere you don't like that
default future and that's a very hard
thing to not have happen I think that is
that's a default way the generic future
would play out what it's decentralized
is that there'll be some degree of value
plasticity and
value change and it will just follow
this large random walk and so if you
think that's a problem then I say you
you do have a problem you have a really
huge problem a problem that seems almost
overwhelming to me I mean in the sense
that it seemed relatively little chance
to avoid it well a couple of points
about that one is its it seemed to me
I'm thinking about this in the past it
seems yes okay
of course future generations will have
different values from our own and we
might be horrified by them just the same
way our ancestors would be horrified by
some of our values and really what you
can do is perhaps say well let's try and
ensure that the next generation or two
are going in a way that we can endorse
because and let's hope then that our
grandchildren will can just make sure
that their grandchildren have fun break
and endorse and so on that's what people
always been doing and people have always
been trying to make sure that you and
grant share their values that and their
value drift we see is the result of you
know those efforts yeah so our problem
is that that that could that is going to
change really fast its technology and
consequent cultural values it was just a
higher level climate is just anything
that you're worried about over the
long-run happens faster when changes
faster so I think you know the high
level bit to know about the future is it
will be world a faster changed in the
past and so any generic concern you have
that would have taken a thousand billion
years in the past will encompass within
less than a thousand years in the near
future so you know that's that's just
because change is speeding up ok can I
could I depress mate just one other
point which is which is connected that
and that is that you refer to
generations trying to limit control and
constrain later generations and applied
to intelligent agents in the future you
seem to be implying that our AI success
really have as much moral right to go
their own way as we would say that our
own children and grandchildren have a
right to go their own way and that
therefore we should sort of we shouldn't
be worried about being about a is being
our successes
did I read too much into what you said
well I usually try very hard to avoid
making moral claims you know I usually
try to say anything else I can closely
related to a moral claim without
actually making the moral claim because
I prefer to keep my analysis space of
facts rather than making bald more
claims than if they find that I can't
support or anybody else can support so I
might more say you some people think
that AI is more plastic and values than
humans are and some people think AI is
likely to drift farther faster away from
our values than say MS or humans would
I'm not so sure of those things I think
we will in fact make AI a lot like us
especially when they take on the
higher-level roles that we usually take
and that will include giving them on the
surface at least values that look a lot
like our values they will change and
evolve in results dispatcher's but then
so will that's the values of our
descendants so I I think there's less of
a difference there that many people see
but of course that's all separate from
the moral value posts i I do think if
you imagine a world full of AI and you
imagine two different worlds full of AI
and they're almost the same except one
is full of empty zombies and the other
is full of lively things with feelings I
must prefer the second world you know
and and I and I and I don't like it
I'm very put off by the scenario of
making sure there are no no moral
problems by making a vast and universe
with only a few humans off in some
corner then whose values matter anyway
you know yeah
okay I think the the next thing that was
written the chat was early but it didn't
seem like a question so unless you my
darling we'll go to Matthias Matthias we
can't you
that's just your microphone might be
muted hello or is it my scribe oh yeah
okay so you have your question please
hello Mitch hi um so oh you did raise
the point that having a universe filled
with lively creatures does seem better
than just having a bunch of humans on a
rock and I agree broadly with you I mean
maybe if we did wind up making a bunch
of paperclip maximises and there were
trillions of them and they were happy
perhaps that's not such a bad thing but
what about saying more negative value
drifts if we wound up creating some sort
of scenario that was say substantially
worse than the Malthusian state surely
that is something we should try and
struggle against even if we have small
odds of doing that like creating a world
where it's filled with unhappy creatures
is there is there a particular scenario
that you think that would generate that
or is it merely the the logical
possibility of it that concerns you I
think it is the possibility yes just
because space is vast and so is time so
it seems like something like that will
wind up happening and that does scare me
a lot and I would want to try and see if
something can be done about that or make
it a little less likely and hence
working on say value alignment and how
to make sure that things don't go
completely and insane
seems like a worthwhile
to do well or do you think it's just so
please me at the at the largest level my
overall attitude is to not think I or
any of us have that big of an influence
over the future so first I expect I mean
we're in a world today where no one's in
charge and our world wasn't the results
of any committee meeting a while ago and
nobody voted for this world and it just
happened and mostly the future is going
to be happening in that same way and so
I don't expect I or any of us have that
much power over where the future goes so
I think my our first priority is just
opportunistically guess where is it
likely to go and then ask given the most
likely scenarios we're on the margin
would we like to push it a bit because
that's probably all we can do so I don't
at all think in terms of like what's the
space of all the possibilities and what
are the worst possibility and how can I
design a world that prevents the worst
possibilities that just seems way beyond
my or anyone's capability to you know to
control the universe that carefully so I
would much more interested in like
particular scenarios by which you think
things could go badly and especially if
they are plausible scenarios well we've
had lots of arms races before so is it a
different kind of arms race than we've
ever had that's concern are the typical
arms race in the past the wing or about
or a different one rating creatures
generating creatures specifically for
violence well that's who we are so you
and I are creatures generated for
violence right
sort of but I feel like there's a
distinction that can be made here
um you we have violence as part of us
yes but surely a lot of us is just built
around sort of I don't know if I call it
stable but some sort of social system
where we don't just try and murder
everyone or take his resources yes but
that's that's the winning violent
strategy is to cooperate within one
violent side to be the most effective at
being violence against others so that is
we are predators and we're social
predators and we're good at being social
predators which means we cooperate with
in our internal alliances against the
outside so and that which we should
expect winning predators and soldiers or
fighters in the future to share those
features now that's the generic winning
strategy got every fight of everything
against everything is a really stupid
strategy and it will always be a stupid
strategy okay sort of I mean I for all
these questions I think we can come back
to them in the sense that like they're
all open-ended questions that there's no
way we can answer it all of it but so
it's up to you how long to spend on each
one and then we can cycle back if we got
time I'm happy yeah backing down then
let's hear what Matthias has come up
with if he has managed to fix his
microphone I can hear extremely with
Mikey
sorry I need to turn it way off
okay sure what is it better now whoever
said that is great okay it's great now
yes it is okay so it seems to me at
least from what I gather reading your
blog posts that you generally seem to be
favored of a very progressive economic
policy where you try to you know
maximize the amount of wealth a society
can generate however you've also stated
here that you very much believe that
everything that was bound to happen is
happening in an ever you know
increasingly fast rate to me this seems
like a very dangerous combination as a
very much like a lot of time to think
about issues such as artificial
intelligence before having the issue
actually arrive it seems to me that you
know a bigger amount of wealth is not
gonna advance the speed of which we can
decide on these questions faster than
the speed at which these questions will
have to be decided upon did you not see
this as a very big argument against a
progressive economic policy
well again I I see myself and everybody
I talked to as pretty limited in our
ability to change the world what I
mostly could convince somebody is to
increase some local growth rate of some
city or a nation perhaps or some
industry I don't have much influence
over the growth rate of the entire world
and if you're worried about you know
learning to prevent problems that the
much easier thing would be to focus on
the people working to prevent that
problem and increasing their budget and
effort rather than trying to slow down
the world because slowing down the world
is just really hard right it's a huge
world and you're really small but if
there's a small part of the world
focused on a problem you could make that
part bigger yes okay that seems very
reasonable that you know as a single
actor it's it's like unreasonable to
expect to be able to fix the issue
through that way and a much efficient
way I much better way with you to try a
different approach let's say you had a
switch where you could simply on a
global scale decide this would you press
the switch to slow down the world yeah
I very I'm reluctant to so first of all
I think actually a lot of the current
energy and and desire to want to analyze
these things is premature that is I
think the world has to get bigger and
more knowledgeable before we're ready to
do much of this problem-solving that
people are eager to do now so in that
sense growing a little faster would just
get us to the point where we could start
to worry about the problems but then at
that point you might be wanting to slow
things down so that you could work on
them faster but at that point I'm not
sure how much you could slow them down
okay sir argument would be that slowing
economic progress down now would simply
just you know extend the amount of
suffering we have to do by living now
instead of the future and we're not
gonna be able to significantly alter
positively alter things such as
artificial intelligence at the current
society we live in now is that correct
cuz we just don't understand these
problems we'll have to do them it's just
not time to be doing much about them
when it is time to do much about that
little because you're near and then
that's the moment you might want to slow
other things down and if you could maybe
you should but um it'll be pretty hard
at that point to slow them down because
you'll be near and there'll be all these
different people out there who could
pursue them do you not feel like this is
like someone and admitting defeat
towards for the same risk of artificial
intelligence you mean but you're saying
because I wouldn't push the button to
slow down the world well it's just that
there is no button to slow down the
world right you know that no no sure
sure sure
practic in practical terms I understand
your I'd very much like to write up a
few of my thoughts in before continuing
the conversation but I found a very
enlightening thank you sure okay then it
seems to be me who sticks in line and my
question is in AI safety there's an
argument for or in extension twist
basically that the future is very very
large you could imagine
I think boström has calculated something
like 10 to the 6 teeth life years
that is possible in the universe and of
course people who focus on existing
AI safety they believe that this makes
it very important but I guess even if
you consider only like one in a billion
chance that you can do anything
substantially prevent an existential
catastrophe then I guess you need to
multiply one in a billion with ten to
the sixteenth or something how do you do
that in a principled way I mean you only
have a limited amount of time yourself
in energy so you mainly get to allocate
your time and energy to different topics
you can't scale up yours by a factor of
a billion you just don't have that isn't
as a knob that's not one of your options
so you can ask where do you want to look
in time and in topic to focus your
energies and the fact that the future
will be very important is certainly a
reason to do what you could about the
future relative to other things you
could do something about and then when
you're trying to help the Foat future
you have another parameter which is sort
of the unlikeliness and drastic Nisour
the scenario that you're focused on so
if you think there's only like doing
little tiny things that hardly matter or
preventing extinction then of course you
might say well sure even if preventing
extinction I can't how much of a chance
on it maybe that's still really
important to do which makes sense if
those are the only two options but I
really think there's a vast range of
other options for things you could do to
think and help about the future you know
honestly the first order of business in
thinking about the future is just have
the foggiest idea what it looks like we
are still at very primitive stages of
even outlining roughly what various
futures could look like and I think it's
very hard to be very useful about
preventing the worst possible scenarios
when you don't even have much of an idea
what the typical scenarios look like I
know that you know as abstractly that
seems like the priority you should find
the worst scenario and then focus on it
but there really is a vast space of
possible work scenarios and you don't
know how to judge which are more likely
than which other ones until you know
what the typical scenarios look like
that gives you much more information
about knowing
worse scenarios are the more likely ones
to focus on okay but actually had a
question while you're swallowing up on
that I'm curious like what sorts of like
like signs you think we'll have like you
mentioned like at some point in the
future maybe you would make more sense
to slow down once we like know enough to
be able to do more so like what what
sorts of like things would you think
would like help us understand this a lot
better than we do now so the scenario is
people are worried about our AIS that
are very agent like and have a very
broad range of discretion today and for
the foreseeable future automation will
have very narrow rules eric drexler has
written about this topic before but and
I think he's right but you know it's we
start with automation in very narrow
roles and slowly move it up to roles
with more discretion
you know similarly if you're worried
about like foreigners coming in and
messing up what's your society you don't
worry very much if they're janitors or
dishwashers or things like that if
they're if their actions are limited to
those roles because it's really hard for
them to come in and screw up your
society by being a bad gen and or a bad
dishwasher maybe as they're a janitor
they get to sneak into a room or
something but that's because again you
need to have more discretion so I again
in the future this the concern if you're
worried about AI being out of control
and causing problems you're worried
about scenarios where they have a fair
bit of discretion and they are able to
choose you know across a wide range of
things some of which are important and
risky but that's a long way away from
where we are now in the sense that the
eye at the moment is has very limited
roles and even most people today can't
do much to destroy society because most
of us on our jobs have very limited
roles in our jobs and that's how we all
keep each other accountable is through
what job we do and what metrics were you
to see who's how doing how well so you
don't really need to start worrying
until the AIS you're worried about our
doing much higher level jobs that even
most people are today there are
politicians they are military commanders
they are investment Brooke you know the
venture capital and they have to have a
big choice where it then the big choice
could go wrong that's when you you could
you humans should even start to worry
about it I guess like I mean people are
already thinking about having like more
autonomy and like military drones and
stuff like that it seems possible that
like more pretty limited autonomy sure
sure no drone out there is that risk of
destroying the world sure for sure but I
mean you can imagine that like even with
like a relatively like low level of you
know you have some like basic
reinforcement algorithm going on or
whatever it seems possible that you
could have something where you know it's
designed to track and like do basic
responses to perceived like border
violations or something like that and
that ends up like escalating into a
local board that's obviously not the
same skills like essential risk
necessarily but it seems certainly
possible that like a relatively simple
system doing relatively relatively
simple tasks could still ultimately end
up sort of feeding into pretty bad
scenarios I mean that's just every
soldier has that right I mean that's
just the risk embodied in any soldier we
already have to say the risk from AI
soldiers are more correlated though
right that's the least one thing I don't
see that necessarily no well assuming
that they're all implemented like with
the same or similar technology at least
within a given military well I mean so
for example if you just distribute some
machine to lots of soldiers and they all
have the same software and then it all
gets invaded by the same virus then
that's a correlated risk across devices
right so that's just a standard problem
we have with hardware is hardware has a
lot of advantages in scale economy and
production and an ability to
transferring thing but it often has
correlated risks and hardware you know
they often in warfare you've found a
flaw in one one tank and that's a flaw
in all the tanks and now you get to
destroy all their tanks for a while
until they figure it out and fix their
tanks right mm-hmm
that's not that's not a new problem
that's the nature of hardware and
warfare but another military example
would be giving a lot of control of your
entire military system like your um like
your neutrally assured destruction
systems handing mozo over to 2 AI
systems and that would be the equivalent
I suppose of having them open and I
having a high status job rather than
being a janitor around right exactly
sure when you're thinking about putting
AI and control the entire military
that's the point you could be start to
worrying well that's very different than
having any AI generator you're not
really worried about the a janitor
taking over the world and destroying it
all by being a bad janitor you worried
about dust be dust beings left in the
wrong places or you know okay okay yeah
okay
Jim also has a question about AI in
warfare yeah thank you can you hear me
yes okay well I work in AI at one of the
big AI companies and often say that the
concern that I'm most worried about has
not so much to do with AI safety in the
sense of super intelligence or AI Foom
but the looming economic incentive to
take humans out of the loop with my
company goes to great pains to say that
AI is augmented intelligence and it's
about augmenting decision-making but if
we well I think I also by the way work
in the DC area very close to George
Mason and I working with a lot of
military folks and I see times coming
when there's going to be more and more
AI controlled battlefield robots and at
a certain point it seems very likely to
me that there will be a very strong
incentive to take a human out of the
loop when a firing control command and
can be made faster by an AI without a
human in the loop
even though nobody in their right minds
once you know armed lethal AI you know
completely in control out of their own
decisions on the battlefield at a
certain point
probably going to be there to make that
happen can you comment on that I make
sense I that is you know as you know of
course our our introduction of
automation in the military has been
quite slow and gradual right
we don't suddenly like introduce a super
soldier who does everything we automate
a particular thing we add more autumn
you know faster automation capabilities
in a particular high in a gun or
particular kind of missile but etc and
about that at some points you will you
will have a capable enough automation
and and speed will be a facet of a high
enough premium that you will automate
the tasks mainly because of speed so
that's that's a little different than
the rest of the economy speed is usually
quite such a premium but in the military
speed is an enormous premium and just be
able to even make pretty dumb decisions
really fast it can be a win and so yes
they will they will put those in the
loop and that will usually win and
sometimes it'll go badly because the
automation is fragile and and not very
robust and so yes that that's the main
trade-off there's a middle wide you know
space of scenarios where it's a win and
then there's the tail of distribution
where it's a loss and you have to be
tracking the tails and the trails are
often hard to see and hard to collect
data on and so you will often make the
misjudgment of estimating the tails
below and they turned out to be higher
than you thought yeah I think the thing
that concerns me there is what happens
in warfare I mean I read something
interesting recently that said that yeah
I guess it was world war two when they
were introducing jet technology
willy-nilly and without the usual safety
concerns that they would have outside of
war something like fifteen thousand
pilots died inside the inside the
continental US during that time because
of the that unsafety and the
unbelievably rapid development I worry
that yeah well that's what I mean in war
you've got high you know yes you'll kill
a lot of people by going fast but then
you'll kill a lot of people by going
slow too and you know that's where
you're stuck in war time young yeah so a
lot of people will get killed as a
result of the enemy using automation on
you and a lot of people on your side
will get killed on the result of using
automation sloppily into fashion yeah
that'll happen
and they don't really see much general
to say about it that's just been the
nature of hardware and war for a long
time agreed
yeah I just think that the one scenario
that does kind of concern me is is one
where you know where we get into an AI
arms race in warfare in time of war that
that seems like the one potentially
likely scenario that trips us off into
completely uncontrollable AI but but but
I think that's about the ability of AI
to generalize that is we would have an
AI arms race in business if we could you
know I'm less worried about an AI race
in warfare because I just don't believe
that even though it's warfare and you
want your ads to be better that means
you can make them better fast that's
certainly true simili in business you
might want your a eyes to be better
faster but they just don't get better as
fast as you want yeah and they don't
generalize as fast as you want to
that'll be true and warfare you'll have
a you know a tank ai and you want it to
be how its Rai and it just won't turn
into a Howard 3a dammit it'll just stay
the tank ai yeah yeah absolutely
and just observing what what we do
within the company I can see that
there's a good handful of researchers
that have little pet projects on the
side working towards AGI but all of the
all of the financial incentives to spend
our efforts on the narrow a is I don't
as those words exactly those those exist
the AGI is you know theoretical at this
point right exactly and for a long time
to come very likely though I've seen
some fairly plausible designs that could
have asleep possible designs going back
50 years people have been looking at
their plausible AGI designs for a long
time it's an old hobby good point thank
you okay Chris had a question about when
boon know that when we'll know when AI
is imminent
yes I'm just wondering if you have any
picture in your mind of what it will be
like will it be just like an
accumulation of narrow AIS in all sorts
of areas of life and so when they were
just sort of turn around and I noticed
for the first time that that we've got
something we would like to call AGI or
maybe will maybe will never say that
we'll do what we did with chess-playing
programs and things and just say okay
well it's what we're doing is it is
amazing but now we've got it I mean
nobody would call that intelligence so
again you're looking for a lumpy
transition to more generality that's in
a sense when you're asking what you're
saying how will we know when AGI is here
or about to be here yeah but it need not
be that lumpy so if you look back over
the last 70 years of computer science
we've definitely had a lot of advances
and some of them at a little bit lumpy
but the vast majority have been pretty
small lumps the our abilities and
computer science haven't actually jumped
in big leaps very often and that's true
looking on one axis of ability as it is
on looking an axis of generality it's
really quite rare to have a big
surprising leap in generality so I think
that's what you should expect for the
future you should just not expect this
sudden moment when everything turns
doubt to be vastly more general than it
was a moment before you should be seeing
the rate at which things were getting
more general and you should be able to
project that into the future and that'll
be a pretty good guide to how fast
things will get how general mmhmm I'm
just thinking um I'm working with a
collaborator who we're trying to do a
children's book explaining AI safety and
it just occurs to me that in a dramatic
story you need a lumpy you need a lovely
story ya know smooth transitions yes and
that's been a problem because a lot of
people's intuitions have been honed off
of science fiction and dramatic stories
where there's the one big lumpy
development that drives the story yeah
yeah I have a lot to answer for
okay thanks okay then my question would
be back in the original ai Foom debate
you made a number of comments I believe
some very bold predictions about the ADI
project called SiC cyc I think the
did not turn out sick didn't seem to be
the strong way forward we have incomes
in there when when the AI Foom debate
was happening psyche was already a very
old project so I I can't have been
making predictions about psych at that
point because it was long past psyche
was I mean I could be talking about
psyche just as a system and the kinds of
system it is compared to other systems
but I'm I'm surprised you could find me
as making a prediction about psyche at
that point maybe I'm but your overall
point seemed to be that the kinds of AI
projects that were built up with a lot
of content were more likely to be
successful compared to AGI projects that
were trying to build on some kind of
architectural great idea
and that that is another way of talking
about lumpiness that is how lumpy our
architectural insights if architectural
insights are not very lumpy then it's
equivalent to there being lots of what I
called content if there's a single
architecture that's really different
from anything before and it makes an
enormous difference then that's a very
lumpy innovation it's an innovation in
architecture and so yet the key question
is about the distribution of lumpiness
is lumpiness in innovation in AI and
computer science more generally and so
you know I have a number of lines of
argument there one is just we have
general data about lumpiness and
innovation in general we have lumpiness
and innovation as represented by say
citations of papers we have lumpiness in
the history of computer science and even
AI more specifically and on all of these
things I think relatively consistently
show that the vast majority of
innovation is in relatively small lumps
and big lumps are relatively rare and
often big lumps get more credit than
they deserve so for example recent
advances in deep learning a lot of it
can be attributed to a big increase in
the amount of hardware devoted to using
methods that we had for a while ago and
if you correct for the hardware actually
the
in you know it's nearly on target for
the kind of abilities you would have
predicted I believe you had in the FM to
paid a you made some kind of outside
view argument to show that you expected
a true AGI to be something like 100
years ago to remember the argument you
made 100 years ago no sorry and 100
years in the future sorry one line of
argument that I've given although I'm
not sure it was in the AF room debate
but I may have been at that time was the
is just a survey that I've made of AI
researchers where I've met them and I
basically asked them in your field of
expertise in the last 20 years how far
have we come
I think it's more reliable to ask people
what they've seen in the past then make
their guesses about the future it's more
reliable to ask them about their this
field they know best and ask them about
broad overall trends and in very large
fields that they don't know very much
about so the usual way of doing things
to ask people how much progress will
they think the entire world will make in
the next few decades
whereas I think it was more reliable to
ask an AI researcher in a particular
field how far their field has come in
the last 20 years and the only
interpretive part I ask them is to say
how far have we come as a fraction of
the distance to human level abilities in
your field and that then they have them
given it to me as a percentage in the
last 20 years what how far have we come
as a fraction of human you know toward
human level abilities in the last 20
years as a fraction and then sort of the
median answer I get is five to ten
percent and then I have the follow-up
question any noticeable acceleration
typically not and then the obvious
extrapolation from that is to say well
then we're talking two to four centuries
yes actually I was wondering if dr.
Henson tell us about well few dr. Hansen
could tell us a bit about your work and
there and whether you're doing any any
research work at projects etc well like
I indicated
the beginning I I have this grant from
open philanthropy and I'm had it for a
little over two years now and the pitch
was that I would analyze an AI scenario
like I analyzed age of M and I basically
said let's let's assume that the
patterns we've seen in software for the
last seven years our reliable
indications of the future so how will
you predict the future of software if
you were just to look at the past
patterns of software and say that that
would continue and so I've been
struggling but coming up with some
insights I think into what I can say
about what the world future world of
software looks like it and more you know
since the time I started that I've lean
more toward the question I've imagined
that nm show up but we also have non M
software which kind of software wins
where and what is the world look like
after the non M software gets really
good are there any M's left that are
that are competitive and can do jobs you
know you know more cost-effectively than
the other kind of software and so I have
a number of things I can say I think
about what that world looks like they
there's not nearly as many things as I
can say about the age of M because that
could say so many things about that
because they were very human-like but I
have a number of things I can say I'm
proud of figuring out and then it would
take you know I don't know if I were ten
minutes here if you wanted me to walk
through the things I think I can say I'd
like that very much I think you entered
it a little earlier about areas where
human-like intelligence is likely to be
successive successful rather and areas
where non-human like yeah I might be
successful right so but right now just I
will defer to the organizer here to
decide how much time to spend on those
sorts of things because that would take
me a little speech a fine time well we
said that we had planned one hour and
the hour is almost up but I'm very
interested in hearing that so unless
anyone objects and then I would like to
hear some more any objections please go
ahead ok so the challenge is just to
think of software as a general phenomena
and ask what general things can we say
about it that we could use usefully to
protect into the future
especially as by comparison with the
software in our brains so that one of
the simplest things I can say it's very
simple it's based on something called
Conway's law Conway's law says that when
an organization has a structure and then
it makes software that has a structure
the structure of the software tends to
reflect the structure of the
organization there are three
organizations that are working together
to make software well the software will
have three modules one for each
organization if you've ever worked in
software that sounds kind of obvious but
it has a dramatic implication it says
that in the future as we replace say
humans with software we could end up
with the world that looks a lot like our
world in the hot largest-scale
structures because the organizations
that replace us with software will end
up creating software that reflects their
structure so today we're in a world with
structures like you know jobs tasks jobs
divisions firms industries cities
nations etc and if we slowly swap out
people for software on each of these
things we could end up with a world that
mostly has these things done by software
but still looks a lot like the world we
live in at those larger scales
so that's just one interesting thing to
notice about the inertia of software
structure a second thing to say is that
today when we look at human jobs we can
see each job is composed of tasks and we
can ask for a task what other tasks tend
to be done in the same geographic area
as that task and those tasks tend to be
jobs that are trying to coordinate with
that tasks so tasks tend to be collated
co-located in space and even co-located
in firms when they are tasks that need
to be coordinated more with each other
and if you look at that network that
network has a clump it has a center of
the network where that where of the
tasks that are highly coordinated with
many other tasks and those tasks
actually tend to be done in city centers
and the most compy ones tend to be done
in the biggest cities and they also tend
to be done higher and/or
musicians and we expect that as we
automate tasks we will automate the
periphery of this network first that is
if you have a task that has very few
other interfaces it's easier to automate
that task because you you you will only
have to change a few interfaces whereas
when you have a task that interfaces and
has to coordinate with many tasks when
you automate that task not only you have
to change how you do that task you have
to change all the interfaces or you have
to coordinate with all the other tasks
that is coordinating with and change how
they deal with their tasks - so that
says that we will automate slowly from
the outside in in the sense of this
network so we will automate rural jobs
before city center jobs and we will
automate jobs lower and organizational
hierarchies before we automate jobs
higher in organizational hierarchies and
the same principle can be thought of and
say even inside a brain or human if you
have a human doing a job when you
automate the job you'll probably tend
more to keep how that job interacts with
other jobs the same and change the
internals of that job and you may even
end up making you know systems that look
like humans except you automate the
insides differ so like if you think of a
human brain is composed of a thousand
modules you might be less you might less
change how those modules interact with
each other and more change the internals
of each model so the general principle
is just when you have this network of
things struck in some sort of
hierarchical structure you more often
change the internals of a structure than
the interfaces and therefore if this is
a network you more turn changing the
periphery of the network relative to the
center and so that says that you know
it's similar to I said before we'll
spend a long time automating peripheral
tasks that are relatively isolated that
don't have enormous impact and the last
things we would automate if we automated
them would be the center of this network
which is the jobs that are most
coordinated in the city centers high
levels and organizations today like you
know marketing management law you know
things like that a governance so so
that's the second thing to say I think I
can say somewhat robustly
and the third thing I have to say which
is the the third and last thing I think
is that in order to ask where do human
software win relative to other software
we need a simple model of what is the
essential difference between the kind of
software in our heads and the kind of
software that we write and my best guess
for that essential difference is to say
that the software in our brains did not
separate out some things that we
separate what we rights offer to date so
that the machines we write today on our
own computers we separate hardware and
software we separate learning and doing
we separate memory and processing these
are just standard things we separate and
we make them in different places and
hint you know we can swap things out
right we could swap out a different
memory of the same processing etc in the
brain
these things were all mixed up and in
particular hardware and software is
mixed up that is you don't have a
separate place you store software that
you can swap into any particular
hardware and each place in the print
brain is both hardware and software so
this has a dramatic implication for the
evolution of the software in our brain
it meant that when evolution was trying
to change the software in our brain it
couldn't do what we do now when we write
new software so today wouldn't we a
human write new software the obvious
thing we do is we start with a blank
screen and we just start writing new
software and then we connect it to other
software as we desire in terms of you
know we don't like to connect things
because that makes things less modular
on the other hand when other stuff
already does a task then that's better
to connect it to something already does
it then we don't have to rewrite that
and that's how we write software but in
our brain when evolution was evolving
your brain it couldn't do that in the
sense that it had a limited set of
hardware and it had a very you know
difficult time adding more hardware and
all the hardware it had was already
devoted to other tasks so all it could
really do was try to cram it a little
more hardware or delete some old
hardware in order to replace it with new
hardware or find some way to reorganize
the pre-existing hardware software
combinations in a better way that it
will allow a small addition to do the
new task
and so evolution having this strong
hardware constraints meant it meant
spent a long time searching in the space
of reorganizations it just couldn't rely
on modularity so much because it just
couldn't add more software anytime it
wanted because that was tied to hardware
and so the human brain is just naturally
much less modular and much better
integrated and so that right there has
dramatic implications for where human
software wins and where it loses
compared to other software so the other
software we write because we can make it
modular much better because we can just
start with a blank screen we use
modularity all over the place as a way
to make stuff work and then report to
avoid problems we modularity is
basically our strongest tool and in
using modularity however we don't search
a long time for the very best structure
we find the first structure that occurs
to us that's pretty good and then we
just go with it and it turns out of
course in the longer run as we have to
evolve a system and add more things and
change it the structure we chose wasn't
that great and this means the system
slowly degrades in its ability to handle
new changes and so we have the Commons
phenomena software rot by which software
degrades as we make it more complicated
and we try to change it to adapt to
changing conditions and it rots faster I
think than the software in our heads
because in the software nur has we just
evolution spent a really long time
searching for really good combinations
of things that work well together that
are robust structures that would
accommodate a lot of new changes and so
this tells us that say the software
that's humans that software that
software writes will probably be even
worse in these regard software squared
if you will and so that'll limit how
much we have software writing software
and humans will will be best suited for
tasks when there's a whole bunch of
things that need to be coordinated
carefully together where modularity
can't solve the problem and software
that we write is much better suited to
situations where modularity works better
where you can just have a bunch of
separate parts
work on separate pieces but and then
staple them together and it kind of
works and so this is also telling you
that humans will last longest at the
jobs that need were a lot of different
things to be coordinated in the center
of the network of tasks were lots of
different tasks need to be coordinated
together and the software we write
ourselves will again work much better on
the periphery of that Network where
modularity works much better and rot
isn't less of a problem and you can just
replace things after they rot and start
all over again and so again we have this
image of a world where there's a lot of
software but mostly human-like minds are
doing the high-level general abstract
tasks where there's a lot of different
little things that have to be
coordinated carefully together so I just
been talking for a while hey I must have
somehow lost I'm not sure if I set it
well or write correctly if there's
clarifying questions this is please ask
okay I have a question if we try to make
a mathematical model of this slowly slow
process of automating these tasks well
if you imagine that let's say just for
the sake of it that 50% of the tasks
have precisely one interface and those
are the easiest ones to automate so we
automate those first and then after all
the easy tasks at the periphery of the
network has been automated then
obviously there is a lot of extra
resources people who are no longer who
no longer need to work at at these tasks
and then there's a much greater
incentive to automate the paths that
have to interfaces and then this this
effect could accelerate quite quickly
well you want to automate them the
question is whether you can again when
we write software that relies heavily on
modularity
it's just very hard to find a really
good structure
doesn't rot fast that can integrate a
lot of things and the human brain
software is is the software that rots
much more slowly and integrates far more
things and that makes it just
increasingly expensive to even try to
automate the center of the network tasks
you would just you you can try you will
try but you won't succeed for a long
time they should they're just hard yeah
my point was that the the incentive to
automate them grows enormously as we
move towards the the center of the
network well I'm not sure if the
incentive relative incentive grows that
much so I mean a key point about the age
of a book that I keep pointing about
about em is in the age of M which is
that once we have brain emulation z'
then hardware advances improve ms just
as well as they improve artificial
software the software right so from that
point on it's only architectural design
improvements that give artificial
software an advantage that is you have
to search in the space of software you
can write to make it better so so far in
you see in our world it's just the fact
that hardware is getting cheaper so fast
that drives the increasing urged
automate things even if hardware it
takes a lot of hardware to do something
it's pretty soon hardware is cheaper and
then you just do it but once you have MS
that stops being true you the hardware
trend no longer drives the change thanks
for taking the time and given us as
insight so it makes a lot of sense just
a quick follow-up if you have time do
you have any thoughts or can you share
any thoughts any way on where you see
your research going in the future
well I'm personally relatively
opportunistic about asking which things
are the most important and where I want
to go with that i I mean I I think I
will write some sort of book based on
this work that I've just described but
I'll put it in a larger context that's
probably more engaging and so I'm you
know asking myself which context to
choose for those things and then beyond
that I you know I'm some
older in my career and I have lots of
things ideas from the past that that
could I could go back and build on so
the more of an installed base you have
of old project ideas and old projects
that you partially built the more
tempted you are to go back and finish
those than to start new things yes so
that'll probably be some I do a lot of
thanks very much which means I'll be
less surprising because I'll be more you
know going back to things you are I've
already done and you can see the sort of
fun question you obviously were very
deeply involved in the whole prediction
market thing I don't know you might even
beep have invented I don't know yes and
I'm just wondering if you have any any
wages bets or prediction market
investments or whatever in the area of
AI ah yeah I just don't think we're
about to see a huge I mean I think we're
near the peak of the current boom and
we're gonna have a bust again we've had
boom and bust cycles gonna go over a
long way back so this is the wrong time
to buy this is the time to sell I see so
whether that's the short run of course
you know these these bump and by cycles
have been roughly a period of 30 years
so 30 years from now it's time to expect
the next peak of the next boom yeah and
have you put any money on that oh well
I'm I'm not buying
there's no sell I'm not buying into AI
at the moment I'm I'm when I'm
conversations like this I say like think
about the long run don't really worry
about you know are we there is this
almost time nowhere it's not if this was
almost time you'd definitely being
seeing a lot of big differences now we
are definitely night not right near the
threshold but the long run is it remains
there and then this is you know a good
time to talk about the long run as long
as everybody's thinking about the topic
now okay thanks okay had a question
I don't think said I was just being a
couple comments thanks a lot for doing
this this was like very very interesting
I'm happy to talk to you guys again if
you want and I'm really surprised you've
done 135 of these meetings that's like
every week for two years or something or
three years yeah yeah okay well yes we
have I only had a question and I think
that'll be the last question for tonight
yes so I just wanted to ask about the
age of M you sort of made a few
simplifying assumptions like assuming
the society would be sort of stable are
you going to follow up with that idea
and try and see what the biggest like
what the biggest branches would be the
most likely places where things would
diverge rapidly the largest sources of
uncertainty are you going to try and see
if there's anything you can say about
that well the the two main drivers here
one is just that when I wrote a Jif em I
just got really obsessed with it and
focused on it and then when I finished
the book I couldn't quite turn that off
and so I kept collecting a file of
additions and then that was put into my
updated paperback version and then even
then since then I still keep thinking
about it because it's in my head and but
the thing of the direction I most often
tend to go is people say they'd like
their fiction in the future in the form
of fixture and and you know could could
I set up a fictional scenario that would
make this more vivid for people and so
that's the direction I'm mostly gone
when I think about hmm things is what
sort of characters what sort of plots
what sort of conflicts etc could I put
in this world so that I could make this
world more vivid for people you know
I've certainly I've got the overall
message from the world that this was an
interesting exercise but that it was
unusually focused on one particular
thing the world isn't that interested in
except for the fact they liked the fact
that there was just an integrated thing
about one scenario
so I'm not feeling enormous pulls and
rewards from people who want to know
more details about the age of them most
people I said say that that was a lot
more detail than they wanted to hear
about any future scenario on there
they're surprised didn't even impress
that I managed to put so much detail
about it and that's kind of a planning a
flag showing that you could say a lot of
detail but still most people said they
didn't really want to know that much
about it so they don't there's not much
demand out there for for for more that I
can perceive although you know if there
was a fictional this scenario I could
imagine there being more demand for you
know a sequel to a fictional stars
because people then like to hear about
that and in the process of elaborating
Victor Lawson re would of course
elaborate some other aspects of it you'd
be thinking about it but that's where it
stands now the idea of doing fiction
about it seems really interesting but I
I would be worried that in the future
people might forget about age of M and
remember the fiction and then when
people want to talk about it they say oh
like that sci-fi book huh well I would I
would I would only be thinking about
fictional books that in fact were
faithful to the world at the age of M
where it was a real story but it was in
that world I'm not gonna I wouldn't have
all be interested in when dramatically
changing the world for the sake of the
fiction not gonna add magic magic and my
people might take the ideas less
seriously if they if they don't realize
that it came from a serious work and not
from a picture like you know I'm much
more worried that it will just forget
the entire thing so I think I think if
there was a fictional star that was true
to the world that is it was a you know
there were characters in conflict but
the world they were in and conflict was
in fact the age of world and world I
described I I would think that was a big
win from the point of view of getting
people to take snart seriously fair
enough but the world needs more books
like the age of them I thought it was
fantastic well you guys are all and
hopefully you know invited to try your
hand at such think that that video that
would be the way that that age of M
would be most validated as if people if
were inspired to do things like it okay
I think there was a nice note to in
this is conversation on thank you once
again Robin Hanson for coming and
answering our questions it's been a
great conversation it's great to meet
you all and feel free to friend me or
whatever I mean I'm not sure I've
actually seen the names of all the
people in this meeting I'm just standing
in a blank Skype screen so and I see
some funny you know Skype names but I be
happy to know the actual regular name of
all you people great thank you
that's terrific yes thank you oh and
before everybody leaves next week's a a
safety reading group we will have will
be on Tuesday the 4th where we will read
the the first half of Mythbusters news
new paper whose name escapes me at this
moment is it about the vulnerable world
the volvo volvo world hypothesis ok so
that should be all take care thank you
have a nice week