Miles Brundage Limitations and Risks of Machine Ethics  FHI Winter Intelligence

so my talk is on limitations and risks
of machine ethics and by that i mean
reasons why it's difficult to ensure
that machines behave in a reliably
ethical fashion
the term machine ethics has
traditionally been used to refer to
near-term ai systems as opposed to agi
so i'm using the term in a more broad
sense that also includes things that
have been called friendly ai
and the basic point of my talk is that
due to the nature of ethics and the
difficulty of acting in a rational
fashion in a complex environment such as
ours it's going to be difficult if not
impossible to ensure that any
any intelligent agent no matter how
much resources it has available within a
practical sense
to ensure that it will behave ethically
so there are a couple of possible
motivations for thinking about machine
ethics uh defined here is developing
computational models of morality you
could do it to learn more about ethics
for human purposes you could build moral
advisors or you could build artificial
moral agents that directly act in the
world based on their ethical uh beliefs
in the context of agi risk you can think
of machine ethics as a subset of
motivation select selection in the
taxonomy that foster talked about as
well as a form of internal constraint
and
sotola at all's uh
taxonomy of ai risk approaches
so
i'm going to talk about the nature of
ethics and why it's problematic to
assume that we could ensure that a
machine would be able to behave reliably
ethically and the general point is that
if we are unable to come up with a
system systematization of human ethics
then we shouldn't be confident that we
can do so for machines
there are various problems that have
been identified with a unified theories
of morality
in the context of ai as well as more
generally yukowski and melhauser and
helm have noted that human values are
complex and fragile and this was also
commented on
by bostrom a few days ago there are also
issues related to computational and
tractability of perfect ethical behavior
there are conflicts between principles
and duties that may arise in the course
of everyday life in real moral
situations as well as the problem of
ranking different choices that are
available to an agent so obviously i
can't give a comprehensive assessment of
all possible ethical theories but just
to focus on two classes of ethical
theories in particular and some general
objections that have been raised against
them if you think of consequentialism as
an ends oriented class of moral systems
and dentology is means oriented systems
both of them have difficulty for
accounting for all possible situations
so there are unacceptable conclusions
they can derive
from a purely consequentialist framework
such as you know they need to torture
someone you know
in order to save a greater number of
lives on the other hand there could be
uh opposite situations in which a
dantological conception of morality uh
would be implausible such as for example
saying that it's uh immoral to kill one
person when you could save billions or
even trillions so the difficulty of
reconciling these different theories of
morality is one problem but there are
also unresolved problems in morality in
general regardless of which theory you
choose and this is going to be a problem
for an agi that's acting in the world at
a large scale because the more of the
world you try and act upon the more
you're going to run into these sorts of
unsolved moral problems that deal with
large-scale issues for example two in
particular our population ethics which
has to do with how we should ascribe
value to uh states of the world and
people on the basis of the number of
them so for example you might
intuitively uh think that it matters
that there are more people rather than
fewer all things being equal so you have
some sort of uh formula for multiplying
you know the quantity times the quality
of these people's lives however there
can be counter intuitive conclusions
that you draw from that such as that it
would be better to have trillions of
people living barely acceptable lives
and then to kill everyone on earth today
what sort of entities have moral value
also could be a difficult problem for an
agi that is trying to think far into the
future
because not only will it have to think
about you know humans as well as
non-human animals and their moral status
but also the possible
the space of possible agis in the future
and the moral status that they could
have
so it's not just a problem that human
values are complex and fragile in the
sense that yukowski and others have
talked about but it's also the case that
they can be internally inconsistent and
incoherent for example savalescu here at
oxford has argued that there are certain
aspects of our folk morality that have
evolved and were adaptive in earlier
times but that are no longer adaptive in
modern society for example a high
discount rate applied towards the future
and our tendency to care more about
those closest to us
there's no a prior reason to assume
there are evolved intuitions insofar as
you take a naturalistic understanding of
morality will necessarily be consistent
and this is a problem for approaches
that try and systematize human
intuitions through some sort of
algorithm
two lines of evidence that can be
thought of in terms of the psychological
literature on morality that are relevant
here first as heights analysis of the
cognitive processes that work in
morality he argues that there are many
he argues that there are many lines of
evidence to suggest that the role of
rationality in morality has been
overstated
and we think that we're more rational in
terms of thinking about ethical
situations than we actually are there
are examples such as moral dumb founding
which is the idea that we're unable to
articulate a reason for our beliefs
often which have evolutionary
explanations but we're unwilling to uh
to give them up even if we're pressed
upon the reasons for it
another line of evidence uh through
thought through
analyzing the brains of people who are
responding to thought experiments such
as the trolley problem in which you are
given the option of hitting a switch
that leads to killing one person rather
than five or the footprint footbridge
situation in which you push a man over a
bridge and then he stops the train with
his body directly often people will give
conflicting answers to these sorts of
situations green's analysis of this uh
of of this dichotomy between people's
resistance to pushing the man over the
bridge and hitting the switch is that we
have an evolved uh opposition to
physical contact and the use of physical
force we have no such intuitions about
about hitting switches even if the moral
consequence is the same he uses a camera
analogy to say that we have automatic
and manual modes to our morality and
that these aren't necessarily
reconcilable
uh i would know that this is you know a
controversial view within moral
psychology and as is the general project
of naturalizing morality
so even if you have a particular moral
system that seems to solve all problems
it's going to be very difficult to
implement it in the world and these are
problems that will apply to any bounded
agent that has finite resources
of both computation as well as finite
knowledge
if you think about the classical frame
problem in a.i and then you apply it to
moral situations you see that there
could be an infinite range of possible
factors that are relevant in a
particular situation in terms of their
long-term consequences this is
particularly problematic for theories
such as consequentialism which are
concerned with the ethical implications
of your
actions over the long term
there could also be domain specific
knowledge and experience that's
necessary
in order to act ethically the
computational complexity of morality is
also important here so gigarenser argues
that we do something called moral
satisficing which is that we have to uh
greatly reduce the space of options that
we consider and the amount of analysis
that we do towards those options with
regards to their moral implications this
seems to be necessary both for a
consequence and a deontological
perspective on morality uh and you know
for example this analysis by reynolds
concludes that
uh for consequentialist and antilogical
ethical approaches they're both
computationally hard uh scaling uh
as a function of m n to the l where m
is the number of actions available n is
the number of agents and l is the time
horizon
there are also environment based
limitations so let's say you have a lot
of computational resources you have an
ethical theory you're still going to
have difficulty acting in the world in a
reliably ethical way there are all sorts
of non-linear interactions in the
environment as other speakers have noted
uh which makes it difficult to model the
environment in a consistent way there
can be chaotic effects as well as black
swan events that throw out the entire
model that you are relying upon in
addition there are intrinsic
philosophical reasons why you can never
have full access to the environment and
to verify validate and confirm models of
the natural and social environment
in addition to all these issues i
brought up there are also problems that
arise as a result of learning and
evolutionary processes of autonomous
agents so as mueller has noted uh the
the autonomous interaction with the
environment that's critical for an agent
to become intelligent will require less
control in order to allow that scaling
up to occur however learning about
ethics in this sort of uh in this sort
of fashion could pose risk for a very
power for a very powerful agent and it's
impossible to explode expose it to all
possible situations
so you might ask how do humans act
ethically well first of all we have a
lot of advantages such as more
computational power than existing
affordable computers as well as lots of
experience both in terms of our lives
and as well as if you think of our
evolutionary
psychology as embodying some moral
experience but on the other hand it's
unclear that we do act in a reliably
ethical fashion most people don't act in
ways that are consistent across a wide
range of situations and everyone makes
mistakes and i personally wouldn't trust
any particular individual to make
decisions for all of humanity
there's specific classes of machine
ethics proposals and again i'm using
that term in a broad sense to include
theories of ai agi friendliness and here
are four categories that you can think
about and i can't deal with every single
one of them but just some broad
limitations of these classes of
approaches can help us think about ways
in which an agi
ethics system could fail
if you take a top-down approach to
morality which is to say that you have
an ethical theory and then you implement
it in some sort of system and have the
system reason based on those high-level
principles about particular cases then
you're going to run into computational
limitations as well as the limitations
of
unified moral theories that i talked
about earlier in addition as yamaholsky
argued yesterday an overly literal
interpretation of utility function could
pose its own
issues
if you take a bottom
bottom up approach to machine ethics
which is to say reasoning based on
particular cases and building up a
theory of morality based on experience
then you're going to have the safety
issues that i mentioned earlier but
there are also some other issues that
will arise such as not guaranteeing that
any particular principle will be
followed there will be no guarantee that
the system will eventually arrive at a
fully coherent or consistent theory of
morality and it might be difficult for
it to explain
the reasons for its actions so i think
that a lot of humans might have
difficulty accepting for example a
system that says that it killed person y
because at time step 789 node one had a
value of 0.81
psychological approaches to machine
ethics take the human
cognitive system as a model and then try
and instantiate it in an agi
the problem of this is that first of all
there's intra and interpersonal
variation in morality so it's difficult
to say whose moral system you're going
to be modeling and whether it will be
consistent in the first place there's
also the inherent problem of the
naturalistic fallacy so it's it's
problematic to say that because humans
reason in this way about morality that
it's necessarily correct and that we
should lock that in into the future in
these powerful systems in addition it
doesn't seem to be possible to resolve
outstanding philosophical issues and
limitations of folk morality in this way
various approaches have been put forward
at this conference as well as at others
uh to deal with ethics in the context of
a particular agi architecture or in
terms of some sort of way of learning
about values and extrapolating them in
the future there are going to be various
limitations of these approaches while
they may be valuable
some things to consider are for example
the reconciliation of conflicting values
both within and between humans the risks
due to acquiring vast amounts of
resources in the course of deciding what
is ethical in the first place so there's
kind of kind of a chicken in the egg
problem there as well as the loss of
important elements of our fragile values
in the process of trying to extrapolate
them in a consistent way
there could be high sensitivity to
initial conditions for example the mood
of someone prior to their values being
extrapolated the prior their prior
exposure to schools of thought many of
which could be consistent with their
prior beliefs and as well as what i
would call the irobot problem which is
that we don't know for sure what the
logical implications of our beliefs and
the particular moral systems which we
instantiate in a given machine are so in
the movie irobot as well as uh not the
novel run around and others by isaac
asimov what happens is that the robots
decide that the logical implication of
the three laws of robotics is that they
need to take control over humanity's
destiny and prevent humans from killing
each other well i don't suggest that
that in particular is going to happen
the overarching issue is that we need to
think clearly about ways in which our
values could be taken to their logical
extreme and whether or not we're sure
that we actually believe what we think
we believe
this is a summary of some of the
possible failure modes of machine ethics
which i've already identified so i'm not
going to read them again there and the
last section i want to talk about is the
insufficiency of machine ethics i think
it should be fairly
obvious why there are going to be human
factors involved in any sort of agi
safety regime and as hibbard argued
earlier in the conference it's it's
perhaps more of a threat how humans are
going to be using these systems than any
uh threat that emerges from the system
itself however these are going to
be tightly correlated
some examples of issues are the
reprogramming reprogramming of agis
which could be possible if they're
widely distributed as well as whether
they're centralized unless you have a
very uh
very wide system of surveillance and
essentially a totalitarian state there
could be domains that are ethically
complex and problematic that an agent is
put into by a human operator in an
irresponsible fashion there could be a
training environment which is deceptive
for the agent and again
dealing with the frame problem will be
difficult because there could be aspects
of the training environment which are
hidden from the agi that lead it to
come to erroneous conclusions
there are also issues related to hacking
that that tie into what uh jampolsky was
saying yesterday so typically there's
been a tripartite distinction in terms
of what's required for trust of a human
some of these factors are benevolence
competence and integrity and i would
argue that when you think of an agi the
analogy between integrity and cyber
security is pretty strong in the sense
that if you want to be able to trust an
agi to be benevolent over time you want
to ensure that the cyber security
infrastructure of the whole planet is
secure that means that even if we have a
wide system of agis that are active in a
reliably ethical fashion a new evil one
might come along and hack all of them
and then you're gonna end up worse than
if you'd never had them in the first
place
there could also be system level issues
arising from multiple agis even if they
each act in an ethical fashion in their
own domain examples of problematic
interactions between agents that aren't
trying to be malevolent are machine
trading in which there can be vast
fluctuations
and destruction of wealth as a result of
act of agents acting in a fashion that
leads to some sort of systemic effects
there could be homogeneity of agents
resulting from the use of the same
architecture or ethical system across a
wide range of domains that leads to it
just it leads to a
decrease in the overall quality of
decision making there could be declining
human wages such as hansen and brings
you or to talk about in terms of the
implications of ai
as well as issues that we don't foresee
now but they could arise from a poor
uh a poor match-up between cooperation
and competition so generally we would
think that an ethical ai would always be
a good thing however there are many
aspects of modern society that depend on
competition and self-interested behavior
we don't know for example what the
economic implications of everyone trying
actively to maximize a global utility
function for others would be there could
be
all sorts of unintended consequences of
this and lastly building an agi
infrastructure that is in charge of many
of the technological systems we depend
on could lead to vulnerabilities to
catastrophic technological failure
so why does this matter for agi risk i
would say that there are a couple of
reasons one is that if you expect or
hope for agis to act on a large scale
then you should be more concerned about
the limitations of machine ethics both
in terms of the unresolved moral issues
that could lead to problematic behavior
for example as relates to population
ethics as well as the fact that if
you're pessimistic about alternative ai
risk ai risk approaches such as boxing
then you should be concerned about the
fact that there are problems with
machine ethics approaches as well
if you think that a single agi is going
to be far more powerful than others then
you should be very concerned about the
prospect of centralizing rural decision
making and lastly i would say that you
could think of a hierarchy of safe uses
for machine ethics in the sense that
generally using it to inform human
ethics is safer than using it as an
advisor which is safer than limited
action which is safer than large scale
action
so in conclusion machine ethics might be
useful in some domains but it's by no
means a solid technological fix
for the problem of ai safety
these three criteria that sarawitz and
nelson have identified for what a
technological fix uh what criteria
technological fix uh should need in
order to actually
qualify as fixing the problem is
embodying the cause effect
cause effect relationship which clearly
machine ethics doesn't do given the role
of humans in operating and developing
and deploying the systems
the effect of the technological fix must
be accessible using relatively
unambiguous or uncontroversial criteria
which clearly ethics is not and it must
contribute to a standardized technical
core which we did not yet have an egi
so my contention is that by thinking
about the ambiguities and ethics and the
difficulty of rational action in the
world we can identify some failure modes
and come up with better systems for
machine ethics i don't have a solution
for you but i think that thinking
systematically about these limitations
could be helpful and lastly i would say
the machine and agi ethics might not be
the best framework as uh hibbert
mentioned earlier it's really important
to think about how people are using
these systems and a better framework
might be to think in terms of ethical
human machine systems
so that i'll open up for questions
uh questions
hello there
there seems to be a fair amount of
anthropocentrism or anthropomorphism in
thinking that um the correct or most
useful kind of agi system would be a
fully autonomous
system
and instead you refer to ethical human
machine systems because you try to
expand a little bit on that
so in part i'm responding to the
literature on machine ethics which is
largely based on human ethics and trying
to develop models of human ethics so i
think it's anthropocentric in that sense
that
people have argued both
implicitly and explicitly that we should
model these systems on say human values
but i'm not sure if
i'm not sure what you mean about the
human
machine system part of your question so
yes
i mean
well we seem to
most people seem to assume that some
kind of autonomous intelligent agent
should be treated on the basis of
thinking of it like a human person
instead
what would you recommend for thinking of
a correct period of ethics that involve
combination of humans and emissions
either as tools or otherwise extensions
of humans
um
i don't have a good answer for that and
i think it really depends on the level
of autonomy of the system in question so
i think today most systems are not
autonomous and you can see technology is
largely an extension of humans so this
so a human machine framing might make
sense today over time as systems become
more autonomous then it might make sense
to think of them as discrete systems but
i think you know it it'll really depend
on the system the situation
and i'm not sure that that i or anyone
has a really well flesh out very human
machine ethics so
i don't know what to tell you though
if i may quickly insert a question of my
own as somebody who's neither an
ethicist or a computer scientist um one
concern that i didn't see up there in
your very comprehensive talk was
the way that our ethics and morals have
been evolving if we for example
programmed an agi with um
the morals of um you know zero id rome
we'd have all sorts of problems
and it seems like
we may have the same problems if even if
we have a comprehensive system that
works now it may not work in 2000 years
or a thousand years when
our understanding of the agi is
understanding the galaxy and the
universe has changed drastically now you
could allow a level of evolvability in
worlds but that would seem to imply a
risk associated um would you have any
comments on that
i think that's a difficult issue and it
and
you know the whole idea of extrapolating
i'm not values that there's a good
answer in the sense of how much we would
want a system or even ourselves to
extrapolate our values in terms of
logical implications so with the irobot
example it's not obvious to me that
you know the system vicky was actually
mistaken in its utilitarian belief that
it should take over humans in order to
protect them however the question is do
we actually
want our our
views to be taken to their logical
extreme and what sort of standard should
we apply to the evolution of morality
and a lot of the machine ethics
approaches have been based on taking
humans as a model and trying to come up
with good models of what we currently
believe but as you say that has changed
over time and you know it really depends
on whether for example you take a
realist or an anti-realist perspective
on morality whether you think there's a
truth of the matter or if you think it's
sort of a human institution and
theoretical construct and
you know
besides saying that it's a difficult
issue and i agree with you that we don't
necessarily want to lock in a
potentially
soon to be outdated ethical system i
think i agree
okay thank you very much
um
what's the short remark and sort of the
question the short remark is that you
mentioned declining human wages and i'd
like to mention that
in his first uh science fiction novel
player piano which he wrote in 1952 sort
of described the contemporary western
society where a lot of people who used
to hold good jobs simple jobs perhaps
but good jobs are now completely out of
a job because of increased robotization
so this is not something for the future
you don't even need agi for this to
happen this is already happening
is just to sort of
strengthen your point there and the
other one is sort of a more general
notion that maybe the problem statement
as you have it is basically an algorithm
to which you input the situation and
then it has to sort of render more
judgment uh
i can only operate by analogy to natural
language processing which is where i do
most of my ai work and
this is how chomsky originally posed the
problem you put in a string of symbols
and you should decide whether it's
grammatical or not and we have come a
long way from that and
every uh working
system is incapable incapable of doing
this this is not what the systems are
doing they're doing much less in in in a
relevant sense uh they try to make sense
of what's given to them rather than
rendering yes no judgements on them so
maybe a lot of these these
problems that are very real you
mentioned are are a consequence of a two
ambitious problem statement so you take
something like i don't know uh uh
the ordinary anti-complete problem so a
large number of them if you're satisfied
with the solution that's i don't know 95
percent uh good then you no longer have
an mp complaint problem you can have a
polynomial algorithm to solve it so
maybe a somewhat more relaxed uh problem
state statement will lead to more
effective
algorithms
so it in terms of the uh
you know
the ambitiousness of the problem so if i
think if i understand your question
you're saying that that
expecting perfection
of ethical behavior might be an overly
stringent requirement is that
it's perfection it's the very idea of
being able to render yes no decisions on
every case in fact when you look at the
human grammar facility humans are hard
put to put
upper bound on this lower bound on this
uh that may be computationally feasible
right
yeah so i i agree that uh expecting a
yes no answer in every case is
potentially impossible or unreasonable
to expect and i think that's something
we should think about when we're
allowing certain systems to gain a lot
of power whether they might uh you know
be find themselves in situations in
which there's no good answer for how
they should behave so i i agree with you
that it's a normally strange
understanding
thank you i
think it's rather common not the
question what they have but in my
opinion it's hard to speak about
machine ethics without referring to a
specific definition of
freewheel of autonomous systems
because without the free will
at the machine level
you cannot judge the decision and the
free will of
decision making
has been already
treated in some papers with
i would say
with very different approaches very diff
different definitions have been
proposed by the authors and i wonder if
you refer to any one of them or maybe
you have proposed your own one
thank you that's a question that was the
first part was the remark the other is
the question thank you so i haven't
thought too much about the free will
question but you know my specific
interest uh in in this research and in
the talk was the question of how to
ensure that machines behave as if they
are ethical and in that sense it's not
particularly important to me whether
they actually have free will or are
actually ethical moral agents in some
higher sense in the same way that you
know i it's not particularly important
to me whether someone actually has free
will in terms of whether they're
treating me ethically so i've been
focusing more on the pragmatic
consideration of ensuring uh safe or
benevolent or uh apparently ethical
behavior as opposed to whether or not
they actually have free will so it's
it's an interesting question but outside
what i was thinking about
okay if there are no more questions then
i think we can move on and i'd like to
close again
you


Stuart Armstrong  Predicting AI... or failing to - Winter Intelligence - FHI Oxford

Thank You people the top does exactly
what it says in the title it has a look
at how we're predicting AGI and its
conclusion is that we need to increase
uncertainty spread those error bars and
add caveats to our predictions now one
hint that this may be necessary comes
from looking at some past predictions 56
where they basically seem to imply that
we would have HDI or something like it
within a few months nine years later
Dreyfus was predicting that basically we
reached about the limit what computers
would be expected to do I think it's
safe to say that neither of these
predictions have been entirely borne out
in practice here's another prediction we
will have a guy in 15 to 25 years it's a
slight paraphrase predicted this various
people predicted it actually this year
quite a lot of other dates many of which
are considerably more than 25 years ago
we should be able to do better than this
so this top will look at age our
predictions timeline predictions and a
few villas and philosophical predictions
asking two questions what performance
should we expect for these predictors
and as far as we can tell what
performance are we actually seeing it's
going to make extensive use of the
singularity's Institute's database of
257 they are predictions going back to
the 50s they got a bunch of volunteers
to scour the literature Internet to find
these predictions so that I didn't have
to do it myself so well what performance
do we expect this is XK CDs cartoon of
various fields in arranged by purity and
everybody sneers on the less pure
subject quite conveniently this is also
fields arranged by how solid their
predictions are let's make a little bit
of space economist about down here and
historians now why is it that for
instance when a physicist says this is
what we're going to see is the result of
an experiment we should listen to him a
lot more than if they sociologist says a
similar thing well it's mainly because
these different fields have access to
different methods from mathematicians
and their deductive logic we work down
through various hardness office versions
of the scientific method and end up down
at the poor historians who were limited
to using nothing but
examples now what should we put AGI
predictions that is indeed where we
should put HIV predictors mainly because
they can't use any of this pretty much
we have not have a single example of the
successful AGI so we're dependent on
nothing but well expert opinion and
expert opinion on its own is a lot less
reliable from the other methods on here
now another thing you might think as we
move down this graph there that as the
conclusions become less certain that the
experts become more tentative more
modest more willing to engage or accept
opposing points of view just one example
let's choose a common
economist here is Paul Krugman waxing
lyrical about Chicago School economist
he is a Chicago economist responding in
an equally generous manner now I should
point out that economics is a field
where average quarterly GDP is adjusted
by plus or minus one point seven percent
on average these are not predictions
this is past data and this is often the
difference between a large growth in a
recession and this is the adjustment
Center past data and if this is the
quality of the data in their field we
should we should expect a lot more
modesty in conclusions and predictions
as I said this is where we are and this
is extremely annoying because these two
people know stuff about economics they
have wisdom they have experience and we
really like to unlock those ideas and
use them for ourselves but as long as
they're sort of saying completely
opposite things we just can't get access
to that
this is the cartoon picture of
disagreements and overconfidence within
our heads we have a bunch of good
reasons to believe what we believe we
also have a large amount of appliances
and rationalizations and all this leads
to a reasonable ceiling conclusion our
opponents are in exactly the same
situation but from within our heads all
that we can see is this so what this
means is that no matter how right it
feels to you inside you must give your
opponent's opinions as much respect as
you give your arms no matter how much
you feel that you're right because
you're feeling this and that that also
exists in their heads then except you
just can't see it the exception being of
course if you have actual objective
reasons to suppose that one person is an
expert and knows more than the other and
when I'm saying that objective reasons
it's not just that everybody gets to
choose what their own objectivity
criteria are fortunately we have some
research into what makes a good expert
opinion it depends mainly on the tasks
there are some tasks of which experts
have good performance others in which
they have poor performance the ones that
have the features on the left you have
good performance the one that's other
features on the right you have poor
performance these are not equally
important on three of the most important
ones or whether experts agree or
disagree whether the problem is
decomposable or not and probably the
most important of all whether you get
immediate feedback
addictions in terms of predicting when
ages going to happen with kind of
probably stuck on these ones so we
should you narrowly expect for
performance from expert predictors that
doesn't mean if you're making your own
prediction you can't do all that you
can't move it on to the good performance
slide as much as possible especially
through for instance decomposing the
problem but very few predictors actually
do that and part of the reason I suspect
is that they're solving the wrong
problem in the FHI we have a special
shorthand
that's predicting grind that's a crop
coffee grinder they religion grind is
easy and predictive insight as hard
grind is something that will happen
because a certain amount of work is done
at it for instance how long will it take
to produce the next Michael Bay
blockbuster well if you think about it a
blockbuster is a huge thing you have
artists your marketers you have actors
you have producers all these people need
to work together and to produce one
blockbuster or attend to the blockbuster
but it turns out that even though it's a
vastly complicated project all that you
need is a certain amount of people to
work at it for a certain amount of time
and the blockbuster will emerge and we
have a pretty good timeline from how
long that'll take we also have a pretty
good estimates for how how many
unexpected delays will we'll get let's
contrast that with insight who would
want to predict when someone will solve
the Riemann hypothesis this is a much
harder task and very often when in age
age our predictions people are
predicting insights but actually
pretending that they're predicting grind
the most typical example of this is
ferrocene carnation
of Moore's law hence AGI which the
argument goes by some year computers
will have some level of Y and a floating
operations per second something around
the hard disk capacity or whatever that
is a level comparable with the human
brain hence AGI
now the Moore's Law is basically grind
people work at it computers get faster
so this looks like it's a prediction
right part of it is is a good prediction
but the most important part is why when
we get that number of neurons number of
fully operational centers will we get a
GI that requires an insight to bridge
that and very often people put up these
kind of predictions with making no
effort to bridge that gap at all but
anyway that's enough theory let's look
at the evidence so this is the
singularity's institute's database of
257 predictions 95 of them were timeline
predictions unfortunately they're not
all in the same format they're not all
well an expert thing by golly a
predictable have human level API by the
Year xxx
I went through them and to each one I
gave a median human level AGI estimate
this is somewhat subjective for some of
the predictions if you want you can go
online and come up with your own
predictions from the data for each
predictor we also access the expertise
of the predictor of presumably computer
scientists know a lot more about this
than journalists or writers so what was
the data look like well it looks like
this there's also a few predictions
about 2,100 here incidentally is true
original production even make out the AI
winters in the middle of the data now
the first thing that strikes me when I
look at this is that it's all over the
place take two predictions you could
easily get 20 years between them it's
not more it seems to bear out so what we
thought that in theory they are
predictors would be pretty poor seems to
be borne out by the data that we have
also there's no immediate difference
between experts and non-experts you can
sort of see small patterns but now
there's there's two sort of full
explanations as to why we should expect
age our predictions to be so poor one is
the so-called nice garan law which is
that basically people predict that an AI
will happen just before they die it's a
rapture of the Nerds a July will happen
and they will be saved there is actually
no evidence for this in the data here
we've plotted the prediction - the life
expectancy predictor these are the
predictions of people who expect to die
before seeing AGI these are the ones
where the API will happen five and more
than five years before they die there is
no strong clustering around zero as we
would expect if the mascara law is true
the second fork explanation is that
everybody predicts it 15 to 25 years in
the future
cynical explanation would be that it's
close enough that people give you money
to research on X but far enough that no
one will be able to call you on your
mistakes until you've moved on
that there might also be some
psychological explanations in that it's
hard to conceive of something that
you're sure will happen but it will take
more than three or four technology
cycles to happen anyway whatever the
explanation there is evidence for this
in the data consistently about third
predictions are in the fifteen to twenty
five year range this is the time to eh I
for everyone this is the time to AGI by
experts this is the time to AGI by non
experts and perhaps more damning of all
this is the time to hei by predictions
that have come and gone hmm
now see how little data we actually have
a few data points we actually have these
curves look really pretty similar to me
so it might be that the expert actually
you know they're talking about and have
extra insights that non experts don't
but there's definite of evidence of it
here now but remember just because the
experts not seem to be any good doesn't
mean it your own guess is any better
your guess is probably as good as an
experts which means what can we do about
this well the first thing is to increase
the uncertainty that helps with
everything suppose you say H is going to
happen at 2040 and you were very certain
about this well what you're saying is
that all those experts are wrong to
noodle confused in the stage you're not
only saying you're right but you're
saying that so many other people are
wrong let's say that you're saying it's
pretty likely
well now the number of people who have
to be wrong for you to be right it goes
down considerably now let's say that
you're saying it's an approximate guess
not something there's broad agreement by
the way feel free to increase the
experts uncertainty bars you can be sure
that everybody is over
confidence now another justification for
why we should increase uncertainty comes
from looking at what I feel is the best
timeline prediction which is the
prediction - for training relations
which you'll be hearing about in the
second session today
it's basically hearing you fix a brain
slice it up scanning and instantiate it
on a computer using less crude
technology and depicted there now why do
I say that this is a good timeline
prediction well because it's very
decomposed the also it's justified why
most of this can be solved by grind it
expects there is Moore's law there is
research being done and there's
justification why after certain amount
of work we should expect certain types
of results there's clear assumption
scenarios integrates new data to a
certain extent and there's multiple
pathways to the same thing my colleague
Andrew Sandberg has run Monte Carlo
simulations under three assumptions how
long it would take to get over any
relations and these are the probability
distributions that he got as you can see
that spread over nearly the entirety of
the coming century and if these are the
uncertainties and error bars for our
best timeline prediction then any other
type of timeline predictions should
probably have error bars at least as big
as this one so what can we say about API
timeline predictions are pretty poor
other types of predictions such as plans
for how to build a GIS tend to suffer
from similar problems now we can
actually get some good ideas about AGI
from philosophy
now this may be a very contentious thing
to say to most of the computer
scientists in the audience because their
vision of a philosopher is probably of
someone who says he named things like
this to which the computer science can
only respond well well not really
and then the philosopher hits back over
the target
intellectual arguments and the
conversation continues in a similar
fashion but the thing to remember
because the philosophers are experts
which means they're massively
overconfident so if you want to take
their arguments you need to add more
caveats or uncertainty decompose the
problems as far as we can do it so let's
just take this in any good else do
anything and decompose it and cabi that
a bit and when it's done that it
basically reduces to the rather
reasonable position that self reference
might be a problem in a GIS and you
should probably keep an eye out for that
and then you can actually get a
conversation going
I mean philosophy has had a few success
stories in the past some of these may
have been considered useful but let's
give some examples of how you can
improve philosophical arguments
this is dreyfuses computers can't cope
with ambiguity paper incidentally on the
65 paper is worth looking at I think it
would have got a lot more respect if it
had been so abrasive and insulting to
all the computer scientists at the time
but there are some genuine
sites inside but let's caveat that
argument that we can reduce it to the
far more reasonable and actually true
argument that using 1965 AI approaches
you can't cover them be guilty let's
have go sees more recent tradition about
go see which is identifying the computer
with the brain may be a mistake
computing isn't thinking if you weaken
this you get the reasonable point Agee
eyes maybe nothing like human brains and
who may go astray by thinking that they
are now I've had to look at a lot of
philosophical predictions for this paper
and in my opinion the best one around
currently is what I'm going to call the
Omohundro Bukowski theory thesis bashing
their two ideas together in its strong
form
it's that behaving dangerous leaves a
generic behavior for high intelligence a
GIS this is kind of a supply and demand
model for AGI it makes simplifying
assumptions as to what these agents are
going to look like but like any like the
supply and demand curve it still has
very useful insights right if we were
fine and now the system we have the
descriptive part which is that many AG I
do not have they release the potential
for unexpected dangerous behavior and
the prescriptive that AGI programmers
should demonstrate to moderate skeptics
of their designs are safe I don't mean
you necessarily have to convince 100 new
Kowski of this but it would be good if
you could convince people sort of not in
your programming Google for instance now
you may have a feeling that this thesis
is wrong but for what I've shown you
before unfortunately your own feelings
don't actually mean
here unless you have some strong
evidence or reasons to back it up
fortunately here there's a very easy way
to destroy this thesis I draw you your
attention to the second class again now
in conclusion so our own opinions are
not strong evidence of anything
philosophy has some useful things to say
one of the main reasons why philosophy
has some useful things to say is that
other methods such as timelines are so
very problematic and finally but
towering above all others and in
philosophy in timeline predictions in
everything I encourage you to increase
your uncertainty and I just want to
thank everybody who's been involved in
this and especially thank anyone who's
had the courage to go out on a limb and
actually write down a prediction
we have time for a few questions first
yours
you're just a random person in society
random person's predictions may actually
be better than predictions they see in
the media those are just I didn't lock
them all but there were some but when
your point is interesting so so so maybe
it's not a 15 to 25 years long maybe
it's a 30 to 40
predictions in philosophy and I would
claim the church-turing thesis on the
philosophical prediction philosophical
thesis because it's connecting
computation it's not something that
could emerge just from say the program
but so for how to make it well I mean
the edge is partially doing that we're
trying to bring philosophical results to
bear on a problem so I think
you should give more money to a park
format um let's see organize more mixed
conferences like this I think I think
it's probably more the job of philosophy
to try and reform itself coming closer
to bring useful results to AGI so
although probably the best thing that
the computer scientists can do is to pay
attention to useful philosophical
results and highlight them and this
might incentivize other philosophers to
start thinking what's practical should
my presentation yet however it refers to
both one is the prediction to hunt the
next presentation but is the question to
you you haven't mentioned term foresight
in your talk neither in your talk nor in
the paper and in fact by prediction you
mean a statement which was issued by
someone backed on intuition but non
scientific research by Thunder
experience on the whole item so and you
have shown a lot of disadvantages of
them however when you say most
prediction or false or not justified
enough have you found any optimistic way
of making such predictions or statements
about AI future have you formally have
you formulated any hint to the
researchers to produce good forecasts
good predictions
yes the the key the key things seems to
be try and check what Europe so what
bring out the hidden assumptions don't
make a breezy don't count on your gut to
come up with something decent try and
imagine how what you're talking about
would come across the would come to
being the decomposing the problem is
probably the single the single easiest
and thing that can be done there are
some decomposed predictions like Bank
rates those prediction for instance for
OpenCog seems very well decomposed Kurtz
was predictions are decomposed up to an
extent but it does the things get more
powerful things get more Marvel and GI
kind of appears so yes so probably so
probably the best thing you can do is
decompose your prediction as much as
possible and what you're saying AGI will
appear here from that'll because of that
make explicit what assumptions you need
to bridge that gap so if you say a fast
enough computer will main AGI happen
put down some reasons for that why is it
today's algorithms are enough if we make
them faster or is it that you're
expecting algorithms to progress at a
certain rate at this point in the
interests of getting a little less blood
into my caffeine stream I'm going to
move on to the last part of this session
but I do encourage you to grab Stewart
and
right


Carl Shulman Could we use untrustworthy human brain emulations to make trustworthy ones

so i need to take advantage of anders
and robin
and i'm just saying absolutely nothing
about what foreign innovation is
uh we'll say a bit
about
why it's worth talking about
so there are arguments that full brain
emulation is sort of an odd duck in the
history of technology
we've got airplanes and horticultures
and there are other kinds
of ways in which knowledge and biology
could inform agi
so you could have something that was
inspired but not closely mimicking over
emulation
there's a survey at last year's edition
of this conference
very small and serious selection biases
so i wouldn't i wouldn't necessarily say
that this is the most likely or default
scenario
but certainly is possible
it's worth considering
and
as robin said it's easier to analyze in
some ways
and a lot of it we might be able to
generalize from
and i'm going to have a little bit of a
different focus so robin was looking
at the inside view
from whole grain emulations operating
within these server farms
these data centers
but you can also look at it from the
outside
and
they wind up very different so if we
take robin's figure
at full tilt growth
doubling time every 2.3 weeks
then if you have a long-term trend
20 dozens of gdp
less than a year
10 kelvin years in perhaps millions of
subjective years for the fastest
m's who are the ones who are advancing
science and technology
acting as leaders
so you wind up that from a
human perspective
while the m era may be big from the
perspective of the m's
it's actually a very brief interlude
before something else
from the human perspective
and over the history of the earth
almost all of the future will be after
this period of a couple of years
and this talk is focused again on the
uh the control problem that nick was
talking about
in question
uh and just to
dramatize the difficulties at human's
eclipse
so consider a country united kingdom
trying to exert effective control over
territory
where you can't send your government
officials
time runs vastly faster
there are
many new technologies that the folk
within this territory have had centuries
to adapt to and understand and you maybe
have got
the wikipedia downloaded to you and
you're trying to make sense of it and
catch up with hundreds of years of
technological advance political change
and figure out
so now what do i do
knowing that my actions
will be will take years of subjective
time has to be transmitted
and of course if gdp is so large
and the center is the historical
correlation
is that military power goes tightly with
gdp
uh so it would be
a surprising break in the history of
governance
if it was easy for existing governments
to maintain authority
over regions within their borders that
or participate in this economy
so pretty difficult
if you were going to do it it would seem
you'd have to have proxies who are
operating at these speeds and these
environments
maybe you don't trust any human in the
world
to
maintain over subjective centuries
uh that kind of loyalty
and you might have further problems with
the earlier brain emulations they may
not be exactly the ones you would want
the reconstruction might cause amnesia
change your personalities
but perhaps you could enhance them to a
level of trust
where even such a
an extreme task
would be within reach
so
we have cover uh three main
topics here
first this question of
going forward towards
a fulbright emulation transition
what are the opportunities to steer
things by way of initial conditions
how did how could those conditions
affect the long-term future and this is
from an impartial point of view
second is a question that hasn't been
addressed very much uh
literature
which is what actually happens to the
unuploaded humans
and this this may be just that uh people
who are worried about future generations
and m's i mean we have a lot more reason
to think
about these considerations
i see the audience here are mostly human
so there may be some interest in that
and last i'll
talk about
the control problem for granulations and
what would be the most off-the-shelf
versions
that
humans that might allow humans to
actually hope to steer
such a transition and
see what we can do to assess that
pleasability
uh so let's start
a natural question why not leave it to
the m's the m's are people too
it is disputed but i would go with the
assumption uh i'm just making these are
moral ancient moral agents moral
patients uh
my talk last year they're smarter they
have a lot of advantages in figuring out
the answers to problems
and it would be silly
imagine neanderthal trying to shape
things
in our civilization today
long dead hard to influence and they
have no stable interventions
move over time
for homegrown emulation is a bit
different
because we have
if this happens we have direct knowledge
because we were able to actually design
them we're contemporaneous
and the technology itself uh opens up
new ways to implement lasting
interventions
and so
yeah and what might we want that they
wouldn't pursue or want to be able to
so these competitive pressures uh
evolutionary pressures
uh that they might not endorse uh i'm
gonna focus more on this section
on the first maneuver opportunities
and the possibility to set up
institutions
and especially the idea of
a singleton solving vocal coordination
problems
uh robin was showing these graphs of the
the talk circles
and as those shrink
it becomes harder to coordinate over
larger scales
and there's an initial opportunity
that the first four brand emulations are
introduced
uh to have this
leap forward have the first foreign
emotions
in a single jurisdiction
uh we're in a single institutional
framework
uh and why might you want that
so a lot of these things tie into the
speed
so
we have
nuclear weapons thousands of them every
so often there's an international
confrontation
cuban missile crisis able archer
uh flareup will happen in taiwan but we
sort of accept it politicians have to
decide am i going to go to enormous
extremes make big political sacrifices
to avoid a little risk
and so they accept some risks
and that's okay over a short time
but over a long time
not so good
there are other factors that recur
occasionally in history
you get
big ideological changes you have
variation
in leaders you have lone individuals
terrorism
and you have new technologies that are
discovered and each of each of these
brings some risk over a long history
many people might
die or suffer out of these things
but perhaps perhaps at any given time
the damage would not be so large
but if you accelerate all this this
process by thousands of times
then a human who's around for the whole
brain inhalation transition
is also around to see
any disasters of this type that lie
ahead in the next ten thousand hundred
thousand subjective years
uh and so if that's not going to happen
then either you have some kind of stable
institutions that make this m world much
less prone to war disturbance violent
turbulence than our history has been so
far
uh where there's some spontaneous
natural natural fall and risk levels
change in technology
and so if that's the case then we might
uh help to create that kind of stability
uh and stable institutions are very easy
and we can naturally merge or be set up
by the ends themselves
then we might try to change their
character just going to skip over this
slide this is just the question of you
can uh can you just move away
from any turbulence
get to the periphery uh we can get to
the periphery maybe for a while
but again if this whole transition and
million-fold economic growth is
happening over a year or two and okay
you can
escape
for a little while but
a one-year reprieve is not
not that great
um
nick bostrom uh again has
ridden on
some of these competitive pressures that
you might want coordination to avoid
uh that there are things that we like
so play romantic love
watching tv that we might have fewer
fewer opportunities for as we substitute
towards work optimal efficiency
and
lots of resources spent on long-term
competitive activities that might have
been acquired so
but that's been pretty well covered in
the past
and there's the
robin made the assumption that we have a
world where whole brain emulation stays
the same
we have no
novel lines no merging no splitting
but there's a bit of tension with the
very high speeds
so if you have
the top of the pyramid so a 16 million
times speed up
then
this is even
even if you have a fairly conservative
estimate for agi timelines uh which i
think is is pretty fair because as you
go to the
to the groups that are less selected for
being agi enthusiasts
you get further and further timelines
uh so probably it should be pretty
conservative for human progress towards
agi
but even if it's hundreds of years
well with very high speed up
100 fold speed up one year one year from
100
000 volt speed up
by two months 53 minutes if you have the
16 million times speed up at the top of
the pyramid
then uh
i mean it's uh much it's shorter than
our q a
so and now the actual speed up you get
wouldn't be quite as fast as the
emulations themselves because there are
other use there are other inputs that go
into ai research uh if you're if you're
using effectively a pentium pentium 480
uh pentium intel computer
and just moved back from the computers
of 30 years hence you know you're gonna
have trouble you have to optimize your
code very finally you can use in
blackboard a lot more
but nonetheless
to the extent that ai research is about
ideas and algorithms and not just brute
force
then you expect
good shifts there
and if you have
again
the movement towards
agi and novel forms of mind among whole
brain emulations
then you have
race to the front presumably by
the fastest emulations
this is just a chart
of the delays between
the front runner and the runner-up for
nuclear weapons and delivery
technologies
and
this is just for the
first atomic bomb
uh
and so
a nice thing about
emulations if you had initially
a singleton other that is if
if you have a lead of say a year or two
or even six months
of
those with the cutting edge brain
emulation technology either the
originals or more efficient
implementation
then there effectively gets amplified if
you're six months ahead but you're going
20 times as fast
then you have a chance to take
do 10 years of research
and
the longer the lead you have
fewer risks you take
so this is uh
these are from the ways in which by
shaping what the institutions first
organizations are under
i don't get to
push on some of these dynamics
so that's all from my impartial point of
view uh weights distant future
generations
whole brain emulations
novel agis
all equally
this would be this sort of
utilitarian universalist kind of
perspective
uh there's also just this
sort of narrow question of what do the
humans do
and who are not uploaded
and again it's maybe of some interest to
humans
from a theoretical perspective and in
terms of what actually happens because
presumably a lot of the decisions and
the development and deployments and
sales
or prohibition
of the technology are gonna be made by
humans
and uh yeah
and uh just
some of these effects on humanity could
be the question of survival uh are
humans affected by wars
ideological violence
other spillovers from the whole
granulation economy or its successors
what happens to human wealth
how is it actually profitable
for humans to try and sell an ambitious
technology into the world
and so on this this area of turbulence
so you don't want to get too much into
the weeds
the main main argument is just that we
witness a lot of violence and
disturbance over history and over an
extremely long subjective period
we have to expect more risk of that than
for a short one
but just to
raise a couple of particular risk
factors specifically brain relations
uh that might lead to an ideology which
is not necessarily anti-human but not
pro-human
so people have huge life support costs
if you have one cool brand emulation
and computation is effective and
efficient
and then you can maybe support thousands
millions of digital clients
and uh
the real really cry is look here are
these millions of poor m's who are about
to be deleted if we just uh
diverted resources from these human
retirees and these millions of people
who get to live a preventative life what
kind of moral monster
would uh will take this attitude
and so to some degree it's to implement
robert nozak's idea of a utility monster
and a utility monster is a creature
that when he eats a human
is so happy it really loves the taste of
people
that the uh the benefit to it outweighs
the harm to the to the person needed i
was something that's not normally
realized
but with such a large difference in cost
of living
you can approximate this and so welfare
and gdp go up every time a human gets
eaten
at least in the direct effect
the fate of the horse
and
this would be the argument
so with the treatment of death which
andrews and robin both went into a lot
so i'm not going to spend too much time
on it
but you have all these pressures to
allow the continual destruction of huge
numbers of these creatures
great economic value of that
tends to promote
uh
ideology that's biological death not
much of a problem so why should we why
should we worry so much
uh if the human population
is either destroyed or uploaded against
their will
there's many humans may disagree or may
not like that
there's the general reasons why you want
to do that
robin had uh 42 hours of sleep on his
weekly schedule for whole grain
emulation and you do that you might do
that if you want to have a continued
activity over time because you need the
sleep to replenish mental processes
reorganize memory but if you're doing
short tests like the plumber
then just eliminate it
and anders at the end of his talk was
discussing uh human whole brain
emulation social distance
a couple of factors uh one is that
we're way behind the trend so if we can
sit if we can there was a discussion
earlier what if the ethics of the romans
had been introduced
keeping slaves gladiatorial games
uh all of these things they'd be very
socially unpopular they would not get
invited to the right parties
they would not not advance in a society
and if you're
hundreds or thousands of years behind
the cutting edge of the fast mines in
the court
then it'll be very hard for you to have
the right in-group singles because those
fads continuously change
and you won't be able to
well a whole brand emulation can be sped
up temporarily to match the fastest
a human can't so again it cuts your
gossip networks
the informal channels of social button
and allegiance and politics
and just you know very different lives
very different situation
so that would be the what uh the sort of
ideological issue
and one reason to think this is
especially important in terms of wealth
uh the role of resources
so which is
uh oil extraction in saudi arabia
so many factors of production labor
capital land resources resources are not
that big today
has a share of gdp
and they don't necessarily become a huge
share of gdp going forward even as wages
go down to subsistence
what they become increasingly
is a source of rents of disposable
income so
masses of malthusian laborers may be
earning a large total income they have
very little disposable income to
purchase luxuries
uh things that might want and insofar as
humans aren't efficient they're not
productive they're way behind the
frontier
sustaining the lives of humans
and the entertainment they might want
the descendants they might want to
produce
all that depends on disposable income
and most of those resources are not
privately held at the moment
so
like the sun
only a billionth of the solar energy
actually lands on the land surface of
the earth
uh the rest is going to space the oceans
uh antarctica and upper national
territories
and so and beyond that there's galactic
resources but
confined to a sort of
solar scale uh for the moment but a very
very large majority of
resource wealth
is unclaimed right now and unclaimed
resource wealth tends to be assigned by
the political process
and political process goes with power it
goes with connectedness
uh it goes with numbers
now if you have property rights either
ownership or part of the economy
bonds
resources
you may be able to have a very nice
retirement if property rights are
enforced even if it's a small amount
relative to the overall gdp
um but again the speed up matters
because you have to not just retain
property rights over 100 years or a
thousand years
of subjective time in order to manage
a few calendar years of possession you
have to be able to deal with millions of
years of change
uh
going back before modern man
from mother
uh and there are various ways in which
uh
wealth tends to be eroded by the broader
society so with property taxes
uh if the property taxes are set at a
uh
efficient level they tend to drive land
to the gdp maximizing use
between people
wealth taxes expropriation inflation
all more popular
against groups that are politically weak
um otherwise
uh now the robin says if the total human
wealth is very small
then there's less
in the way of spoils to be shared
but on the other hand
there's less political clout to defend
against random noise like if as
regulations and policies change for
other reasons and then incidentally
affect
humanity or it becomes just a a moral
issue those are ideological reasons
before
um
yeah so i've been
highlighting some of the
neglected
negative features there
but
decisions might not be made by uh some
democratic vote of the people of the
earth or the most powerful governments
of life
but if individual corporations are only
positioned to decide will we create
whole grain emulations will we unleash
them into the world
and then see this economy we get a lot
of profits right away depending on how
much we're able to appropriate out of
their labor production
uh and so is that a good deal for
microsoft shareholders
so
mostly uh share ownership is pretty
diversified
uh microsoft is held by thousands of
institutions millions of people
and indeed has a share of the world
population uh which is relevant for the
political process
uh you're you're getting above 0.1
percent there
uh you start selling nuclear controls
over emulations
you have some downside risk
of
getting knocked out by turbulence from
the fast center
and you have to divide the profits among
your many shareholders
so
on a pure wealth basis
if the human share
of gdp going forward is going to be much
less than 1 000 it might be actually a
bad deal and expected value for
microsoft shareholders
to be the ones to sell whole grain
emulations to the open market
if you adjust for risk on the downside
it might be more extreme
although if your facebook and mark
zuckerberg
voting control of the whole company that
maybe decides well
just go ahead social media
next level
but broader picture here
is that significant risk
apparently
human elites losing power
i can make that concrete
and inclusive democracy
almost all the voters very quickly rms
the only candidates for office who can
actually follow what's going on in
society rms
and so existing human officials uh human
governors are not in a position to
compete
an authoritarian regime it's that
problem from the beginning
of uh can you have little territories
uh within your jurisdiction that have
almost all the military power are
operating super speed and just uh cut
them out
entirely from governance and
tricky
so to the extent this is happening you
have the politburo
is considering
so some companies are working on whole
grain emulation and they're just going
to publish it open source
and do we want that
given that we expect that we're going to
be out of power
relatively soon human electorates are
going to lose their current position
and generally relative terms big losses
with some absolute risk of absolute
losses
so it seems like there'd be a
significant incentive
for decision makers looking forward if
there is any way
in which they can stabilize this
transition
uh have more control over it
i'd be pretty motivated to find a way
and which brings us to the control
problem
uh and so nick gave a whole panoply of
different uh methods under capability
controlling the sum of the ones that are
relevant
so
resetting
very important
automated surveillance
and since robin was discussing if you
have mind reading and direct
psychological alteration
as potential to create superhuman levels
of loyalty uh and loyalty to arbitrary
things
and that we have
we have some instinctual drives
reactions to food and sex and heat and
cold
but no one is born with a
deep instinctual attraction to the
unitarian church
or the government of liechtenstein
but if you have the ability to
um to observe your own reactions
different stimuli
set up programs to react to that and
stimulate the appropriate regions then
you can have such reactions attached to
arbitrary stimuli
and proud to get a degree of stability
which is very high and then have those
stable emulations implement some
procedures that bind their own
successors
but how would you get there
again there's this
competitive pressure
depending on the lag time if developing
those more thorough
thorough means takes years
it's very difficult to coordinate around
the world all the corporations academics
governments that can do things so maybe
maybe that doesn't happen and we push to
the competitive scenario pretty quickly
but
again perhaps even if
whole grain relations aren't being
unleashed to do their thing unencumbered
maybe there's some intermediate speed up
you can get that is
if humans would take years to develop
some safety measures maybe 10 years
which is too much for them to manage
and the actual lead time that they have
could you make use of the untrusted
system that you have at the start
using just the the prudent methods
in order to
find your successors
and
some of the the easiest methods would be
you have your human team
you have brain emulations
uh which are limited in their speed and
numbers
uh and doing compartmentalized tasks uh
basically generating abundant amounts
and then this can be called and used for
various purposes
again yeah ethical issues
very very serious
but i'm running out of time so
the mechanical turk human-based
computing there are a lot of techniques
for integrating the ability to call on
humans easily rapidly
kind of tasks you could put it to
anything that can be put into a formal
proof
verification system is going to take a
long time to make a proof longer to
formalize it but you can hand off the
work of formalizing it to the whole
brain emulations
and then all you need to trust is your
formally proven proof verify this this
helps for maybe
some things math some algorithms some
crypto
but not everything complex software uh
it's something easier to build your own
based on
the algorithms idea than to exhaustively
be sure that there are no backdoors
neutralization problems
and just in developing the direct
programming motion control methods uh so
uh models for the more fit more
efficient emulation migrating
psychological alteration
you have now your masses of experimental
subjects potentially
that others did not have very helpful
and just miscellaneous research tasks
it would not be surprising if you can
get a tenfold speed up
these sorts of sorts of things and
leverage any initial lag in
convolutional technology
now
if hardware is the strongest limitation
that maybe have less of a speed up
initially
changes some of these scenarios
and also the
the fine
the the level of uh
local paranoia in the multiple
compartments
by neumann month api
may be great overkill
but it shows how far you might be able
to go even with
quite severe restrictions on the
capabilities of any clump
any team
and so
the
the slides are not up but the
paper will be following
and so
thanks everyone
uh max before we get to the questions
um saint anne has
said that people who are checking out
today need to
have already checked out
um
so if you are checking out today um can
you just go and check out
um relatively quickly there's a place
you can leave your stuff at the fortress
lodge
in the meantime
right
questions
uh
yeah
progress within uh club emulation uh
world to the
actual progress that we have had over
thousands of years
how much do you think the
comparison would actually
have to be slowed down uh or would be
like stop linear but for two reasons one
is that
the lack of access to physical
experiments that are like on the same
speed scale and the other is uh
the lack of uh
program like
uh progression of generations but if you
have same people around
and you foreign kind of robust brain
emulation technology we're talking about
presumably people will eventually figure
out how to create artificial babies or
scan more
uh on on progress yeah it seems you
would get a very different
uh profile of technological advance so
if we arrange the fields by the extent
to which they are
brain and chalkboard uh scientific
literature versus
uh fermilab huge complicated equipment
that takes years of physical
construction rare elements etc you're
going to get a huge skew
of progress towards the former end
there'll be lots of advances in
mathematics
lots and lenses and pure algorithms
fewer and brute force
optimization of parameters
uh way left
you may have a lot of
theoretical physics advances but then
be unable to test which theories
are right likewise with
working with
chemistry
some some kinds of engineering
biotechnology all of those will be much
slower
but
if we're looking at the the areas that
would seem to the most promised to keep
going areas that progress would be
skewed towards
those areas of mathematics computer
science photography
seem especially especially relevant
and with
towards agi another thing if you advance
sufficiently far on that
then you get
advantages in all of these other areas
which
will be much less than you have in the
ivory tower fields
but then you you deploy them and have a
a slow a slower effect on those lines
but still very fast from a
outside perspective
like political turmoil willing
yeah so that's the question how how
confident can we be
whole brain emulation
uh and whatever follows from that in
this very
very rapid sequence
is going to be much much much much more
stable than sigma being historically and
i think it it very well could be much
more stable there are conceivable ways
in which that could go but from the
perspective of a decision maker who's
deciding
well do i really want to rush this
forward for a possibility of slightly
greater profit
than it's their uncertainty over will it
be much safer or will it be
uh something vaguely continuous with
historical rates and subjective types
it seems to me that one of the dangerous
things we could do would be to try to
solve the control problems
of the relationship
having been born out of this situation
where a bunch of neanderthals trying to
control and limit their capabilities
might be exactly the kind of thing that
would make us to make them less friendly
stewards of the remaining 10 20 million
humanism
and so potentially there's a version of
the control problem which is just let's
not try to solve it but let's think
really carefully about which
brains we emulate first and like you
know who the first 10 or 20 people we
sent through this transitioner what
personality types we want
in order to have a friendly society at
the other side um
how
much have you thought about those two
things
of initial yeah
the thing you strongly want to do and so
from a personal perspective i'm really
pro with the upload rights
um indeed so if with an option
and didn't involve uh
being copied on mass briefly and then
wiped out as some slightly more
efficient version uh appeared on the
market it would be something i might
might want to embrace
um but
you can you can adopt the perspective of
emulations are people too
and even
going forward we like
even
many or most humans to be able
to upload and move into this position
and yet also want to create some kind of
very stable order
uh that will be able to maintain that
initial commitment
of those first uploads going in
uh who maybe don't want to have this
radical transformation into
the the full reaches of the competitive
scenario
but whether they're able to hold that
together over time in the face of
competition
it depends on their initial condition so
i would if you were having a project
like that i would imagine you would want
to find volunteers
who are very
who are very keen on the the long-term
prospects you can get after you've
stabilized the situation
which then allows for say a more equal
distribution of the right to be uploaded
uh allows for a more
sustained even distribution of wealth
across uploads non-included humans and
new
agis and other kinds of beings that come
into existence
and to
to the extent that you can separate
separate that uh that was the way to do
it
um but there's there's a question of
with people working on such a project
brain emulations working on such a
project of developing better control
measures perhaps including known forms
of agi
uh
better forms of
mind reading and loyalty verification
would they see that as
look we're working under very harsh
conditions
as we accepted in order to achieve
this important thing and help the rest
of humanity and ourselves uh going
forward uh or have we been like drafted
into this project by people who don't
care about us and uh
this is perhaps the the disadvantage of
rushing over ethics
but uh
yeah so they
yeah so
i i could endorse uh the solution to so
make sure that you have only volunteers
who can who have philosophical views
that are compatible with this make sure
that they
incredibly have critical expectations
that they will be greatly rewarded going
forward uh their rights and will be will
be respected do every everything you can
to make things congenial to them
and uh sort of
valorize and honor them because they're
saving your butt
hello there um i'm eric scruff
uh
i would like to criticize the
presentation a little if you wouldn't
mind
no
sure i just a question well you seem to
be
thinking or suggesting that it's a bad
thing
that the human elites
are going to be retired well personally
i don't think that's a bad thing at all
uh i'd just like to disclaim that i'm
not very attached to the position of
power of the public bureau
i do think it's something that if i'm
thinking descriptively
about what's likely to happen well well
politburo or the white house i'm not
fond of either
anyway and also as somebody who is
well very wary of dangerous ideologies
you seem to be suggesting a very
unhealthy mixture of human
exceptionalism which i interpret as a
kind of racism
and
well totalitarianism some kind of
obsession with controlling every kind of
ai out there
and um
totally free unregulated capitalism
which i also interpreted as highly
unethical
so well instead of these why aren't you
working on thinking of what kind of
social
political changes would be more
acceptable for both mankind and machine
kind without you know this seems while
this slide
presently disregards the rights of our
minds children entirely i think
why is why this approach because you
know dr strangelove
would have loved your plan but he's not
a very reasonable person well in fiction
but this is not science fiction it
should we
i don't want a dystopia
why the why do you want a totalitarian
world i don't understand it
i think i'll just
disclaim uh endorsement of unregulated
capitalism uh totalitarianism
or
differential moral status for
organizations and humans
um
i wonder if you well so what's your
bottom line in terms of probabilities
you would assign say a conditional on
whole brain emulations being the first
form of machine intelligence to arrive
that the initial creation of this will
be under the control of
one agency whether it's one state or one
research group rather than sort of
simultaneously merging in many places
that's one
and b
the probability that if there is this
one group that controls the launch of
the whole brain and malaysian era
that they will be able to get an outcome
at the end of it that
avoids an existential catastrophe
if or that avoids
a kind of hyper competitive market
scenario of the type that drop in
outlines
yeah
so i'd say
we should have very wide confidence
intervals on these in general
so the kind of anticipation
involved and
for for seeing such things having an
organized effort
to
create that create the technology
that won't necessarily happen if things
are coming out of
a scientific project that just starts to
work better faster relative throughout
the sunlight not to happen too much if
things develop
in an open source fashion in the uh
in the scientific literature and so on
then again
you don't have uh much
de-synchronization
uh so the
you say
that
there is a a sizable chance that you
wind up with one agency
more than
more than a quarter but less than
three-quarters
would be uh being my assignment there
and
going forward of
do you avoid
the
hyper-competitive
evolutionary outcome given
an initial starting point
of a sink of a single
global governance mechanism
my guess would be that first that these
any uh any additional safety measures
one is adding on here i don't
necessarily have that much marginal
effect
beyond just the starting point uh if you
have
people uh the whole brain emulations
who have plenty of time to work on
things
and given that people
in general
want to
avoid outcomes
that they
see as horrendous
given
lots of time
smarts and resources
i i would not i would not bet on
severe systematic error
or screw ups
just because i mean if you have a very
predictable error and this is the
question of
why are all of these agents unable to
predict it given the all the other
evidence that they have
okay i'm gonna have to cut off the
questions there
unfortunately and so join us um at uh
six for the um
for the final keynotes
of the conferences and uh just give a
thanks to carl martin
you


Stuart Armstrong Manchester AI debate

hello and welcome to this talk entitled
should we or not create strong AI an
important question connected with that
is could we create strong AI and if it
is possible could we actually decide not
to on the subtitle of the talk is yai
compares with plagues in nuclear war
which might give you an idea of the
direction where I'm coming from at the
future of humanity Institute where I
work we define existential risks as
risks plausibly likely to annihilate
earth-based intelligence life or
permanent curtail its potential amongst
the the central risks this or top five
we're looking at it for the moment are
no particular order pandemics synthetic
biology nanotechnology artificial
intelligence and nuclear war not
included on this list of things like
asteroid impact and environmentalists
simply because in relative terms these
are just not dangerous enough so today
I'm going to be focusing in on the
artificial intelligence and you might
take askance why is this considered to
be dangerous why is it indeed an
existential risks well first things are
any questions what do the experts say
let's have a look at them unfortunately
experts don't have exactly the best
track record this is the dartmouth
conference in 1956 essentially
predicting that artificial intelligence
could be made over the summer this is
Dreyfus nine years later I'm suggesting
that the top of the achievements of
computers was imminent I think it's safe
to say that neither of these predictions
have been entirely borne out in
practice here are some more I
predictions AI will be developed in 15
to 25 years you may want to guess when
this was made well in fact it was made
by various people in 2012 also 2011 2010
2009 2008 2007 six and a whole host of
dates all the way back to the 1960s I'm
hoping we can perform better than this
and the first question is to ask why is
why the prediction so terrible well
let's have a look at other fields where
predictions exist this is the xkcd
cartoon which arrangers feels by purity
with the purest field sneering of the
lace Pure Ones by a convenient
coincidence this is also approximately
things arranged by predictability how
strong and accurate the predictions are
if a physicist tells you something about
the result of an experiment he'll be
quoting three significant figures if a
sociologist or psychologists say
something the predictions will be a lot
less accurate and a lot less likely to
come true I've added an economist and
historians on the graph now why is there
such a difference in quality of
predictions what mainly because the
different fields have access to
different tools mathematicians are lucky
enough to be able to use deductive logic
others have stronger or weaker versions
of the scientific methods down to the
poor historians who are reduced to
nothing else for past examples but where
should a I prediction lie on this graph
well there is a convenient hole down
there in the left and indeed AI
predictors lie down here because they
don't even have past examples to rely on
since no one is actually built in AI so
they're relying on nothing but to expert
opinion which
considerably worse than any of the other
tools here so the question arises when
our experts good when index which would
give predictions this is from James
shantou who noted that the performance
of experts tended to vary more depending
on the field in which they were working
rather than what which particular expert
you had the quarter their quality or
their training so for instance are in
medicine anesthesiologists would be
quite good and many of the mammogram
interpreters who wouldn't because their
fields have different things of good
performance or poor performance
especially where feedback is concerned
now feedback is probably the most
important thing distinguishing a good
field feelers was a good front port one
two other important ones are weather
experts agree or disagree on their
stimulus and other aspects of it and
whether the problem is decomposed or not
where AI predictions are concerned were
probably stuck with this where almost
all the features are the ones that
should lead to poor performance
interestingly enough one of the major
ones that could lead to better
performance is what the field is
decomposable or not and it could be
decomposed but unfortunately very rarely
is so this is the theory what do we see
in practice well here I've plotted
various prediction dates for a ice
arrival made by some experts and non
experts and also the data which the
prediction was made you can distinguish
cheering's original prediction here and
the AI winter here where no one was
talking about AI anymore and the thing
that strikes me on looking at this is
just how spread out they are the
difference between any of those two bars
is 20 years and they are just spread out
all over the place no
difference between experts and non
experts and the real indication that
there's any sort of convergence on any
value some genuine sign of expertise now
this is the cartoon version of
disagreements and overconfidence because
we've seen that I experts strongly
disagree when we reach an opinion we
base it on a lot of things life
experience evidence you did arguments
and a variety of other stuff and let's
be honest some also some biases and
rationalizations and this reads leads to
a reasonable conclusion what about the
other people the people who disagree
with us well they're doing exactly the
same thing except when within our minds
this is all we can see so no matter how
much if we feels that our estimate is
correct and that earlier the else's
estimate is superficial this cannot tell
us that we are indeed correct because
that is what we expect to see in whether
or not we're correct so that means that
just because the AIS are disagree I
experts are disagreeing over the place
that doesn't mean that our own
intuitions are any more correct our
performance and our prediction is likely
to be just as good as an expert's which
means utterly terrible but let's look
into AI itself more why could it be a
potential risk well don't think of the
Terminator which is basically just a big
muscle and no brain as this picture
shows here the dominant species not the
one with the big muscles this is the
model of a chimp rain or a picture which
in brain next to picture of a human
brain to scale chimps have a population
about 200,000 and use basic wooden tools
humans have heavy industry and nuclear
bombs and we've spread across almost the
whole surface of the earth and since
we've I've munted our power
computers aren't elected with computers
we've developed hydrogen weapons I
landed on the moon and had unprecedented
economic growth so the question of what
could happen with intelligence is if we
have an AI that takes the next step up
what transformations it could write it
could do my preferred model of what sort
of AI you could get with purely human
level artificial intelligences is you
could create say a super committee of
the AI Edison Einstein George Soros
Clinton Oprah Plato gurbles Steve Jobs
and Bernie Madoff give them vast
databases and network them together
running at thousands and thousands of
times human speed this entity you could
create with just by copying and training
human level eyes would probably consider
that the internet and the human race are
just useful resources for whatever its
goals are its goals okay it's one thing
to say it's powerful but might it not
have positive goals or potentially what
we would want is that the AI would have
a tag where kill all humans is false and
help our humans it's true except of
course the problem is that these are
undefined and trying to define what
these means is immensely complicated and
prone to a lot of potential disasters
for instance the goal of preventing
human suffering which sounds very nice
and effective how would a I interpret
this well this is the single kill all
humans is the single fastest and best
way of preventing human suffering okay
that's not what we meant but what we
said so let's be a little bit more
sophisticated keep humans safe and happy
okay
I think you can see what this is going
this is in Tomb everyone in underground
concrete coffins on heroin trips and the
AI will fight you if you try and prevent
it from doing this because any other
possibility any other outcome will not
be the maximum way of keeping human safe
and happy Andy I'm a perfectly well
understand this is not what we meant but
it has absolutely no reason to care now
some slightly more sophisticated
versions have the AI de Deus human
preferences from observation rather than
trying to program them in now this isn't
may not be quite as dangerous but
there's definitely a risk that if we
unleash this an AI takes it literally
that we get a future of the entire
universe that sort of looks a bit like
this
anyway I said that there were the AI is
a domain which is very difficult to
predict that's true you could say quite
a lot more about how the a.m might
develop then about sort of specific
timelines this is what I call these
simplified Omohundro you kowski thesis
that behaving dangerously is a generic
behavior for high intelligent a is for a
variety of reasons to do with how the AI
would work on itself how it's again with
unclear goals and how amassing power is
almost always a good thing for the AI to
do whatever its goals because it gives a
great chance of achieving its goals and
if say human safety is not fully
programmed in humans would might just be
an obstacle or tool for the AI to
achieve power for whatever goals it
actually has now this is sort of the
simplified economic supply and demand
equivalent it's a good starting point
now you need to carry it a bit which is
that many AI designs have the potential
for unexpected dangerous behavior and
with that claim that goes a normative
claim that AI programmers should
demonstrate to moderate skeptics the
design is safe now you might disagree
with this thesis even if it's caveated
refined and narrowed form despite the
evidence and the arguments there but if
you disagree there's something very
simple which you can do which
demonstrates a moderate skeptics that
your design is safe if your design does
not pass this bar then what are you
doing messing around with it in the
first place but anyway to sort of
summarize of AI is there potentially
extremely powerful I don't want to claim
that it's certain that they will be but
there are great uncertainties here and
the great uncertainties cannot allow us
to say they won't be extremely powerful
but they're necessarily week the
probability of them being extremely
powerful is worryingly
hi there is extreme uncertainties as
I've said it's probably inevitable if a
eyes could be developed our then with
the commercial pressures and military
competitive pressures it's probable that
someone might that someone would given
that they can in one country or in
another they're potentially extremely
dangerous as we've seen and they're very
few people working on a true AI safety
there's some at the future of humanity
Institute where I work the summits miri
which is a Californian group and there's
a few other scattered ones but it is
very small and so I conclude this by
pointing you to the websites of these
organizations I have a booklet called
smarter than us that presents these
arguments in a polarized form Nick
Bostrom the head of my Institute has a
much better and thicker book called
super intelligence which I strongly
recommend that you look at and thanks
for listening
you