The AI Control Problem

[Music]
recently I finished reading super
intelligence paths dangers strategies by
nick bostrom published in 2014 an
astounding insight on the speed of AI
development occurred while looking at a
table on the book regarding AI game
mastery many AI experts figure that the
game of go would not be mastered for
another decade citing the progress made
up until that point however this
prediction was shown to be ill informed
when the world champion of go Lisa doll
was beaten by alphago on the 9th of
March 2016 what this shows is that the
field of artificial general intelligence
can have overnight break first and the
time left until artificial general
intelligence arise cannot be predicted
in any certain manner while this does
not mean that Meiji is just around the
corner it does mean that preparations
the creation of one do need to begin
sooner rather than later since I've
already made two videos on this topic
you can see how close we are and
whenever an age I will be friendly or
not in these videos here my purpose for
this video is to talk more on the points
highlighted in the book super
intelligence namely about the control
problem and what should be done about it
the AI control problem deals less about
how to control an AI and more about how
to make the goals of one aligned with
humanity's values to their ways of
species do not face an existential risk
however to do this is much more
challenging than one might expect
Steve on one hunter and expert in the
field of AI outlined four basic drives
most intelligences including advanced AI
systems would follow to achieve any
specified goals
these are self-preservation since in
order to achieve any of its goals it
must continue to exist in order to best
achieve them efficiency meaning that an
AI system will want the maximum amount
of use of all matter it requires the
nexus resource acquisitions meaning that
to achieve its goals having the maximum
amount of resources available to all out
to best achieve them and finally
creativity the ability to think of new
ways to achieve any given goal
these basic drives mean that the
potential for catastrophic risks
especially in the domain of resource
acquisition is possible with an AI
programmed of any given goal and poorly
specified ethics any AI that wishes to
require resources including the matter
that makes up your body would not do so
out of malice but out of necessity for
furthering its original goal while no
rational programmer would create an AI
like this on purpose as super
intelligence would understand the
ultimate purpose of its goal better than
any human and would disguise its
intentions until it would be absolutely
sure it could achieve it in the book
Boston refers to this as the treacherous
turn so how can we solve the control
problem there are two main areas of
focus in this regard capability control
and motivational selection methods
capability control is controlling how
much influence super intelligence is
able to exert on its surroundings in the
book four main methods are given boxing
which is when an AI is shut off from the
outside world to the best of our
abilities or to place the AI in a
simulation which it believes is the real
world so we can test it to see what
we'll do in a given specified scenario
this method may seem smart however an AI
may discover that it is boxed and will
act friendly until we deem it safe to be
released at which point the treacherous
turn occurs and it's true intentions are
shown when it is too late
other capability control methods include
tripwires which would detect an
attempted escape or certain actions and
shut the AI down however an AI system
may be able to detect and evade these
tripwires
given in high enough level of
intelligence or predictive ability
limiting the capabilities of the AI such
as less powerful hardware or inefficient
software is known as stunting something
an AI seems unproductive however as
competing countries attempting to create
powerful AI systems could be careless in
regards to this method or the AI may
eventually improve its own software and
bypass this control method anyway
finally there are incentive methods to
not act against the interests of
humanity controlling the capabilities of
the AI would only be a temporary
solution since even if these methods do
successfully imprison the super
intelligence once another one is created
the control problem
need to be solved all over again the
more logical solution would be
motivational selection methods these
involve aligning the goals of the AI
with the values of humanity nick bostrom
outlines four main methods to
controlling the motivations of AI
including direct specification which is
creating an extremely detailed goal
outlining all aspects of it and in great
detail however as as the most free laws
of robotics are proven this method is
unreliable due to unforeseen loopholes
or exploits within the fine print of any
given goal another option is to
masticity which is giving the AI
unobtrusive self restrictive goals the
third method is augmentation using an
already known method of ethics such as
that of the human brain and using it as
a base or super intelligence however the
ethics of a human based system may be
flawed due to our nature and we also do
not know how the human brain would react
becoming infinitely more intelligent
than any other human has ever been
before finally there is indirect
normativity this method is using
indirect rules to specify goals hence
the name instead of coding an AI with
specific goals we could give it a
general principle to follow that places
our goals to be the same as the AIS the
most challenging aspect of most of these
methods outlined above is how to specify
human values what makes humans as a
species happy and how do we code it
since the first super intelligence we
create will likely be the last we need
to make the first time we try to teach
it human ethics will also be the only
shot we get at it so there can be no
read or tense ultimately to solve the
control problem will be a matter of time
and research even in the time this book
was written many more methods and
insights into the value alignment
problem have been suggested but whether
the time it takes to solve the control
Rollem is 100 years or 5 the best time
to start is now there are some awesome
organizations doing preliminary work on
the topic which I'll leave some links
through in the description until next
time thanks for watching
you


203. Roadmap to a Roadmap

yes
so i know that
uh you've all read the paper and come up
with some
really great probing questions so i
won't rehash all that in detail
but just briefly to set the stage of
what
prompted this paper i felt like
a lot of the discussion in the
literature around
arms races toward ai and potential
arms races didn't take into account
the relationship between what we refer
to as
the surface area of the problem and
how nation-states actually make
strategic decisions
so i felt by exploring
how that tension could play out we could
open up
some profitable areas for exploration
on what sorts of incentives
states may face over the coming decade
or more
and namely find some arguments for why
the incentives
that major powers face may be
sharply discontinuous at some point
so that's the gap
that we intend to fill with this and
we're
really looking forward to diving into
your questions unless
mathis you have anything else you'd like
to add right off the top
um yeah no so i think what drew me to
the paper the topic is the sort of
intersection of
a questions around uh scientific
uncertainty and scientific processes
around the eye and sort of outside and
inside views of of timelines and
and what's involved in certain
trajectories of agi development and then
connecting those sort of
epistemic or scientific questions to
uh strategic and governance questions on
the other hand and there's a lot there's
a lot of moving parts in there and a
number of uncertainties
but that makes an interesting basis also
for um
understanding what are the sort of
relevant parameters to think about
the intersection of
either yeah so the agi development and
progress in the coming years or
perceptions of such development and pros
and progress and how that will feed back
into um
yeah the problem of ai governance and
the problems of
sultan addressing ai safety
okay great so uh i should also add that
zoom has this raise your hand
or feature or alternatively if you have
a question please
type it type the question in the chat um
and then i'll try to maintain some kind
of queue
of questions but the first question
would be
from this list here which is when in
the situation where agi
might not be a manhattan project away
but something that i here referred to as
ati
light could be a manhattan project away
which is
a an agi that can that is much less
general
only capable of um doing a particular
strategic
uh task and whether that would be
sufficient for for state access to um
to pursue a manhattan project against
my sense of it is the answer to that
will differ
among the six strategic cognitive
domains
that are listed here intelligence
amplification
obviously has huge implications for
super intelligence and spiraling effects
that could have absolutely decisive
strategic outcomes so for that one
absolutely that could justify a
manhattan project
strategizing i wasn't quite sure what
was
meant there could the question or maybe
flesh out a little bit more detail
uh so uh i was one who posed this
question and i took this actually from
nick bostrom's book super intelligence
where
there is some more description of
subtasks in
um under the heading of strategizing
it's uh
you know long term planning and uh
modeling
different actors actors and their
incentives in order to
obtain uh distance strategic goals this
kind of thing
yes so that seems to have
some forking along
different domains there are some areas
where
strategizing in in nearer contexts
would not seem to imply a very general
intelligence and others in which
it absolutely would when it requires
some very abstract world model or
representation of
how many different factors come together
similarly social manipulation
at a high level would likely require a
very general kind of intelligence in
that way
i can see how technology research
in in some areas might not but
the overall strategic impact of
ai that can productively advance let's
say
valuable tech patents would be quite a
strong
incentive hacking is an interesting
subject there because
most real life hacking
has a human component
the most effective hackers are generally
the ones
who don't train some million dollar
super cluster to crack something by
brute force
but rather they offer someone
mail enhancement pills and
play on their human insecurities somehow
and fish them
so there are some components of hacking
that would not imply a very general
intelligence at all that is the the
more brute force mechanistic algorithmic
approaches and then some
which are in fact not much different
from
social manipulation
i i think it's also interesting though
or yeah
very even so there's the question of uh
aj light as on the
technical level or sort of like how
useful or capabilities but also there
could be a political level the question
of well
would it induce a manhattan project an
imagine project of course is sort of a
a it's a sprint not just out of
um if we would like to have it it will
be useful
uh but it's a desperation of like we
need to have this before the other side
gets it or before anyone else gets it
and i think in that sense it's
interesting to look at these
um because i could imagine some
capabilities amongst these that
are undeniably useful but i'm not sure
if if
um interestingly so strategizing for me
is an interesting one because if you
have a system that can only do
strategizing
um i i i at a human level
and it doesn't have intelligence
amplification so it doesn't get better
in a weird way
uh i can see that hitting diminishing
returns pretty quickly uh
it's in the sense that if you uh in the
same sense that a
a military doesn't hire a a
a thousand people to sort of come up
with a with a yeah a strategy at some
point it's just a number of generals in
a room
and so um i could see
if it was just our system helping plan
campaigns and stuff
um having having agi level capabilities
on just strategizing
saves you labor costs but at some point
you don't need more and at some point
perhaps adding
100 more systems doesn't meaningfully
improve
your military campaign beyond just
hiring
yeah 30 generals and so
i guess it's interesting that like
there's some of these where and again
intelligence amplification it's hard to
imagine you
having that without it sort of really
cashing out into the other capabilities
at some point down the line
um but if it was only intelligent
amplification
and it was sort of like oh yeah you just
got um
um yeah very very narrow
improvements on a very narrow other task
that doesn't really generalize beyond it
of course you're not really talking agi
uh in the first place then
um but i could see that not necessarily
triggering a manhattan project rush
technology research and economic
productivity would really depend
on whether a states believe sort of
um so if
i'm china and i believe the u.s is about
to have an agi light
in technology research whether i'm one
to
pursue a manhattan project towards a
agile light technology research
capability depends on what i think
is in reach if i think that
capability will allow the us to crack
fusion power
well i mean that air like that would be
great and it would really be i mean that
would be disadvantage
for us it would be advantage for us but
if i think that this capability
in technological research will allow the
us to within the next five years develop
sort of unbeatable aircraft
or a way to overcome the chinese sort of
nuclear deterrence that would be
the grounds for if i was china to start
a manhattan project
rush yes i would
also push back on the premise of the
question a little bit
in that arms race decisions are made
under profound uncertainty
it's very hard to know in advance
how limited an agi light would be and
personally
it's hard for me to imagine what sort of
evidence
could strongly persuade china that they
almost
certainly can sprint to social
manipulation in five years
but almost certainly cannot sprint to
technology research in five years
at that level of abstraction i think
our prior should be that intelligence
is too fluid to make predictions like
that with any kind of confidence
okay great uh i think david had a uh
a question here
would you like to repeat your question
yeah so
i had a quick question about your
commentary
on the uh social persuasion um
superpower uh in particular uh though
to some extent it generalizes to the
rest of them
you said that at some level that
requires a
high level abstract world model
um which i agree it does
for doing persuasion the way humans do
it but isn't that an
unnecessarily anthropocentric view
on persuasion i could easily imagine
uh gpt's uh n style
text predictor um
becoming extremely persuasive under the
right circumstances
and um uh
if i am correct about that would that
change her answer at all
i think actually so it's interesting i
mean you also have some some thoughts
on that so um it's it's an interesting
thing because i think there's a
degree to which you're right that you
could have persuasive systems that are
involved into even like gpn or even i
mean there's currently studies on the
potential uh to use gpt3 and even gpt2
in the past
to produce ideologically aligned content
or radicalizing content and i think
there's a study that came out a couple
weeks ago
yeah that showed that that you can use
it to create
ideologically tailored content that
conceivably since humans can reliably
distinguish since gpt3 content and human
generated content
would be as persuasive as any propaganda
at least this
i guess there's a there is a sense in
which
there might be a harder degree to if
we're talking about either like
a general persuader so in a very
different paper um
the um uh policy this is
the zidarata so nick bostrom allen
default and carrick flynn i think that
is this
i think it's now called a vector fueled
approach anyway they discussed the
idea of a super persuader which is a
system that can persuade anyone of
anything
and that i think would be something more
than just
targeting people with sort of a preset
ideological content it sort of implies
that you have a system that will be
able to turn me against my family in in
a sense or turn me into
uh and i guess is a
higher level of i think that that second
capability might
require a role model but it's true i
think it might be the case that for many
practical purposes of
yeah ideological conversion near-term
systems scoot work
yeah maybe john yet also some but
uh yes i i sorry can you can you put the
sorry can you put the name of that paper
into the chat yeah i'll
uh put it on my rating list sure
okay i think persuasion is still
rather poorly understood from a
neurological
standpoint we don't yet have
a scientific basis for
saying what the strongest form
of persuasion might look like is it
possible
that a gpn could find the right words
to convince anyone to turn against their
family in
a tweet-length number of characters my
intuition says no
but the science on that is still
i think just so immature that
we have a lot to learn
okay i'll go to the next question here
is and that is the fact that nuclear
weapons are actually
a really really ethical kind of
technology
in that if you expend half the effort of
a manhattan style project
then you get half a bomb and that won't
work
so it's a very discontinuous
technology whereas we expect that
something like tp3 seems to improve
continuously
in the more compute you add to it the
more the better it becomes
and that seems like it's the fact that
there are
half an agi might be uh
have some ability to do social
manipulation
and some ability to do uh uh economic
to be economic productive um and there
are a number of other
advantages here gwen have some where you
can
use once you have this model then you
can use it to train other models
um and how does this change if it turns
out that agi
that it's a more um continuous case
where
you put in a bit of effort and then you
get enough money for it to pay
for itself quickly
yes so we think a manhattan style
project makes sense for something that
is more continuous than nuclear weapons
but not fully continuous there's at
least good
a priori reason to believe that there
will be
some discontinuities on the road to
agi for example once ai
is able to read english with strong
comprehension
it can gobble up an enormous store of
very rich conceptual symbolic
information
that it currently doesn't have access to
that is to say
sort of reading all of wikipedia and
remembering it perfectly
so computers have advantages inherently
in
in speed and memory that will be
unlocked once the
natural language processing and some
other cognitive faculties
reach human level so i see a bottleneck
there
that should get us thinking maybe this
would be a discontinuity
yeah i think this gets us a little bit
into the debates over
to what extent we sort of
would follow the scaling hypothesis of
that we
gptn will basically get us to agi in
which case
yeah that seems that we're maybe we're
in a way already
yeah on on a runway or it's basically
yeah we you could just scale it up and
all of it's gonna be continuous
or whether we expect a sort of more
um uh yeah
it's an assembly of pieces and perhaps
each of the individual modules
will have some sort of limited
uh domain area yeah
faculty so you have systems that can do
yeah
language generation you've got some
systems are able to do limited
causal reasoning then um
but that and those are there might be
incremental benefits but that
in some sense you only get a large leap
in benefits once you put them all
together and to
the extent if you want to stretch the
analogy you could argue that there were
limited benefits to nuclear research
even before you had a nuclear bomb in
the senza it
yeah you could get people uh work in
nuclear power you got
it could feed into sort of nuclear uh
like radiation medicine
um i i agree that those aren't quite as
salient in that sense um
i think it's a question of whether
the scenario here is indeed that like
it's continuous in the sense that oh
each of the individual capabilities is
already really great
and then when you really put get to agi
it's just a cherry on top so it's like
50 better or whether we're talking uh
yeah
just qualitatively more powerful um but
it's yeah it's an interesting
distinction i think also there's a brief
science
question there that nuclear
at least after the sort of manhattan
project era
other states have faced the question of
like well you want to build a nuclear
you might want to have a nuclear
breakout but the
finishing line is no longer just having
a opera operable nuclear weapon but it's
having a deliverable deterrent and so
you need to raise
not yeah to having a single nuclear
weapon but to having uh something that
can be
survivable anyway
thank you for the answer so the next
question is from ali
ali would you uh post your question
please
uh i think i did um
i'll just the last thing i posted that's
basically
what i was yeah that's probably the
question i would ask
sorry that was with the um
the decaying of the of the
infrastructure or
i'll just read it out if that's okay
yeah suppose you tried to make an agi
and fail because you were five years too
early
five years hence would you be in a
substantially better position than
someone who had no experience
at attempting to make an agi of course
you means
the relevant sort of entity like a state
a massive corporation etc etc
i think that this is somewhat
interesting in the case where you have
say
continuous progress or where you have
something like
the time scales to agi are not really
that long
so say if it's in the next 50 years and
it's plausible that you could just make
a mistake 10 years and be 10 years too
early
so in a sense you could argue that um
well without having sort of the shorter
five-year timeline
uh one of the cases that we sort of
refer to is that in the
80s darpa was running a um
yeah a project in pursuit of what
they didn't call it agi but we would now
call it agi and they were
and they failed um and
i'm less clear about sort of
where the the infrastructure the
expertise of that specific project went
in the succeeding decades
but i could imagine a lot of it sort of
was diffused or at least
folded into other projects
in the defense sector or elsewhere
i guess the interesting case then here
could be something like
um darpa trites in the 80s to make a
um agi or like a yeah human lovely
but they failed but what if russia
during the 90s had made a breakthrough
that seemed that they were very close
how easy would it have been for the us
to sort of reassemble the team and be
like oh hold on we were actually
yeah uh pretty close it seems
so it seems that as john clark says that
um
having had the infrastructure
and the the trained up people um
that if you went right into a wall but
someone else
finds a piece of inside that
yeah makes the surface area much larger
you could just re-mobilize
those materials but of course that does
imply that
you haven't sort of like poisoned the
well and and um
people are just unwilling to spend money
on this or to accept this
um as a new project that's actually yeah
i
would need to think about this more in
detail because it's an interesting
scenario
so john do you have you have written
something i guess that's your answer
um it seems plausible to me
um the only thing i would just add
is that if
an agi manhattan project fails
the reason would likely be that
the surface area was too small
and so a lot of the effort will in
hindsight turn out to have been wasted
so that that's something to at least
keep in mind
as a plausible reason for the failure
which may suggest that
the actual progress made toward
fulfillment of agi would be smaller than
you'd expect just from the number of
dollars spent
okay so the next question uh is
uh with some of the historical examples
you you pre
in your paper you have some um examples
uh of projects like either
or uh the international space station um
but there's another kind of reference
class you could do with uh
things like uh if the
if ati could potentially give a decisive
strategic advantage to
either you or your rival then uh you
might look at things like um
total war in the second world war where
germany and the un and
the soviet union were both uh faced with
this particular choice and
indeed chose to devote many many
resources
to this um and that was one
other kind of possible parallel the
other was with uh dreadnoughts
where uh the introduction of a new
technology
uh suddenly spurned and an enormous arms
race what do you think about these two
kinds of
uh alternative parallels that is a
great question uh i don't think
that the dreadnought parallel is very
instructive here
because as revolutionary as dreadnought
was it was still vastly more incremental
than nuclear weapons or the prestige of
landing first on the moon
or what we think is likely with agi
dreadnought could still hit a mine and
go right to the bottom
also very key is that the arms race
in battleships in the early 20th century
was asymmetric between the royal navy
and the imperial german navy what i mean
by that is
strategically all germany had to do
was tie up the royal navy
britain had this vast overseas
empire with shipping lanes in
every ocean that it had to defend
and by contrast germany didn't have
anywhere near that kind of
footprint that it had to defend over the
seas
thus germany could lose a battle
tactically
and win strategically just by
reducing the relative advantage
that britain had so i think dreadnought
would have
been decisive if germany's main goal
had been winning a trafalgaresque
showdown in the north sea but that just
wasn't their goal
so for that reason i don't see
dreadnought as revolutionary as it was
as being that kind of
game changer
um oh sorry oh uh yeah uh
so and the other question the other
parallel where uh the manhattan project
possibly was existential but world war
two for the united states
surely was existential or almost
existential
and they were willing to devote
apparently 100 times as
many resources to winning the second
world war so that seems
to imply a much large your willingness
to
to invest when it seems necessary i
i disagree with that analysis the
manhattan project i think
only cost that little because the us
didn't know
how to allocate more resources to get
the bomb sooner
if they could have gotten the bomb in
1943 for 10 times as much money
it seems quite likely that they would
have done so
also i i would add that the manhattan
project
would have been existential for the u.s
if germany had gotten there first and
that was
the single driving worry behind the
project
in the first place so
if um if the
if the us hadn't taken it as seriously
and germany had gotten the bomb first
especially
if by a margin of several months or a
year or two
there's a very good chance that the u.s
ultimately would have been
heavily atom bombed carved up enslaved
and put under
totalitarian rule by my lights at least
that's pretty existential
so it's a i think it's an interesting
question over like what it what makes
technology existential what makes it
perceived to be existential and so
there's an interesting case
so at the end of world war ii where
there is the debate
um essentially over um
what would be the relations between the
the yeah or
the the western allies and the soviet
union
where there were advocates at the time
including uh controversially
um the uh so russell
slater's pacifist who advocated for a
nuclear strike against soviet union
uh in order to um yeah
um prevents sort of what they what they
fear would be the inevitable nuclear
arming and then a nuclear war with the
soviet union
um and at the time stalin
stalin perceived this to be
um so at least somewhat existential i
think there's sort of quotes where he
instructs his physicists or like we must
have the bomb or it's really
the exact phrasing escapes me um
but there are interesting cases in in in
later eras or nuclear armaments
where um there's interesting cases yeah
and strange cases sometimes of nuclear
restraint
and some sometimes you could argue all
that was the case because of um states
being willing to
like all we can get under the nuclear
umbrella of some of the great power so
why do we need to do it ourselves
and sometimes there's cases argentine
and brazil i think we're running
nuclear programs in the 60s and 70s um
i mean it was venezuela but so basically
they they had fought
a war in the late 1800s there were some
rivalry and some sort of territorial
contestation but at some point um
it's sort of the program's cost exceeded
the the
sense so it's an interesting question
under what circumstances
especially you could argue that there
might be under peace time circumstances
that states might be less prone to
rush whereas under
10 geopolitical circumstances and
especially during wartime
just these types of sprints are more
or more plausible
okay so the uh next question here
is um how large do we actually expect
the uh
the surface area to be um in
in the case of tpt3 which seems like
it's currently our best guess at what
could become
agi at least in the near terms um there
are uh
apparently quite a lot of things that
can be done uh
outside of the core researchers who are
working with it
in the paper that itself they they point
out many many things that could be done
and it seems like they could um easily
uh
scale very very widely in particular
with something like fine tuning
which seems like it would require a lot
of effort but potentially also have a
very large impact
so there is certainly oh go ahead
no no the question was if we are already
at the point where
one in a thousand or something could uh
could
could participate potentially uh in a
atp in
when you say participate you mean in
some kind of a decentralized
collaborative effort no rather
is in a manhattan style project where uh
everybody who has a an education or know
how to program computer they start
working on
fine-tuning the sludge model i oh i i
see yes
so i don't know go ahead
there are certainly ways of improving
gpt-3 probably
by a lot it is deeply unclear
whether they can scale that all the way
to agi
it is possible but i think the most
useful signal there the best
evidence is that as far as we can tell
openai has not been seeking
vastly larger resources for such a
sprint
and because they know more about their
own technology
than we do at least as
our prior that tells me that if they
knew how to spend
the money they would be
yeah i think i think this question might
er i think this question might also
partially reduce to simply
which of the three scenarios that we
discussed also do we believe or run
and do we believe sorry if if you follow
the scaling hypothesis
then it would in a sense be a functional
like
we have the underlying architecture now
it's a question of
of scaling it up sufficiently and in
that case you could argue that um
it's we you can do a manhattan project
the surface area is well basically
i guess from a scientific point of view
irrelevant because you don't need
deep certificate insights you can just
scale up
the the um you can arbitrage
money for compute for performance
um if there is a more sort of ambiguous
one where okay we feel like this is
getting
closer so i feel like it's the question
of
um can one in a thousand people
participate at what and so
which people can be trained or retrained
to
contribute to chip production or at
least elements that feed into that
which people can uh how
what is the threshold for contributing
to the
yeah production of or provision or
labeling of of data
that's the bottleneck um
whereas if we think that the the
underlying
bottleneck is something very sort of
fundamental or theoretical then
you could argue that well the um
[Music]
it might be still a fairly constrained
amount a number of people that could
participate with that but on the object
level here
i confess i've also been impressed
by yeah by searching for the gpt3 i
think we
wrote the initial version of the paper
just after
gpt3 came out and it's been interesting
since then seeing some of the broader
debate
um yeah within the community
um and i'm very curious to see where it
goes in the coming
year and answer how far things like
these can be scaled up
so i should just say here that we will
have jared kaplan
the person who uh predicted that gbt3
would continue to scale
will be joining the reading group and
presenting uh just like this basically
uh on the uh 29th
uh i uh sometimes in november i can't
actually remember precisely when
so for the next question this is a a
picture from
the gt3 paper which basically seems to
imply
a a scaling law between
how much compute you throw at a model a
language model and how
much validation loss you get and gwen in
his
uh analogy uh compares it with a
with the human level with basically 0.7
bits per byte of validation loss it's a
somewhat of stretch analogy in my
opinion
but possibly not totally off and so
that's a graph here that you can
literally put into well from alpha and
then it'll say that you will hit the 0.7
level
at 60 000 times more compute
so if you could just buy that i'm not
quite sure you could actually do that if
someone would
sell that but that would cost 28 of the
gdp of the united states to do
right now and so uh if the manhattan
project
ran over like three four years or
something like that then
we are we basically have a road map
right here or we will have it
very soon with this does this compute
anyone
uh convince anyone um or um
this kind of argument or do you think
this is possible
so you mean that sort of given the the
rate at which computers is
doubling uh we would be at a
the u.s could buy this for a a few
percent of gdp within a few years
is it clearly more and then you'd have
you could have a manhattan
project type expense like like right now
28 percent of the gdp the u.s is sort of
immense but you could have a
manhattan style expenditure within a few
that's interesting
actually 720 billion dollars is
less than four percent of u.s gdp
sorry that's me looking up things wrong
sorry but that's
well it's interesting because it is
equivalent to the u.s
department the uh military expenditure
their budget
of last year uh which is maybe yeah not
in gdp
linked um but it's interesting
to see that type of that's a lot of
money
but yeah jungler
um what was
the remaining element of the question so
the the main
is why do we need a roadmap to roadmap
when we have
a roadmap here
well what i would say there is that this
is
not an engineerable road
map um but
if this hypothesis is correct broadly
speaking
then we're already on the runway
and if if someone wanted to spend
this amount of money today they could
get there in a small number of years
so i hold open that possibility
all i've been trying to express in this
session
is that our priors should weigh against
that
sorry could i ask a rather naive
question
with all this compute what is it you're
computing
i mean what is this machine
what capabilities is it supposed to have
the uh the gt3 uh uh is
currently uh uh basically doing text
prediction
so you're you're putting yes why is that
ai i mean that's
you know big big computing problem but
hardly its repertoire of compared with
even a
small mammal is pretty pretty trivial
that's that's of course where it is
right now that it is uh it's making a
lot of mistakes
but potentially you could just uh ask it
for uh
what how uh could you have a plan for
taking over the world and then
um it will just say the next words and
the next words will be a
feasible plan to obtain a decisive
strategic advantage
what while or the next words would be
what the trading
data or the trading corpus like what so
what the people on the internet
think would be the plan for strategic
advantage i guess
yes but it remains deeply unclear
how strong of insights could be
distilled
from this huge corpus that's an area
of ai research that still seems quite
under theorized so i think that's an
example of one of the things that could
broaden the surface area of the problem
if we got
a better sense as a research community
how well that approach could scale
whether you could read every word
that's ever written and come up with
time travel or whether there are some
hard ceilings
that cannot be overcome
by text prediction i i suspect the
answer is
more toward the ladder that there are
ceilings but
the fact that it's not infinity
still leaves a whole lot of room
for discovery of just how high or how
low the ceiling is
i mean anecdotally there's interesting
cases where they had
systems that were trained i think on
like material science discovery
and then they were able to um
what's the phrase so retroactively
predict
certain types of compounds that were in
labor were actually discovered by humans
but they were the system could predict
them several years in advance
based on sort of the uh so
there is a i guess a domain focused
precedent for if you had a system like
that it could actually come up with
strategies that are not just reproducing
what we've come up with
yeah it comes up with sort of strategies
that are missing from our
landscape but um yeah i i i'm quite
uncertain about
about this
and i'll just flag that this gets at a
broader open area of research
which is around the limits of
intelligence
itself what i mean by that
is in some domains on
some questions it's clear
that scaling up intelligence
will not increase performance for
example
in chess some positions are just lost
positions assuming your opponent plays
even competently it doesn't matter
whether you have
alpha zero alpha zero omega alpha zero
omega plus plus plus
if your elo rating is 5 000 or a million
in a lost position you're not going to
win and the reason is
in chess the decision space
is very tightly restricted by the rules
of the game
if the i ai came up with a plan
to distract the other player and
switch the pieces around when they
weren't looking that might
work in a certain sense but the rules of
the game
say that's not a valid winning strategy
but in other areas like physics
discoveries
it's unclear how much the application of
ever greater intelligence can let you
keep knocking down walls and making
advances and so one of the areas
that i expect to get a lot more
attention over the coming decade
as we get closer to agi is
really trying to get a firmer sense of
what intelligence allows us to do
in an abstract way and what sorts of
limits
that implies
okay nevermind
so there is here uh some weighing up and
down with uh here there's
an extended quote from your paper here
which weighs some some factors
up and down and the uh
and there are advantages to having a
roadmap but there seems to be some
strong
disadvantages potentially in that a
roadmap would help the
runner up if there is a roadmap and
according to the roadmap
the united states will have a decided
strategic advantage in seven years
that seems like something that would
cause china to
uh try to to beat them to it and cause a
uh
an arms race
yes so first i'll just say our citation
there
was intended to credit yudkowski for the
term
fire alarm rather than to imply his
agreement with
the broader point around that so
apologies if that was
misleading um we are in fact
taking a contrary position to yudkowski
at least in the broad strokes
that a road map could constitute a
fire alarm also
we're not saying we should build a road
map
our position is it's dangerous for
someone to have a road map
and therefore we should assess how much
that danger is growing that's the road
map to the road
map and the road map is what would let
the winner win
let alone the runner up
okay so um
uh that's not so many questions from the
audience um but i guess
uh we'll go to uh one further back
here this is a bit more technical with
the questions
so here we have three scenarios that we
are approaching the runway we are on the
runway or there is no runway
and there are
no compelling reason why we have
priorities why
we think there'll be no runway and we a
lot of
actors everybody acts like we're not on
the runway
um so this suggests that either we are
approaching the runway
or we are on the runway at least that's
how i understand this quote
but shouldn't it be more uh why is this
uh and as a chance that we are
either here or here does it i
don't think does that make sense
yes our intent uh
by that quote was to suggest
that um if the second scenario
holds then nobody knows it yet that is
if they only spent the money they could
get there soon
therefore if we're not seeing them
spend the money that
is not suggesting that there's
no runway just that they don't perceive
there to be a runway
so does that help clarify that helps us
clarify yeah
and so the second part of the question
would be that these
three uh factors that the researchers
act like this and the companies act like
this in the states act
like we're not on the runway it could be
an information cascade where
everybody is actually sitting with some
private information that looks like
it should point towards that we are on
the runway but they are updating too
much and all the others who seem to
appear like
we're not on the runway i like
yeah i like that i mean it's interesting
there's a um
interesting argument gordon has also
made and maybe you've discussed
uh also already where he suggests that
um
to open the eye has bought into the
scaling hypothesis
and sort of wants to um pursue that but
his point is also that
um that they need the scaling hypothesis
to be true they so open ai needs us to
be on the runway already
because if not if not if there are sort
of deep underlying breakthroughs then
there
have a much poorer chance of competing
against deepmind's approach or
some of the other major players and in
his
reading at least deepmind uh and other
major laps
are loath to sort of um
yeah sort of pivot away and and
now really pursue the scaling hypotheses
that may not be
so from other services questions i've
had there are actually a fair number of
people within deep mine as well that are
yeah quite enthusiastic and sort of will
point to the success of gpt tree like
systems
as a rational for pursuing um
that more but it's interesting to see um
amongst the yeah the different labs
which of them
um so yeah how do they weigh
the evidence coming out of their own
labs how do they weigh the technical
evidence
coming out of others labs and how do
they weigh
the sort of the perceptions or the um
yeah the claims from other labs and
other teams
that would be an interesting yeah model
i guess there could be a scenario where
there is um where everyone sort of seems
to
believe well no one else is panicking
and moving out into the um
in relation and really scaling up and
therefore
what we have here must not be sufficient
to to get to agi
um then it's an interesting question to
what level of
like if the scaling laws don't break at
what level would the
nerve or the confidence of different
actors that this is not going to go to
agi
at what point does that break
yes i see the information cascade
story as being more likely if actors
believe
that secrecy is hard
that is to say if you
think there's only a very small chance
that someone can
sneak an agi manhattan project by you
in secret you'll be more likely to
trust the public signals they're giving
off
as revealing the
quality of information that they
actually have and
i think that would be an interesting
area
for further research building on this
paper
of what the signals would actually be
for detecting a sprint project
of this nature how plausible could it be
that a major effort like this could be
undertaken
by a nation-state without the broader
research
community knowing and i suspect
that the answer would differ
substantially
between democratic and authoritarian
societies
but i don't know and i don't have a
strong enough understanding of the
chinese ai research community to
to understand how wide that gap
is between how well the u.s could keep
this secret
and how well the communist chinese
government could keep this secret
there's an interesting question i guess
also there which has to do with
relative states sort of uh competitive
positions so there's like the debates
over
um there's a lot of these basically is
there an arms race in u.s and china
and if so who's ahead of it and there's
an interesting c-set report
that came across um a while back that
was dirt looked at uh well
depending on your definition of ai um
you will find that either china is a hat
or the us is a hat and if you're talking
about robotics
um the us is ahead if you're talking
about uh natural language processing the
u.s has ahead if you're talking about
location systems and
china's ahead um and i
i'm curious if there's situations where
um so if you imagine a situation where
uh the us sort of imposes more export
controls
and and blocks down the export of high
performance computing chips
china and if china is not able to
advance their native production base of
high performance computing the us will
have a significant compute advantage
and china will not and it would be
interesting to see how that shapes
their domestic perceptions of what will
be the
like whether we're on a road map and
what are the likely pathways
where you could imagine chinese
researchers not
not sort of selling selling research
projects that are less compute heavy
to their to their sort of leadership
because they're not never going to
compete
on compute anyway um
but that is this is a lot of speculation
like i would be
interesting to um yeah explore frederick
could i pitch in with a slightly
different perspective on this
um because i realized that this
discussion is
is more political than than technical
but um all the way back
when i first read super intelligence
it was very striking to me
that nick had some
suggestions for how we might achieve a
super intelligence
but there was one missing and that was
that
somebody might come up with a theory of
general intelligence which
would seem pretty implausible but from
where i sit
at the intersection of a number of
cognitive sciences there's a
there's been a convergence of thinking
across
ai and cognitive science and
neuroscience
philosophy and various other things
out of which there is actually quite a
general framework
for understanding general cognition
human cognition anyway
and and which is which is mechanizable
and my own interest has been applying
those ideas in
in medicine and i don't know if this is
an agi light
but you know i think i could persuade
you
that we could build a an agi light that
could
do pretty much what the doctors of the
world do
for a large proportion of medicine um
it's not a compute problem it's a
problem of having the right theory
so we're trying to say what you're
saying sorry i would take too long
it'd be boring to go into it but it it's
it's about
that's what it's just a line of work
that i've been pursuing
i think as i say another at another time
i
i can i think i can persuade you that we
could build an agi light that
that could pretty much cover the whole
of medicine and the constraint isn't
computing
these these are not difficult things to
compute um
it's uh it it's knowledge
what what do what do the collection of
all healthcare professionals in the
world know
and how do you capture that and they
understand it of course they don't just
statistically understand it they
understand it in a deeper
deeper sense but but it seems to me that
as far as that agr light is concerned
for medicine
we're already on we're already on a
runway
um and it
wouldn't it wouldn't be billions of
dollars to do it
whether anybody's prepared to fund it
for medicine as opposed to
warfare i don't know
so that probably sounds a bit um i'm
convincing because i haven't
well so i find it interesting so there's
an interesting research project that
i've
sometimes wanted to look into and this
also relates back to the earlier point
we're having it's
like um could we be on a runway and
enter
with actors if if they don't know or
don't realize it and there are history
there is historical precedent with sort
of like
um i think i think alan davifo and his
research agenda
refers to um penicillin as a case where
there was a
sort of underlying breakthrough but it
took a decade for for people and funders
to realize that that was lying around
and so it's an interesting question of
where when and where
there are uh what what's the phrase the
sort of the um
the debate over the 20 dollar bill on on
in central and sort of like well other
scientific or theoretical
um yeah uh
yeah low-hanging fruit or dollar bills
lying around actually that aren't
yeah recognized here in certain states
that would be something for the model
but i think i think what you're
describing john reinforces
what we were talking about a little
earlier that
agi is still under theorized that
there's still a lot of room
for breakthroughs quite apart
from on the computing side
that could then unlock
what seems to be an overhang on the
hardware side
and and that's some of the evidence that
leads us to expect
discontinuous progress
um yes i think i think what i've
perceived and i've i'm i'm pretty
old-school
ai um is that over the last 10 15 years
uh we've kind of decided
probably in the 90s that good
old-fashioned ai
had failed it couldn't deliver um and
and the last 10 years or so it's all
been
brute force various kinds whether it's
reinforcement learning or the countless
different types of
big data analysis and so on and so forth
um
and i think the assumption that the
old-fashioned ai has failed is wrong
um and and i was really
struggling to understand why
anybody would imagine that just putting
more and more
um compute power
expensively into a very large machine
would deliver what
i use the phrase any kind of interesting
repertoire of
capabilities other than predicting text
which is um don't see where that goes
but i do see where building a machine
that could basically advise on
any topic in in medicine um
i i do know how to build that um
it doesn't it it requires resources but
it's not compute resources
it requires coordination i mean i i
a few years ago i i i had
a similar idea of a manhattan project i
didn't want to call it manhattan project
i called it the greenwood project but
you know you could bring a bunch of
people together from a lot of different
disciplines
um and that you could build this thing
um
and if you can do it for medicine you
could probably do it for a lot of other
other fields as well as i said that
that's not the question you're asking
you're in
you're interested in a political
question um
rather than a technical one but but i do
struggle with the way
the the political debate political
discussion
what it's grounded in um it's not kind
of grounded in anything that i can
understand as a as an agi
with a broad range of capabilities
i'm sorry i'm sorry if i'm um
being i don't want to be critical and
just struggling
no not at all
just to clarify a bit on text prediction
yeah i think what people mean by that in
the context
of gpt 345
n is that if humans
can output only through a linguistic
channel
like in a touring test then a text
prediction
task is functionally equivalent
to human intelligence
but what remains under theorized
is whether the kind of text prediction
architecture that gpt uses
could actually achieve the performance
that human cognition does but in terms
of the underlying task of
getting a prompt and then answering that
prompt that's the fundamental similarity
between what our brains do and what
a gpt style transformer does
i still struggle it sounds just so
similar to
you know if you had enough monkeys and
typewriters you'd end up with
shakespeare um
you wouldn't and if you did it wouldn't
be interesting because the monkeys
wouldn't understand anyway
in a sense you you may be proven right
on that
john in that it might turn out
that the curve does bend that scaling
gpt's up takes us
further than where we are now but still
lands well short
of agi and in that case then
you're just adding monkeys and not
making progress as you scale up with
more compute beyond that point
we just don't know today in
october 2020 how
how that question will turn out
okay um so i'll
move towards the next question i hope
that answered it john
and that's um we had an interesting
discussion
i'm still struggling okay so um
this is another quote from your paper a
broadly
set of metrics for assessing how close
we are to the runway would reduce the
risk of
compromising ai safety and that's
of course one thing it could do in that
if people
believe that we are not in an arms race
situation
it um then there would be no
not so much of a rush but it could also
do precisely the opposite trigger and
arms race
um and if people can see that building
this
roadmap will possibly turn into an arms
race
then there seems to be two kinds of
interest there are the ai safety
concerns
um which motivates uh don't publish a
road map if it will uh if it will turn
into a
an arms race and then there are these
strategic uh concerns which say
do publish and roadmap if it will give
us a strategic advantage
um and it seems like um
statesmen care much more about strategic
concerns
than ai safety in practice so they will
choose that
i think the difference is that the
ai safety community
has a lot of inherent disadvantages
compared to nation states in that we are
decentralized we are not
particularly well-funded
and we don't have the same strategic
imperatives
impelling risk-taking that nation-states
do therefore they have
very strong incentives that will
press them into this arms race
no matter what the ai safety community
does
on the other hand taking
this road mapping seriously can i think
very meaningfully
increase the ai safety community's
ability
to counteract those dangers or mitigate
them so
basically my answer there comes down to
the difference in base rates
that the
the danger is already so manifest
with nation states already dominic
cummings
the chief adviser to the uk prime
minister
has blogged about agi
as a potential arms race super weapon
target
so in the context of that level of
awareness
already being present among the most
influential
strategic advisors in
cabinet-level decision-making that
suggests to me
that the horse is already out of the
barn when it comes to not
spooking nation-states into an arms race
i see it as quite likely
that an arms race will
take hold at some point in the next
decade or so
really the thing that we can control as
the ai safety community
is whether we're ready for that arms
race whether we have
plans for how to mitigate it how to
steer state actors
in safer directions i don't think we can
prevent the arms race
but if we at least have this discussion
early
we can make the arms race relatively
less risky i don't think that's a
terribly encouraging thing to say
but it's honest
well thank you for that answer it's
getting a bit late here
so i'm thinking uh does if anyone have
any uh final questions or comments
well while people think about if they
have any final questions then i would
like to
say thank you to you uh john clark levin
and
to matthias mars thank you for coming
here it's been a pleasure hearing you
and uh i've learned a lot and i think a
lot of us have learned a lot
so um thank you very much for coming
here
we appreciate it very much and uh good
luck with uh
future papers and whatever you decide to
do thank you thank you for the
questions also very yeah very very
provocative and interesting yes there
was a question to me from stephen hoover
i think
i think it was uh more of a comment uh
well okay
i'd like to i'd like to pick it up with
you outside this meeting thanks a lot
um i do have a question for the authors
if you guys have a moment uh yes
so i i think i was confused when you
guys were talking about
uh the intelligence uh problem like
you you had phrased it about something
like
theories um we don't have good theories
of like the boundaries of intelligence
for instance
i think you gave examples of problems
that like might not be solved you could
think of it
like uh two doors with equal probability
of events one of them is bad one of them
is
um good but you can't
redo it so you just have to make a
decision and if you don't have any other
priors or anything else to update with
is that a good characterization of what
you guys are saying and then if so i
have a follow-up
but if not maybe i misunderstood
so i think so i think um
[Music]
the the idea that there would be
situations where your
the ground for decision to serve limited
or information available to make
decisions is limited
to an extent that just adding more
intelligence in certain directions
will help you make better solutions
so like in the case that you described
you would you would have a yeah 50
chance and and um
so so i think that i think that that
would be our the case that there's
limits
to the performance gains that
intelligence can buy you in those cases
but what would be your
i'd be curious to hear your follow-up
yeah i
i think that that's like i certainly
don't disagree with that i was curious
why you found that to be or why it was
mentioned because i didn't think that
that was as
interesting of a problem like it seemed
like
you could formalize a problem that like
does not have a solution
and that's fine but it like doesn't have
a solution so i
was wondering why that would be
interesting uh from your perspective
well it it gets to parameterizing
how transformative agi or super
intelligence would be
for example in the social manipulation
domain
that would be much more dangerous if it
were possible
to come up with a tweet length character
string
that could get anyone to commit suicide
that would be vastly more dangerous than
one that
merely is quite persuasive
it's like it yeah it does imp does
skill a persuasion sort of keep scaling
beyond
human levels or is it like does it reach
an asymptote
beyond which it's uh
yeah being vastly more intelligent at
some point doesn't buy you a lot
more ability to persuade people to kill
their families or
yeah it sounds a bit that's right yeah
it's a it's a dark analogy i'm sorry
but no no i mean that that makes sense
in a sense
in a way where you'd ask like what would
you be concerned about that that's
intelligible i think that answers my
question
so to to characterize your point it it
narrows your scope on what to be
concerned about
in a sense yeah i think i think that
doesn't of course there might
there might still leave a fairly large
range of
yeah problems to be concerned about so
things that so i think
to come back to an earlier point even if
you can't do
perfect persuasion of anyone of anything
having a capability that can do
fairly reliable radicalization over time
can still have big societal impacts and
that of course
gets more powerful the more you can
change people's minds
further along with a shorter message so
and at the limit you get john clark to
the yeah
a complete sort of character reprogram
in a tweet-length message
um but of course even
long before you get there there are
things like okay well if you have a
system that can
spew out um tax that is basically
eighty percent likely to to convince the
median voter
to vote for the party that you would
like them to vote for then that is
already
plenty politically disruptive if not
and that would actually that could be
enough
to spark a manhattan project because if
the us worries that china has a
like i mean seeing the debates already
right now over sort of
russian election interference and
information campaigns and this
information
and so if one state believer was about
to get a ai capability
that gives even just his or her 60
chance at yeah making people vote for a
selective party
that seems sufficiently yeah powerful
capability
um that that could yeah that is quite
um yeah politically uh
if not existential threats it's quite
dangerous
um and i see a question over
um so i think
we we mentioned darpa's attempt because
that was i think a reference we also
made in the paper
um specifically to do with the um
strategic computing initiative i am
less familiar with the fifth generation
um
japan's fifth generation project i got
the impression that it was
so yeah based on expert systems i get
the i do think that
they at the time so it's interesting i
think at the time it was presented as oh
at the end we'll have a conversational
robot
or a conversational service system i
think people were not were thinking
maybe relatively small in the sense of
oh and won't that and won't that be a
great secretary
um to sort of stereotype a bit rather
than
in darpa and sort of military context
more like this will be the the general
that will like bring our armies to
victory and so i think
to stereotype a little bit um and not
being very familiar with the japan's
program i think it's a difference
between someone proposing
human level ai because that will will
get will
that will do great for entertainment or
for
um yeah domestic services
versus a military research institute
proposing a strategic advantage
but it's an interesting comparison
okay great are there any other final
questions
well um then uh oh let's see just this
moment
that was a new move no certain to me
okay great and so the uh um
the final thing um just before i say
goodbye to everyone would be that
next week we'll be discussing uh shane
legg's
work on intelligence uh stephen will be
giving a presentation there
and there are uh a um
a really real subset of the people uh i
will uh
send a message on skype to everyone
about what we read next week
so that is all for tonight again thank
you very much matthias
and john clark it's been a pleasure
having you thank you for hosting and
thanks to everyone in the group for
these great questions and a very
stimulating discussion
great bye bye bye


AI and Value Alignment | Jaan Tallinn

[Music]
has been my work over the last 10 years
trying to bring these communities
together to have this open discussions
about topics that have been taboos and
yeah thanks for that I grew to have been
able to create such a positive
atmosphere I see like a lot of smiles
around me in as I Shane said that this
is probably the most important topic to
have how these debates about and I think
it's kind of like my relationship with
Jurgis mid-to-upper there you are
I think it's like exemplified like a
good example of that spirit like me like
whenever we meet like we have like a
huge debates and like I'm kind of I
caught myself like yelling at you every
month no sorry about that
yet like every time we kind of leave we
are like I think we are gonna getting
better and better friends as a result
thank you
so yeah without further ado my talk my
favorite piece of parenting parenting
advice is to that when you want to
explain some picture kids treat them as
adults and don't dump down your answer
because its first of all it's like bit
of mischievous fun to watch them to try
to decode what you just said but more
importantly kids abilities increase over
time so if you dilute your explanations
you're likely to keep just constantly
undershooting interaxial intellectual
capabilities now my original plan for
this talk was to look back what has
happened in AI safety since the last fli
conference now key positive feedback for
achievements such as all the technical
value alignment work that has been done
take stock of the current challenges and
generally kind of congratulate the
community for coming together and
during majoring so quickly I even had
this prop here like I created a trough
where I where I would use this shirt
because I bought it from Puerto Rico
after my luggage was lost in transit but
then it occurred to me wait a minute
wouldn't that parenting advice apply
here as well wouldn't I just completely
undershooting the intellectual level and
the progress of this community so I just
read my draft and solid hour a new topic
we need a way to figure out what
humanity wants when I talk about value
alignment on in public one question I
almost always get is whose values are we
talking about and it is in response time
usually mumble something about the vast
majority of human values being so
obvious that we don't even think about
them as Stuart Russell puts it everyone
right lift the right leg or my own
favorite everyone likes our planet to be
roughly at room temperature that is not
to say that we don't have a problem with
aggregating our values we do a massive
one we don't know what a complete set of
humanity's values are not nor can we
simply ask people either since Staniel
Kahneman has pointed out there's a
difference between what people say what
they value and what they actually value
also our values are clearly a moving
target because they keep evolving over
time what's more we don't have a great
track record in coordinating on what we
already know is valuable the League of
Nations failed to prevent World War two
and although it although the UN has some
successes on under its belt such as
eradicating the smallpox and fixing
those on layer it's widely considered
well somewhat ineffective at this job
also free markets and democracy although
seemingly better than the alternatives
seem limited in their ability to steer
the world towards a bright future
especially if you think about the recent
events indeed I think that the biggest
disservice that capitalism has done to
the world is that it has created a false
sense of security in technological
progress not to mention the Tony
philosophical paradoxes with value
aggregation and even potential
dependencies on some unknown laws of
physics when we get down to the sort of
nitty-gritty of how the aggregation
value aggregation should work yet there
is hope first of all we now know much
more about morality human values and
game theory than we did when say the UN
was established second various new
technologies seem to favor global
coordination for example the internet
and Mobile's have connected the planet
cheap satellites and other sensors will
create an explosion in transparency for
better or for worse and the invention of
crypto economics has introduced a new
regime it's now possible to have
worldwide consensus about a piece of
data without trusting any central
authority to maintain it now I have many
examples but just to give one example
about what can be done with crypto
economics it's now possible to create
global decentralized courts and I got
this idea from Vitalik who is also here
in audience on block chains that resolve
conflicts by enlisting random people as
jury and then came terrifically
incentivizing them to produce opinions
that society in general would find fair
not to mention that the continued
advances in AI and techniques such as
inverse reinforcement learning and
approval director agents by paul
cristiano seem extremely relevant here
so therefore I'm proposing that we start
designing explicit mechanisms to
transparently and robustly
aggregate global opinion about what a
good future should look like the
mechanisms have to be open and
transparent blockchain style to instill
trust that their purpose is to serve
everyone in fairman
they have to be robust in the sense of
being hard to gain corrupt or otherwise
defecting and this probably requires
incorporating philosophical meta
principles such as veil of ignorance
that is you could only benefit from the
system as a random member of humanity
not as a particular person in a
particular position so basically I'm
advocating extending the technical
approach that has been very successful
in planting the frontier in AI safety
thinking to the problem of global
preference discovery about they are
safety thinking I think someone in
Cicero ffh I gave this wonderful
metaphor that the process that has been
unfolding in the last few years is like
sort of high-altitude bombardment by
very gonna prominent people the
canonical example of course Elon Musk
and Stephen Hawking followed by ground
troops of safety researchers so like
some people who are gonna still shooting
at airplanes might be doing so from
position that all it has been claimed by
safety research active sea safety
research of course all this new
technology can also make things harder
one man's coordination is another man's
collusion we have to be careful not to
catalyze criminal activity or worse
paint humanity into a corner but
introducing terrible Nash equilibria on
a global scale not to mention the clear
and present danger of various AI arms
races computer Nick just talked about
both literal and figurative we
absolutely need to avoid this and I know
this think topic has been on day misses
mind for a long time and finally the
looming AGI limits our time budget we
have lost over half a century since the
original warnings about AI value
alignment by Alan Turing and Norbert
Wiener a certain hope that we have still
another fifty but I know that several
experts in this room are much more
optimistic about their AI timelines and
thus pessimistic about our remaining
time budget last month there was a
workshop but FHI you know
we're one of the sessions was about what
to do if the value alignment won't be
solved in time and it had this eerie
atmosphere of a science fiction story
featuring an alien fleet in orbit aliens
who couldn't care less about humanity
and in a room full of decision theory
experts trying to find a philosophical
loophole that would allow humanity to
keep just at least one galaxy 100
billion one galaxy as a consolation
prize for the losers this 50% star
systems work for every human alive today
with that I want to illustrate to
talking two things first even if you
mostly screw up things might still turn
out to be pretty okay in the end and
second worst thing we could do is to
continue playing our usual political
zero-sum games when losing 50 galaxies
per second
luckily a transparent reference
discovery mechanism might serve as a
ladder for Humanity to climb out of the
arms races and other bad Nash
equilibriums it might also help with a
problem that many of you personally feel
society doesn't necessarily trust you
with the power you have over the future
grant that they might trust him more
than the politicians come on that's a
pretty low standard of course there's a
valid reason for that mistrust history
is littered with catastrophic tragedies
caused by individuals or movements that
amassed too much power I should know
that having personally experienced the
tail end of one such tragedy now imagine
if there was a way to credibly
demonstrate that I'm working towards a
future not just what you personally
thought was a good idea but towards the
future indicated by the global
preference discovery mechanism it's sort
of what open area people have been
talking about but on steroids finally
having a strong selling point for
humanity's values should be a great tool
for philanthropists effective altruists
and politicians who
we want to improve the human condition
and of course ultimately we want the
mechanism to converge into something
that can safely guide and superhuman AI
two years ago standing in front of this
conference I compared AI development to
launching a rocket initially you mostly
worry about having enough acceleration
but eventually steering should become
your primary concern the Sumer
summarized my current talk I would kind
of extend extend this metaphor to say
that now that day I research as a
process producing even more powerful
engines and the steering systems
designed by AI safety researchers is
also progressing it's about time to
start plotting or eventual trajectory
crucially the trajectory planning must
be globally transparent and fair because
everyone everyone will be on board thank
you
[Applause]
[Music]