244. Democratising Risk 1

hello and welcome to session two hundred
and forty four in the ascii.com reading
group
tonight we'll be discussing the first
part of the article democratizing risk
in search of a methodology to study
existential risk by carla so creamer and
luke kim
carla so creamer works at the future of
humanity institute
and you camp at the
center for study of existential risk
this article is a couple of months old
and we'll only be talking about the
first part
of this article today
hopefully we'll be able to cover the
entire rest next time otherwise perhaps
we'll have to split into a couple of
more sections
first i would like to say thank you to
the authors for critically examining the
arguments for what we do i honestly
believe that it's really important to
get critical views on
both ai safety and existential risk work
in general because this is a
an area where
it's just very difficult
for a person to be able to explore both
sides of the story and that's why i
believe it's very valuable that other
people are taking the time even uh
whether they uh uh
do it as like red teaming or
because they sincerely believe that
there are problems i believe it's really
really valuable
and of course since i'm summarizing
something that
i
don't necessarily agree with that means
that
the summary when i summarize things i
omit a lot of qualifiers and things and
that might make it so much strong manage
and i apologize for that
also
at the time when i was
writing this
there was a story that russia had placed
the cheating nuclear forces on high
alert and mitchell's gave one in 20
probability of people dying in nuclear
explosions
so um
that obviously biased me a bit more
towards more immediate short-term
considerations with regards to
existential risks
and finally um i
will if the authors or anyone else see
this they will notice that i'm extremely
critical i am very critical with all the
texts last week we have some rather good
work from anthropic that i also engage
very critically with so it's not just
this
so the first argument being presented is
regards to the standards of uh research
in existential risk studies and it is
arguable that it should be held to a
high standard and
i am
this can be interpreted in multiple ways
there is
the
symbol saying that we want the research
to be of high quality i certainly
believe it should be and i think just
about everybody would want that but
standards doesn't actually mean that one
of the ways
we
analyze standards is regards to how much
do we put in
and from the public side the funding for
extension research from research into
extension risk is extremely low
and that should affect our standards
like if we imagine that you cut the
budgets for the hospitals with 50
then the standard of care you would
expect would be much much lower
so
in this way it's it's not entirely clear
whether they are they're just talking
about the uh the quality or they are
talking about the standards
and the arguments given for requiring
high standards are threefold first it's
an ambitious field and it could affect
the lives of many and finally
scholars
who changed the trajectory of global
society
and this to me is
affecting the lives of many that's the
argument for high quality basically
and the other two perhaps could be
construed as
related to the standards but it's um
somewhat more confusing to me at least
perhaps some of the same uh confusion
happens when we talk about the
challenges that essential risk studies
faces
we start out with how can it be
inclusive of all our preferences and
visions of the future and of course
that's something that in alignment
research in particular has been studied
and discussed very very extensively like
uh on a bit more on the object level
like how can we make a super
intelligence that's aligned with our
query extrapolated or whatever um
and another challenge is avoid baking in
subjective assumptions
uh into our analysis and that's
something that i think researchers in
general can't avoid making this kind of
uh analysis completely objective is
probably too high bar um and
probably not really a a desirable way in
order to get the best possible estimates
then how uh there's the argument that it
will affect people who don't share the
values of the researchers well essential
risks are of course unique in that they
affect everyone in a very directly
direct way
um
so again we are seeing somewhat of a a
difference between the subject and the
the object in the meter level three
other considerations also make this
clear how will researchers come
conduct complex risk assessments deal
with uncertainty and different levels of
possibility and evidence and this kind
of thing and this is
i'm when i read these texts i become in
doubt because um
on the object level people who are
working with trying to uh
forecast ai timelines or something like
that they obviously need to deal with
uncertainty and that's like the core
thing the object thing that they are
actually working on
um
it is also possible just to state that
existential risk studies as a field
needs to have special ways special meter
practices around this um
that
they for some reason needs in addition
to these object level considerations um
but
if that is what the authors are trying
to say then we need some kind of reason
like why is this in particular a
requirement for existential risk studies
and for like cancer research and
whatever
another perhaps more interesting issue
is we want to ensure that um these the
studies
is misused for dangerous actions uh in
ai risk the classic dangerous action is
the two proper sparse of a model of
capability and alignment research
and
where it's well known that there is a
substantial overlap between the two
in general ensuring known issues is
super super hard
one challenge that the author don't
mentioned is of course the central
challenge and that is to reduce of
existential catastrophe and i think that
it would probably
have been wise of the authors
for reasons i'll show later to be much
more explicit at this point
it is indeed possible that there is some
kind of trade-off between what level uh
the risk is uh democratized
and and toward a level
it is uh
the risk is just lower
um
perhaps we'll get to that later but it
was not in the first three chapters
we'll instead talk about the techno
utopian approach
the technotropian approach is based on
three
pillars of belief transhumanism total
utilitarianism and strong long-term ism
i think i need to clarify one thing uh
again it's possible that i've totally
misunderstood this but when they when
the authors use the word approach i
think the word that they are actually
meaning to use is argument like this is
an argument why extinction is bad
and so the techno-utopian argument
against
existential risk is that
as some kind of technological maturity
we could have a huge amount of
utilitarian value and if we don't get
that then that's almost as bad
opposition is an existential catastrophe
and so we have a really strong moral
application to ensure that we get to the
second utopian future even through
exceptional actions
and in in this case uh
uh as
if this is an argument against ex
existential risk then it
it is just one
of many i think there are many possible
ways to argue against
extinction like you could argue that
people's performances are being violated
and things like that so this is one
argument among many
but it is especially in the sense that
essential risk uh studies originated in
people who were talking about the
technology natural approach and that's
actually somewhat odd i uh
i noticed i'm a bit confused like why um
why don't other moral philosophies uh
call out and explicitly say hey we
should really ensure that the long-term
future ends up in a nice place
because as the authors point out there
this is something that kind of came
about in this in a
strange way where people were exploring
the taking utopian approach and um
from that uh
realizing that
it was very important to look into
existential risks and then of course
figuring out that there were substantial
problems
um but
it
i think at this point
to me at least um
the origin of the field
is
of
uh questionable uh importance i'm not
saying it is of no importance it's just
that the burden of cruise flies squarely
on the on the authors to show that there
is a problem because um
in general shooting the messenger
afterwards isn't really relevant like
one analogy i just came up with is that
there is some astrologer who might be
crazy maybe not might have other moral
failings but he told you to look up and
you look up and there's a meteor heading
for you and so if that happens um
then the goal should be to deflect the
meteo and whether that astrologer is
actually crazy or in the art group or
has
evil or whatever that doesn't matter
very much the focus is on on the
asteroid coming for us right
so for me personally i don't care very
much about this argument against
existential risk um but i think uh in
the interest of this uh of um getting
into this i expect that i will be
defending this techno utopian approach
somewhat more than what i actually truly
believe
also with regards to
the origin
later in the article
toby or is actually quoted as saying
that other
other value systems ethical material
theories can uh
be used to argue that extinction is bad
and
i agree that but it's uh
it's still strange that the utilitarian
toby or is saying this and not like are
you starting like if virtue ethics
implies that it's important to avoid uh
extinction you would expect someone else
to have figured that out over the past
two thousand years
so what are the arguments for focusing
on the chicken youtuber approach
um
the authors argue that it is by far the
most influential approach or argument
within this field
unfortunately they don't really present
any arguments for this
and
so from my personal intuition which
might as well be good um
i think if you ask especially as you get
closer to the op-ed level
especially as you get close to the
object level you will find people with
who don't really care that much about
ethics like
why is nuclear weapons bad if you ask
someone involved in
disarmament why nuclear apocalypse is
bad they might not have a good answer
for
you uh excuse me
um
some people will probably try and make
an argument based on like altruistic
considerations and some people might
also
have your plane uh self-preservation
which i think is a perfectly fine
argument for
trying to avoid extinction
the problem is this is an uh
the the moral values in the chicken
utopian approach might be embedded in
the analysis
and that could be problematic to me
certainly if it means the um
the uh analysis suffers from it and the
authors argue that they will later not
today unfortunately but in the later
part they will show examples where um
the technology and approach leads to
conclusions which in fact increases
catastrophic risk and that would of
course be really bad and i
i really like if this is called out
um
let's go back to the
definition of existential risk here i
found boston's original
definition of an existential risk
one that threatens the premature
extinction of earth originating
insulting life or the permanent and
drastic destruction of its potential for
desirable future development the
building is in the origin
and um
this is described by the authors as um a
techno-utopian definition
so of the three pillars
um
total utilitarianism as far as i can
tell there is no social utilitarianism
here there is perhaps a bit of
transhumanism in that there's an earth
originating intelligent life
there is perhaps a bit of long-termism
if you really
about you know desirable future
development but um
but calling this uh the uh this uh
technical utopian is to me uh
really really questionable it's uh
um yeah so um
one of the things the authors argue is
that there are no other definitions than
this at least not uh
in vice produce and i think that's true
but i also think that a lot of the
practitioners of the field just in
practice um
take the first half of the definition
like the bolded part and then they that
replace earth originating intelligent
life with humanity so existential risk
is once that threatens the premature
extinction of humanity
at least i think that's how people do i
don't actually know
the authors make some rather strong
claims about how this is embedded
including that it characterizes almost
every existential risk text with
significant public profile i think that
is
depending on precisely what it means
that the argument characterizes um
that seems rather wrong one of the
examples that i think people would
expect
to do this would be the book super
intelligence by nick bustrom which does
include this argument against
existential risk but hidden away in
chapter six and um
and put inside one of these boxes of
custom views to say that like this is
optional reading is not really necessary
for the argument to carry
um
there's also an argument it's also
claimed that there exists people in
extension risk study who subscribes to
this technology approach and i think it
is certainly clear without any doubt
that there are people who accept this
argument but
the authors don't actually claim that
they're that it's the primary argument
and um
i mean
it that should be possible to find out
right you could literally ask bostrom to
reveal existential is most bad because
the
people die or because then we can't have
this glorious future
so moving on a bit more with the
definition of if we take the last part
of the definition which i think is the
one that um
[Music]
carla and luke are really uh
attacking like the permanent and this
drastic destruction of humans is
potential for desirable future
development
they claim this as an example of
something that would be covered by the
existential risk definition humans
persist sustainably and
equitably for the next billion years
without major technological progress
now on the face of it uh
that sounds desirable so by definition
if it's uh something that means we can't
have a decibel future then by the
definition it's not really covered
the word desirable does indeed imply
values and the classic thing
like disciple can mean different things
but um
the thing that it could mean is that the
people who are there are happy to be
there
is it possible that there is something
in this definition that isn't really
covered like is it possible for instance
you could have like ecologically aware
living sustainable and equitable like
communism but without the desirable part
um i think it is possible it is possible
to have
uh in theory to think that you could
have a society like
if you are a utilitarian i am a
utilitarian and then whether it's
desirable as a society depending on like
other people that are heavy seems to me
almost like an action but i recognize
that for most people
most people are not utilitarians right
so something like that might indeed be
what people want
um to give an extreme example someone
like pop pot would be one who would be
happy with an ecologically aware
undesirable uh communist
society and i think that that's of
course a very extreme example uh but uh
i i think there are a number of people
like most people are hugely serious
right so there must be a lot of people
for whom the ethics don't aren't really
based on people being heavy
um i think in practice although that the
reasons why
uh
the reasons for the destruction of of
the potential of humanity is almost
certainly going to be something that is
really bad
i think what boston is thinking about is
like something that kills almost
everyone destroys the planet or a
particular stable or authoritarianism is
like the the thing that he's worried
about
and
of these things like total destruction
of the totally
and stable authoritarianism these kinds
seem really really bad for me
so
i'm happy to include this as an
existential risk but i recognize that we
are making a valid judgment here
the next complaint is on transhumanism
um that there is value in exploring
posthuman and transgender modes of b
uh boston has uh uh
there's a quote from boston here that if
we can't do this that may itself
constitute an existential case
sensitivity so question is using the
word may and the authors are
concluding from this that preventing
existential risk is not primarily about
preventing the suffering and termination
of extension who
uh but it's focused on uh trying to get
to this techno utopia i think it's just
plainly a
much too sharp um
conclusion you can't really from
boston's carefully may conclude that
this is what it's primarily about right
there are many reasons why extinction is
bad
strong long-term yes
the thoughts that um
the um
the values the the
the value of the long term uh
can and often took an extreme degree
will overshadow the values right here
and now and uh the uh
is accurate but that it doesn't give any
clear guidance on when we should process
the living humans of today
and i think it does in fact
it does give we should prioritize uh
humans today to the extent that this
help us in the long term which in
practice almost always will right and
and
in particular uh you there are it's very
easy to find principle guidance from
consequentialist utilitarians uh it's a
lot harder to be a perfect consequential
utilitarianist
but they they certainly know what what
to do right you just calculate
um so
yeah here's the definition of
long-termism that might be an ethical
imperative to select the choices that
are that will have the best effect on
the long-term future
and that's often
reducing existential risk but there are
some underlying assumptions
three are mentioned here then we will
have continued technological development
that will settle the stars eventually
and the future people will be heavy
and i think uh the the full taking
utopian approach assumes all three but
just a strong long term isn't just
this doesn't really matter that much the
key thing is will people in the future
be heavy
and that's uh something that needs to be
um investigated that has been
investigated and listened and should be
investigated more because it is in fact
rather important
the extent to which this is settled is
uh called into question by the authors
sign three
words one what's wrong with human
extinction on the survival of humanity
now i haven't read those um but to the
extent that they call into question
whether or not extinguishing is bad i
want to claim that it's an extremely
niche
and that it is just completely accepted
among just about everyone that
extinction is bad period
representation um the technology
argument is not uh representative like
at all of what humans right now believe
that doesn't necessarily say whether
it's right or wrong um
actually according to paganism we would
expect it to have at least some
uh to be some evidence
the authors claim uh recently that it's
risky to rely exclusively on one
unrepresentative approach
and um
it would be but it's not true that
exercise studies relies exclusively on
this argument like the
there are some people who believe it but
uh
relies exclusively is
way way way too strong
he also suggests that we should have
empirical studies on what are human
intuitions on uh on extinction
and
i mean it's possible that uh there
indeed exists a large amount of people
who would be happy about existential
risk but we need before we call for uh
like a lot of opinion polls or something
like that we need some to have some kind
of um
uh
[Music]
intuition uh like my uh right now my
intuition is that almost everybody would
prefer that there not be nuclear war um
and uh
we need
some kind of argument some kind of pulse
must have been made somewhere for the
authors to say that this is something we
need to be investigating more detail
they don't try to argue that it's um
that the utopian approach is not
representative
here is the world transhumanist
association in 2007
was 90 male and had a median age of 30
to 33 years old
i uh i i actually think it's way less
representative than this um and i'm
surprised that they also couldn't find
better arguments like the meeting age is
33 that could describe like literally
everything right um
so um but i do actually believe that it
is not representative um and depending
on how you ask you could get at least
total utilitarianism is something that
you can ask in your opinion impulse and
people will often say it's a good idea
but
depending on precisely how you ask it
could also be a bad idea
uh i think the reason why people don't
give consistent answers in uh opinion
polls is just that they haven't thought
a lot about it um but but i want to
emphasize here that i accept and i
believe that the
second utopian argument is very french
in humanity at large
another perhaps most serious charge
against extinction risk studies is that
of elitism
which is it is claimed to be an
illegitimate
field and the definition of ill-test is
that the researchers and champions are
granted decision-making powers in
society
uh
and um
i will claim that it is that the people
like bostrom annie and zukowski are
indeed not granted decision-making power
in our current society but that leads
into what
scott also has called a bravery debate
where everybody is saying that they are
like
very brave people standing up to the
other people who are in fact the
decision makers and uh that doesn't
really uh
i mean
and i would like to avoid getting into
that um
it's true depending on like if you're
literally pro uh extinction then yeah
you might be brave to stand up to the uh
extinction people and feel like the
anti-extinction people are
controlling the debate uh at least um
but but to go from there to uh to say
that these people are decision makers is
clearly true
and i think the others also
don't make the strongest version of that
argument in that they are more like
scholars but still claim that they are
rapidly and intentionally growing in
influence
and to argue for this they
have a few examples of some politicians
who pay lip service and say oh yeah
that's extensive risk and then nothing
more i think
to show that they have any kind of
influence in society you need to do way
way more clustered investors and then um
an argument here that extension risk
studies is four steps removed from
neoliberalism in that existential risk
uh studies is
according to the authors very influenced
by the techno-utopian approach which is
again
related to something they call the
californian ideology
which is related to
neoliberalism
about um
ideology
so
like to me this is um it can't be that
pervasive a cultural
thing mean if if i've never heard about
it
i think it is in in practice in order to
go
that way you need to go like one more
step to say your the techno-utopian
approach is related to silicon valley is
related to capitalism is related to
neoliberalism but this is a really
really bad argument right with like five
steps of separation you can literally uh
i believe you can connect anything to
anything and
the authors don't even um
argue why new liberalism is bad they
just use it as an applause light
and
i think this is really a really poor
argument unfortunately
what are the risks of this
non-representative view well granting
influence
to uh of potentially of our future to a
tiny minority is problematic
and i would agree if this what was what
we were doing if we were giving
influence of
um of humanity to nick bostrom that
would be something we should think about
uh but it's not like we are granting him
influence he's just basically writing
some stuff
and seeing who is written right and and
almost no one else is thinking about
this
and um
if we try to implement policies that
reduce existential risk well then there
might be other interests that are um
that are overshadowed
um
and and this would be true indeed if
nick washroom was implementing policy
but he's not implementing policy the
people who are implementing policy are
not considering it at your risk at all
and again the people who decide what
risks we should take and what risk we
shouldn't take is probably narrow right
now uh and i agree it's probably narrow
but again the people who are actually
discussing
deciding what uh whether we should build
more nuclear weapons do not include
eliezer youth castle
and finally um one of the things that
people in the
working within the techno utopian
approach have been writing about is
moral uncertainty and that is something
that the authors i think is really good
and should continue and i also think
that's good so i want to end on a
positive note
that is all for today thank you and see
you next week


196. Ben Garfinkel on Scrutinizing Classic AI Risk Arguments

Hello and welcome to session 196 in the AISafety.com
Reading Group. Tonight we’ll be discussing
the first half of the podcast “Ben Garfinkel
on Scrutinizing Classic AI Risk Arguments”
by Ben Garfinkel, of course, together with
Howie Lempel, Robert Wiblin, and Keiran Harris.
Ben Garfinkel is a research fellow at the
Future of Humanity Institute and at 80,000
hours, we have Howie Lempel asking the actual
questions, as well as Robert Wiblin and Keiran
Harris supporting. This is a podcast, and
this link here also includes the transcription,
transcript, and published on the 9th of July,
recorded about a year ago. So, some new developments
like GPT-3 are obviously not covered. We’ll
be discussing the first half, up to the 1:43:11
mark, mostly starting at the 50 minute mark.
One thing I should point out is that it is
not precisely defined what constitutes the
classic AI risk arguments. So I’ve chosen
to mostly define that as the “AI foom debate”,
as well as the book Superintelligence. But
whether that’s entirely correct is a bit
unclear. You could also argue that the classical
period was before 2015.
I’d really like to say as the first thing:
A big thank you to Ben Garfinkel for making
this podcast and doing the work he’s doing
- trying to really look into if the arguments
that we have for AI Safety are sound. I really
strongly appreciate that. Also it is 1 hour
and 43 minutes of podcast, so when I summarize
this I have to do it by removing qualifiers,
so if Ben Garfinkel says something like, “Arguably,
A -> B”, then I’ll summarize it as just
“Ben Garfinkel claims A -> B”, and I’ll
give a counter-example, and I won’t even
include my qualifiers. So I’ll do that throughout,
and that means this is kind of a straw-mannish
presentation: Not of Garfinkel is actually
saying, but what someone could say if they
were making stronger statements. Another thing
is that Robert Wiblin claims that Ben has
probably scrutinized classic AI risk arguments
as carefully as almost everyone else in the
world. One problem that I had when I made
this presentation is that podcasts aren’t
really the best medium for having a text which
you need to engage very, very deeply with.
A formal article can often be much better.
And there were sometimes some sentences that
I wasn’t able to parse, and that is of course
also problematic. There is some substantial
chance that some of my objections here are
based on misunderstandings. I think Ben will
agree that it’s very very important that
people write this thing down, and if Rob says
that Ben is the one person in the world who
is best positioned to do so, then I would
really, really appreciate it if this could
be written in a more formal way, in a precise
way. Right, let’s get on with it.
There is a general thought I have with this
and it’s related to counter-factual value
of the classic AI risk arguments. So we ended
the previous talk, Andrew Critch’s article,
with this quote: “The most useful form of
pessimism is one that renders its own predictions
invalid by preventing them”. And here, of
course, we have Nick Bostrom and Eliezer Yudkowsky
making these kinds of arguments. And this
has, indeed, had an effect. And I believe
you can trace a direct line from Bostrom and
Yudkowsky’s argument, to certainly OpenAI
and quite a bit of DeepMind’s work. And
I think a lot of the insights we have into
where is the state-of-the-art of the AI right
now depends on these two organizations. And,
in particular, our epistemic state is also
strongly influenced by organizations like
LessWrong, Future of Humanity Institute, MIRI,
and their work. So counterfactually if we
didn’t have that we would be in a situation
where we had much much less information publicly
available. And that would of course make things
seem much less smooth. So in that way, there
are two questions being mixed up somewhat
here: Were the arguments correct when they
were first written down, first made? And are
the arguments still relevant? Because now
we are actually doing something about AI safety,
and do the original arguments still hold?
And of course that is what I’m mostly gonna
focus mostly on here.
The podcast starts with a description of AI
and AI risk, and why effective altruists should
want to focus on this. And it’s quite good,
I really like it actually, and I think it
makes a great case that AI is important, neglected,
and tractable. And he also seems very positive
on AI safety more broadly. He points out that
the case for AI safety has been broadened
recently by some new arguments about political
instability and lock-in scenarios. I don’t
think we… I think we can find them in the
book Superintelligence but they’re being
given low priority there. And it certainly
seems much lower than what the effective altruism
gives them right now. Ben Garfinkel however
returns to the classic arguments and the importance
of understanding those, and really thinking
hard against the weak points in those. In
particular among the classic arguments, it’s
the Bostrom/Yudkowsky arguments that’s been
written down in a very very formal way, and
those should probably be prioritized in a
way. And that’s something I really, really
very strongly agree with. I believe that figuring
out… this kind of skepticism is something
that I’ve engaged very very much with and
very very deep with.
So one of the objections that people come
up with is one called “painting a concrete
risk scenario”. And this is something that’s
not being done very much. And one thing that
I noted here was when I linked to the transcript
on Facebook, and then Facebook added a picture
to that, that wasn’t actually in the transcript.
So I looked where that came from. It’s a
meta tag, and unfortunately that indeed includes
a very concrete risk scenario - A dystopian
vision of how we would end up if we actually
lose. To me that was NSFL - not safe for life.
But I might be more emotional than most other
people with that regard. So the argument here
is that the descriptions of AI risk don’t
really seem rooted in reality. And I think
most of them are not. I think Life 3.0 actually
contains something that is quite detailed,
and, you know, seems quite grounded. And this,
the argument goes, is not really true of pandemics
or climate change. And that, I’m not really
sure. The descriptions of existential risk
from global warming in particular, the ones
that I’ve seen, don’t really seem to be
very grounded in reality, and not very fleshed
out at all, compared to the picture of, like,
a modest increase in global warming. So this
seems like an odd thing, right? We have an
existential risk which you can’t really
describe in concrete terms. And, that seems
odd. But we do actually have theoretical reasons
why we should be unable to predict how this
would go. And this is related to The Singularity
and the problems of figuring out what someone
who is smarter than yourself would do.
Some of the existing skepticism have a different
focus, that the original arguments don’t
engage with how AI research is today. Because
we’ve had the deep learning revolution from
2012 onwards, and AI was very different in
2008 during the AI foom debate. And so, I
think the book Superintelligence suffers a
lot less from this than the AI Foom Debate,
partly because it’s newer. It seems moderately
agnostic towards what methods could be used
to achieve AGI, and he does... Bostrom is
quite clear that Machine Learning is the most
likely path to AGI. And so the thing we have
mostly seen is that Machine Learning has improved
far more than people expected. And whether
the other approaches toward AI are stagnating
or if they are also improving towards AGI,
I think that’s a good question and really
hard to say, because all the focus is on Machine
Learning, because things are moving really
fast there. Ben Garfinkel makes the following
claim: “I believe reinforcement learning
techniques only get about two paragraphs in
the book Superintelligence”, and so I looked
that up. And I think this is kind of an example
of the problem that I have with the fact that
this is a podcast. Because if this was not
a podcast where Ben has to make things up
as he go but, you know, if this was an article,
then of course Ben would have looked this
up and, you know, CTRL+F’d the document,
and found over 26 places where it says the
word reinforcement, and he would see that
there is a subsections called reinforcement
learning on the book, one page with two paragraphs,
but after that there is a lot more about how
to use reinforcement learning techniques for
value learning. And it goes into quite a bit
of details actually. And I think this here
is the formula that Bostrom ends up with in
the book Superintelligence. And I think it’s
fair to say that if Bostrom had gone substantially
deeper into reinforcement learning theory
than this, he would have lost, well, more
than 99 percent of all readers, really. I
don’t think it’s reasonable to expect
Bostrom to go deeper into reinforcement learning
than this level, basically. So apart from
that, there is more to machine learning than
reinforcement learning, so his history of
AI emphasizes neural networks, and if you
think, neural networks and reinforcement learning
are, you know, recently related techniques,
then I think it’s not really far away from
how you would write it in 2020. But of course,
yeah, machine learning really took off in
2012, and I think the book was finished in
fall 2013, so he just got the very start of
a revolution there. And this might not really
matter a lot because the problem is that,
sure, it doesn’t engage that much with how
much is actually machine learning. But this
is an argument that never actually cashes
out into anything. So you can’t say or use
this as a reason for why other things in the
book Superintelligence are not true. And Ben
Garfinkel is, of course, realizes that this
is not a very good argument. But he’s still
sympathetic to people who react dismissively
to AI Safety arguments, in spite of the fact
that the arguments don’t really cash out
into anything. I’m substantially less sympathetic,
right? If you have an argument that doesn’t
cash out into anything then, I mean, then
it’s just a poor argument, and if the people
who are working with AI are aware of this
criticism, and don’t really engage with
it, I think it’s something you could criticise
quite strongly.
Another potential problem is, one of the analogies
for, certainly, the intelligence explosion
is the evolution process, in particular the
evolution of hominids. This is something that
hasn’t been written very thoroughly yet.
Ben says he’s not aware of any more than
a page long piece of writing that uses this
analogy to try argue for this discontinuity.
So the most detailed piece about this is written
in AI foom debate. I tried to count the number
of times wherever they said “chimp” and
“hominid” and “evolution”, and this
is written in like, many, many places, but
in a somewhat verbose and informal sense,
trying to use this as an analogy, and “thorough”
is not the… the AI foom debate is quite
meandering. It’s also something that’s
discussed over 4 pages in the book Superintelligence,
and this book also contains references to
other people who have been working with this.
I think more importantly, the book Superintelligence
argues that this is a rather weak analogy,
and probably you can’t use that for very
much. And this is the reason why I and many
other people… We are busy people, right,
we don’t really have the time to put a lot
of work into something that we don’t expect
will lead to anything. And that’s something
that I think should... the podcast is… would
be nice to have more clarity here, that the
evolution argument is actually not really
in any way central to why we could fear an
intelligence explosion.
Another analogy that hasn’t really been
explored according to Ben Garfinkel, is how
much compute does the human brain use. Like,
could we get timelines by looking how much
would it take to emulate a human or something
like that. And compare that with how much
compute do we have right now. And this is
something that, I would say, actually a lot
of people have explored this really well,
notably Robin Hanson in the book The Age of
Em has written a lot about this. And, unfortunately
from this, I’m not really very happy because
sure, if we can get some lower bounds, lower
bounds would be worthwhile, but they seem
really, really weak. And I don’t think this
analogy is actually going to be very useful.
Ben Garfinkel says that maybe the fact that
the arguments haven’t been written down
is something that caused him to disregard
them too much. And actually, no, I don’t
think that. I think it’s quite fair to not
value arguments that haven’t been written
down as carefully as, for instance, the book
Superintelligence. But the book Superintelligence
in fact has been written down in a way that
is certainly sufficient. And this is why I
believe that this is the key piece of writing
we should focus on, maybe focus even less
on the AI Foom Debate for instance.
To get into one of the main points that Ben
Garfinkel makes against the classic AI safety
arguments, and that is what’s called Brain
in a Box scenario, which is a specific AI
development scenario that is purported to
be implicitly implied in the classic AI risk
arguments. And as you might be able to see
from the screen, I think we’re at a straw
man fallacy here, in that this argument is
actually not one that is central at all, almost
not mentioned. And I had to dig deep into
the old sources of the AI foom debate to try
to figure out why this would be relevant,
and if anyone is actually using this.
Brain in the box scenario is that there is
a time where we have some narrow AI progress,
roughly like what we see now but nothing that
really transforms the world, and then relatively
abruptly we have one AI system that is very
cognitively similar to a human, to a brain.
And from that, we get an intelligence explosion.
That is my understanding of the Brain in the
box scenario. But I might be wrong here, and
Ben Garfinkel describes it with roughly with
these words, but he doesn’t make a reference
to anything, he can’t do that since it’s
a podcast and not an article. So I tried to
see where I could find that, and I tried to
google for it, and the best thing I could
find is the Foom debate. Where after the actual
blog post, there was an in-person debate where
Eliezer Yudkowsky described this, and he uses
quite different words, for the brain in a
box scenario. There’s nothing about it that
would take a day or a month, it just takes
a while, and the thing this Brain in a box
can do is reprogram itself, and whether it’s
cognitively similar to a human is not mentioned
at all. Similarly, Nick Bostorm has this vision
of an Intelligence Explosion and talks a lot
about continuous improvement. It is not abrupt…
this intelligence explosion. And the thing
the AI system is doing is improving itself.
It’s not discussed at all whether it does
anything else. One thing however that will
be discussed later is the Concept of Deception,
that one thing the AI system might do is to
start to conceal its abilities. But apart
from this, “very cognitively similar to
a human” is not something that is described
at all. So if I look, Eliezer Yudkowsky is
using this brain in a box but Nick Bostrom
is not. So you could argue this if you put
the emphasis on BRAIN in a box, then it sounds
like the focus is on that it is like a human
brain, and I think another way to state this
is to focus that it’s IN A BOX, so if you
put the emphasis there, then it’s not just
the mathematical object, it’s scalable and
you have two boxes, things like that. And
I think the second interpretation of the words
brain in a box as is the one that are used
by Eliezer Yudkowsky.
Is this discussed in Superintelligence? Well,
not really. That’s of course problematic
if you are scrutinizing the classic AI risk
arguments, that it’s not included in the
classic AI risk arguments. So there is nothing
here that address the relative plausibility
of something like the brain in a box scenario,
compared to something that is more smooth,
or present an argument like why you would
expect something like a brain in a box scenario.
So part of it is clearly wrong because chapter
4 of the book Superintelligence is called
the Kinetics of an Intelligence Explosion,
and that is indeed precisely this. So the
way that a single AI system undergoes an intelligence
explosion is indeed described in a very, very,
very great detail here. So I think that there
is some kind of misunderstanding here, and
Ben Garfinkel actually means something slightly
different. So if I should take a guess, one
of the things that Ben Garfinkel might be
putting a lot of weight here is whether what
is sometimes called a “seed AI”, that
is able to improve itself but not able to
do very much else is really able to do other
things than that. Bostrom in the book Superintelligence,
he doesn’t describe this, he doesn’t really
care about this. But whether that might be…,
I think right now, with GPT-2 for instance,
it seems like these abilities actually are
correlated to such an extent that it’s quite
reasonable to expect that it might be able
to do poetry, if it’s capable of writing
computer code. It’s also possible that Ben
Garfinke doesn’t mean this, but is talking
about an AI system even earlier than this.
So before it can self-improve. In this case,
he’s talking about this very early stage,
that’s something that’s described in the
earlier chapters, You can find some of this
in chapter 2.1 and parts of chapter 3, but
I’m really guessing here so I’m quite
unsure precisely what Ben Garfinkel means.
Howie Lempel actually tries to put things
back on track, asking the question: Assume
that among the things that these narrow AI’s
are good at doing, one of them is programming
AI, and so you end up with that leaping to
human level AGI and then take of from here.
So trying not to focus on the very broad,
cognitive things that a human can do, into
just the task of programming AI. And unfortunately,
Ben Garfinkel dodges the question and instead
talks about that if you’re trying to do
science, then there are actually a lot of
different tasks in this instead of a single
task. For instance you need to create new
hardware, well, if you need to do that physically
then you need a very long list of skills.
And that is undoubtedly true. But the thing
Bostrom is worried about, and Howie Lempel
is asking about, is the “simple” task
of actually programming the AI, in particular,
improving the AI itself.
There is some talk about feedback around the
human levels, whether the AI can outweigh
the contributions to AI progress for all the
other AI systems. Again this is a very very
broad frame, like AI progress is much broader
than just improving a single program. Ben
Garfinkel believes that if it’s able to
do that, something interesting must have happened
before. But if that does happen, then the
risk is indeed substantial. I guess I could
make the intuitive argument here that I can’t
prove, I think, but: Just about every program
in the world can be improved with moderate
effort. And from that reason, I believe that
with moderate effort compared to the amount
of work that went to actually creating the
program. From this claim it seems quite clear
that it is something we should expect that
the AI itself will be a program that can be
improved.
Another vision of the future is the Comprehensive
AI Services model by Eric Drexler, where we
see capability increase without increase in
generality. And there may be really strong
arguments that specialization may be favored
over generality. And we might be able to see
that in AI. In a different world this might
be something that we see where something like
GPT-3 doesn’t happen, but apparently when
you make something that can predict text like
GPT-3, then apparently it can do both poetry
and SQL statements. And whether we’re talking
about something that is really general or
specialized, in this case what we really care
about is the ability to improve one particular
software program, and that is something that
doesn’t really require a lot of generality.
Another thing that would be different in the
comprehensive AI services scenario is that
we will build narrower systems because of
safety. The book Superintelligence doesn’t
really assume that the person who is building
the seed AI that is undergoing the intelligence
explosion really cares a lot about safety.
So is it something that is likely to happen?
It remains to be seen. Ben Garfinkel probably
believes that, seems to believe that it is
more likely we’ll have something very weird,
a mix of things. And I think that trying to
predict the future is very hard, and the future
is going to be weird in general, but we don’t
really care about the future in general. We
care about whether this particular AI, the
first one which is capable of making an intelligence
explosion will actually do that.
Another scenario is called the smooth expansion
scenario, which is not… it’s hard to figure
out precisely how much weight Ben Garfinkel
places in this. But that’s where we slowly
see an increase in how many relevant tasks
the AI can do and how general the AIs are,
what time-horizons they are working at, how
independent are they. Once you see the first
system that can do everything a human can
do, which is basically the brain in a box
scenario, then maybe they are already better
at most things. That might be true, but in
particular we care about the 6 cognitive superpowers
in the book Superintelligence. Those are the
ones that are strategically relevant. And
the others are mostly not relevant. Of course
in particular, very very concretely we care
about the AI improving itself. Ben Garfinkel
here has a statement that says: the first
system that can do everything a human can
do might be preceded by superintelligent systems,
and that’s kind of, just wrong by definition
of superintelligence.
If we are in a world where AI development
is more smooth, then we might have a lot of
other… yeah, this is a direct quote from
transcript and I’m not 100% that I understand
correctly, but if we are in this smooth world,
where AI is improving gradually, than people
are not so likely to be caught off guard because
we can do work ahead of time, we can build
institutions, we will know about specific
safety issues, in particular because we might
have seen some of them before, things like
specification gaming. We’re seeing that
already and we might see more of those. I
think specification gaming is probably quite
likely something we’re gonna see more of,
but in particular we’re caring about the
problem that’s called the treacherous turn.
I think Ben Garfinkel would return to this
in the second half. But I’ll just quickly
say here that finding low level versions of
the treacherous turn... that seems really
difficult to have that happen before we get
software that is capable of, you know, improving
a particular software program. And so, there
is some more discussion about whether the
capability improvement will be smooth in this
way, and I believe that it could be smooth.
But this conception of that the AI should
try to hide its own intentions, that might
be a candidate for a strong discontinuity
in the safety landscape.
Another model that is implied in the classic
AI risk arguments is the race between capability
and alignment. The argument goes something
like this according to Ben: And again, there
are some of the… some of the sentences from
the transcript that just doesn’t make sense,
so again, it’s possible that I’m misunderstanding
this. But this model have a steady creep of
AI capability increase year by year, and I
think this is strongly not what the classic
AI risk arguments say because in those, AI
capability doesn’t increase quite gradually,
it is by that we have an intelligence explosion.
At the same time, the capability/alignment
race model has the AI goal progress, the alignment
basically, happening in a much more uncertain
fashion. And this creates some kind of deadline
in that we get capability before we get alignment,
then bad things happen. And we need to figure
out what goals should the AI have, before
we have, as Ben calls it, “extremely intelligent
AI”. And actually, as I read the classical
AI risk arguments, we don’t really care
about the point, extremely intelligent AI.
We care about the point where the AGI is able
to self-improve. Still however, this deadline
metaphor is one that is commonly used, it’s
just we have a lot more uncertainty about
how fast the AI capabilities will increase.
The deadline metaphor has a lot more uncertainty
in the classic AI risk arguments.
Now for one of the key points: The entanglement
between the capabilities and goals. Ben says:
It’d be hard to argue against the idea that
there’s a deep entanglement between advancing
of goals and making AI act in a way we’d
intuitively regard as intelligent. And, no,
that would actually be really trivial to argue
against, because if we think about places
where we see capability improve, we are thinking
about things where we have a benchmark, like
chess, for instance. The goal in chess is
to win, and in 1950 the goal in chess was
to win, and in 2020 the goal of chess AIs
is to win this game of chess, right? And if
we have benchmarks, something like ELO ratings,
then these benchmarks often imply that the
goal is fixed and if we have something...
we also talk about capability improvements
like, say, image recognition or, you know,
all these kind of games that are commonly
talked about when we talk about that AI improves,
then in all these examples there is not any
entanglement at all between the goals being
advanced and the capability of the AI. They
are completely disjoint. And even if we take
some more general things like self driving
cars for instance, right, Tesla probably when
they are programming their self driving cars,
then they probably do something, likely do
something like, let’s say, cars should do
like the average human would do - except outliers,
and in particular except the outliers where
the car crash into things. And they probably
do something like, you know, a bit more complex
than that. But the real challenge of self
driving cars is in the capabilities. The real
challenge is to have a model about what’s
going on around the car, to have an actual
robust model of that. Ben claims that making
something more capable and the after project
of making it have the right goals often seem
really deeply intertwined. And it’s not,
like, it’s two separate goals. So selecting
the right goals is an alignment problem, essentially.
To me, these are quite different. If you try
to create an AI, work on the goals of an AI,
then you try to say “work on this, and this,
and avoid this” etc. where the alignment
problem works in a much more indirect way
where we want the AI to, you know, work where
you have a pointer to the humans, the masters
of the AI, and say it it should do what we
want, and try to specify that in a robust
way. And this is quite different. And I’ll
argue for the disentanglement in two parts:
the first is that even if we have a lot of
progress in alignment, then that won’t help
in capabilities. We won’t get AGI just because
someone comes up with the perfect solution
for the AI alignment problem. Because if you
take an AI right now, something that’s implemented
in Python or whatever, and you say, “Oh!
It should do what I want!” Then the AI might
try doing that but it will obviously fail
because we don’t have the capability to
make an AI that can look at a human and figure
out what it is that we want. And a lot of
the techniques that are used in AGI, like
“AI safety by debate”, if you try to implement
them with current AI capability methods then
that’s obviously going to fail, because
we don’t have AGI yet.
The other example that Ben uses is with a
house cleaning robot. Programming a house
cleaning robot is pretty hard right now, because
it’s hard to specify a goal. And I will
argue that, no, that’s actually not the
real reason why it is hard. But first let’s
talk about narrow and general AI, because
this is something that the classic AI Risk
arguments stress. And cleaning a house: is
that something that can be done by a narrow
AI or general AI? One classic test of whether
something is an AGI is the coffee test by
Steve Wozniacki, whether you can go to a random
house and make a cup of coffee, and it seems
strictly easier than actually going into a
house and cleaning it. So this house cleaning
robot is probably something we’ll only have
once we have AGI. And that’s of course the
reason why we don’t have cleaning robots
right now. Not in particular because we can’t
specify the goals, but because we don’t
have the, we can’t make it do what we want.
Ben Garfinkel’s vision of this house cleaning
robot problem is we could do it with some
kind of reinforcement learning where we hand-code
the reward function, but if we make the reward
function too simple we get some side effects,
it will ignore things that are not there.
And it’s really hard to capture everything
we want in a reward function. I actually believe
it is going to be rather easy to do this.
Not really easy, but compared to all the challenges
of a housekeeping robot, I expect this will
be a really minor one. You know, if you just
say “dust minimization”, sure, that won’t
work, so you can’t literally solve it in
five seconds. But if you, like, spend the
day on it, you can get much much farther.
In particular, as long as the AI is narrow
then this kind of hand-specification is probably
going to work really well. So, the major difference
that I see here is we need to just specify
some approximate instrumental goals like “don’t
knock down the vase”, and in the alignment
problem we’re trying to precisely specify
the final goals of the AI. And those things
are really, really different. And, yeah. And
then Ben Garfinkel makes an argument that,
sure, this is a problem for AI safety but
AI safety is more. Real housekeepers sometimes
set the house on fire, and avoiding this kind
of problem is something that is relevant for
AI safety. And I strongly disagree here. I
believe that this is something where, which
the classical arguments for AI safety make
a big point about saying that this is not
what they’re about. This is not about ensuring
that a self-driving car doesn’t hit a pedestrian
or something like that. This is, this is not
related to this at all.
In Ben Garfinkel’s model there is an unhappy
valley that is required in order to have a
disaster. And the first is that if there is
no progress on alignment, well, then we won’t
use an AGI. If there is a lot of progress
on alignment then we will use that AGI but
it will also be great because there it will
be aligned. And the unhappy valley is where
we have enough progress on alignment, and
can get capability, but not enough that it
is actually safe. And my objection to this,
I believe I’ve stated a number of times,
is that the framing is much too general: improving
just one specific piece of software, and we
might even have a fixed benchmark. Ben Garfinkel
makes the following statement here, I think
it’s like, quite likely to be false, that
is that: only malicious or insane actors would
make an AGI pursue a narrow objective. And
I think a narrow objective that can be specified
really precisely is to make a profit-maximizing.
You have a software program trying to… you
know, make sure that you get as much money
on this particular bank account while not
breaking any applicable law. Or something
like that. I think that is eminently plausible
and we will get AGIs pursuing very very narrow
objectives. And also the fact that, you say,
only insane actors would do this seems unsafe.
Well, I think the vast majority of AI researchers
and AI users, they are not convinced about
the merits of AI safety at all. And I think
it’s not just insane people who don’t
care about safety. A lot of really smart people
are just basically… plainly not convinced
of the arguments.
Finally we get to the analogy of strawberries
on plates. According to Ben Garfinkel, Eliezer
Yudkowsky posed the following challenge: how
do we get an extremely superintelligent system
to move a strawberry to a plate without blowing
up the world? And this kind of framing doesn’t
really conduct the way that machine learning
research works. I am not really sure what
the word “conduct” means in this particular
sentence. I tried to look up whether that’s
actually something that was said. It’s not
in any of these classical things. The best
place I could find was Eliezer Yudkowsky’s
Twitter where he has pinned something that
was very similar: “Place onto this particular
plate here, two strawberries identical down
to the cellular but not molecular level.”
Which is, I think, I think that’s quite
different but other people have used this
analogy, and I found someone claim that Eliezer
Yudkowsky had said this, but had a dead link
for the quote. So it’s quite possible that
he at some point said this. But I think the
key thing here is the framing doesn’t relate
very much to the way standard machine learning
research works. And it’s indeed on purpose
because the purpose of this strawberry on
plates problem is to show that instructing
a superintelligence is very, very different
from what we’re doing with our current machine
learning. And this state of machine learning
techniques cannot be assumed to help with
this particular problem.
That is all for the first part of Ben Garfinkel’s
podcast. Thank you for watching and see you
in two weeks.


AI's Game Playing Challenge - Computerphile

so go is a very very
simple game in terms of the rules
but it's very difficult computationally
um
there's an enormous
depth of complexity that comes out of
the very simple rules which is uh
i think part of what makes people love
it so much as a game
so to understand why it's hard for
computers i guess you have to go back
a whole bunch and talk about just how
computers play
games in general these sort of
turn-based strategy games or
uh that kind of thing and start with
something easy like um notes and crosses
for the sake of internationalization yes
tic-tac-toe i like notes and crosses as
a name because it's descriptive you know
but whatever yeah we're going to look at
notes and crosses as a game and i'm
going to keep calling it that i
apologize but you don't have to
apologize i just i just want to make
sure people
people understand what we're doing right
right right right that's all so i'm
talking about this game
where you draw your octathorp and then
one person
you know goes here and one person goes
there and so on and you alternate
placing
noughts and crosses and you have to get
three in a row to explain notes and
crosses
we should play it an even simpler game
so the idea of the game is we're going
to take it in turns to choose
left or right
and i'm going to try and get the highest
number possible and you're going to try
and get the lowest number possible it
starts off my turn so if i choose left
for example it goes down to here
and then it's your turn and you get to
choose left or right and then which
number it ends up on is the outcome of
the game so i want the highest number
possible which is seven
you want the lowest number possible
which is one failing that i'd prefer the
five to the three this is a perfect
information game what's the difference
what's what's an imperfect information
right right so so a perfect information
game is just a game where all of the
players have all of the relevant
information so this is like that nuts
and crosses is like that chess
go these sorts of games you do get games
that are not like that for example poker
there's hidden information you can't see
each other's hands so
you have to if you want to write
something that's going to play those
types of games obviously it has to work
differently to take into account that
uncertainty the question is
should i choose left or right and how do
i make a principled
decision
now i want the seven so in principle i
should be going this way right you would
think i want to steer towards the seven
but on the other hand
at this point it's your choice you're
not going to choose the seven because i
know you want the low number you're
going to choose the one
so if i'm choosing this node
i'm effectively choosing the one because
i can predict you'll do that whereas if
i choose the right then your choice is
between a three and a five so you are
going to choose three which is better
than the one so i should go right i'm
trying to
maximize
the minimum value
yep i'm trying to make it so that the
best choice available to you is as bad
as possible for you and as good as
possible for me
so this is this is called minimax
because i'm trying to minimize the mac
well you're trying to minimize the
maximum i'm trying to maximize the
minimum if you see what i mean yeah um
and then purely based upon who goes
first as to who wins this and right
right yeah this is absolutely unfair
game yeah i started off saying i want
that i want the seven so maybe i should
go left to get the seven right that's
one thing i could look for the highest
possible payoff and try and steer
towards it that seems like a reasonable
way of playing the game not on something
this simple where you can see it
obviously fails but in a more complex
game you might think that look for the
best outcome and try and steer towards
it yeah but you've got to bear in mind
that your opponent is steering away from
it
um
or i could look at it and say ah that's
the one i definitely want to avoid the
one so i can't possibly steer left i'm
trying to avoid the bad outcome so i
should go right or you could look at it
and say okay how about this entire tree
what's the average here what's the
average goodness of steering left or
right if you take an average on seven
and one yep obviously gonna get you get
four four because that's sim plus one is
eight
if you take the average of three and
five you also get four so in this case
they're the same does that mean they're
both equally good well no because one of
them i end up with a one and one of them
end up with a three so so we're
basically saying here okay there are
different ways to evaluate the choices
that you make yep and most of them don't
work most of them aren't any good okay
um but there is one way that works which
is basically what we did which is min
max and so it's a recursive function so
let's try
a more complicated game
i'm going to generate some random
numbers just using python to get us
a list and then shuffle that list and
there it is
so i'm just going to draw the tree in
before it was obvious that i kind of
designed
those choices to make a point here it's
random so it's a real game now in many
ways a better game than nuts and crashes
because somebody's can actually win
probably whenever you see a tree in
computer science you should always kind
of be thinking
about recursion which you've done a
video on before right so so i dive back
into this same piece of code because a
tree has this kind of fractal structure
where ideally you want to have an
algorithm where you do the same thing at
every part of the tree and then you can
have one algorithm that processes the
whole tree
so
the problem with recursion a problem
with recursion is
uh infinite
recursion where you have this sort of
loop of nested
stuff that never ends you need a
degenerate case you need a case
where the answer is easy
and in this case it's the bottom
right
so if we are at the bottom
first move by the maximizing player
second move by the minimizing player
third move by the maximizing player so
here
which way should i go i should go with
whichever one is bigger it's the seven
right
so we can effectively call this
a seven
it's not the end of the game
but the value of this node is
effectively seven because if this player
directs
us to there the game will end with a
seven so you can do that in each case
so this one is a five because that's the
bigger one this one is a three and this
one it's the six
so now
these nodes
which before you couldn't do because
they didn't have any values
now these look a lot like the nodes
below so here now we're being the
minimizing player so this will be a five
and this will be a three and now we're
being the maximizing player again
who wants to go for the five
so you can see here
that as the maximizing player the best
score i can get is the five that's
assuming that we're all playing by the
same decision-making strategy right it
helps that it's the optimal
decision-making structure in this
situation okay um
but
you don't have to make the best choice
right if you make a mistake
i have values on all the other notes of
the tree i can make probably do even
better
but
the way that min max works is you end up
with the making the best possible play
you could make
as bad as possible
um
and you're trying to do the same to me
this is a game you can play with your
friends it's a it's a huge fun for
everyone if you can
do min max in your head
and
obviously if you have you can put 16
numbers at the bottom 32 whatever
um
but you'll notice of course
the more numbers you have at the bottom
the more work you've got to do
to figure out all of these into
intervening notes and uh
it's tractable for some real games that
people play like naughts and crosses so
i could i i'm going to try it so if you
take a game like knots and crosses
tic-tac-toe which is so simple that
human beings reliably learn to play it
optimally you start off at the beginning
your naughts you have nine choices right
because you can go
one two three four five six seven eight
nine those are your options now it's the
crosses turn and you there you have
eight options because one of the squares
is taken so
one two three four five six seven eight
and here there's eight and you see just
quickly this becomes ridiculous but no
game of knots and crosses lasts more
than nine turns guaranteed you run out
of spaces we go along
blah i'm only going to fill in
part of the tree now the point is what's
the degenerate case
because we're not trying to get numbers
so what you do is you say this isn't
going to be right
but let's just say as an example for a
moment yes
but it's that so it's a win for naught
so if we're playing nords we give this
board a score of one because we say one
we've won excellent a different board
where the crosses have a win
that's a minus one because i really
don't want that and then any state where
the game is over but nobody's won it's
surprisingly difficult i accidentally
won it
this is how hard this game is there
what what can i do all i do is win so uh
so okay
that's a draw yeah that's a draw so
that's a zero if it's your turn and you
have a move that can win
then that board state is a win
and therefore you just propagate that
up
indefinitely exactly the same way as you
would before yeah so you can see how you
can plot this out and it's much too big
to actually do
but if you write a computer program to
do it
it's not difficult it's a short program
it doesn't take long to run
and nodes and crosses is
a solved game right
so let's draw the thing for
chess how much paper have we got
yeah let's draw a chess
okay
yeah
i'm not going to draw
chess why are you not going to draw a
chest well you know it can't be that
much more difficult kind of
yeah so we need more paper
uh we need a lot more paper to draw a
chest um
and the issue is the thing the thing you
can see the the thing that's so
different here between the simple game
the very simple game and notes and
crosses is this thing called branching
factor which is
basically at each little node how many
branches does the tree have
simple in this game it's two here we
have two choices here we have two
choices here we have two choices you
always have two choices this is a longer
game
it has more turns but the branching
factor is still two every turn you have
two
choices you can make in noughts and
crosses okay firstly it's more
complicated because the branching factor
is higher secondly it's complicated
because it changes
in the first turn
you have nine choices the second turn
you have eight and so on so the
branching factor actually reduces as the
game goes on which is one of the things
that makes it easier to compute so let's
think about it let's think about chess
then in chess
the very beginning let's say you're
white you're about to start
you've got
eight pawns you could move you can move
them forward one so that's eight
possibilities or you could move
it forward uh two
so that's sixteen plus you have
two knights that are allowed to move
because they can jump so
each of those has two it's another four
i think it's twenty i don't really play
chess but i think i think it's 20. i
think you start off with 20. so i would
be drawing one two three four five six
seven eight nine two eleven twelve all
the way up to twenty
um
and then as the game goes on
much like in naughts and crosses
the number of possible moves available
to you legal moves uh varies right
so presumably first it goes up
yeah as the game opens up early on you
get more moves available because you've
got sort of more space and things can um
things can move to several different you
know positions and all that kind of
thing
um
and then past a certain point it starts
to go down again because pieces get
captured
um and when obviously when you have
only a few pieces on the board you don't
have as many moves available to you
but on average when people play chess i
think the
average branching factor is about 35
but
chess games generally go
longer than nine turns right
so
when you're calculating the number of
nodes in your tree
every turn
you're multiplying by the branching
factor
and that quickly gets completely
unmanageable so you can't just min max
on chess like this
uh it's computationally
completely
infeasible
which is why chess was considered
such a big milestone for a long time if
you could make a computer that could
play chess you know you're not just
doing this very trivial brute forcing
thing like you're doing for notes and
crosses you oh if a computer could play
chess it would really have to be
thinking right you can see how here we
were taking the end states where we knew
the value of the game and then
propagating backwards in time from there
to give value to these board states that
we didn't know the value of before the
problem is there you have to go right
from the very end of the game
which in chess you you can't do there's
just too many possibilities um
so you need some way of giving boards
values
that isn't just propagating backwards
from the possibilities from known and
you know checkmate states um
and luckily in chess there's a lot of
things you can do about that um the most
obvious one is just
counting up
the pieces
you just say well a pawn is worth you
know one point and a queen is worth nine
you do some analysis of how games tend
to go and you you try and figure out how
good the pieces you have left are
basically and the position of them
obviously is important
but you can get a good it's a good
heuristic it's a good approximation to
just add up the the scores of the pieces
and say well you know this team
uh has
both their knights
and still has their queen and this this
only has one knight and has lost their
queen so it's really obvious that this
team is winning
um and therefore that's a good state of
the board and you don't need to be able
to see
all the way to the end game to know that
it's a good state of the board so the
point is because chess's branching
factory is so much higher you can't do
this thing where you work backwards from
the end states the branching factor is
too high you have to start from where
you are
and look forwards and say okay
hypothetically if i went this way
what would that look like what boards
would that allow me to get to and then
if from there i went here how would that
look and so on you don't know whether
you're heading towards a win or not and
this is where the heuristics come in is
that they let you put some kind of
numbers on these notes like you know do
i still have my queen in this note and
that should guide you towards winning
positions because there tend to be more
winning positions in situations where
you have your queen but it's always a
heuristic you you they're not
the the chess ai is not seeing forwards
in time to a win they're seeing forwards
a certain distance and seeing that it's
a good position to be in
according to and it obviously it's not
just what they have on the board like
that's an oversimplification um there's
a lot of different heuristics but the
point is
you're going to have to evaluate a very
very large number of boards so you want
something you can compute quickly
because the more boards you can evaluate
the further ahead you can look
and therefore the better you can play
even in this game if you're playing with
someone and you can think more moves
ahead than them that gives you a huge
advantage because you can force them
into a into a situation where the best
move available to them is quite bad
and they didn't
think to steer away from that because
they didn't look as far ahead as you
what this number means is if i go here
and then play perfectly from that point
on what's the worst score i'd end up
getting this algorithm assumes that your
opponent is always going to play the
best move that's available to them
because you're always prepared for the
worst possible scenario so you're only
ever pleasantly surprised
chess is a different ballgame it's got
more complicated maneuvers it's got
different ways of branching as well as
larger branches so how does this all
feed back into our original point of go
looks pretty straightforward to me
right okay go right
let's draw go
okay we're going to need more paper
significantly larger universe to put
that paper in go is difficult for a lot
of reasons actually
the first reason that makes go difficult
is the branching factor is very high
it's generally
more than 200
um compared to chess's 35.
um so every turn
there are at least 200 possible moves
you could be making
um
and so
if you imagine me drawing this tree with
200. come on then let's draw it all
right
let's do it 200 one i'm gonna i'm gonna
run out of pen
two three four i'm gonna go in between
five six seven eight nine three do you
know what
i think life's too short rob yeah you
know you're right
yeah this margin is too too narrow to
contain so um
it's not gonna happen right it's not
gonna happen
um and what's more
even
you know it wasn't going to happen with
notes and crosses but you can do it in a
computer
you can't do that with go you can't do
that with chess even and you really
can't do it with go and so even a lot of
the cleverness
of tree search and like pruning and all
of the
really neat
uh algorithmic improvements people have
made for ways of navigating these trees
that allow
um
this type of old-fashioned ai to be
extremely good at playing chess better
than any human at this point
um
that approach is not going to work with
go
and was never going to work with go and
so
go became the great oh my god past tense
go became
the great uh
the great milestone right after chess go
was
the big one
so right we were going with why is it so
hard
the branching factor is huge
that's one thing
but also
you don't have a lot of the tricks that
you have with chess
you can glance at a chess board as i
said and just add up the pieces and say
well this you know white is probably
winning or black is probably winning
in go that's not the case
in the middle of a game of go between
two good players
other good players could not necessarily
tell you who's winning right now
because the specifics of who's winning
depends very sensitively on
the precise layout of the pieces it's
not just a matter of who has more stones
on the board or who has more territory
or who has more points or anything like
that it's
sort of an emergent property of the
pattern of the layout
um
which means you don't have these easy
shortcuts you can use to evaluate a
position not only are there
ten times as many possibilities to
search there's no obvious way to even
tell if a position is good once you once
you're looking at it which is well this
is why this is so impressive what's been
achieved
by alphago
um
to beat the best human players
at a game where
all of the standard tricks of the trade
won't work and you have to really try
something new
head mounted display that tracks where
you're looking at various other things
so it overlays
and this can represent the other paddle
so we've got two objects with the
identical interfaces one to represent