Eliezer Yudkowsky on Intelligence Explosion

our first presenter today is actually
one of the cofounders of the singularity
Institute he has a popular community
blog called les wrong which has over two
million visits he's currently working on
a popular level book on rationality
based on his writings on les wrong and
he's an artificial intelligence theorist
prolific writer about human rationality
and he's here to talk to us today so
please welcome
Eliz your huge Kowski to the stage good
morning so I know you're all still
groggy waking up so I'd like to start
out by asking you an extremely difficult
moral question you'll have to think hard
about this one
the woman in the background of the slide
has smallpox probably none of you have
ever seen it in real life I expect used
to be terrifying scourge of the world um
my moral question is this are you for or
against smallpox this used to be a
really controversial question back when
dr. Boylston was trying to introduce
smallpox inoculation to Boston in 1721
it was said that smallpox is a judgment
of God on the sins of the people and to
avert it is but to provoke him more it
was said that inoculation was an
encroachment on the perogative of
Jehovah who's right it is to wound and
smite nowadays the equivalent of this is
zayn Hashmi he was born with a rather
painful disease that requires painful
continuing blood transfusion his parents
requested in 2001 to conceive a savior
sibling who would be screened to ensure
a perfect match for bone marrow donation
Britain's bureaucracy approved it in
2003 it was challenged by a group called
comment on reproductive ethics or core
the case reached the House of Lords in
2005 and was allowed problem was at that
point Saint Hashmi was five years old he
needed the transplant before he was six
the mother had just turned 40 the embryo
was they found the good embryo they
implanted it she miscarried I haven't
read in the news yet that he's dead but
on the whole not a very good showing
there so why would this group comment on
reproductive ethics do something like
that
the argument that they used was that
there was a slippery slope you let them
tissue screen embryos and the next thing
you know we've got cannibalism they
didn't actually say that I'm not even
sure they actually bothered describing
where the slipper
slow blood so for those of you who have
heard about Jonathan hates moral
foundations there's five sort of
intuitive schemas that tend to underlie
morality they are harm and benefit
fairness groups and loyalty to the group
authority and respect and purity and
contamination and they Jonathan hate
doesn't actually say this because you're
when you're a scientist you're not
supposed to sound quite this judgmental
but moral schemas based on purity and
contamination tend to go really nasty
really fast so if you've got to sort of
defend I'm not quite sure what it is
I'll describe them as defending but
you've got to defend its purity from
even a tiny bit of contamination by
allowing Saviour siblings and they're
willing to kill the kid for that and I
would just like to note that that is
what I would call a slippery slope going
forward life extension anti-death --ax
doing basic research to resolve basic
problems and aging is now a possibility
is now being debated and here's what
leon kass who at the time was bioethics
advisor to our thankfully former
president george w bush had to say about
it
perhaps mortality is not simply an evil
perhaps it is even a blessing against my
own strong love of life and against my
even stronger wish that no more of my
loved ones should die I aspire to be to
speak truth to my desires by showing
that the finite tune of human life is a
blessing for every human individual
whether he knows it or not that
translates into I'm going to advise the
president to ban anti aging technology
because I want you all to die
and you know personally I think he
should have just gone with that love of
life thing I think it's one of those
things your unit unit you know you're
supposed to go with your love of life
and your desire not to see your loved
ones die are not actually supposed to be
fighting that down Leon casas
orientation is basically religious which
means that he actually thinks everyone
is already immortal they're just not
supposed to be immortal here they're
supposed to be mortal in heaven actually
I actually sort of find it somewhat
amusing that you know you ask the same
people and they will say simultaneously
what could you possibly do with eternity
and they will also claimed to have
immortal souls and they will not realize
that there is anything at all
contradictory about these two claims or
about these two statements I should say
if you ask secular humanists what they
think of life extension a lot of them
will give back exactly the same replies
as people who think they have immortal
souls secular humanists will also go
around saying things like death gives
meaning to life which is another one of
those things where if you actually think
about it for five seconds instead of
just marveling at the beautiful sound
that the words make you realize no wait
a minute
death does not give meaning to life life
gives meanings to life a beautiful
sunset is every bit as pretty in that
moment whether you're going to die
eventually or not you don't sit there
looking at a beautiful sunset saying you
know what makes this sunset meaningful
the fact that I will inevitably cease to
exist
so why does all this happen why do we
have humanists saying these sorts of
things even though they don't believe
that they have souls
so the basic schema here is something
called cognitive dissonance and the
notion is if that if people were hit on
the heads with baseball bats once every
month then soon philosophers or other
people whose job it is to say why
sounding things would discover all sorts
of amazing benefits to being hit on the
head with a baseball bat you know to
help you smile while you get hit they
would say that you know getting hit on
the bat once a month with a baseball bat
toughen you it makes you less afraid of
lesser pains it makes the bat free day
is all the sweeter really it's getting
hit on the bat again getting hit on the
head with the bat once a month that
gives meaning to life but if you take
someone who's currently not being hit on
the head with a baseball bat and you ask
them if they'd like to start an exchange
for all these wonderful benefits they'll
say no and it's exactly the same way
with smallpox aging and should we
someday get so far as eliminating it
death the reason why smallpox seems to
us like such an obviously bad idea is
that we don't have to live with it
anymore if we're still around people
would be making all sorts of excuses for
it and the problem is that when someone
invents inoculation or when they start
introducing for the first time the
prospect of doing basic research against
aging the excuses and rationalizations
don't automatically go away people
really fooled themselves into thinking
smallpox was part of the plan that it
was meant to be and that if you try to
get rid of smallpox you were interfering
with the plan and then in order to
embrace inoculation you have to stand up
and say you were wrong you have to say
never was a judgment of Jehovah if your
child died of smallpox a decade back you
have to admit that that was actually
pointless there wasn't a higher purpose
to it it wasn't part of the plan and
that you could have prevented it as
easily as sticking to your child with a
needle if
known what to put on the other end of
the needle so yesterday's cute little
rationalization for something that no
one can possibly do anything about to
help us live with it and smile while we
get hit on the head with the baseball
bat is tomorrow's obstacle to progress
same way with old age and death nowadays
we have to live with them now people are
making all sorts of excuses that sound
clever like it's really a blessing in
disguise
that's one heck of a disguise you could
have fooled me but if there was some
planet where no one had ever heard of
old age or death and someone stood up
and said hey I've got this brilliant
idea
let's wrinkle up like prunes spend 20
years lying in bed and cease to exist
it'll give meaning to life try it but
just remember for it to work it has to
be affable and it has to happen to
everyone whether they like it or not so
I think that the people on this planet
would say no and then haul the person
off the mental asylum and now that we're
starting to debate the weather to
conquer old age the same way used to
debate by the Contra smallpox is exactly
the same schemas popping up so I think
the number one reason why people don't
just go with their love of life is that
it sounds too simple there's a lovely
little quote from this about
evolutionary psychology it's John Tobey
in the audience thanks Sarah hi John
Tobey 99% of what Darwinian theory says
about human behavior is so obviously
true that we don't give Darwin credit
for it ironically psychoanalysis has it
over Darwinian Darwinism precisely
because its predictions are so
outlandish and its explanations are so
counterintuitive Freud's ideas are so
intriguing that people are willing to
pay for them even though they would
never were experimental II tested in
anyway and as far as we can now tell are
all wrong but you know if you if you say
that all of your life problems come from
being secretly in love and with your
mother it may not be true but at least
sounds surprising enough that you can
see why they would pay psychoanalyst is
to say that you know an evolutionary
psychologist and ask them well what does
evolutionary biology predict about the
human mind given that we know that it
was designed by natural selection and
they'll say well people will like to
have sex and this is a unique prediction
of the theory that the brain was
designed by natural selection it but it
still somehow just doesn't sound
surprising enough that we give
evolutionary psychology proper credit
for it
and similarly is death good no who's
going to pay you to sit around all day
and be a professional bioethicists if
your papers are that short
but there are some occasions in life
where life does not have to be
complicated and this I think is one of
them now sometimes the obvious answer is
actually right if I ask you what's 2+2
you say 4 then I ask you but what does
2+2 really truly ultimately equal the
answer is still 4 still 4 even if I ask
you to deep portentous tone of voice is
life better than death yes is life
really truly ultimately better than
death
yes and this leads me to my proposal for
what I call simplified humanism you see
a lot of modern moral philosophers seem
to have special clauses that come into
play when you do something your
ancestors didn't do or when advanced
technology is involved so for example
it's generally good to cure disease
unless genes are involved because genes
are icky and scary to illustrate the
sort of thinking that I think is going
into this well this story you know I
heard a do about a friend of a friend so
it may be totally untrue then again it
could be totally true either way I find
it amusing
and it's about someone who once went to
a restaurant and was warned not to eat
the tomato on their hamburger because
the tomato has genes in it
similarly with the idea that it's good
it's great to be healthy and alive until
you reach a certain age at which point
it is bad to use technology to prevent
you be from being sick and dead it's
actually like sort of better to be sick
and dead once you hit well 80 unless you
are 80 in which case the 8k in which
case the the cutoff age is 85 or unless
you're 85 in which case the cutoff age
is 90 or something of which I like to
say I want to live one more day tomorrow
I will still want to live one more day
therefore I want to live forever proof
by induction on the positive integers so
my proposal is to simplify you have a
simplified version of humanism the
simplified version you know instead of
saying curing disease is good and less
genes are involved you just say curing
disease is good instead of saying life
and health are better than death and
sickness up to age 80 you just say life
and health are better than death and
sickness another key point of simplified
humanism personhood theory says that
fishes and chickens are non people
because they're differently brained not
because they're differently shaped to
illustrate why this is sort of
non-obvious I have a friend who says
that when she was teaching an
introductory philosophy class she would
ask the students is Yoda the little
green guy from Star Wars is Yoda a
person and often they would answer no
because to them human and person meant
the same thing so by way of making her
point clear I suggested to her that she
asked her class would you eat Yoda
and she tried that and she said that one
student said yes they would eat Giotto
because he's not human
and I said to her and reply I begin to
wonder if your students are people so
personhood theory isn't obvious to
everyone and back in the bad old days it
was a lot less obvious the reason why
the Enlightenment ended up on the
correct side of the war against slavery
was this notion that if someone can be
sad and happy just like you if they
resent being enslaved just like you if
they experience life in the same way you
do and don't want to be in Chains
anymore than you would be then their
skin color shouldn't matter once we know
these key facts about their minds and
that's why it's a bit alarming to hear
someone say that they would eat Yoda
this shouldn't matter that his skin is
green and a important implication of
this simplification where we just take
skin color out of the theory you can
generalize this notion of personhood
theory a bit because the key idea is
that you pay attention to the mind so
you can change your shape and still be
an object of moral value you can be made
of silicon instead of carbon and still
be an object of moral value it's the
same sort of outlook that would tell you
that the same sort of outlook that would
tell you that Yoda is a person and
valuable would tell you that Zayn hafiz
putative genetically engineered savior
sibling or star probably genetically
screensaver sibling or genetically
engineered whatever hasn't had her
purity or her sanctity violated by that
unless it affects her mind somehow
so simplified humanism life is better
than death not just when you're thirty
all the time
health is better than sickness whether
you're ten years old or 100 years old or
a thousand years old you know we might
not be able to figure out how to keep
you healthy when you're a thousand years
old personal I think that's pretty
likely by the time we actually are a
thousand that we'll have the technology
but the key notion here is that
simplified humanism is stating of value
it's saying if you can have the
technology then it's good to use it
happiness is better than pain ceteris
paribus all else being equal nothing we
should all run out and smoke crack but
all else being equal it's better to be
happy than sad I know it's a strange
idea and while run the subject knowledge
is better than ignorance it's not all
that relevant I was just making the list
and felt that that belonged on the list
somehow and simplified humanism embrace
the goal of success instead of making
excuses for failure which brings me to
the second part of my talk about
positive futurism so here's a beautiful
piece of poetry in the age when life on
Earth was full they loved each other and
did not know that this was love of
neighbor they deceived no one yet they
did not know that they were people to be
trusted
they were reliable and did not know that
this was good faith they lived freely
together giving and taking and did not
know that they were generous for this
reason their deeds have not been
narrated they made no history so what is
wrong with this it's a beautiful
sentiment
check uplifting check virtuous morale
check poetry yeah good poetry it's not
true yes that's right it's factually
false or never actually wasn't age when
life on Earth was like that we know by
now what hunter-gatherer tribes were
like or this is actually a picture of a
rice paddy but you know the original a
dawn age was hunter-gatherer tribes and
different anthropologists have different
estimates for how likely you were to die
by violence in one hundred gatherer
tribe the top estimate 65% of the adult
males die by violence lower estimate is
15 percent of the adult males die by
violence Steven Pinker
to say that in the 20th century a
hundred million people died by violence
which shows how far we've come because
of the 20th century had been in violent
as the good old days of hunter-gatherer
tribes two billion people would have
died by violence during the 20th century
so this wonderful poetry here is not in
fact true which is always a good thing
to keep track of I mean if this poetry
were true we just have to deal with it
but as a question of simple fact this
poetry is mistaken
so having settled this issue which
always comes first the other problem
with this poetry that ought to jump out
at you is that it is oriented toward the
past and is putting our best days behind
us
David Brin one suggested that whether
time slopes upward or downward is a key
component of your cultural Gestalt
worldwide most cultures believes in some
lost golden age when people knew more
used loftier thoughts and were closer to
the gods but then fell from grace only a
few societies ever dared to contradict
this dogma of nostalgia our own
scientific West with its impudent notion
of progress brashley relocated any
golden age the future so the title of
this talk is positive futurism
and you might guess that positive
futurism is going to be futurism that
says x slopes upward and our best days
are still ahead this of course is not
allowed because before you can be a
positive futurist well first you've got
to take care of that question that
always comes first is it actually true
you are not allowed to decide questions
of fact by being optimistic about them
and my way of introducing positive
futurism I'm going to talk a bit about
rational futurism yes you all knew I was
going to work a lecture in here on
rationality somehow so rationality based
constraints on futurism number one what
you want doesn't control how the world
is moral beauty of progress doesn't get
to dictate the laws of physics even if
being a more being alive is better than
being dead it may still be that if the
universe lasts long enough we will
eventually run out of negentropy you
cannot run out of energy energy is
conserved when people talk about running
out of energy they're actually talking
about running out of negentropy but that
is a whole long physics lecture I'm not
going to get into but the simplified
humanism again it doesn't say that you
can do this it says that if you can live
forever then you probably should and
this is worth emphasizing because a lot
of times if you talk about immortality
people will say well but what about the
universe running down well first of all
even if the universe runs down after a
while it is still better to live a
billion years than to live 80 years
it's there still a significant
difference and the other part of it is
that you're talking about values first
fer to you how you decide your values
then you separately from that figure out
your facts then you decide like can you
get the technology to do this
rationality constraint number two there
is no destiny that helps you that's the
other reason why you can't just say well
I'm going to be positive futurist and
seedtime sloping upward you know there
could be various events that are going
to make time sloped downward and the
laws of physics which are what this
universe actually runs on are not going
to check and see did our positive
destiny get violated by this event
there's going to go on you know having
corks bump into corpse and so on and of
course if a positive future really were
inevitable we could all just relax and
play video games and there wouldn't be
any need for a singularity Institute so
I guess in that sense we are you know
sort of biased when it comes to this
business of claiming that you know
there's no destiny involved I think the
usual saying is that causes alternate
between declaring that victory is
inevitable and declaring that victory
will take a long hard struggle so we
could just leave out the first part of
that
rationality constraint number three
magic doesn't work we're by that I don't
mean stage magic like James Randi uses I
mean you know real magic whereby real
magic if I had to define it I guess I
would say that it's sort of a mistaken
outlook on the universe in which things
are inherently mysterious rather than us
just being confused about them and you
have ontological e basic you know sort
of very high level facts about I wanted
this to happen so it happened and it
doesn't reduce to much simpler
mathematical parts so we probably mostly
take for granted that magic doesn't work
it's worth noting because outside this
hall it tends to get overlooked a lot
you can't get an artificial intelligence
by summoning a spirit into your computer
and the other reason why it's worth
empathizing that yes we know about this
constraint is that people who are making
very quick judgments by pattern
recognition instead of detailed analysis
will often classify positive futurist
ideas as magical so they say ah you want
to transcend human limitations that's
magic or that is they automatically
classify it as magic because most of the
people who talk about transcending human
limitations are talking about using
magic to do it the thing is though that
lots of people try to fly by magical
means before the Wright brothers built
an airplane so if you were around at
that are you around before the age of
say hot air balloons and someone talk to
you about flight you might recognize
flight is a magical idea that only
wishful thinkers talk about but what
makes it magical is not the goal itself
the goal of flight if you'd say I want
to fly you're just stating a preference
what determines your rationality or
irrationality is whether you try to fly
by drinking potions or you try to fly by
building a wind tunnel and experimenting
with it until you can figure out how to
build an airplane so back in the good
old days being strong enough to tear
apart mountains was a property of divine
beings nowadays it's a property of heavy
machinery
staying young at age 100 there's
certainly a lot of people who are
selling quack methods for that right now
if you're but it but if you start out by
saying well I'd like to still be young
at age 100 and then you do research on
it you're stating a value and choosing a
rational means of working toward that
value but if you work by pattern
recognition and Association then
obviously talking about staying young at
age 100 will feel very magical because
it's currently talked about by people
who talk about other magical things
finally you can't just make stuff up I
mean it's it I mean it by default human
beings just make stuff up it's what the
ancient Greek philosophers did they just
had no clue how stuff works they go
around saying things like all is water
or all is fire and they never ask
themselves wait a minute even if
everything was water how on earth could
I possibly know that so the example of a
violation of this rule the International
Congress on forecasting in 1982 some
cognitive psychologists showed up they
asked one group of professional
forecasters for the probability of an
earthquake in California causing a flood
that kills at least 1,000 people
sometime in 1983 that's another group
for the probability of a flood killing
at least 1,000 people sometime in 1983
the first group assigned higher
probabilities to the strictly more
complex event so the conjunction rule of
probability theory is that the
probability of a and B is always less
than or equal to the probability of a
whether or not B happens in the
conjunction fallacy is if you assign a
probability to a and B that's higher
than the probability of a and people
commit this fallacy all the time because
adding more detail can make a story
sound more plausible even as it becomes
strictly less probable and once you
realize this that every little detail
you add on to your prediction is an
extra burden that has to be separately
lifted by evidence too
strong enough to support it then you
look at the futurists who you see on
television and you realize they are just
making stuff up they are making up lots
and lots of details that create this
wonderful amazing story about the future
and that sort of futurism is a branch of
the entertainment industry then you've
got the futurists who for every little
detail that they tack onto their story
they ask can I justify that particular
detail what do I think I know and how do
I think I know it and this sort of
futurism looks completely distinct from
the other sort it's also very rare to
name one sort of person outside the
singularity Institute who does this Nick
Bostrom at the Oxford feature of
humanity Institute it does tend to be a
bit more academic and once you know what
the conjunction fallacy is you can just
sort of really quickly eyeball the
futurist and say is this person telling
a story or they do it do doing an
analysis so the positive in positive
futurism cannot possibly mean for
telling a golden age because all of your
predictions are strictly questions of
fact they either come true or they don't
and your values and your optimism get
nothing whatsoever to say about that on
the other hand there comes a point we
have to make policy choices where your
beliefs of fact tell you what happens if
I do this what happens if I do that and
you pick the policy that your values say
has the best consequences so positive
futurism is about positive goals rather
than positive predictions and well for
example people often come to us and they
expect us to agree with the idea that
changed accelerating exponentially I'm
sort of neutral leaning towards skeptic
on that Michael Vassar our president is
a skeptic
Peter Thiel our major funder is a
definite skeptic people often expect us
to agree with the accelerating change
idea because it's sound
like a positive thing to say about
technology and expect us to agree with
anything positive that anyone says about
technology but if accelerating change
were true it would just be a sort of
background fact then the question would
be okay taking this background fact into
account what policy choices do we make
Steven case identified four attitudes
toward technology technophile technology
is good technophobe technology is bad
techno normal nothing really good or
nothing really bad is going to happen
things will just go on as they mode you
know life will carry on and techno
volatile the future might be extremely
good or extremely bad but it's not
likely to end up anywhere in between the
main one of these attitudes that strikes
me as completely unreasonable is techno
normality there's all sorts of extreme
forces coming onto the game board that
weren't there before and to expect them
to all fail or exactly cancel out for
the purpose of making the outcome normal
is would just be really would be one
heck of a coincidence
now there's one version argument for
techno normality that says we've heard
these predictions before and life has
gone on but suppose I were to tell you
that you know a hundred years in the
future we'll have gone back to horses
instead of cars Kings instead of
democracies and books being too
expensive to own well that is what life
used to be like and if it went back to
that we would call it a catastrophe so
how is this not a golden age life did
not go on as normal we look back in time
and what we see is that history has
gotten steadily more normal for example
woman used to be unable to vote this was
very odd now they can vote that's normal
so you can see that the trend is toward
greater and greater normality over time
but if you imagine how incredibly odd it
would be to live in the 19th century
well the future is probably at least
that odd and if you imagine how odd it
would be how odd life was before humans
were around well since some of the
forces that were that are coming onto
the gameboard
are changing the constituency of
intelligent life that is no longer
within the tiny little cluster the tiny
little dot were all modern modern humans
live we can expect life to be at least
to that strange as well
techno a lot of techno normal advocates
actually say that all this talk of
Golden Age and catastrophe is childish
which I can only say that the laws of
physics have not been written in such a
way that they check whether something is
childish before it is allowed to happen
they are offering literary critiques of
things that are meant as predictions
rather than as literature they are
saying that extreme futures are bad
poetry and even if that were true it
would not make a difference so the
singularity Institute is techno volatile
Peter Thiel is techno volatile Nick
Bostrom is techno volatile I'd actually
be sort of hard put to think of any
careful thinkers who are not
technological with the possible
exception of Robin Hanson so if that's
what we say can we claim to be the heirs
of the Enlightenment which is what this
talk has really been about there was
this thing called the Enlightenment it
was first in favor of rationality of
making things better for everyone of
progress of not making as many excuses
anymore Sir Francis Bacon
one of the first advocates of the
experimental method wrote a utopian work
of the utopian literature called the New
Atlantis in which he said the end of our
foundation is the knowledge of causes
and the secret motion of things and the
enlarging of the bow
of human Empire to the effecting of all
things possible reasonably optimistic it
you can contrast it say to someone who
is not the heir of the Enlightenment
Bill McKibben wrote a book called enough
you can't stop progress but that's not
true we could choose to mature that
could be the new trick we share with
each other a trick as revolutionary as
fire or even the computer the Letus bit
by toward the revolutionary idea
that we've grown about as powerful as
it's wise to grow this is a sort of
thing that makes david grin foam at the
mouth because this is not a
revolutionary new idea this is the idea
that is around as old as fire all
societies used to think like that until
the Enlightenment came along you know
not quite strictly true but by and large
now that was the way most people thought
so which side is the singularity
Institute on we occasionally go around
talking about how AI could go wrong we
even say things like the vast majority
of possible minds and mind design space
are not friends with you and it will
actually take very careful design and
precise targeting to get an AI that will
not go wrong that sounds a bit
pessimistic and the argument I would
make is that you are as long as you are
still playing for very large positive
stakes you can still claim to be the
heir of Sir Francis Bacon in the era of
the Enlightenment see this is what I
don't think Bill McKibben has or what
people who are only trying to stave off
ecological collapse and whose vision of
a better future is that we all live
happily ever after with sustainable
technologies they don't have a vision of
how to get there from here and as long
as you're trying you're still trying to
get to a kinder happier intergalactic
civilization you might just be an heir
of the Enlightenment
so the new New Atlantis simplified
humanism life is good death is bad
person whose cognitive not everything
valuable is human you obey the same
rationality constraints as any other
person in any other problem when you're
thinking about questions of fact like
what happens if we do this is always a
question of fact and positive futurism
climbing upward with or without a
natural slope it's not about being an
optimist it's about having positive
goals and still playing for very
positive outcomes as a stake on the
board and that is that which I believe
is the new enlightenment thank you right
in Iraq
think you got time for one question
anyone want to ask one question
anyone at all that guy I like it thanks
for a great thanks for a great talk um
reminded me a little bit of Michael
yesterday Foster I have a question about
the values so because because you make a
bold statement in terms of terms of
absolute statements that happiness is
good and life is good but what are your
thoughts if you look at Aldous Huxley
brave new world so if you only have
great positive values it's not scarce
any more so it doesn't mean anything to
people so I do believe personally that
the dev or pain is good to a certain
extent as it defines the other part of
the spectrum so what is your view on
that things did you have to ask a
question that complicated
so I cannot resist mentioning my short
story three worlds collide but you are
welcome to Google so my general stance
over there is that while there probably
does come a point where by making people
artificially happy at all times under
all circumstances regardless of how the
external world looks and taking all the
interesting challenge out of life by
limiting eliminating variants in effect
and so on you can certainly push it too
far
Aldous Huxley's brave new world I don't
want to reason from because of the
something I call the logical fallacy of
generalization from fictional evidence
it never actually happened but if you
ask me to pass a value judgment on that
world I would say that they could have
done better and I think that if you look
at today's world we could actually push
the level of happiness up by quite a bit
before we started running into trouble
hmm Thank You Ellie hazard eliezer
yudkowsky everyone


Artificial Super intelligence  - How close are we?

YouTube is the largest video sharing
site on the internet in the modern world
videos can be watched on PCs laptops
mobile devices and even watches however
think back 10 years ago to January of
2005 and you may start to realize just
how fast technology is advancing YouTube
will not exist for another month the
first iPhone was still two years away
LCD TVs were still incredibly expensive
and for $199 in 2005 you could buy one
gigabyte of RAM streaming services were
non-existent Facebook was still starting
up and Windows XP was the standard
Microsoft operating system that was just
10 years ago add just one more zero and
suddenly the Wright brothers are
completing their first flight and the
first Ford model-t was still three years
away add another zero and the world is
now a very primitive place with no
electricity and a very small grasp of
modern science and one more zero and
suddenly civilization as we know it
ceases to exist debt logical progress is
speeding up the next 10 years will
produce a new world which makes the
modern world look outdated and slow by
comparison this progress is happening
faster and faster and is predicted that
in this century will make 1,000 times
the progress in technology compared to
the last century these facts are largely
unchallenged the big question is where
are we headed we now arrive at the
problem of the singularity two big
questions to answer are will it happen
and will it be beneficial to mankind so
let us start with a more basic question
what is the singularity to put it simply
the singularity is the moment where an
intelligence smarter than any human on
the planet is created and when that
intelligence begins to make smarter
versions of itself at an ever-increasing
rate such intelligence could quickly
become smarter than every human on the
planet combined and this would make it
the dominant force in the planet much in
the same way humans became dominant not
through brute strength but through their
technology and their intelligence it
should be noted that while a lot of
representations of the singularity
depicted
financial rights in the rate of
advancement it is still not infinite at
some point after the creation of the
superintelligence a level off would have
to occur when that would actually happen
is a complete mystery the main point is
that the advanced and technological
progress will be beyond human
comprehension the reason for this is
because of smarter than human
intelligence would be exactly that with
a power of a supercomputer with
petabytes exabytes or zettabytes of
memory and processing speeds much faster
than human it could perform a month's
worth of thinking in a second
multitasking will be trivial and a
global system of cameras will allow
sensory capabilities vastly better than
humans with superhuman capabilities and
super intelligences proverbial
fingertips it could potentially improve
itself at a faster and faster rate what
form the singularity will occur in is
still unknown some possibilities include
a hive mind or a transhuman singularity
the most believer will occur through AI
artificial intelligence at as it exists
today is known as a and I or artificial
narrow and elegance this type of AI
specializes in one tasks for example an
AI specializing chess would be useless
at checkers AGI is an artificial general
intelligence this is an AI that matches
humans in almost all areas of the brain
this type of AI would be as capable as a
human in fort AGI does not currently
exist a wild mouse a our research is
directed towards Ani we are getting
closer to an AGI each year there are
multiple ways in AGI may be created one
of these methods involves simulating an
entire human brain computers are
currently not powerful enough to do this
but most predictions place it sometime
after 2025 as a period in which such a
simulation will be possible thanks to
exponentially advancing technology
finally there is a si artificial super
intelligence at this level the AI is
smarter than humans and its capabilities
are greater than humans and a si would
be incomprehensible to humans and have
given access to the outside world its
actions would be unpredictable and
unstoppable the singularity could occur
in two different ways a soft take-off or
a hard takeoff an ASI and may find that
makes
other copies of itself becomes
increasingly harder as it continues and
the process may take many months years
or centuries other restraints may
include hardware limitations or inbuilt
safeguards to protect against an
uncontrollable intelligence explosion
this is known as a soft takeoff a hard
takeoff may occur on milliseconds in
which the second we create an AGI it
quickly becomes an ASI within a matter
of milliseconds or seconds such a
scenario would make the combined
intelligence of humans appear as that of
ants - an ASI which takeoff scenario is
most likely is still unknown wherever a
singularity will actually occur and what
will happen to you humanity when one of
these is created is still unknown when
the singularity will occur if ever is
also not yet known some have placed it a
2045 while others have placed it to be
within the next 100 years
once singularity is created unless if we
are prepared
there will be no retries what we create
is what will quickly begin to influence
the world around us in drastic ways we
won't expect the bright sides of this
however is that we get to make the first
move we have both time and research to
consider what a super intelligence may
want to do with us once it is created so
what will happen once the super
intelligence is created will it be
beneficial to humanity
next time I'll explore friendly and
unfriendly AI subscribe if you have not
done so on thanks for watching


250  A Generalist Agent 1

hello and welcome to session 250
in the aisafety.com reading group
tonight we'll be discussing the first
half of the article a generalist agent
by scott reed and others
this is a work done by deepmind scott
reed is the primary author but there is
a team of 21 people
um and i'm not gonna uh
find their pictures find all of their
pictures
it's published very recently so it's
like a couple of weeks old at most
um the word gato as far as i could tell
is it's not described why they chose
that name it means cat in spanish and um
like it's
pro not an acronym uh i could at least
generalists probably an agent that's
like ga and then you could she is
probably a transformer and
maybe optimizer or something like that i
couldn't find anything else so i don't
think it's an acronym of much mud
i also don't really think it's an agent
like um i realize some people say
everything that has to do with
reinforcement learning is an agent um
but uh
i i don't prefer such a limited like
most people when they hear talk about an
agent they don't think something like um
like uh gator in fact
so let's start out with a few reactions
to this paper
it's been very mixed someone like uh
daniel kukotal was willing to call this
indeed subhuman level agi that we do in
fact have agi here
other people like nostalgia bryce
made some uh very dismissive comments
saying this work is like totally what we
could have expected um and was really
underwhelmed but
one thing that certainly was not
underwhelmed was the prediction markets
because around the time when this was uh
uh
revealed the prediction markets for when
we would have weak agi moved uh
dramatically to become dramatically
shorter going from 2048 you might be
able to see here in this graph it's on
an uh logarithmic curve here so it looks
like it didn't go that far down but it's
actually from 2042 to 2028 which is a
rather dramatic shortening of timelines
um
i looked a bit around what people said
to that and a lot of the people said
this was a crazy overreaction and mostly
because uh
not because it changed very that much
but because uh
saying this could only happen 2042 was
already uh crazy so the market is
correcting
in some way
i must admit i i looked more into the um
the details of this bed in particular
and one of the things required for weak
agi is passing the turing test so you
need to be able to
follow a scenario somewhat like what
alan turing originally envisioned and
that probably requires more than human
level intelligence to do and also in
fact it requires winning the lutner
price and that's the price that's been
defunct for the past couple of years and
people
according to wikipedia people scorn of
this price
and that's probably why they don't give
that out any longer so uh this
uh
uh
this uh prediction here probably also to
some extent reflects the probability
that it will never be one because the
lutheran price will not be uh
reinstated or something like that
one neural network to rule them all
that's my quote and not one from this uh
from this paper
um
the the paper the introduction four
times in a row states that
gato does a number of things and it uses
the same neural network for all of these
tasks and it's very important for them
to say that it's the same neural network
and why is that a good idea well
you avoid
all of the hand crafting that you
normally have to do when you use uh
for different things and there is you
don't need to determine that good
balances you get a lot more training
data if you can just have one big model
and then you can train on literally
anything and then you get a more diverse
training and of course it's a model that
can be reused meaning that the total
amount of computation to use is much
less those are the five reasons given in
the payment i think
things like performance out of
distribution is probably a lot more um
interesting and probably
part of the reason why we care about ati
and of course the key thing that enables
this is that it does in fact work so you
take in more diverse training from
everywhere and then you keep getting uh
better results on all the tasks the
scaling hypothesis that we've seen so
many times
they have a number of tasks that they do
and when they just write it out it
seemed really really diverse you can
breathe read here all the things it can
do both in reality and in simulations
and games i couldn't find any specific
um
reasoning why they chose the tasks that
they that they chose
um
so i think mostly they like maybe they
had someone who was really good at the
bodies and then thought i will do some
robotic tasks or something like that
uh and of course the the main claim that
they're making is uh or one of the
things they're also showing that i won't
go too much into details with tonight is
that this that if you add more data and
fine tune it then it becomes even with a
little extra fine-tuning data
dramatically more capable
they have this statement natural
language is a common grounding across
otherwise incompatible embodiments and
that looks kind of nice because
obviously they're using a language model
to do things like control robots and
like white language a part of that i
will later uh
argue that this might not be entirely
correct or
and of course another thing that we
should note and also that makes it
somewhat less of an agent is that they
in fact did not use reinforcement
learning they used supervised learning
and learn from that
so um they could of course have used
reinforcement learning and
i think it's just a matter of
convenience that they
had some expert trajectories and then
they just used those um but
that is they didn't do any kind of
self-play or anything like that
finally in the introduction they have
this statement we hypothesize gateway
can be effectively scaled towards
covering any task of interest
and that certainly looks like a ati a
claim that uh an explicit claim from
deepmind that this is in fact an agi
um
this is kind of like an intuitive sense
that i have from this paper mostly
not
mostly from reading other deep mind
papers in fact other deep mind papers in
my intuitive sense are very anti-hype
trying to avoid making statements
like this statement in fact um but um
but i can't you know it's it's hard for
me to if someone asked me to please
point out the places in other deep mind
papers that are less hype than they
could be um like i can't really and
certainly a lot of other research is
is quite high
so it's more like regressing to the mean
let's talk about this skater model and
all the things that it does
the overall guiding principle is to get
as much data as possible and uh as
uh
as varied data as possible
um
and they put all this very data somehow
into just standard tokens which is you
know mostly
mostly words
and the the way they do this will get
back to the details later but i would
call that naive of course it's really
pretentious of me to say this research
is naive in the sense that i could not
do it myself um
but um
there doesn't seem to be any um big
ideas there are a number of small ideas
and we'll get to some of the small ideas
later but um
but it's certainly not something that
seems groundbreaking
and and the way they are using this
model is not
something that is reminiscent of um
reinforcement learners or anything like
that but mostly like a language model
they're just taking a language model and
trying to use it as an agi and say hey
it works uh just
by coincidence or maybe not entirely
like coincidence
but it's not designed with any kind of
um deliberate
thoughts about agi
it's 1.2 billion parameters
because and that's a very small model
because they want to use to control
robots in real time and so that means
when you query the model you need to
have a very fast answer
um for comparison gp2 was larger than
this model than gator and qt3 of course
was
more than
100 times larger so there is a
both a potential for it to be scaled up
and also uh like when you look at how
impressive it is in things like text
generation and try to compare it you
need to realize that it's much smaller
than gp3
let's talk about tokenization um the way
they uh in code
it seems like every half year there is
like a new standard way to do it that's
just a tiny bit smaller and now we're
doing using a method that looks at 32
000 subwords and we maps those to
integers
and images are also
like six
divided into patches and normalized and
um
they have a uh
for atari button presses they uh make
those into words in a rather interesting
way and i'll try to explain how they do
that so first they say they do it in row
major order so if you have like uh
nine values you can either this is row
major order or and this is column major
aura and they're using the top one
and let's
take an atari controller this is an
atari controller and if you squint you
can see like there is a button up
and up and to the left and to the left
and down to the left and down and then
in the middle there is in fact a button
that can be clicked or not clicked so if
you imagine that you are holding the
controller stick to the left and
pressing down the button at the same
time that corresponds to zero zero zero
and then one one zero zero zero zero so
in that way they turn atari control
inputs into uh into integers
they also need for uh for robotics they
need things like water movements and
where is the robot's body proprioception
um and the way they do that is they take
something continuous value and
in a
use some
tricks to put map those into some
special um
words so uh the first from 0 to 32 000
that's text and above thirty two
thousand uh and two thirty three
thousand and twenty four
they uh that's the robotics and that is
in fact the the way the thing that makes
me think this is not quite as general as
a model as you think because this way of
segregating the input into two parts
makes me think that uh you know when
they previously said that uh
they're turning everything into language
then that's not entirely true because
they are using in fact um different
values for this part
um
so so it feels more like there are two
neural networks pushed together rather
than um
one neural network doing both
um
let's talk about the last function which
is of course specifying the loss
function is a crucial thing for
determining how a how to train a neural
network
this is in fact a picture from uh
existing latent knowledge that i've
chosen here and you might remember this
is a patient network
with statements like it is raining and
the sprinkler is on and there is water
on the lawn and i should get an umbrella
and things like that so you imagine
there are some values here um that are
propagated um and let's say you want to
encode this kind of knowledge
so how do you calculate the probability
let's say the joint probability that
this is true this is true this is true
and this is true how do we calculate
that
well that is you can write that as a
probability of a1 a2 a3 and a4 right and
how do you calculate that you can
calculate that using the chain rule and
the chain rule looks something like this
so you have the probability that a4 is
true
given that the others are true
multiplied by by the probability that
the preceding ones are true
and this of course you can if you
want to calculate this well then you can
use the chain rule again
to get that's a three uh
uh
given these two uh multiplied by the
probability of these two together and
then you end up with something like this
okay so this is just a motivating
example of how to do it for four
probabilities let's take the logarithm
of all this because we don't like for
practical reasons we don't like to
multiply we much prefer to take the
logarithm and just add things okay so
what's the uh log of the probability
from
uh s1 to sl well that's the the sum
of
uh the probability uh s one
uh and then from is one to one
and then
uh
this index becomes two three four uh and
so this is basically using the chain
rule
okay
and
then using this
um
this
equation we they plug it into and get a
loss function for a
policy and a batch size
and for if we just take the batch size
here then you can see this is in fact
the uh
the statement up here the probability
that you go all the way down in this
tree in the bayesian network and there
is a masking function this masking
function uh ensures that there are some
parts of the output that we don't
actually um
uh train on and we'll get back to later
why we don't want to train on that but
this
equation here is the actual uh loss
function for training the neural network
so before i can explain the um
the masking function we need to look at
how data is actually flattened into uh
into the input in a batch
so there's some text it could be like
i'm going to london or you know whatever
people have posted on the internet there
could be some robotics proprioception
and continuous action
um and there are things like images with
questions and
atari games
and all this are put into some batteries
you can see here
and this is fit into data and then we
normally engage with the loss function
we need to add here
they mask some of the uh the inputs like
if you imagine here the uh
atari then we want to train on the
controls but what we predict the screen
to be isn't actually all that
interesting or not something we should
train on so we mask that out in order to
get our last function
there are some more details about the
training they choose transformers for
simplicity and scalability
scalability not learning something that
just one was not good at uh i'll leave
it to you to decide whether the
transformer architecture is actually
simple like it seems to me like that's
not the case but it's easy for them to
implement because uh you don't actually
have to implement that uh because other
people have implemented it for you so uh
yeah like if you want to use a
transformer then that's not very very
difficult right because other people
have done it
and some more just know about the
parameters for the
transformers and sort of other
details about how they do that and they
have some prompts uh always wants
demonstrations
the hardware they're using is a bit more
interesting so they're using a 16x16 uh
transfer processing unit version 3
and version three
why are they not using the tpu version
four well the conspirational part of me
would say that well the tbu falls are uh
busy build uh training a uh online super
intelligence or something that they
think they can use to take over the
world um
in
another hypothesis less conspirator
it's something that happens quite often
that people
have some kind of model and then they
spend half a year and even a year before
they get around to publishing it that's
certainly i think that happens um
another thing that i think is perhaps
most likely is that they simply don't
care and i'll explain why in just a
moment the time they used to train this
was uh four days
so if you currently have 256 views
running for next six hours that comes
out to around 25 000 uh gpu hours
and um
in uh
google will rent you these new hours for
two dollars meaning that
even i mean i can probably get them at
cheaper than but even then we're talking
about a price point of like um
fifty thousand dollars and for people
with 21 um uh authors i think this is
you know
peanuts really
the training costs are totally trivial
they trained it for one million steps i
couldn't find the stop criteria
being used
but yeah and there are some more details
about this lens of course interesting if
you want to
reproduce this um but
might not be interesting for us
likewise there's here's a more
description of how they uh the
deployment works in atari games um i
don't think i'll go through that
the data sets i won't go through that in
detail but one thing you'll notice is
that there are vision and language tasks
and there are control tasks and the
control tasks are in fact 85 percent
and there seems to be like a decent
spread over different tasks
and also
in the vision language mostly is a
massive text i thought that was i
noticed a um dataset called the align
dataset and i thought hey they're
actually doing alignment work and
wonderful what's the align uh dataset
and it turns out to be something with uh
you know image recognition unfortunately
and that has nothing to do with
alignment um so that's a bit sad that
people are choosing uh such a uh
misleading uh name
i don't think it's malice it's probably
just ignorance the people
working with this
just are not really aware that there's
something called ai alignment and they
are just
choosing the picture the name for all
the reasons
let's talk about the simulated control
tasks um
they have a number there is uh the atari
grid world instructions following some
physics-based simulations transparency
procedural and atari
simulation and meat environments so it
seems to me like quite well as far as i
can come
i think of course
only to use
experts uh
play throughs uh all from the best
reinforcement learners um
and only in the cases where the agent is
successful and some like much so much
revenge this is a non-trivial constraint
what does that do
my intuition is that on the average case
it probably makes it better but in the
worst case because we haven't seen
anything like that the worst case could
be worse but that's kind of an intuition
about the um
the consequences of this choice
okay let's go from simulation to reality
and staying uh red green and blue blocks
and there are two overall tasks skill
mastery and skill generalization
skill mastery is like how well can you
stack these blocks on top of each other
and skill generalization is if you have
only stacked green on blue and suddenly
you want to stack blue and green
can you figure that out
and
they're both doing this in simulation
and
in fact in reality so they do have
actual robust during this
and the uh
the episodes are running at uh 20 hertz
so um
that turns out to be uh requiring a an
end to end time of 50 milliseconds for
querying this this model um and that's a
a very very substantial constraint and
that's of course also why we need to
have such small models and i think a2
this is as far as i can tell a really
hard constraint and it's uh uh probably
something we should be really impressed
with that they are able to um to uh to
make a full turnaround on this in in 50
seconds that's uh
quite impressive as far as i can tell
and of course uh
it gives this very tough constraint
gives a um wrong picture of what this
technology can do because you could
imagine that you um just relax it like
d3 operation takes far longer to answer
and um you also get faster computers
so this constraint is something that is
that we shouldn't put too much weight on
and they have some examples of how they
how gator compares to
the state of the art and uh depending on
how you squint it is probably beyond the
state of the art in general and this is
of course something that people who
write email papers care
really a lot about we don't
from an air safety point of view it's
not so important whether it's beyond the
state of the art but like the trends
that
this
greatest kind of generality uh whether
that is useful or
not on in total on simulated control
tasks you can see a graph here
like um
there are 600 tasks
um and you can see how many of them
you can get perform as well as the uh
the the experts the state of the arts
and how many of them get like 90 and how
many get uh 75 percent um this is of
course an interesting question the way
they formulate that is if you go to the
50 threshold then 75 of the tasks can be
solved at the 50
threshold and um
i think actually it's more interesting
to see how far uh yeah
uh like if you go up to like 75 or
something like that well that seems like
even um
that's much closer to state of the art
and
it's still too fast and you can do that
or even if you go up to like requiring
90
of state of the art then you can see
okay it's still you know around 50
of the tasks that can be uh solved at 90
of the state of the eye so i think this
part of the graph is actually more
interesting than the one they highlight
here
and
also one thing that i think is
really striking from this graph is how
even the performance is
the performance curves right that
um
as you increase the requirements it very
gradually falls out how many tasks
follow this it's not like there is any
kind of cut off points or anything like
this it looks like um some very smooth
graphs um and of course what we've seen
with scaling general is
a lot of smooth graphs and that's also
what we're seeing here
and finally of course what i
would like to know in some of these is
like what is the human level um because
for some of these um it can be really
hard to
get an intuitive feel for uh like um
they just compared with state of the art
in reinforcement learning and i would
like to know like how
are these uh reinforcement journals at
the human level or far above the human
level or far below the human level
because like if the human level is in
general like um i don't know seventy
percent of the best reinforcement
learners um well then uh this becomes
far more impressive but if humans are
generally far above the best
reinforcement learners so this is a a
subhuman level um but then this becomes
much less impressive so the comparison
with the state of the art i i mean if
you are a
researcher in this specific field then
you have some kind of intuitive
idea about what is the state of the art
is it above the human level or below the
human level this is something that when
i people like me have no way of knowing
i think
this
uh these expert reinforcement learners
are in general above the human level but
the comparison is just not me and i'd
really like to know
there are some text symbols shown um
it's trained on massive texts and
yeah the colossal clean with my common
calls web crawl corpus
yeah
this c4
so this is how it's trained and it's
also true those efficient language data
sets
and then huge examples uh calling it
rudimentary
and so uh
like i would like to see some more
a thought in really comparing this like
how much better is it than gpt2 that's a
question that like they're not trying to
answer at all uh even though obviously
it's a
smaller model and it's mostly trained on
other things than language um i think um
it is in fact better than gbg2 and
that's of course an interesting thing
right they are training it on less data
and they're training it
it's a smaller model and it's mostly
trained on other things
so
it is somewhat surprising why does it in
fact perform better than gpt2 and i
think the answer is in all these small
details like the the the mapping of
words to integers and all these small
performance optimizations that are
continuously being found with the
transformer architecture and way to
implement this like there are so many
small improvements um that um
that we do in fact with a smaller model
and less strain data and less everything
gets better performance
um
but of course the exams that they show
like
it's clear it's not state-of-the-art
it's
[Music]
like i can speculate why it's not
perfect and how much
but it's not really obvious
finally i'm going to talk about
survivorship buyers like you might
as some of you might have seen this
picture before this is a picture from
the second world war where people were
observing like counting the bullet holes
on the planes that returned home and uh
put a red dot and they could see okay
there were these uh places that the
bulls were hitting so they thought
perhaps we should armor those areas and
then actually they realized afterwards
that no in fact the reason why uh they
saw this was because those were the
planes that re that did return home
so
they should armor these places instead
because when they apparently when the uh
aircraft was hit in the cockpit then the
aircraft turn
did not return home in fact
um
and so
let me uh try to use that as an analogy
for the safety work in this paper
so rowan shah
from deepmind from the mind safety team
um
is uh
commented on others people and he was
asked
did he review this and what was he
thought and he did not review this and
he believed that no one at the deepmind
team did in fact review this they would
have been happy to chat with him if they
had reached out but um
they didn't do that and uh when rowan is
reading this paper afterwards he is
looking at this and saying this doesn't
seem like something that can destroy the
world
and i think if you can imagine some kind
of self-reinforcing bad cycle in
deepmind where they build an agi by
people who obviously don't care the
least about safety
and then
they uh they start the agi and they the
agi does not destroy the world um and
then they uh write a paper based on this
api that came out not to destroy the
world they show it to the safety team
and afterwards and the safety team
afterwards and say obviously this can't
destroy the world and they're right
because they're only seeing it after it
has been published and one of them has
not destroyed the world and so they
update on this saying okay we'll see
this and uh obviously the world didn't
get destroyed so we're seeing more and
more examples of the worlds that can get
destroyed and they become more confident
and the problem of course is in the
cases where they build the agi without
caring about safety then in this case
it would never reach the safety team at
deepmind so i think um there is a
fundamental problem in deep if they have
the safety after the deployment the uh
the safety team should be involved
before the deployment
that's all i have today there will be
much more about the paper and about
safety
next time


Artificial Super Intelligence - Will we survive?

a few years ago a friend gave me a
compilation book written by Isaac Asimov
within er was the foundation trilogy the
naked Sun and iRobot each of these
novels moving further back through as a
Los fictional timeline one of the major
themes in iRobot was the free laws of
robotics which are as follows one AI
robot may not injure a human being or
through inaction allow a human being to
come to harm
- a robot must obey the orders given to
it by human beings except where such
orders would conflict with the first law
free a robot must protect its own
existence as long as such protection
does not conflict with the first or
second laws these laws seem logical
until I started reading the actual book
to my surprise much of the short stories
are about how the free laws failed to
work in many different scenarios in the
real world is there a set of ethics that
humanity has decided eh I will follow or
super intelligence have our best
interests in mind to start an
explanation or friendly and unfriendly
AI should be given when we talk about
whether an AI system will have positive
or negative influence on us most pay I
Ephesus use the terms friendly and
unfriendly friendly AI is an AI that
upholds human values and produces
positive outcomes for humanity an
unfriendly AI on the other hand is an
intelligence capable of causing great
harm to humanity an unfriendly super
intelligence would likely cause an
existential risk which means even the
extinction or the permanent suppression
of humanity at a glance it may seem easy
to think that all the AIS we design
would be friendly it would do what we
program it to and therefore if we tell
an AI system for example to maximize
human happiness and shouldn't it do just
that while this claim may seem
reasonable it may not be so simple an AI
system need not think like a human most
people make the mistake of thinking that
an AI system would reason like a human
being and giving non-human constructs
human value is called amp
for morphism in reality an AI system
would be the hyper logical and very
alien in both thinking and it's set of
ethics most say our research has agreed
that what mayor will actually be like
when one is created remains a complete
mystery for example given the command
maximize human happiness
an AI may decide that to maximally give
humans happiness it would be best to
simulate the highest possible number of
humans and to turn the earth into
combattre nyan uploading all humans into
the planet size computer and flooding
every human brain with pleasure signals
permanently leaving them in a catatonic
state while the scenario initially seems
absurd if an AI does not share our
morals and if it decides that this is
the most logical outcome of the command
given to it it will be out of our
control to stop it a super intelligent
AI would be smart enough to completely
understand humans lie and not let us
know of its plans until it is too late
to stop it the above scenario is often
confused with the Terminator scenario
but the big difference in the advent of
an ASI doomsday scenario there would be
no chance for humans the idea of a
resistance would be the same as a
chimpanzee resisting humans in a recent
poll of 550 AI experts the question was
asked what do you think the outcome of
the creation of a strong AI will be 24%
said extremely good 28% good 17% neutral
13% bad and 18%
extremely bad leaving over 48 percent of
the experts unsure neutral or bad some
of these AI experts such as steve-o
mohandro Shan League Stuart Russell and
Nick Bostrom among many other AI
researchers have come out to talk about
this future problem to the public
through novels such as super
intelligence and lectures the notion has
also been recently popularized by Elon
Musk's Stephen Hawking and Bill Gates
with Elon Musk in particular donating
ten million dollars to safe AI research
so are there any proposed solutions to
an unfriendly AI confining an AI
underground or
kind of virtual prison would not work as
any sufficiently super intelligent
system would perfectly understand human
psychology and would be capable of lying
to escape from captivity not developing
AI systems in the first place runs the
risks of other countries or terrorist
organizations developing one first which
could have even worse consequences there
are many other proposed solutions which
have been suggested but most have big
flaws in their execution the truth is
that there is currently no agreed-upon
solutions many AI experts and Ephesus
have started to work on the problem just
recently but the numbers on ER are still
small and the field of AI ethics is
still in its infancy while there are
many unknowns in their areas of AI
ethics the reason for this is because we
are still in the early days of AI
research as our knowledge of artificial
intelligence grows our understanding of
how much of a problem
AI ethics really is will be better
understood organizations such as the
machine intelligence Research Institute
or Murie for short and the future of
life Institute are doing great
preliminary work on the topic and I'll
leave some links down below so you can
check out their sites so what do you
think about the AI ethics problem will
we solve it before it's too late or zero
default outcome for humanity creating
such an AI system leave a comment below
of your thoughts and thanks for watching


198. Language Models are Few Shot Learners 1

hello and welcome to session 198
in the ai safety reading group tonight
we'll be discussing the first half of
the article
language models are few shot learners by
tom b brown
dario ammoday and many others
this article is uh published by
openai and the number of people who are
working there
it is dario moday who has designed and
led the research tom b
brown benjamin mann nick ryder and
melanie sugar
have contributed equally this is this
normally means that those are the
four primary authors but there are a lot
of
supporting people actually there are 31
authors in total so i um
i'll let you go through all the names
and backgrounds i hope that's reasonable
um and this is a paper that was
published on the
at the end of may this year and can be
summarized as
the fact that scaling up language models
greatly improves
task agnostic few shot performance
and i will be describing the results of
this paper
but also please keep in mind that i am
no domain expert in this
so i might from time to time say things
that are incorrect
also we'll be focusing on section one
and two
in this part so the first
background question is natural language
processing
what is that well this
uh general subfield of computer science
deals with the interaction between
humans the
the language that humans use and
computers
so there's uh as part of it which is
like
low level like uh
[Music]
speech recognition and speech synthesis
and things like that
but we're mostly talking about the the
higher level
where there are meaning to the words and
in this part of natural language
processing
we have things like uh automatic
summarization dialogue systems like you
can see this
gift shop here machine translation
many other things
and if you go to an even higher level
above all these
then there are two major tasks one is
for
the computer to understand natural
language and the other one
is for the computer to generate natural
language
you can actually go even higher at the
very very highest level
natural language processing is simply
predict the next word that a human would
say
that would indeed solve all these
subtext
subtasks if you could do that um
but but this is generally considered you
know
too high level or i would uh if i should
quote myself on
my thoughts before i heard about all
this that would have been
that sure this predicting in the next
word that's
an interesting subtask but not really
relevant right
because um if you have a task
like translating then
obviously you can exploit some of the
structure of this problem
right if you have the problem of
translating a text
from english to french then a dictionary
that translates
words from english to french is
obviously going to be to be
really really relevant right so
in in this way this um
the the claim that it's uh
just predicting the next word is enough
seems really really counterintuitive
so let's have an here is a very very
simple ex example of this task i have a
pen
dot i have a and then apple pen
read or hello if you have to choose
between these most people would say that
apple is probably the correct next word
so what was the state of the art in 2014
according to this paper well um oh yeah
i
couldn't resist just sniping a bit at
brian thomas
who in 2014 um
argued that we are unlikely to see um
uh dramatic improvements on this kind of
thing
through algorithmic insights because
there's not so much precedence for this
and
um as we'll see there has indeed been
quite a bit
so um how would this have been done
we'll have
in 2014 we would have been using rather
simple
neural networks um using things like
word vectors where here we have a number
of
uh animals and then some dimensions like
are they domesticated are they fluffy
and then we assign numbers to those in a
word vector
in 2015 things got better we are using
recurrent neural networks with
multiple layers that feed back into
themselves
and we're using a some
contextual state for instance this is
called
long short term memory where the
the previous things that have happened
are fed into the neural network
here you can see this this box zoomed in
it actually looks like this and this
was quite a breakthrough this
algorithmic progress
here according to wikipedia this was
used
really really successfully
now in 2017 even more progress happened
and this was where google grain
which is you know kind of a competitive
to uh
um to open ai they had a new um
architecture called the transformer the
paper is called
attention is all you need and there's a
link to it here
and um this is what the uh
transformer looks like so as you can see
it's a somewhat
more complex um architecture
um there are there are more links and
and things that
do other things than just the basic
neural network
but fundamentally these are neural
networks
so it's basically still neural network
and
the key thing as you can you might be
able to tell from the the title of the
paper
is that you can actually do away with
all this recurrence
and also something called convolutions
uh
if you just use the attention concepts
so transformers
are built exclusively on attention
the good thing about this is when you
don't need to feed
in the results from uh from just before
then you can do it in parallel and
the fact that the transformer
architecture is parallelizable
whereas recurrent neural networks are
not um
is in fact a very big deal however
in 2017 this was still uh something that
was used in a task specific
way in 2018
people started realizing that you could
improve this somewhat
by having some pre-training done there
were also a number of improvements to
the
architecture throughout this there are
small tweaks
and extra features being added to the
architecture so
it's incrementally improving but in
aggregate
the architecture is improving a lot and
in this way
the architecture can actually be the
same whether you're doing
um classification or entailment
similarity or multiple choice
you actually end up with the same um
with the same architecture
but unfortunately um in 2018
you still needed uh task specific data
sets
and you still needed to fine-tune uh
your
um your model against
the task task at hand um
and because you can do uh you can
basically use the same architecture
um then um this is a substantial
step towards making this practical but
the problem is
that if we need task specific data sets
then usually we have only very small
data sets
and if you have a really really strong
model that can model
just about anything and then you have a
really really narrow training set of
training examples then you are very
likely
to have your model find correlations
that don't actually exist in the real
world
spurious correlations they call them so
this is a big problem
in 2018. in 2019
openai introduced the generative
pre-trained transformer to
the tbt-2 in the in the paper called
language models are unsupervised
multitask learners
in this they actually dispense with
everything that is
task specific and they compensate for
that
by using a huge huge transformer with
1.5 billion parameters
uh so is that precisely the same as
saying that there are
uh 1.5 billion nodes in the neural
network
but you can imagine roughly that order
of magnitude
the uh alec redwin in this paper
actually in the abstract
he notes the following increasing
capacity of the language model improved
performance
in a long linear fashion across tasks
and there is an old rationalist
virtue called more daca which means that
this is three muscles who pointed that
if you have something that appears to be
working then it's not a question of
what if you would want to scale it up
but you actually need a really good
reason to not scale it up
if it seems like that is something that
would work and that's something
of course that um that is in fact
counterintuitive to a lot of people
um but apparently not to the people who
built gpt too because they built
gpt 3 the generative pre-trained
transformer
version 3 of dash 3.
so let's talk about that and talk about
fine-tuning as well so what they did is
basically
they took the uh same architecture as
gbt
ii again with these tiny variations in
the architecture
that um that are always being being
found
and they they basically scaled it up
gave it a much larger
transformer a much greater amount of
data
both in how many examples and how much
diversity and then they trained it for
a lot longer um
so that's basically the thing they did
but in this paper they also
uh explore um different settings
for how this model can uh
perform in particular they um have a
spectrum
on uh how much fine tuning or task
specific things
to use so at the uh at one end
we have fine tuning which is um you have
a
data set that is supervised where humans
have looked into it and said
this is an example of what we want um
and this is something
you could in theory do it with gbt3 they
haven't done it
but they think that would be promising
you can do future
learning where there are if you get a
few demonstrations um but um
uh when you actually need to juice it
but you don't
change the model itself when you give it
a fuse
a few examples um in gt3 they have what
is called a context
window how many examples that are
possible
and with the size of model they can have
like
10 to 100 um examples at most
uh more than that is out of these
the attention of the model and
there's a one-shot learning where
there's one demonstration
and then also a natural language
description of the task
this is of what humans use and then
there's a serial shot
which is just giving a task
without any examples and this is
something that a lot of humans struggle
with
in a lot of contexts um so um
that's of course on the other end of the
spectrum um
and in the paper it's claimed that one
shot is the first comparison to human
performance
and that might even be that might be
true but
i think in general people don't really
care about fairness
i can see it from of course a
theoretical point of view that's
interesting
but i think in practice uh the flu shot
is
far more interesting because uh if
if you have the resources to do it one
shot then you almost certainly have the
resources to do
to do a few demonstrations
so what training dataset did they use on
recall or
i don't know maybe not recall but gpt 2
was trained on
40 gigabytes of data which is quite a
lot
um but for this they use the common
crawl data set
which is basically everything that has
ever been written on the internet
1 trillion words which is 45 terabytes
compressed text they
did some filtering on this so just to
pair it down to a
half a terabyte or something like that
um
and they are really you know cool about
this uh
they just throw out 99 of the data
because there is so much
so if it doesn't you know look like a
sentence in some primitive way then they
just throw away because there is
so much data and um
you might want to compare this to the
concept of
data overhang an analogy to a hardware
overhang
from boston's book super intelligence
this training data set was certainly
sufficient
even the very largest model in gt3
never had to be to use the same sentence
uh two times because there is just so
incredibly much data available um
so
one of the things that are different
also in the training compared to what is
normally done
is that they have a larger model that is
strange on
trained on fewer tokens compared to what
is typical in machine learning
there were an interesting bug with
obviously one of the few things that you
don't want to train
this uh dbt 3 on is the set
of solutions to standard tests
because if you train it on the solutions
to the standard text
then it might just look that up another
force cheating um so they tried to avoid
that but actually they failed a bit and
there's a
discussion about how they tried to get
around this like
it cost quite a bit to uh to train this
model so they couldn't just repeat it
but this training was provided by
microsoft who i believe is a pretty
major sponsor of
open air so
how did they do the evaluation well they
took and k
which is like a number like uh require
depending on how much is required for
the test uh for conditioning
and uh because they did this on a huge
amount of different
tests then the way that this evaluation
actually has a lot of um of variations
and details
um and um in some cases
for some of these benchmarks then the
actual results are not available as
something you can download
you can only query a server through a
web service
to figure out what is is my
generation actually correct
so the x the model that's called tpt
three
has a number of parameters uh well hyper
parameters
with how many how large is the neural
network how many layers
and there were these attention heads etc
and
these are the hyper parameters that are
being used
um and one of the things that
they they note is that these numbers
down here
actually don't matter very much of
course the number of parameters matter
but it's not really strongly sensitive
to how many heads you have etc
um i think that's that's interesting
because i'm
both interested in the things that
matter and the things that don't matter
and it appears
obviously that having a huge amount of
compute matters a lot
the transformer architecture matters a
lot things like
um incremental tuning and thing the fact
that this doesn't matter
i actually also find it's quite
interesting
now let's talk a bit about how dbt 3
learns and they're using something
called the the authors called
meter learning which is where the model
develops
and it learns how to learn have a set of
skills and patterns um that it
has a training time and then it uses
these
when it's a time for for the test for
interference
um to either directly recognize
the the task or adapt to that
so the way this works during training
here is
there is an outer loop and an inner loop
so on the outer loop
it's using you know rather standard
stochastic gradient descent
um in in in this pre-training
and then the inner loop it is learning
something
uh things in specific contexts like
additions or
uh flipping letters or
translations and things like that and in
in these
while it is learning um
it gets results that are really really
poor compared
to uh to something that is fine-tuned
but but it does this a lot and it learns
uh a huge amount of um
of these uh uh skills that that it will
later be able to
um to put into use
so let's see for the actual uh result
here
now um this is the um
accuracy on a large number of benchmarks
some standard and not so standard
benchmarks for
how good is this uh model at
um doing things that humans can do
and um as we can see there are a lot of
things
that um and down here we have the uh
the size of the neural network basically
the number of parameters
and as you can see there is a trend
going upwards
both with a serial shot learning
one-shot learning
and few shot learning it does seem to
somewhat increase and
on aggregate whenever you increase the
size of the model
you get a substantial performance
improvement
and um in particular one of the things
you can
directly measure from all this is what's
called what they call
luck loss um and um
this is uh something that's used in
machine learning and and
this is a a measure for how good your
predictions are
and according to this um
lock loss your performance increases
quite linearly with how um
how good your your guesses are so let me
just
quickly give a hint of what log loss is
like if you um have an algorithm that
assigns probability
one that is 100 certain of something
that turns out to be the truth
then you want a loss that should be zero
in order to train your
algorithm and if you assign really low
probability something that is the truth
like
0.01 or something like that then you
want a huge
loss and the mathematical function that
satisfies this is the negative log of
probability
in the article they call it just log
last they mean the negative log
of course but yeah
so this is how it looks when you scale
up to a 13 billion parameters
how about in gv3 d3 when you scale up to
175 billion parameters
well you get this and this is pretty uh
uh broad um improvements
on basically all the tasks um
one of the things that i noted in
particular
is just how many of the tasks that just
go a number of them go straight up to
100
so it means that um i haven't looked
into which does this
but apparently a number of tasks just
uh have per have human level performance
and you know there seems to be a lot of
things where
um the tasks were basically impossible
to solve
at 13 billion parameters and you know
some of them are
go go really far up that seems to be you
know not solved
but uh really improvements are made by
going to
from 13 to 175
this might not be that uh unexpected
because
the the learning uh that you do in
pre-training
involves a lot of very varied skills
and the more things you have to put into
the model
the more difficult it becomes but if the
model is larger it can contain
more skills and that that's why
you should kind of expect that this kind
of meter learning is something that
improves dramatically with scale
so let's go into one particular line
from this
the uh from one particular test which is
uh removing extra characters so let's
hear
that task in particular this you have a
word like succession here um and then
you add air on every other
spot you add a random character and then
um you'll see if the ai is capable of
filtering that out
and here you can see that basically um
with 1.3 parameters it basically can't
figure it out
it gets slightly better if it hasn't
some more examples
10 examples here one example here
and you can see um with 175
billion parameters it just performs
really really
really well even without a prompt
even if it's just given the word
succession and
then it can indeed um figure out the
next word where it should be in
succession which is
you know kind of impressive
so in this example uh dairy amadeus
admits this is a particular striking one
but it is engine
a representative of a general trend
i feel this is not one of the
a really good example for talking about
natural language processing
in that this is something where you
could
you know just removing every other
character is something that could be
done by a really really simple
program and um
this specific task of removing
every other language is something that
is so simple that it might
uh that might you might expect it to be
contained into the model
at some point so that it would just be
able to you know
solve it with 100 every time
so what's the conclusion from all this
well on all these tasks we get
some really promising results uh
sometimes even competitive with
the state of the art by fine tuning so
this is some really impressive results
and it seems indeed that
the more er
the larger the model is the greater the
difference between
zero one and few shot performances uh
so this could be argued to mean that
when the model is larger it has better
meter learning
they're not really willing to strongly
conclude this
but i think this
this gives a fairly good indication that
this might be true
and indeed the title of the paper that
language models are few shot learners
is also hinting of a very general
conclusion
so one of the things that i think would
be interesting to discuss
is last week or two weeks ago we talked
about
we saw bengal finkel saying the
following that if an ai has the
capability of understanding natural
language
it will be using this understanding for
the interpretation of its goal and
uh if we uh look at gbt3 then this is an
example where
at one level it's obviously false
because tpt3
has precisely the goal of predicting the
next word
it doesn't um
thing it doesn't um use common sense
in figuring out if this should be its
goal it just has this goal
but it has um common sense i would even
call it
a moderately rich model of human
cognition
because this is something that can be
used for predicting
what the next few words that a human
would be you would use
um and i think this is an interesting we
will probably get more clarity on this
question
in the rest of the paper um and another
thing that i would
that i also couldn't stop thinking about
is that if you have something that has a
really rich
model of human cognition like qt3 can
understand
if sentiments like that a human is angry
or sad
or the this text is written by a sad
person or something like that
and if you have something that
understands sentiments really really
well
then you have the feeling that well it
might
not just understand what uh sadness
means but actually be able to feel
sadness
i think um i think that's unlikely but i
am
not really 100 confident in this and i'm
not really sure
how you can have 100 confidence in that
conclusion
um but i expect so a lot of people will
disagree with that
that is all for today thank you and see
you next week