Edouard Harris - Emerging problems in machine learning: Making AI “good”

hey everybody welcome back to the
podcast i hope you're doing well um my
name course is jeremy i'm the host of
the tourist data science podcast and i'm
also on the team
over the sharpest minds data science
mentorship program and
we actually took a a pause a break last
week we didn't release an episode i
wanted to open this up
just by explaining why we paused and
what we're doing
a little bit differently going forward
so first off the reason for the positive
we're working on the next series of
episodes
those episodes are going to be focused a
little bit more on questions surrounding
how we deploy
ai systems for the benefit of humanity
as our ai systems have become more and
more powerful
increasingly it's become important to
start asking questions like
should we actually deploy an ai system
to solve this particular problem
or what could go wrong if we start to
deploy increasingly powerful
ai systems in this sub-domain there are
all kinds of questions
around safety both in the short term
where we start asking questions like you
know could an ai system
accidentally recommend a course of
action that might lead to
harm to human beings or damage to
economies
all the way to more fundamental and
perhaps more concerning long-term
questions
like rai systems actually reflecting
human values
shaping humanity into a form that we
want and our ai systems
actually safe from a more existential
standpoint is there the possibility
that an arbitrarily intelligent system
might have
implications for the continued existence
of humanity itself that we might want to
consider
sooner rather than later and so that's
going to be the focus going forward
a lot of questions surrounding ethics
bias in ai systems
and the should we shouldn't we questions
around distribution deployment
so i'm really excited to dive into this
whole series we have such great guests
lined up
and it was so hard to pick one to start
that i ultimately ended up sitting back
and thinking about the person who i knew
who i'd be most comfortable with talking
about
all these topics not just ai ethics not
just ai bias not just the technical
details
of ai alignment but somebody who cover
all those bases with me and i landed on
my brother ed who's been on the podcast
before
he is a co-founder with me of sharpest
minds
and he's actually got a whole bunch of
experience on technical alignment work
as well as startup work seeing
algorithms deployed in the wild
and because of his breadth of experience
and depth of focus in the ei safety area
i really decided he'd be the best person
for me to open things up with so i hope
you enjoyed this conversation
it's going to be a little bit more off
beat than some of the ones we've had in
the past
but if you have any feedback let me know
and i just can't wait to release the
next couple episodes
they were exploring a really exciting
topic here and it'll be great to share
these thoughts with the community and
get your feedback as well on how we're
doing
so without any further ado please enjoy
the show
hi ed thanks for joining me for the
podcast thanks so much for having me
appreciate it
oh yeah no problem at all um so jeez i
think there are so many places we could
start with this
uh ai is a really big space and
questions around how ai should be used
how it should be secured
are starting to pop up obviously that's
the theme of this entire series of the
podcast and what we'll be focusing on
for the next few episodes
um i guess maybe to begin with
we could talk about so many different
things but you're more focused
on this idea of ai alignment getting
machine learning models to do what we
want them to do
in different forms making sure that
their performance isn't uh dangerous
essentially to human beings and that it
optimizes in a direction that we want
can you can you introduce the
idea the field of ai alignment like why
should we
be spending time and effort focusing on
aligning our machine learning algorithms
yeah so this is something that i have
been thinking about more recently
as systems have gotten more powerful and
as we're seeing the advanced from year
to year
the more powerful the systems get the
more important it is that they are
doing the things that we want and
alignment is important because it's not
always
obvious that a system
is going to be doing the thing that we
want and it's not always obvious that
even if the system
starts out doing like what it looks like
what we want
that it won't end up doing something
really bad so a good example
is uh social media feeds um algorithms
that
keep you coming back to twitter and all
of this stuff maybe you think about
what is it that twitter the company
wants from you like
you know it's uh it's money but like
it's pretty benign right they they just
want you to click more
and and like yeah so they're gonna you
know to at the first level
you think of is like yeah the algorithm
is going to give you uh
posts and tweets that you're gonna click
on more so you're gonna
stay there and see more ads that's like
at the first level
but the problem is that if you train a
general system to do this
one of the things that the system will
discover
is that hey i actually from doing this
kind of testing myself on the people
that i'm showing these tweets to
i can discover that i can actually make
human beings
more predictable by showing them certain
kinds of content
and one of the ways that uh that that
that these algorithms are making us more
predictable to them is like
a politicization of content if if i'm if
i'm more politicized
uh i'm my cliques are more predictable
than they are when my political position
is at the center so one of the things
that's actually happening is that
these algorithms purely through the act
of trying to make more money
are actually um pushing people to
different ends of all these different
kinds of spectrums different political
issues and all of that
purely because they're like hey i'm
trying to make you into
something that's easier for me to
predict and people who are more extreme
politically
are just like more predictable because
they tend to have a lot of correlated
views
so this is one of the the ways in which
like
it's actually kind of scary and you can
see how it like creeps up on you because
this happened over years and years
but you can imagine this like this is
already having an effect on the world
and as these systems become smarter and
smarter we should
expect more similar things to be happen
to just begin to happen
right okay so the primary problem here
is we have algorithms that sometimes
are too clever in the sense that if we
don't think
very very carefully about what we want
them the kind of world we want them to
create for us long term
they'll find solutions that we've never
even imagined
could be solutions to the problem like
if the if the solution
or the problem rather is predict what
i'm going to click on so that i can
you know generate more engaging content
than really like
changing my users mindset to make them
more predictable
to turn the political spectrum into more
of a binary so it's a smaller
dimensional problem and easier to
dimensionality reduce um is that that
sort of becomes like
its own pathology right i mean that
that's at the core of this right
yeah and it's uh it's the sort of thing
that again creeps up on you you can
give a system a set of instructions that
you think are perfectly benign
and totally makes sense for you make me
more money with more clicks okay you
know
pretty benign um and it's only with the
benefit of hindsight really that we can
look back and
say like oh my god if only we'd known
at the time that this was the force we
were unleashing upon the world
this you know we might have done things
differently but uh
at the time it was not possible to
predict that
uh that this would be the outcome and
this is the problem is that
uh when you're dealing with something
that's smarter than you are
you pretty much by definition you can't
predict
what it's going to do because it's
smarter than you are um
if it if if you were smarter than it
you'd
you'd be able to predict it but because
it's smarter than you are it can predict
you and not the other way around
and i guess it doesn't even necessarily
really i mean i think this is one of the
problems in
when you're talking about ai safety you
know what is smartness what what does
intelligence mean i think these terms
are sort of ill-defined
um there are certainly senses in which
it would be very hard to
argue that the twitter algorithm is
smarter than a human um
other senses in which narrowly
interpreted you could say it is smarter
than a human but
there's definitely a threshold where the
two kind of start to interact and
compete
and one algorithm starts to outstrip the
other the twitter algorithm starts to
outstrip your human brain and starts to
change you
more than you change it right yeah i
would say that um
i would say that algorithms can be
smarter than us currently in narrow ways
but we should be concerned about the
possibility that they may become smarter
than us
in more general ways and of course there
are other things to be concerned about
before even that point is reached okay
so yeah let's let's get to because i
think we'll get to the
the general concern about general
intelligence and where these more and
more advanced systems might go
in the shorter term you know you
mentioned twitter um
obviously a lot of people have talked
about uh things like ai ethics and ai
bias in ai systems right
and this is i think a real theme of the
day because we see it around us we can
see systems
that operate in ways that have biases
that will surprise us in many cases
they seem to reflect the way that the
world currently is
with its current failures failure modes
and it'll tend to reinforce those
failure modes because it's been trained
on that data and it'll make predictions
that kind of mirror that data um is that
uh
is that a like i guess you're more
concerned about the long-termism side of
things but
could that be an issue for the long term
as well
uh to an extent so one of the things
about the kinds of failure modes and
like bias in ai
is that in a manner of speaking this is
the sort of thing that is happening
to an extent because these systems are a
little bit too dumb
um they haven't been trained in general
enough ways um
the the further in the future risks
and probably eventually the bigger risks
are going to be what happens when
systems are too smart but
the risks the the risks of bias are
definitely real
now today and there are current um these
bias risks are kind of a turbocharged
version of uh the kinds of
of what you essentially get when um you
know when people build software
uh they'll generally build for
themselves they'll
test the flow of clicks and and things
that they
naturally expect other people to do
they're going
to like everyone is egocentric right so
we all
basically build for ourselves um and and
that's perfectly fine but
what can sometimes happen when the data
that you collect
is uh biased in a particular way is like
it's not the algorithm's fault the
algorithm is just
running on the data that's been given
and it can just like totally forget that
oh you know there's
uh there's there's a bunch of people
that like uh don't have
names written in uh latin characters or
that uh you know don't have a particular
skin color or whatever
and so uh you actually can get of course
these situations where
an algorithm is very bad at dealing with
cases
that the developers didn't think of and
the more power again we give to these
systems
the more costly those mistakes become so
i would say that's currently the status
of
bias and ai right and i guess there's
one i mean people talk a lot about data
set bias that gets a lot of attention
i think one other really important
aspect is feature selection bias
because when if you think about it like
human beings collect features about
the rest of the world as we navigate our
environment those features typically are
things like sound waves
they're things like smells there's
things like vision and touch
those are the features that evolution
has engineered for us
because we see the world through that
lens we are unable to notice certain
things these are things for example like
the neutrinos that fly right through our
bodies all the time that we're
totally um insensitive to and and yet
that account for a you know a large or
not unconsiderate fraction
of the amount of energy that's kind of
that we're bathed in any given moment of
time
so i don't think it's an exaggeration to
say that our environment goes mostly
unseen
and even the things that we could in
principle see were so myopically focused
on one like
tiny fraction of our visual field that
we don't take in you know the
overwhelming majority of the information
around us
when it comes to our machines i think
you know we're doing something similar
we select features
when you tell for example if you were to
tell a um a credit card
uh pricing algorithm like the um the
name
age uh occupation and skin color
of a person you've just caused it to
look at the world through a certain lens
and that biases it to find certain
correlations
more easily than others or to fold
complexities and nuances
into one high-level core strain feature
um like is is that something that uh
that you
see as well as an issue or do you think
that the um the data set bias is a
bigger problem
so what i would say there is that uh
there are
two kinds of biases that
can arise from feature selection the
first
is an omission bias where you don't give
the system
an informative feature like you give the
system a feature that is not
informative with respect to the
conclusion
the second is more like
like labeling bias and so in the event
where
uh you give a system you know the skin
color
of someone for a credit card assessment
um if you
if you were actually training the system
on like
a truly you know ground truth
informative data set
in theory that skin color thing
shouldn't end up mattering like if you
have a perfectly general
intelligence that's running on that
system it's going to
uh ignore skin color at the extent that
skin color needs to be ignored
but the problem is that when you have um
when you're training that system
on labels that have been put there by
people who themselves have biases which
you know we all have
then that bias itself gets induced onto
the system
and this is the sort of thing that like
you know one encounters
uh in terms of uh potentially
like an algorithm delivering a verdict
for example if it turns out that
the verdicts that it's seen before are
delivered by
you know there's a significant sample of
those verdicts that are biased against
you know who knows the height or the
skin color or the hair color or whatever
of the person
then you know this again the system it's
garbage and garbage out
the system is going to learn what you
teach it and so uh it's going to have
the same
exact biases induced on it so i guess
that's interesting i i mean i guess i
see
the case of biased data as one part of
this
i don't think it's the case that um you
know
if you i mean you talked about a
sufficiently generally intelligent
program that would look at this
information including skin color and so
on and then go about
drawing the right kinds of conclusions
um i don't think that's actually the
case i think if you
um if unless that system has access to a
fuller set of features
it will tend to see things myopically
through the particular lens that it's
given on the world
and sometimes that lens even if that
data set isn't biased even if
it detects just what's happening in the
world um you know if
we talk about skin color or we talk
about gender or whatever there are
correlations between skin color
and tendency to pay pay down debts or
whatever
um and those are reflected in you know
the biases
if you will of a poorly engineered
system but
what's really going on under the surface
is if you if you kind of part that out
if you control for other variables all
of a sudden
we expect i don't know that anyone's
actually done this but i would highly
expect personally that skin color would
would iron out of that equation but only
if you control for these other variables
so i think it's only to the extent that
you add to that feature set
that you can actually de-bias the system
in that way yes
yes this is correct um the this this is
basically
this is i think another way of looking
at uh what i was saying in terms of
their
sins of omission that you can make so to
the extent that you have
given the system enough enough ground
truth data
to actually break down those variables
that that ultimately make
these uh these these like gender and
other
uh these confounding features i guess uh
irrelevant
to the extent you've given the system
that extent of data
uh it will it will learn that those
variables are irrelevant
um but uh but yes if you if you make
those omissions
may actually learn on variables and
exhibit a bias as a result
yeah i think it's it's always
interesting as well that the extent to
which all this challenges too are our
notion of free will or
maybe illusion of free will because i
can imagine as well
if you take this process to its limit
and you keep fine-graining and adding
more features and adding more features
eventually your model becomes so
sophisticated
they can account for i mean almost like
in the absurd limit the firing patterns
of every neuron in your brain
and then you can predict your behavior
with super high levels of certainty and
all your
agency kind of disappears um i mean i i
see that actually kind of as an
interesting
part of the journey we're going to have
to make as systems become more
sophisticated
uh i think i think they're that that's
true to an extent but uh there's a limit
to that
like there's so it's the same as
predicting the weather
um systems that have like even even if
you're not thinking about oh there's
quantum mechanical uncertainty on top of
everything
even if you're not thinking about that
um classical systems that have chaotic
properties
have like you can't really predict them
beyond a certain number of time steps
the way it works is that
i think if i recall correctly uh it's
something like
your however much accuracy you have
in the beginning um you
the accuracy that you have uh later
like goes down as one over like the
square root
of the time so basically after a certain
period of time
even if you have a perfectly
deterministic model of how stuff's going
to happen
if your accuracy of measurement at the
beginning
basically propagates and gets much much
worse over time so
there will always be a limit to the
extent to which very intelligent systems
are able to predict the future
uh at least that's our our best current
understanding now
uh but you don't need to be able to
predict the movement of every single
atom to be able to do
a lot of very instrumentally effective
things that could be dangerous
yeah and i think there's actually maybe
the
the sidebar to this is there's there's
already been a story in the uk about
um the government trying to not the
government sorry i think a university
trying to predict students test scores
given that they weren't able for
whatever reason i think it might have
been coveted related
they weren't able to actually write the
test right and so you've got
students who you might say arguably are
rightly upset
that their agency has been stripped away
um in this case it seemed as though the
algorithm
was non-performative it overstated as
often machine learning models do
nowadays but it overstated its accuracy
um or downplayed its uncertainty however
um i think there's an interesting
question as to okay well what if that
hadn't been the case
what if the algorithm had been super
performative
um where would we be there and
yeah well uh if you think of it
in a certain way the test itself
is an uncertain evaluation
of your own like it's a noisy evaluation
of your own skill
and so if you have a good day on the
test you should be happier
like you've been you've gotten like an
extra bonus whereas if you just had a
bad day had a bad night's sleep whatever
it is some sort of mix of stuff
you maybe you should be rightly
indignant that this test didn't
correctly assess your level so
the i think that ultimately it's it's
it might be more of a semantic and
comfort zone issue than a real issue
if you have a perfectly unbiased
algorithm with a known uncertainty so if
your algorithm is
unbiased and well calibrated because if
it's unbiased and well calibrated
then i can at least look at my test
score or my predicted test score
and say well there's a 50 50 chance
i should be mad about this and a 50 50
chance i should be happy about it
on the other hand we all have a natural
discomfort
in having a computer uh judge our
futures
so i don't know that we as a
civilization will ever get over that
necessarily right which i guess starts
to invite some
some of the parts of this conversation
that i think are more forward-looking
um so as we as we keep developing this
technology you alluded to this earlier
the more powerful these systems get the
more important it's going to become for
us to know
exactly what we want and this idea of
being able to build systems
that exactly what we want being able to
communicate that to them being able to
make sure that they're actually
doing exactly what we've asked them to
do is known as ai alignment
um what i'd like to do is ask you can
can you describe ai alignment serve in
your own words and then
um maybe can you provide a description
overall like
maybe the outer alignment versus inner
alignment problems as well just so
people are
a little bit more familiar with them
yeah so roughly speaking
the problem of ai alignment is the
problem of getting an ai
to do what we truly
want it to do there is a lot of detail
in that description and even a big part
of the alignment problem that's still
open
is figuring out exactly how much detail
there is
left to figure out in that description
but uh right now people are
people tend to break down alignment into
two sub
questions like you said there's outer
alignment and inner alignment
so the first problem is getting
uh getting the system getting this this
big ai
to actually try to do
what we want it to do and so this
you can think about as uh the problem of
i found a magic lamp i rub the lamp
and a genie comes out it's not the genie
from aladdin it's not like this
you know really like nice genie who's
like trying to be helpful and stuff
it's a it's a genie that like will try
really hard
to misinterpret your wish as hard as
possible
and the genie is also much smarter than
you so what you have to do
is you have to figure out a way to word
your wish
so that you absolutely pin down that
genie
so they absolutely cannot misinterpret
your wish in
any meaningful way and they're like ugh
i guess i'm gonna do the thing
this thing because like you've left me
no other choice so
so this is interesting because you
immediately frame this as an adversarial
thing which i think is interesting uh in
and of itself
but the implication is that this machine
is trying this machine this ai or this
genie is trying to misinterpret what
we're saying
which i don't you know is not the case
but it seems it does seem
at least as somebody who's been in the
space and worked with you on a lot of
the stuff
it definitely does seem as though in
practice you have to it's almost like
defensive driving
you have to assume because this thing is
so powerful
so much more intelligent than you are if
if you're building super powerful
uh super intelligent ai systems that
it's going to find a solution to
whatever problem you presented with
that is going to be way too clever and
that will involve
doing things that you can't even imagine
right like is that is that where this
adversarial implication comes from
uh to an extent yes it's i think the
simplest way to put it is
that there are way more ways to
misinterpret a wish
than there are to correctly interpret a
wish and the smarter you
are the more ways to misinterpret the
wish
you're going to be able to think of so
the genie that you're facing
is able to think of a lot of ways to
misinterpret your wish
whereas you're like no no i want you to
interpret it correctly
in one of these so can you give an
example of a wish actually that might
help flesh this out
yeah yeah uh let's say uh
i wish to i
like something simple like i wish for
uh i wish to be happy or something like
that um
okay like you wish to be happy if you're
facing a person
who really wants to be helpful and and
who has all the constraints of a person
you know they'll ask you like what do
you want like i'll make you a cup of
coffee whatever you know makes you happy
in the moment
um but if you're facing a super
intelligent genie
uh what the genie is gonna say is like
first it's gonna be like okay well
you know how do you define happiness
there's a lot of fuzziness there um
is it that i'm like smiling all the time
well
if that's how the genie thinks what
happiness is it's gonna like
you know grab onto my face and like
force me to smile forever and
great it's done its job but i'm like no
no like don't
don't don't do that please stop but but
that
the point is it's not listening it
doesn't want to listen i've already told
it uh make me make me happy
it thinks that making me happy means
making me smile
and more or any number of other things
right
it could be juicing you up on drugs it
could be any any yeah
yeah like oh like then i'll just inject
you with cocaine or like uh like i'll
i'll drill into your skull pull out your
brain and like drop you in a vat of
endorphins and like that'll do it or
whatever it is
uh but the problem is once you've given
the genie that very first directive
from that point it also has the
incentive to prevent a change in that
directive because
it's going to like it doesn't want once
it has that directive
it's got this goal and it's going to try
to do everything it can to accomplish
that goal
one of the things that will reduce the
chance
of that goal being accomplished is if
the genie's own goal is
changed by you afterwards so if you
are able to tell the genie oh actually
no no like
don't do that like smiley face thing
that's like really scary please don't do
that
that reduces the chance of you ending up
being
happy and forcibly smiled uh and so the
genie
before you tell it stop smiling but
after you told it
like make me smile is gonna be like okay
now that i've gotten this directive i
have to like
prevent my directive from being changed
defend myself
against any attempt to change it
including by the guy
who just asked me to do this thing so
like
i better just like you know force this
guy to stop talking or like
just quickly kill him and uh make his
face smiley anyway
and like that's that's happy or some
some crazy stuff like that
um this is the level of
misinterpretation that
we're talking about it's why it's
dangerous yeah a lot of this reminds me
of
you know the early days of or the early
days
my first interaction with computers
right i think everybody experiences this
when you first start to code
you realize that man this machine is
taking me
absolutely literally like i tell it to
do something
and i mean i remember having this
frustration you write some code that you
think should work
you think it should work and then you
hit run and then it breaks and there's
an error message and and then you
what do you do get frustrated not with
yourself but with the machine
because your brain is telling you oh the
machine's screwed up i know what i meant
but the machine the machine
misinterpreted it and
i think that there's a similar kind of
um
error that gets made when people look at
ai and start to think of it
as if it's going to be default positive
as an outcome there's a lot of
anthropomorphizing that people do where
they imagine like
oh it'll you know but it'll get that i
don't mean like
this horrible way of achieving the thing
that i just asked it to do or
you know i won't have a genie take me
literally is the assumption
and anyway it just it strikes me that
that that really does reflect like my
own attitude when i was starting to
learn how to program it's like a totally
understandable
way of approaching these problems yeah
the machine
has no sympathy intrinsically on
except to the extent that we give it
sympathy and
we don't understand sympathy well enough
to give a machine sympathy
so that's like and that's that's one
part of the problem
but it's like it's a part of the problem
right okay
so we've established that it's not going
to be trivial
to train this uh this advanced ai
in order to get it to do what well what
we want i mean that's the other thing
right like
when you tell a machine i would like
i i would like to be happy um there's so
many things that get rolled into that i
mean philosophers have been trying to
figure out what it means to be happy for
thousands of years with no success
and now we're going to try to quantify
that complete uncertainty
and feed it to machines i guess that's
hint to get another part of the problem
here
but suppose we got past that and this is
what
you were saying refer to as the outer
alignment problem
um what so what's the the next the next
layer of difficulty as as hard as that
was
what's the next piece here yeah so the
next piece is
what's called the inner alignment
problem and this
is more like the genie you're talking to
might be made up of a lot of internal
components
and those internal components might be
working
against the interests of the genie so
this is kind of hard to picture but
maybe a good way of thinking about this
kind of anthropomorphically is
you think of a company let's say that
instead of a genie you know you were
making a request to a company you know
make me happy and the company charges
you money or whatever
make you happy and something like that
so the company
might well you know have a pathological
tendency to make you happy or whatever
like like twitter does i mean i love
twitter but it does have this uh
this algorithm problem but what's
interesting about a company is that a
company is made up of
people that all are
partly aligned with the goals of the
company uh the people work together to
accomplish something that more than
what any individual person in the
company could have done but they're not
all rowing exactly the same direction
there are
uh people inside the company that are
more aligned than others some people
really deeply believe in the company's
mission and they'll put in like all the
hours they need
they're ambitious in the same direction
all that other people are
have a variety of different motivations
they just want to you know get home at 5
pm and see their kids that's that's
one motivation that's perfectly fine
there's other motivations like
some people are going to uh you know be
working
uh just for their own ambition their own
personal ambition to rise to the ladder
they have no interest
in the company's own success or outcome
um and other people are just
like they're literally freeloaders like
they're literally doing crimes and
embezzling the company
and like you know selling twitter
accounts to hackers or some crazy stuff
like that it's gonna be a tiny tiny
fractional number of people like this
and the uh the entire company itself
the goal of that company is to keep them
all aligned and doing the same thing
but when we take that
framework and transport it over onto
something that's an ai
it's not actually clear to what extent
uh it like it's not clear what it takes
to get the ai to get its internal
components all lined together
you can think of it as it's it's
actually it's the same problem
basically is the outer alignment problem
it's as if
the genie itself contains a bunch of
little genies
in it that are trying to solve the sub
problems but
those little genies might be acting
against the the main genie and the main
genius to think carefully about the
instructions it gives to its little
genies
so what are some examples of um of sub
problems that might lead to the
formation of these
these by the way these little genies
right i mean in the language of ai
alignment these are miso optimizers
that's usually what they're referred to
as so a misa optimizer is it's kind of
like an
optimizer within an optimizer so you
have this machine learning algorithm
and then within it there are a whole
bunch of sub problems that have to be
solved in order for the overall
algorithm
to work so if you're doing computer
vision right part of the part of the
overall vision
problem of computer vision is for
example um
i don't know recognizing edges and
corners right so you might have a little
misa optimizer and one
that specializes in edges and another
one that specializes in corners
and these kind of can take on a life of
their own to some degree i guess that's
the concern
at least and to the extent that they do
that they want to protect
themselves in the same way i guess this
is what you're saying with outer
alignment right and
in the same way as the outer aligned
agent doesn't want to be reprogrammed it
wants to retain its original purpose
these mesa optimizers don't want to once
they've latched on to their sub problem
they don't want that sub problem to be
changed and so
they desperately want to preserve their
own structure right
yeah so one example of this if you go
back to like the genie
putting your brain in like a vat of
endorphins and whatnot
um the genie like you you give it the
instruction like maybe happy
and the genie is like oh i'll put i'll
put his brain into a vat of endorphins
that's what i'll do
um then what it does is it assigns to
one of its
sub-components the sub-problem of okay
like in order to make this work
we're gonna have to figure out like a
lot of stuff about human neurochemistry
so like get working on that
and we've got a bunch of other sub
problems too and then maybe the thing
that's
working on human neurochemistry has some
version of the thought process of
oh wow man this is a hard problem i need
lots of computers
like more computing power than i
currently have in order to solve this
sub problem
okay i'm gonna like convert a lot of the
atoms in the world
and the solar system and the galaxy into
more computers to make sure that i've
really
really well solved this sub problem of
figuring out
human neurochemistry so then i can pass
that solution upwards to like the main
optimizer that's solving the big problem
um and oh like uh maybe the big
optimizer
is is using computers over here
but i want those computers to solve my
sub problem and so you get this kind of
fight happen
um it's actually you can kind of feel a
similar thing happening within your own
mind
so if you're deciding you know hey i
want to
um do i want to eat like that cookie
over there
it doesn't feel like you know you have a
one
coherent loss function for your entire
life and you're trying to compute
expected values over it instead it feels
like
there's a whole like separate thing in
your head that's dedicated to solving a
sub problem so you have like the hunger
module in your head
that's fighting you for control and the
hunger module is like yes yes go for the
cookie
but you know you've got a higher level
process that goes like
oh well hang on how does that fit into
my life goals of not getting fat or
whatever your life goals are
but that that it feels like you're
you're fighting a
part of yourself in that decision right
right
and so um well the
uh there's so many places we could go
with this having introduced me these
optimizers but i think one thing that's
worth
pausing on and noting here is um first
off
this stuff can sound can sound
understandably a little bit out there
and wild right i mean we're talking
really about machines
that to some degree want to take over
the world i think it's worth pointing
out
that or at least you know in some modes
of operation
they will tend to if they are
sufficiently powerful have sufficient
compute
have access to enough data eventually uh
develop ambitions like that
this it does sound wild but it is a very
very well established
um what's known as an instrumental
objective
of machine learning models in this kind
of stage it's something that
at the very least uh very serious
researchers focus on as a problem i mean
this is considered a very serious issue
in the alignment community um can we
talk a little bit about this idea of
instrumental objectives
what is an instrumental objective can
you define that
yeah um so well so so what you said
earlier about
basically we we think that
uh all systems that are smart enough
uh or the vast majority of systems that
are smart enough
will want to after some fashion take
over the world which sounds like
a very extreme claim and the reason
is uh exactly this this idea of
instrumental goals and instrumental
convergence
the idea is that uh there are certain
resources and there are certain actions
that are good bets
to take and to seize no matter what your
goal is
so if your goal is uh to put my brain in
a vat
you kind of saw how oh like you know the
sub sub-optimizer wants
more computers to make sure it
absolutely nails the problem and gets it
right
that's kind of a part of it um in order
to solve a hard problem you need lots of
computers and
when when there are no real limits
placed on your power what you can do
there's no reason why you wouldn't just
convert the entire earth
and every particle of mass that you can
get your hands on
into a computer like again there's
there's nothing
approaching sympathy that is even in the
concept space of of something like this
so
you could certainly argue that's what
humans are trying to do already i mean
we are quite literally converting as
much of the world as we possibly can
into compute resources which we're then
deploying to solve our own instrumental
goals
and yeah and and even that is with
you know we we have limits to our
capabilities and even we have
sympathy for the animals that we're
killing the trees we're destroying
the environment like you know we not all
of us do but like enough of us do that
we're like hey guys like we need natural
preserve national parks all these sorts
of things
that's because we to an extent value
these things
enough that we're you know not willing
to maybe utterly annihilate the planet
to get them
um but if our goal function were
different
uh we wouldn't even care that much and
then yeah who cares about the rainforest
who cares about
this and that and that's that's
ultimately what
what we would be dealing with and you
know imagine the the the plight
of an you know uh the uh
an african lion or something faced with
that degree of destructiveness which is
far worse than even the degree of
destructiveness that we've
imposed on lions already it's just like
it's incomprehensible
um just zero sympathy totally mechanical
and utterly destructive and this is i
mean one famous
example or how you get to this is this
idea of the paper clip uh paper clip
optimizer basically you know the device
that's told hey make paper clips
and just without any other context it
goes oh cool there's some iron like in
the ground there's some iron in your
blood there's some iron here some iron
there it just collects iron from
everywhere and
you destroy the world making paper clips
um cool
so i guess one last note i guess on the
instrumental
convergence idea there are instrumental
goals or
a number of different instrumental goals
that many people consider to be
plausible
things that machines will tend to
optimize towards
as a side effect of trying to optimize a
main
objective function um can you can you
list a couple of these
i know you've alluded to a number of
them but maybe just so people can
see something yeah at a fundamental
level based on you know based on the
laws of physics that we know
uh of today uh you know matter and
free energy um you you want stuff
and you want the juice to run it uh so
in other words
as as much put put matter together to
build lots of computers so you can think
um as fast and as effectively as
possible
uh and uh and and grab the energy
sources that you need to actually run
those computers
and uh and there are other instrumental
goals like uh survival
generally speaking um if i
like i can predict that if i survive my
goals are more likely to be accomplished
than if i die in in most cases if i have
a certain set of goals like
if i'm around to continue pursuing them
those goals are more
likely to get accomplished so
self-preservation is one of these
instrumental
sub goals and um
uh a goal like goal cohesiveness or goal
preservation is also uh an instrumental
sub goal so
um i if if i remain alive
but the goal that i'm given in my brain
is
changed then the previous goal that i
had
is also less likely to be accomplished
because i know i would be no longer
working on it after my goal has changed
and so i'm going to seek to prevent my
goal from being changed
so i get you know in in at least like
uh as as we kind of expect we get
basically one shot
to set the set the goal right um because
otherwise like if we're like oh no
no wait wait then you know presumably
the system's gonna be like oh no hang on
hang on
i got this i got this like i i got this
goal and i'm gonna accomplish it
well so so that and i think that
reflects um to some degree
maybe correctment firearm maybe your
tendency towards one of the camps in the
ai alignment debate
so there's the camp that says guys we
get one shot at this
we gotta make sure that we get our goals
right and there's another camp that says
uh guys we should be focused on
robustness making
models that account for their own their
uncertainty with respect to the goals
that they pursue
and trying to be courageable in other
words invite corrections
and be robust in that sense um i just
wanted to
kind of flag the distinction between the
two because it is two branches of the
ecosystem would that be fair to say yeah
the argument
starts to get a bit technical in in
those areas uh i would say that
that both are uh viable directions and
certainly
you know there are probably going to be
phases you know before we make some kind
of super evil genie that can destroy the
world
where we definitely want to be
more in learning mode than in ah i got
this exactly right
mode um so that we can actually look at
like okay
like what is systems that are like
pretty smart and like maybe
dangerous but not necessarily
instantaneously destructive
what do they do after they've been given
a set of instructions
can we is there a way we can make them
okay with having a switch flipped so
they're stopped
all that sort of thing cool okay so
um having laid that foundation i think a
lot of this will be
um well it'll probably be old news to
people who are familiar with the ai
alignment
ecosystem but maybe new stuff for folks
who are joining us from the last series
where we're doing more but data science
career focus stuff
um one thing i want to do is because you
and i have talked a lot about misa
optimizers
um i think one of the really interesting
things that the
concept of mesa optimizers does is it
does give us
i think a pretty interesting perspective
on
what humanity is what life is
uh what the universe is really and i
wanted to explore that if you're if
you're okay with it
just kind of airing some of our uh our
private conversation
conversations out here a little bit this
is part of the reason why i wanted to
invite you on the podcast
um so are you are you cool with that
yeah yeah of course
okay great so i guess maybe um
i'm trying to think of a question to
start us off with but um maybe i'll just
provide a framing and you can take it
from there so
we've discussed the idea that
um that essentially every organism in
the universe
can be seen as a misa optimizer
so essentially what's happening is the
universe has a whole bunch of atoms it
has a whole bunch of
photons particles all kinds of different
degrees of freedom
and it's running this experiment it's
it's
training on whatever reshuffling of
particles will randomly tend to occur
so it's kind of you've got this
interaction between particles that's
going on over time
and over time it seems that the jumbling
of particles is tending towards
self-assembly of complex systems
and whatever the final end state of the
universe is like it seems like we're
tending towards one of greater and
greater and greater complexity
to the extent that humans don't wipe
themselves out to the extent that we
actually survive to the point where we
create a self-improving ai system
it really seems like we're iterating on
our way to
something like an intelligence explosion
as it's kind of
been referred to in that framing
we're here to compete with all these
other misa optimizers around us
so we want access like you were saying
you know you've got that mesa optimizer
that's specialized in understanding
human neurochemistry so that it can
so so that the overall problem of
putting your brain in a
vat of endorphins can be solved well
humans are focused on solving the
problem
of being really really good humans and
surviving and
and kind of uh propagating maybe i'll
park the thought there because i
yeah so uh i think that's that's it's a
good
uh a good place to start so
we and you know all the other animals
and organisms
in the world um we we are
we're all uh the the
we're all on the inside of a giant
optimizer that's evolving us
right so evolution is kind of the big
process that's
uh that's trying to drive us towards uh
genetic fitness um so we're you know our
our evolutionary directive is
um have lots of kids
relative to the size of the population
like increase your genetic
representation in the next generation
that's our evolutionary directive
what's interesting is that uh especially
recently
humans like really don't seem to be
following that directive very well um
if we were actually following that
directive if we were actually like
all right like i want like as many kids
as possible
compared to like we would be acting
quite differently um we would not be
using birth control for example
um we would probably not be using
condoms uh there's a bunch of other
things that
uh we we will be having more kids as
opposed to you know fertility drops and
stuff like that
so this is a signal that like something
has gone wrong there's something
mismatched between
evolutions like slow optimization
pressure
and like the stuff that we're doing and
learning to do um we
are actuated by desires
like hunger uh sex drive
fear uh all you know anger all these
sorts of things
that correlated very well
to our inclusive genetic fitness
for a very long time um these these are
really complex adaptations
they evolved over millions and millions
of years many animals that are unrelated
to us
probably feel them like uh animals
probably feel fear and all of that sort
of thing
so something something is happening
there's an increasing amount of space
between evolution giving us like
telling us what to do in a certain sense
and
uh our own internal drives telling us
what to do and
our internal drives were able to satisfy
those internal drives
in ways that evolution would look at if
it was a person and be like hey whoa
that's pathological
like that's you guys are you guys are
doing something wrong here
you're optimizing too much for your
hunger you're getting fat and like not
having kids what's this
you're optimizing too much for your sex
drive and having sex with condoms you're
not having kids
it almost feels like from evolution
standpoint and evolution is like the
human
trying to get the the code and the ai
right so that it doesn't go off track
and what's happening now is like
evolution programs you to have a whole
bunch of kids
and then instead of um instead of
getting horny and having sex with people
and having kids you start to watch
pornography and evolution goes
whoa that's not what i meant like you
can't cheat like that
and so yeah yeah and that's the key
uh evolution isn't actually programming
us to have kids
evolution found an easier way to do it
evolution was like
oh like if i just like put these
little like actuators on my thing
the organism is going to figure out how
to have kids from there
the problem is when the organism gets
really really smart it
it figures out how to satisfy those
desires without
doing the thing that evolution wanted it
to do
it transcends that initial loss function
basically and
i think so you alluded to more and more
daylight between what evolution wants
humans to do
and what humans end up actually doing i
think it's worth exploring like
what the source of that daylight is
because i mean correct me if i'm wrong
based on our conversations i think i
know i think we're aligned on this but
um i think it comes down to
computational capacity right eventually
um the organism is capable of running
more computation
than evolution itself you're able to
essentially you have enough compute
power
that natural selection isn't the main
thing that's changing you
now the main thing that's changing you
is your own thinking your own cognitive
capacity
and so over the course of a human
lifetime you can actually change
yourself
much more than evolution would allow if
you were like a water buffalo or
something
yeah i i think that this has to do
with how dense versus how sparse
uh the internal versus the external
feedback signals are
and how many processing steps so can you
elaborate on that
what do you mean by dense and smart
sparse here yeah so like
evolution sends you a really sparse
fitness signal
it sends a fitness signal that's like it
it happens like every 20 or 30 years
like
you know do you have kids do they have
kids like every 20 or 30 years
um and and like did you die in that time
before having kids
that's like that's a very smart signal
it doesn't it doesn't come it doesn't
like
punch you in the face very often but it
punches hard when it does
whereas your own internal processes send
you
like in that 30-year time span you might
get
god i don't even know maybe i don't know
you get
you get like hungry god knows what like
three times a day and whatever that is
in 30 years like
thousands you bump your elbow and and
you yeah
scratch your knee and yeah yeah that's
it that's it and so
uh what happens is like you you have
degrees of freedom in between those like
bump like pulses of evolutionary
pressure signal
to adapt yourself around your own
like brain signals so the way it works
is that evolution
is providing feedback pressure on the
way your own
uh your own signals are arranged so
evolution manages the balance between
like
how hungry do you get how horny do you
get how mad you get how happy do you get
all those things evolution manages that
on a time scale of decades
but within your brain you have a set
arrangement of those signals
and you're internally optimizing and
processing on a time scale that's many
tens of thousands
to like millions of times faster than
that
and as a result you have the space
to be super focused on those signals
in a way that evolution doesn't really
account for
um our processing power is greater is
is now too great
to be ruled by these signals
and even like it's it's to the point
where
you know it's plausible that soon we'll
be able to alter our own dna directly
and just completely decouple ourselves
and so we'll be following
we'll we're still slaves to evolution
but we're
we're just like from evolution's point
of view where these misshapen
pathological slaves that are no longer
slaves to it directly
but that are slaves to like the
clumsy clunky things that it built well
it's interesting that you're saying from
evolution's perspective right but
evolution of course
is whatever whatever evolution gives
rise to
i mean by definition we are the goal so
so to me this is exactly that difference
between
um well basically this is
the outer alignment issue um something
is being optimized for in this universe
it's clearly not biological evolution
that was that was what was being
optimized for for a period of time of
about
14 billion billion years leading up to
the present day
but in the last few hundred thousand
years human beings
have started to decouple to lift off as
our cognitive capacity has created
daylight between what evolution wants us
to do
and and the timeline as you said on
which evolution gives us feedback
and the timeline on which we're able to
get feedback from our environments
and that's not by any means the end of
the process right this is what brings us
back to
the ai discussion there's an extra step
here where we
shift substrates where we're no longer
running computation on biological
hardware
but we're running computation on a
substrate of silicon rather than
cells and to the extent that that
happens you start to relax a lot of the
constraints that make humans so
incredibly slow compared to machines and
we might have the same paradigm play out
at a an even tighter time scale yeah
this is right
so one of the issues with computers
is that yeah they're they're so much
faster than us like
they're as much faster than us in terms
of
like the serial depth of a computation
like in other words i do this
then i do that then i do that the number
of things that they can do one after the
other
they're as much faster than uh than us
as we are
faster than the evolutionary process in
terms of its ability to
you know switch out genes and test out
new uh life forms and so forth
and so as a result uh you know
you can kind of make the analogy of like
oh you know i build this thing and it's
so much faster than me
i give it these kind of clunky coarsely
defined
goals that correlate pretty
well to my goals at first but then it
optimized and optimized and optimizes
get smarter and smarter and like
very very quickly perhaps is able to
satisfy the course goals that i gave it
in a way that you know i had no idea i
was like oh my god
this is not what i meant stop stop stop
but by then you know we don't we don't
care evolution
might be thinking the same thing about
us but like we don't care about it we do
what we want
yeah and just as we just as evolution
seems to stand still
relative to the pace of a human life a
human life will appear to stand still
relative to the pace of the evolution of
this artificial intelligence i mean
whatever form it takes um yeah so i mean
i think
that's a big pro in favor of your
earlier argument that we have one shot
to get this right
um i guess one of the concerns too as we
as we start to look into this domain of
um or time domain where ai is starting
to pick up
more and more progress is being made
more more general systems are being
built things like gpd3 but not only that
we're going to get more and more
advanced systems very soon
um we we start to get in this domain
where
we're at the mercy of like the the least
ai safety aware company that
wants to design a program right i mean
this is really this is the domain of ai
policy where you start to say okay how
do you get the game theory to work here
how do you prevent um some foolish
company
that that decides to say oh i'm not
particularly concerned about ai
alignment
um it doesn't you know i'm not worried
about this how do you prevent them from
then you know taking some very
irresponsible action in this direction
do you have any i know this may be less
so your your area but
do you have any thoughts on uh on the af
policy side with respect to that game
theory
well yeah so on the economics of it like
um there's a lot of there's a lot
there's a lot of talk about
democratizing ai and that's good
um that's good up to a point i think
when you're start you're starting to
talk about these
very very big and extremely capable
systems that start
to potentially present dangers not just
like
they could take over the world but
danger is like long before that like
they could be abused by someone to
send you know massive amounts of crafted
spam or
just all sorts of all sorts of novel
kinds of
risks um so
they're so the the economics of it are
that
um like
the the closer to a monopoly you are
the more margin your business has and
here i'm talking about it as as it's as
though it's a business because like
these models are being built to be sold
as apis and all of this
and we see the more margin really you
mean the more profit and like
so that's actually uh that's part of
what i mean when i say margin like that
margin
can go to profit it can absolutely go to
profit and up till this point it's gone
to profit
but it can also go to safety it can go
to safety investment
so when you're you know a monopoly you
can
let's say you charge like a 50 profit
margin 50 of the dollars that come in
are yours to keep and you can do
whatever you want with them yes you can
use them you know for on your balance
sheet and like oh now i have more money
that's great
but another thing you do with them is uh
commit them to
keeping your systems secure and safe and
that takes an increasing amount of
investment
um one of the issues of having like a
widely competitive landscape with lots
and lots of these different models being
sold is that
um they are uh when you compete
against when they compete against each
other each company and each
you know each each company with with
each model uh they they compete their
pro their margins away
and uh in capitalism you know
that's generally seen as good um that's
that's good because
it means that uh consumers are paying
less uh
if a company is mean or evil or whatever
i can switch to another company
uh and it's you know cheaper for
everybody in the case of a system that
has the potential to be dangerous
uh it's what you kind of want ideally
is like one monopoly that's spending all
that's
margin budget on safety um and
not having too many with with limited
amounts of margin they're competing with
each other
so they can't spend it on safety um this
is this is potentially
a risky arrangement to have yeah there's
there's definitely
i i've seen arguments for uh what this
is called multi-polarity or or
uni polarity essentially the idea of
like how many different poles how many
different
companies or organ organisms sorry
organizations
um have you know active ai efforts or
cutting edge ai efforts
and and how does that affect the risk
landscape i've seen arguments that say
you know multi-polarity
might be more desirable um i personally
lean uh definitely in
in the direction that you've just
discussed there i know we've talked
about it a fair bit
um i think another pretty convinced oh
sorry did you want to add something
i'm just going to say you you might be
able to make multi-polarity work if you
had um
you know an organization that was purely
dedicated to safety
very well-funded and also had a lot of
internal
transparency into what all the other
companies were building
right that might be a way to work but
that
seems hard at least in the current
arrangement of things yeah
i know openai put together a uh piece of
policy work
researching basically like moving
towards a more transparent way of
developing
ai internally within organizations a big
part of the game theory behind this is
trust
you know if you're if you're google and
you're working towards a super powerful
agi that could potentially be dangerous
and you know facebook tells you
they're working towards that as well but
you and and you both claim to each other
that you're investing a lot of effort
into safety to make sure nothing
horrible happens
like how much should you trust that
that's actually happening at the other
company
and they're not actually just taking all
their margin and throwing it
out competing you to develop a product
faster and that's
yeah and that's uh and that's like one
of the good scenarios
um these are these are two companies
that are uh you know
culturally uh culturally aligned
exchange employees a lot
are based you know roughly on the same
strip of the bay area uh they're they're
extremely similar companies in a lot of
ways uh and and
you know how does the trust picture
change when you're dealing with
companies that are
from different countries don't speak the
same language have never exchanged
employees
have different governmental structures
and fundamentally different value
systems like
it becomes quite different uh different
and more difficult for trust to be
established in that case
yeah and i think the i mean maybe the
most appropriate analogy
is um well it depends how you take it so
people especially those who argue for
multi-polarity one common argument
focuses on the more mesoscopic domain
between
super powerful systems and and current
day systems where
they say look uh there's the risk of ai
being deployed
to weapon systems now it may be good
that you have multi-polarity
because who knows how the cold war would
have unfolded if just one country had
access to nukes
and no other country could engage in
mutually assured destruction with them
thereby making sure that nobody would
ever fire their nukes you know maybe
something like that would happen with ai
i think there are compelling reasons why
that's not the case or at least
yeah i've taken that view in the past
but yeah sorry
i think this is basically the uh
capitalistic argument for competition
that
uh under all of the circumstances that
we've seen
pretty much up to this point uh
competition has been a good thing uh
it's been good that
uh different you know a lot of different
organizations that similar technologies
they can compete with each other and
keep each other in check and so forth
um and this may continue to be true with
ai
up to a certain level of capability
but i would suspect that there's going
to come
a level of capability beyond which
safety investment becomes more important
than
capability investment and that's the
point at which
you basically need a lot of margin to
commit to safety and so
some arrangement that's equivalent or
isomorphic to a monopoly
yeah and i mean certainly we see that
with like you know tabletop bio weapons
or bioengineering where
designer pathogens are going to be a
plausible reality
fairly soon um and and likewise with
nuclear weapons right i mean you get to
a point where
okay it's it's time to stop publishing
results out in the open i think open ai
itself to
to their credit as as at least from my
perspective
has um and they've taken a lot of flag
for it they've come out and said look as
we start to approach
greater and greater capabilities we're
going to have to start bringing some of
our work in-house
and not publishing as widely um they've
been more and more cautious we heard
with gpt2 their first step in that
direction
now gpt-3 is kind of entirely in-house
none of it is
is public domain like it's not just a an
open source uh
model um anyway it's gonna be
interesting to see how the field evolves
and develops obviously there's a lot of
controversy
with any decision made in the space i
think given the stakes though one of the
one of the key things is really like
targeting policies that are robustly
good
as we've you and i have seen um as we've
engaged with you know
everyone from policymakers to ai
alignment researchers
it's so easy to make moves that end up
doing more harm than good if you're not
really really careful in this space
so i guess what i wanted to do was close
off
this conversation with just an appeal to
people who are listening
you know if you're interested in ai
alignments ai safety
uh anything like that if if you find the
risks that we've been talking about here
compelling and interesting to work on
um then then check out some of the links
that we'll provide
with the blog post in the description of
the video um
because there are a number of
organizations you might want to check
out just to kind of
see what your options are where you
might be able to contribute but
look looking at things through the lens
of what's robustly good
you know what's what's likely to really
not do more harm than good
under unexpected shifts because things
can go wrong in surprising ways when
you're talking about cutting edge
technology
um anyway just something i wanted to
throw out there because we've run into
some of these issues ourselves
yeah uh i think that that makes a lot of
sense um it's
important first to it's one of those
it's it's sort of a
first do no harm um type of thing uh
it's
uh it's it's a hippocratic oath i guess
on a bigger scale
um but yeah i think that's uh that's
where
the first effort should be directed is
like how can we do stuff
that avoids uh bad consequences going
forward and keeps
everyone uh in a in a nice state
awesome okay well ed i really appreciate
it um i think
what we'll do is i'll link obviously
your own personal website
um in the description of all this stuff
too uh
anything you wanted to share i guess do
you mind sharing your twitter so people
can follow you
sure yep uh i'm at uh uh
neutrons neurons or neurons neutrons i
actually forget sorry man i think um
oh shoot i think it we'll
neurons uh so you can hit me up and uh
drop a follow there
fantastic all right well thanks so much
ed uh i really appreciate it
great conversation my pleasure
we'll keep it going offline cheers
cheers