1:AGI Safety: Evan Hubinger 2023

I am Evan humier I'm a safety researcher
at anthropic and I'm going to be talking
about AGI safety so
just a little bit about me before we
start so I said I'm currently at
anthropic I do empirical and theoretical
AI Safety Research there uh before that
I was a research fellow at Miri the
machine intelligence Research Institute
and I did other stuff for that at open
Ai and other places
okay so uh essentially what we're going
to try to talk about is I want to teach
you over the course of this whole uh
sequence how I think about existential
risk from artificial intelligence
so what does that mean so existential
risk we're sort of imagining a situation
where humanity is in some sense
disempowered where uh the you know the
future is no longer sort of in our
control in some respect that could be
Extinction it could be every single
human is dead uh or it could be you know
some other scenario where we lose
control something there where maybe you
know there are still humans around but
effectively those humans have you know
no control over the the course of the
future
okay and we're going to be focusing on
prosaic artificial general intelligence
so what does that mean so prosaic means
we're imagining a situation where we get
to very powerful Advanced AI systems
without sort of any fundamentally new
advances in how we're thinking about
intelligence
uh that means we're essentially going to
be looking at machine learning uh the
sort of you know current broad Paradigm
of how we do Ai and imagining that that
is in fact the thing that gets us to uh
very powerful AI systems and this sort
of notion of very powerful is you know
approximately something like this notion
of uh artificial general intelligence
um we're not really going to rely too
heavily on this sort of generality
notion but essentially the idea is you
know uh some system that is
approximately as capable as a human
across a wide variety of domains
um
and so that's sort of what we're what
we're looking for we're going to try to
understand you know what these systems
might look like how they might be
developed uh and why they might be
dangerous
and the hope is that you know this
sequence should be accessible regardless
of you know what sort of prior knowledge
you're coming into it with uh so you
know we're gonna really try to cover as
much as we can uh you know to help
people understand
okay so this is the broad outline for
the sorts of things we're eventually be
talking out of the course of the whole
sequence
um today we're just going to be doing
one and two
okay okay
so machine learning so I think
um any situation where you want to try
to understand what is you know going to
happen with you know current very
powerful AI systems and you know what
those systems might do in the future we
have to start by understanding what it
is that the you know the mechanism that
actually produces the system how does
machine learning work uh you know
fundamentally so
now I have here what is sort of our
prototypical machine learning algorithm
this is going to be you know sort of
what we're what we're thinking about
this is a little bit intimidating
um that's fine we're gonna sort of you
know I'm gonna really talk through you
know everything that's going on you
don't have to understand what I put up
here but essentially what this describes
is the process of machine learning which
is we have some function and that
function is parameterized so there's a
bunch of parameters that describe how
that function operates so you know for
example we could think about a linear
function is described by two parameters
it's described by The Intercept and the
slope
um among other ways described lines and
we're imagining that you know those
parameters associate those parameters
are determining how the function works
and we get to search over what possible
parameters we want to find the function
which results in the behavior we're
looking for
and in practice the way we do this is
via gradient descent or we have this
parameterized function this function
that has you know in practice well
Millions potentially billions you know
many many possible parameters that
determine exactly how it operates and we
search over the space of the settings of
those parameters to find a sort of
parameter setting which results in good
behavior on some uh data distribution
and some loss function that sort of
describes what we're looking for on that
data distribution and we do that search
for your gradient descent where we at
each individual Point sort of calculate
the derivative of the loss and and step
in that direction
okay so so how do you think about this I
mean it's a it's a you know structurally
complex algorithm I think the best way
to sort of conceptualize and think about
what machine learning is doing is by
thinking about lost landscapes
so what is this so well we can imagine
you know if you look at each individual
parameter as we're sort of varying those
parameters and searching over the space
of possible parameters
um we can plot you know how well does
each particular setting of parameters
perform uh on our on our loss on our
data you know what is does that
particular setting of parameter result
in good performance bad performance what
does it do and so that's what we have
here we have a sort of dimensionality
reduction we've looked at a couple of
particular neural networks uh you know
sorts of uh parameterized functions that
we often use in practice and we want to
understand for those particular
parameterized functions for each
individual setting of parameters how
well is it doing so you can see for
example over here there's a bunch of
these sort of values and nooks and
crannies where you know individual you
know parameter points might do really
poorly you know like out here in the red
or they might do really well you know
down in the valley in the blue
um and the process of grading descent is
effectively rolling down these slopes so
we have this sort of lost landscape that
describes you know what the different uh
parameter settings do in terms of how
well they perform and we're searching
around this space sort of rolling down
trying to find these values that do that
have parameter settings that perform
really well
there's a couple of things that I that I
will talk about about this this sort of
particular setup because it's a little
bit
um disingenuous in a couple of ways but
yeah question first
so what exactly are we looking at here
um I don't see any like x y z axis
um descriptions and also yeah this
these parameters which layer are we
looking at in these different networks
um yeah
um I'd just like to have a clearer
vision of what I'm looking at here yeah
great great question so uh a couple of
things so first is
um
in terms of like what were the axes uh
we can think about the the depth here
right is describing the loss it's
describing and the loss just means how
well do you perform on the data set so
um we'll talk about you know some
examples of you know what that might
look like in just a little bit but
essentially where you know we have some
set of data and we want to find a
function which does something on that
set of data we have some desired
behavior and and this sort of
essentially is the you know the depth
here is determining how good is the
behavior of that particular set of
parameters and the axes are different
parameters so you can imagine I think in
practice these particular case is a
dimensionality reduction but um you can
sort of think about you know this being
two parameters so we have you know one
parameter on X and one parameter on y
that describe you know two specific
parameters that we're searching over so
if this was like a linear function right
and we in fact just had two parameters
overall you know then you know one of
these would be all the possible values
for the slope the other would be you
know all possible values for the
intercept and we would just be looking
to see you know which combination of
slope intercept values results in the
behavior that we want
um and so that's sort of what we're
looking at here in in fact
um you know like I said it's not quite
that so it's a dimensionality reduction
and also
um you know because in practice you know
the number of parameters is really
really big uh we have we're you know
millions billions of parameters in these
sorts of largest Networks
and so you can't just sort of represent
it you know in this sort of
three-dimensional you know you need to
be able to see in millions of Dimensions
to really sort of understand what was
happening here but
same sort of thing you know replicated
across many dimensions you know in each
individual parameters that we're looking
at there's going to be you know some you
know situations where we can sort of
fall into valleys in that parameter
um and and and and you know these these
same sort of shapes arise now there are
some important facts about the fact that
it is so large dimensional
um because it is in fact really large
dimensional there are some things that
sort of your intuitions about lower
Dimensions won't quite work so you know
one of the things that we'll you know
we'll see is that it really you know
changes the volumes of these basins so
you know we could think about you know
this you know falling into this
particular area you know all of these
sort of things around here if the you
know we follow the gradient and we just
sort of fall down they're all going to
fall into the same Basin here
um but thinking about volumes of basins
in very high dimensional spaces is sort
of counterintuitive because volumes
rapidly expand with radius when you have
very high dimensional spaces so you know
if we you know in a you know
two-dimensional space right you know
it's proportional to r squared the
three-dimensional space it's our Cube
you know what an n-dimensional space in
a million-dimensional space you know the
rate the volume is proportional to you
know R to the millionth power and so uh
you know we end up in this sort of
situation where some things can start to
get a little bit counterintuitive
um but but essentially the idea is that
um you know we're trying to you know
look for these sort of you know
desirable basins Yeah question
uh yeah go for it oh just to summarize
the blue part is where the loss is
better or like lower and that's where we
end up right and then
um yeah perhaps as a follow-up question
why the heck is the thing on the left so
rugged and like that's a very good
question I haven't talked about that yet
I will talk about I think it's not I
think it's kind of an interesting uh
thing to note I'll just mention briefly
I think it's not super important but
basically what's happening on the other
side here is this is a particular
instance of a case where these are two
different architectures where they're
different functions that have been
parameterized in different ways
um
in particular the actual difference here
is that the smooth one has something
called residual connections and the
rugged one doesn't have residual
connections
in practice what residual connections do
is they they substantially smooth out
the landscape now it's a little bit
disingenuous to say that this is like
you know fully smooth uh because it's
very zoomed in so if you were to take
this picture and like you know Zoom way
out on the Lost landscape you know even
in the smooth one you would start to see
you know you know more you know valleys
and stuff uh you know as you look at
multiple different basins but what it
does do is it makes it overall
substantially smoother right so at the
same scale we can see that with residual
connections things are much smoother and
I think that is one way of describing
why we use original connections right
I'm not going to talk about all of the
individual different architectural
features that modern you know neural
networks use but I think one sort of
General point is that when people build
you know machine Learning Systems when
they choose their architectures what
they're trying to do in picking you know
what that that parameterized function
looks like is they're trying to get lost
Landscapes that look like that one on
the right they're trying to get these
nice lost Landscapes that have good
properties
um in terms of being easy to optimize
over having good Minima that can sort of
fit all the data
the problem is that that task is really
tricky because we don't actually
understand the sort of mapping between
how do I choose a function and you know
a parameterized function and how do I
what sort of lost landscape do I
actually end up with
um and and that sort of Disconnect can
be quite tricky a lot of times
um so yeah we'll talk about that a
little bit Yeah question so uh you said
that as the dimensions go the volume of
the basins go like r squared r to the
33rd and so on but what is art are
generally what do we mean by the basins
growing like in as proportion of the
football
of the whole parameter space
the proportion is going off the space
from which you end up in one particular
point yesterday yes this is a really
good question so I haven't I haven't
talked too much about like what I mean
by Basin yet but essentially we can sort
of think about it like this gradient
descent right likes to roll down these
Hills you know it finds situations where
you know we start you know maybe at some
random point so I'm setting up
parameters it doesn't really describe
any particular you know meaningful
behavior and then grading descent you
know looks at this lost landscape and it
goes down it tries to find these points
that actually have good behavior and a
base that is sort of described as the
volume where all of the points roll to
the same point right they all sort of
roll into approximately the same area
and so if you think about you know here
we have multiple basins right we've got
this like Valley on the left and then we
have this like big bass on the right and
those are you know if you start on one
side you end up in different situations
you know on this side they're also the
same base and this is just looking at
one basin
um and so R is just sort of the
characteristic size the characteristic
length of the Basin you know for example
you could think about it as the distance
that it takes to roll into the basement
um
it's getting a little bit technical it's
not you know super relevant to you know
think about all this but basically you
know we're thinking about okay we have
this you know big we have this
parameterized function it has this
really big space of all these possible
parameters uh and in that space there
are these basins that describe various
different algorithms that the you know
could be implemented by that function
and uh we sort of want to understand you
know which of these basins do we
actually end up with when we do this big
search
um and we'll talk a little bit in a
little bit about you know what are some
factors that actually make that
determination
is there a concrete example of like what
these parameters might look like in
order to help us get an intuition of
what this kind of system actually is
yeah that's a great question
um I'll talk a little in a little bit
about some like actual algorithms that
the functions might Implement and what
those might look like
um in terms of what the parameters are
they're just numbers right you know uh
they're just uh you know they're just
floating Point values that you know they
we use the Matrix multiplications and
all of you know in not just in you know
all sorts of various different ways that
there may that the function makes use of
them I mean you can think about like you
know I don't want to go into too much
detail on like exactly the structure of
a neural networks because I think it's
not super relevant I think the basic you
know some you know we'll talk you know
one thing that is relevant is just like
you know they do sequential computations
so uh the you know there's a particular
computation which is performed uh and
then you know based on the results of
that computation performs another
computation and each in each computation
is parameterized so you know exactly
what computation it performs is just
dependent on whatever the values of the
parameters are that they determine how
it does it um exactly what those
parameters are doing and how they work I
think is not super important
um
yeah I mean and and many of them are
used in different ways you have biases
you have you know I mean we're talking
about like Transformers you have
attention heads so there's a lot of
various different parameters that do
different things but basically it's just
some massive parameterized function
um and one of the things also that I
think is you know maybe important to
point out is that when you have very
very very large networks as the sort of
size uh increases generally they become
capable of implementing you know
essentially almost any function and uh
the the functions that which you can
Implement increase as you increase the
size so with really large um you know
sets of parameters you can start finding
almost any algorithm which is uh you
know could theoretically be implemented
um and we'll see later that's really
important the fact that as you increase
the size of your network as you increase
the number of parameters the number of
possible algorithms which you can
Implement increases and you gain the
ability to implement new fundamentally
new sort of structures
okay I'm going to move on I think that
maybe this is getting a little bit
confusing and hopefully it'll get a
little bit less confusing as we talk
about some more stuff so
um
okay great so here is a concrete example
um of you know a situation where you can
have a sort of machine learning system
and it finds different basins
so we're going to uh here's what we're
going to do we have these sort of two
shapes we have the blue blocks on top of
triangles and we have the red Pac-Man
with a cape
and the idea is we are going to try to
sort of train a network that is we're
going to search over possible you know
parameterized functions until we find a
setting of parameters that in fact is
able to when given the you know Blue
Block always you know result in Blue
Block and when given the red Pac-Man
with a cape it always results in you
know packing with a cape
um and then we want to see we want to
ask what happens if we take that uh you
know function that we've just learned
and we see what it does if we give it
you know the swapped colors so we what
if we gave it uh you know red block on
top of pyramid or a blue Pac-Man with a
cape and the interesting thing here
right is that there's sort of at least
two different algorithms which you could
learn which would do a good job on this
particular data right there's at least
two basins uh that describe different
sorts of ways that you could fit this
data you could learn a color classifier
that classifies any red shapes as you
know one thing and any blue shapes
another thing or you could learn a shape
classifier you could learn the shape
classifier that classifies the Pacman
with a cape in one bucket and you could
classify the blocks on top of pyramid
another bucket
and so we can sort of think about this
in the Lost landscape setting as
representing two sort of distinct Basin
that describe uh two different possible
algorithms that you could find when
you're doing this search over uh you
know function parameterizations
um and when humans do this when you ask
humans to sort of you know what would
they sort of classify humans generally
will go with the shape though it's a
little bit unclear there's sort of you
know a reason for this you know in
practice you know if I see a red chair
or a blue chair
um you know they both sort of
functionally serve as chairs and so we
you know we tend to classify objects
based on their shape but a fact
match on this task uh we'll almost
always pick the color classifier I
almost always learn to classify the cut
you know the red things it's one bucket
and the blue things into another bucket
and that's interesting right you know it
tells us something about what it is that
determines you know which basins are
selected for you know which
parameterization uh you know which
settings of the parameters are the ones
which actually get uh implemented in
practice uh you know by these networks
which which sorts of ones do they find
and in this case at least you know we
know that what they find is they find
the color classifier
and and this sort of difference you know
the like okay there's multiple different
possible algorithms which could learn
the data but in fact we find one
particular algorithm which fits the data
is in some sense the key to what makes
you know machine learning so powerful it
makes it work so we think about uh right
so we can go back to you know thinking
about these basins right we can describe
each of these different algorithms it's
occupying you know these different
basins in the Lost landscape now in some
cases you know there will be algorithms
which don't even exist in our lost
landscape because our model sort of
isn't big enough to implement them but
in cases where our model could
theoretically Implement either one we
have these multiple basins and you know
the machine learning algorithm has to do
some Basin selection that determines
which of these different algorithms that
could fit the data do we actually end up
finding
okay there's some research that shows
that you know sometimes there are
symmetries between basins
um but most the time we're sort of going
to be imagining that we're going to be
looking at a sort of decimetrized basins
where we have
um you know different basins describing
functionally different algorithms
yeah
uh you described as like the algorithm
choosing which Basin to use but isn't it
the case from what you said before that
we start out at some random point and
then we don't live with any control over
which Basin we wind up in it's just
going to roll down help yes that's a
really good point I think that
um
in practice it's a little bit unclear
the extent to which you always end up in
the same Basin or you end up in
different basins depending on the
initialization it really varies based on
the particular machine learning setup I
think one thing I will point out though
is that some things are very over
determined so if we think about the
color versus the shape classifier in the
previous example
the the fact that you learn the color
classifier in that case is extremely
over determined you never learn the
shape classifier when you start uh
training from scratch on that task
and we can talk about you know why that
is but I think that the thing I want to
point out that's important there is that
um even if I randomly initialize
Sometimes some basins are so much larger
and so sort of preferred by the grading
consent process that we effectively
never find the other basins so in that
case um you know and why is this well we
can think about you know previously
right like I was talking about you know
what determines the base and volume
right well in these really really high
dimensional spaces Basin volume you know
can vary drastically between different
basins because
um even small differences in this you
know radius of the Basin can have
massive changes on on the total volume
and so because of that
um you know you can have cases where you
know some algorithms occupy like 10 to
the 20 more volume than other algorithms
and there's sort of no chance no sort of
you know chance uh that you could ever
find the smaller Basin uh when that's
the ratio that you're dealing with
but that's not always the case sometimes
there absolutely are cases where there
are multiple different algorithms that
you you know that are sort of both
plausible and it'll depend on the
initialization you know which random
point you start with uh what you'll end
up finding
does that also depend on the data set
that we're provided like for example if
we provide different colors of the shape
so if we know a priori that we're
looking for a shape classifier not a
color classifier for whatever clyrus and
then all the confounders could have been
advanced
yeah that's a great question so uh if
you uh give it a bunch of instances of
you know cases where you have the same
shape but a bunch of different colors
and you tell it to find an algorithm to
classifies All Things based on shape and
ignores the color you absolutely can
learn shape classifiers
um but the question and so it totally
depends on the data set
um in this case though we're sort of
asking well you know what if we don't
give it that information what if we sort
of don't say whether we're asking for
color we're asking for shape we sort of
just want it to figure out what is you
know what is the best algorithm for
distinguishing these two things and then
we sort of ask you know what what it
learns in that case
um and you know I'll talk about a little
bit but I think that that distinguishing
you know the ability of the machine
learning algorithm to pick you know
which algorithm does it like better to
solve the the problem is really critical
to why these things are able to do what
they're able to do because
um you know you can imagine if they just
memorize the data exactly at every point
you know that would do a good job at
fitting the data you know any data you
give it if it just memorizes every data
point it'll always be able to do you
know 100 perfect performance but but
it's useless right you know a 100
memorizer that has no ability to do
anything coherent on any new data points
um doesn't do anything you know
structurally useful for you
um and so the fact that machine learning
doesn't do that that we don't just learn
this you know memorizer that we learn
something that has an interesting
structure that actually implementing
something you know relevant like you
know distinguished based on color
um is what makes it powerful and useful
is it possible for
to to implement the same algorithm that
with different weights such that I mean
if we are in such case we have different
rates but basically the same algorithm
does it mean that we have learned into
the same vessels or into different
vessels bits which are equivalent in
terms of plus yeah that's a really
really good question so in fact there's
a there's some research that shows that
um that fat the fact that there can be
cases where the same set of Weights will
implement or start different sets of
Weights will implement the same
algorithm
um is a really important facet of which
basins sort of end up being larger and
which base instead of being smaller
which algorithms sort of end up being
favored because if you have an algorithm
and the same basic algorithm can be like
implemented in a really really large
number of different ways then that
algorithm becomes much easier to find
and so it becomes favored by the you
know the gradient descent process uh you
know it becomes you know that sort of
base and becomes very large because
there's all of these different sort of
effectively equivalent parameterizations
which result in the same uh you know
functionally equivalent sort of model
and so we think about something like the
color classifier one of the ways we can
understand you know why does it learn
color is well color is you know an
algorithm that is sort of relatively
easy for it to calculate based on the
RGB values that it's sort of taking in
um and there's a bunch of you know other
stuff that it you know doesn't it
doesn't have to it doesn't really matter
you know it only takes a small number of
parameters the rest of the parameters
can be set to essentially anything
there's a bunch of ways to you know
calculate the color looking at different
pixels looking at different places on
the image and so it's not sort of it's
It's there's so many different ways to
implement it and it's so simple based on
the data that it sort of input that it's
receiving
um that it sort of ends up being you
know a substantially larger base in the
sort of favored by default
um yeah
so one thing I read last year was this
idea that the algorithms will most
likely to be selected with the most
compressible because they could be done
in the fewest parameters letting the
other parameters be basically whatever
they wanted is that related to this idea
of pace and volume yeah absolutely so uh
I think that we're going to talk a bunch
about why Simplicity is you know a
really important component of you know
what what's the lacks which algorithms
you end up learning and simplicity you
know functionally is essentially the
same as compression it's just you know
how you know can I can I take it can I
you know find some really functionally
simple algorithm
um that is able to explain all of my
data that's sort of what a compression
is
um
and so compression is absolutely you
know important part of what's Happening
Here Right
um now how it maps from you know symbol
algorithms to large basins and things
which are favored by greeting descent is
a little bit complex there's a bunch of
different things that go into that so
one really important facet is what I was
just talking about which is the fact
that when you have
um you know a sort of structurally
simple algorithm it leaves a lot of
parameters un you know untouched or like
not relevant or there's like a bunch of
different ways to implement it which
means there's a lot of different points
in the weight space which all correspond
to effectively the same algorithm and
that and that you know means it has a
very large Basin um so that's one of the
factors but there's other factors as
well one other factor is that we often
will do explicit regularization where we
will actually like take the function and
we will explicitly say you know we want
functions which do better on the metric
of being small and simple
um and the reason we do that is the same
reason we did the like residual
connections right in the previous thing
where we just it's we really you know
the reason that machine learning is so
powerful the reason we want to do it is
because we're hoping to get these simple
struct you know structurally simple
algorithms out of it and the best way to
get those structurally simple algorithms
is by you know using these techniques
that help us find
um you know structures that
um in fact result in these you know
simple algorithms and so that's why we
do suffer great realization that's why
we add things like residual connections
to get these um you know lost Landscapes
that have you know the sort of simple
algorithms favored
I think I'm wondering about now is when
we got these bases is there a way to
tell which algorithm they will implement
my current model of this looks like oh
we've got these points on like these
basins in parameter slash lossland the
in the last landscape and then those all
kind of show which parameterizations
perform well on our different tasks so
for the classifier uh the the log basins
um will basically result in a classifier
does dot has a high accuracy
um but the way the way it does that is
still kind of obscure to us right does
it do it by Form classification or like
well through the the colors and now my
question is is there a way to actually
tell just by looking at the Basin what
kind of algorithm it is implementing
that's a really really good question and
I think the unfortunate answer is
oftentimes no
um you know if only thing I know is okay
it's some setting of parameters and that
setting of parameters looks like it does
a good job on the data that's often all
I know you know I know that in some
sense you know okay I know that this is
this what every algorithm I found
corresponds to a large base and it
corresponds to some sort of structurally
simple algorithm but I don't often know
what algorithmic corresponds to now
sometimes you can figure it out so in
the shape you know color example it's
pretty easy to figure out because we
just give it a new example you know we
give it a case where the shape and the
color are different and we see what it
does right and so we can tell you know
in that case you know what it's doing
and sometimes you can do that you know
to tease about different uh differences
and algorithms but sometimes that gets
really hard you know there's just a lot
of various different things to test on
and it can be very difficult you know
form hypothesis about what it's really
doing so we'll see in a little bit you
know an example of a case where we can
do transparency we can actually sort of
look inside the model and you know see
what algorithm is influencing in a
particular case but that can also be
really hard because oftentimes you know
interpreting what these parameters are
actually doing is really difficult
because you know they're not selected to
be interpretable they're just selected
to be whatever you know is the low point
on the Basin you know whatever set of
parameters and fact results in good
performance and so you know there's no
reason that they we would be able to
understand what they're doing in some
cases we can though in some cases we can
see ah it's implementing this sort of
simple you know algorithm that we
understand let's see an example of that
in a little bit but you know that's not
always the case sometimes we can
understand and sometimes it's really
difficult you know hopefully if you know
the fields of transparency progresses we
can get to the point where we can sort
of always look at a you know set of
parameters and understand what it's
doing but in in general right now there
isn't really a way for us to do that
okay so I I talked about this already
but the thing that is so important about
you know this Basin selection about this
idea of you know figuring out which
setting of parameters you want is you
know structurally this is the thing that
makes machine learning uh you know so
good what makes it what it is because if
I imagined an alternative process right
that you know just took you know uh you
know I saw these red data points right
and I was like okay here's how I'm going
to fit the Red Data lines right I do
this you know crazy blue you know line
right that goes up and down
um you know we know that that line is
wrong right wherever this data came from
wherever you know these I collected
these red data points from it probably
wasn't from that distribution it
probably didn't come from a line looked
like that blue line
um and we know that because you know
something like Occam's razor right you
know in fact real world data you know
real world patterns that actually exist
um tend to have these simple
explanations they tend to have you know
uh generating uh procedures behind them
that they have some structurally simple
pattern that describes what's going on
and so the magic of machine learning the
thing that's so powerful about it is
that we don't just find any function
that fits the data right we have these
procedures for finding simple functions
that fit the data functions that are
sort of structurally simple explanations
for what's going on you know this green
line here
um that are actually likely to do a
reasonably good job if we give it a new
data point
um and so you know what's simple means
is a little bit weird right so it's not
the same as always what symbol means to
a human so if we think about the you
know color versus shape classifier you
know humans will often pick the shape
but the the you know the model will
almost always pick color
um and so you know it's not quite the
same but it is you know something that
is you know very important here because
it is we have selected our machine
learning models we found the
architectures that in fact result in
finding things that are simple in the
sense that they do a good job simple in
the sense that they actually fit real
world data they actually describe real
world patterns
okay
okay so I promised an example of a sort
of you know transparency of a situation
where we can actually look at what these
sorts of simple algorithms you know in
fact look like in practice when they're
implemented in you know a neural network
um so this is an example of a case where
we took you know a very large
parameterized function in this case a
convolutional neural network and it was
trained on uh imagenet which is a class
uh a problem we were trying to classify
a bunch of different uh images so you've
got cars and cats and dogs and you have
to be able to distinguish between each
individual one and tell uh you know
which is which is which and so we've
trained a very large Network to do that
task and we want to know what's it doing
right
um in this particular case is you know
we're trying to understand how it
classifies carbs so it has one
particular point in the computation
um where we can sort of ask okay what
image would most you know cause this
particular part of the computation to be
large you know to really uh you know uh
activate
um and we find this sort of you know
image that sort of looks like a car and
uh you know the conclusion is okay this
is sort of where it's doing the
computation that determines is the thing
a car or not and so you can sort of
think about this image as the maximally
car image this is you know the image
that this neural this neural network
thinks is you know the most possible car
and um an interesting thing from this is
that uh you can sort of see I think if
you if you sort of squinted this uh what
it's doing right like what what actual
algorithm is it implementing uh for car
detection
um if you have not seen this before you
can try to think a little bit and guess
as to what it might be
um so I'll give you I'll give a couple
seconds to just sort of think about that
um but I will reveal the answer because
what we can do is we can actually look
at the inputs so we can see okay because
the algorithm you know operates uh you
know the particular parameters function
that we found it operates in these
sequential computations it first does
one computation and then it does another
computation we can see you know okay
what are the computations that happen
before this and what are the sort of
images that maximally activated those
computations
um and we can use that to help us
understand what this what this
computation is doing
and so if we look uh we can see and I
think it's pretty clear based on this
what it's doing so the images on the top
are sort of what it's looking for on the
top of the image the images on the
bottom or what it's looking for on the
bottom of the image
um and it's pretty straightforward it's
looking for Windows on the top and
wheels on the bottom and that's what a
car is to this neural network it's
determined the algorithm it's found this
sort of structurally simple algorithm
for finding cars which is look for
Windows and then look for Wheels yeah
those blank things in the center are
those just not shown or does that mean
the algorithm doesn't care what's in the
center
it means it mostly doesn't care it's
saying that uh whatever is in the center
of the image it's not really looking at
it's not using that to determine uh you
know whether it activates or not
and so uh yes you can see in this case
it's really looking you know very
concretely for two very specific things
it's you know we have in the previous
computation we you know we did some
computation to determine what a wheel is
and it did some computation determine
what a window is and looking to see you
know is there a window on the top and is
there a wheel on the bottom
okay so this is pretty cool right it's
showing us you know this concrete
example of you know something you know a
little bit more complex than just the
you know um you know the color the shape
example where uh our neural network you
know we found a particular
parameterization by searching this big
parameter space to find some setting of
parameters that is you know uh you know
has some large Basin and does well on
the task and we found this structurally
simple algorithm right we didn't just
find some you know memorize every
picture of a car you've ever seen we
found some algorithm that actually you
know he's doing some structurally
interesting thing
um that is able to generalize right this
is an algorithm that if I give you a
random new picture of a car it's often
going to succeed you know sometimes it
has you know cases where it might fail
you know maybe I've removed all the
wheels from a car and while it's still a
car
um and in that case you know maybe this
would this would fail but in general you
know this is a this is a structurally
you know uh simple algorithm that is
actually performing the task in
generality
and that's what we're trying to get
right that's the thing that machine
learning is doing it's trying to find
these structurally simple algorithms um
that are able to solve the task in
generality so that you know you can find
algorithms that actually generally do a
good job
okay
so um
he I want to talk about another thing
that's a little bit interesting about a
particular fact about this you know
notion of simplicity because I think
it's sort of counterintuitive
which is that this sort of notion of
Simplicity right of finding these sort
of structurally simple algorithms to
solve tasks is something that actually
uh when we when we build larger networks
when we build bigger networks with more
parameters and more data
um well when we build larger Networks
um we often are doing it for the purpose
of finding simpler algorithms and this
works right when we have
um larger networks we actually find that
they do often learn structurally simpler
algorithms so this is a little bit
confusing right so you know okay they
have more parameters right so in some
sense you know a large network with more
parameters it you know it requires more
things to describe and so it must be
more complex right
um and in some sense that's true right
it is the case that if it has more
parameters it takes more things to
describe it's sort of more complex but
it might be learning some algorithm
which is structurally more simple so you
can think about let's say I had a
network that all it could do was you
know one sequential computation right it
can't learn if it can only do one
sequential computation it can't learn
something like look from Windows on top
of Wheels because that's an algorithm
that really requires you to be able to
First do the wheel and the window
computation and then combine those into
the you know windows on top of Wheels
computation and if you're only doing
something very you know short and simple
you can't learn that algorithm or you're
going to have to learn some sort of uh
you know you know maybe more Brute Force
you know memorize these sort of thing
and that sort of brute forcing
memorizing algorithm isn't structurally
simple in the way that the windows on
top of Wheels algorithm is and so in
even though the windows on top of Wheels
algorithm maybe takes more parameters to
describe the actual sort of fundamental
algorithm that it's implementing is
simpler it's doing something that is
sort of structurally uh you know simple
and we can think about this as a sort of
Occam's razor sense right so Occam's
razor says that in practice real world
data is described uh you know well by
the simplest algorithm that fits the
data and that's sort of what we want
right if we believe Occam's razor and we
have some data and we want to do you
know find the real world pattern behind
that data then we want the simplest
algorithm that fits that data and when
we have larger networks you know we we
sort of increase our search space now we
can search over even more possible
algorithms
um and that unlocks our ability to find
algorithms that are sort of structurally
simpler than any of the algorithms that
we previously were able to search over
and so we can think about it as well
we're still trying to find the simplest
algorithm but now we're trying to find
the simplest algorithm in a larger space
and that means we can do a better job
because the simplest algorithm in a
larger space can be simpler than the
simplest algorithm a smaller space so I
have an example here of what that looks
like in practice but here question
so does this mean that in general as
models get larger they actually become
easier for us to figure out what they're
doing and what's going on
yeah that's a great question and the
answer is maybe I think it's really
unclear so
um uh Chris Chris Ola who's the head of
interpretability at anthropic has a sort
of hypothesis about this which is you
know what happens is first as you have
uh you know networks like uh you know
something super simple parameterized
functions like linear regression right
where it's just two parameters it's
really easy to understand because it's
so you know the uh function itself is
sort of structurally simple
um but then as we introduce more
parameters it starts to get less and
less uh understandable
um but the idea is like like like you're
saying as we keep adding parameters
it'll start to get more understandable
again because
um you know we're starting to learn
these sort of simpler you know human
understandable algorithms one concern
though and this is sort of what Chris
hypothesizes will happen after that
point is that at some point you start to
learn algorithms which are sort of
simple in a human sense there are
algorithms which are structurally simple
and their algorithms are structurally
simple in a way that we understand them
the sort of simple algorithms that we
learn that we use right when we try to
understand things in the world but at
some point you start to surpass the you
know the best algorithms the human use
uh you know the best algorithms that we
use that we understand and you start to
get to algorithms that are structurally
simple in some meaningful way but in a
way that is not the sort of algorithms
that humans often use to sort of
classify and understand the world and at
that point it's sort of unclear you know
maybe at that point it starts to become
less interpretable again
um that's all conjecture though you know
we don't really know so
um what we have seen is that you know I
think
the sort of broad Strokes of what I just
you know sketched out about the the sort
of starting part of that curve seems to
be broadly correct where a lot of really
really early algorithms are very easy to
understand and it gets harder and then
it gets easier again
I think one tricky thing is that there's
been a lot of that sort of work was done
originally on image networks like with
the you know windows on top of wheels
and it gets trickier for other sort of
models like language models which we'll
talk about uh later
um
but um but you know I think I think it's
unclear and you know it's you know the
best answer I can give is just you know
more research is needed you know we need
to you know if if we do more
interability and we start understanding
more about you know when is it the case
these systems are understandable we can
learn more about this and right now you
know the best I can offer is like some
speculation anecdotal evidence you know
what sorts of things we've seen and what
general Trends we don't really know what
we're gonna what we're gonna find
question oh you've got it already great
um so I'm going to interpret the graph
on top
um it looks like so it looks like at the
beginning perhaps the model is
underfitting and then there's this point
around like 100 or 110.
um where the loss begins to rise again
and then it at a certain point begins to
drop off and I guess I'm wondering is
the way to interpret this that there's
like two competing forces
um like the one where you described
where a larger
model is able to disappose
um search over more algorithm space
um and so you have like overfitting
happening at first and then this search
for this ability to search for more
space takes over at a certain point yeah
newer fitting stops is that so so I had
now for a while and I haven't talked
about it so that's my
back so let me let me try to give a
little bit of how I how I think about
what's going on here so
um yeah so first just very briefly so
what's on the top once at the bottom so
on the bottom we have training loss so
what this describes is it describes how
well does the particular setting of
parameters that we found do on the data
that has seen so far so on the data
we've actually given to it on the data
that we actually you know have
um available uh that we're sort of
training it on how well is it doing
and we can see that as we sort of change
the size of our of our model in a this
is a particular type of size we were
holding everything else fixed we can see
that as we increase the size it gets
better at fitting the data right the
data that we've actually given it uh you
know given to it is able to fit you know
we're able to find some setting of
parameters which does a good job on that
data
okay but now we have to ask the question
right how well does it do on data it's
never seen before and in this particular
case this is a translation model so
we're trying to get it to do a
translation task
we want to understand you know if I give
it a you know pair of sentences in you
know English and French that it's never
seen before how well does it do on those
sentences suppose the ones it has seen
and the answer like you were pointing
out is well it's a little bit weird so
you know it starts out doing very poorly
because higher loss is you know worse
performance uh because you know it
starts out at some random algorithm and
uh you know when it when it just has
some really small embedding Dimension
right when it's just a you know a tiny
model it basically can't Implement
anything useful or interesting right and
so it's doing the best it can but the
best it can is not something very very
you know meaningful but as we increase
the size of the model
um it gets drastically better and so we
find you know an algorithm that is sort
of able to sort of fit and understand
what's happening here and then it gets
worse again so we start you know getting
into a regime where the performance you
know degrades and then as we keep
getting larger it starts improving it
so what's happening here so a couple of
things so first let's start with what's
happening in this First Basin or that
this first sort of uh you know this this
part here what's happening in this part
is structurally where uh we started with
a you know we have a situation where
it's not possible uh you know to
actually find any setting of parameters
because our thing is so small that fits
the data perfectly
um and so because we can't find anything
that fits that data perfectly it's sort
of first it's sort of forced to find
something which is simple uh via the you
know just the brute facts that it can't
implement the memorization right even if
it wanted to memorize the data exactly
even if it wanted to just you know
memorize every you know sentence it's
seen it can't do that because then it's
not big enough right it doesn't have
enough uh you know parameters to
actually memorize the data and so it's
forced to implement this sort of simple
uh you know some sort of symbol
algorithm instead okay but then we give
it more parameters right and what it
does is once we give it a just enough
parameters that it's able to fit the
data exactly where the training loss
goes to zero you can see that that is
the point where the green green
corresponds to green and purple
corresponds to purple that's exactly the
point where we get the worst performance
on the the test data set that sort of
held out data that's never seen before
so what's happening well what's
happening is when we get to this point
where it's just barely large enough that
it can find something that is able to
fit all the data well in that situation
there's basically only one thing that it
can learn it can learn to you know it
doesn't sort of have to have a choice
from a bunch of different possible
algorithms where it can pick the
simplest one that fits the data there's
just one algorithm it's just the you
know essentially a memorization
algorithm right it's just just barely
large enough that it can memorize all
the data points and it can't do anything
else it's not big enough to be able to
learn some you know structurally simple
algorithm that is able to sort of uh you
know solve the data in a meaningful way
and so because of that we just end up
with this poor performance but if we
keep making it larger we get better
again because if we keep making it
larger now there's a lot
of all sort of have a good performance
and it can pick from among them you know
whichever one actually is simplest
whichever one actually you know has the
largest Basin whichever one actually is
sort of effectively doing you know
something which is structurally simple
in a way that is able to generalize to
these other you know new settings that
it's never seen
so I I will point out that you know a
lot of this is is not super well
understood
um you know a lot of the stuff I just
said is is some amount of disconjecture
I think that a lot of it though is
pretty well supported by the literature
on this question uh this is this
specific phenomenon is called double
descent
um and another thing I will point out
also is that in practice you don't you
don't always want to be in this regime
so sometimes you'll want to be in this
Basin you know in this case here where
you're early on and you're simple
because you're just sort of too small
and sometimes you want to be in the
other case where you're simple because
you're too big
um but in either case you're sort of you
know selecting it to be simple now one
thing I will say is even when you're in
this sort of early Basin where you're
sort of simple because you're too small
um as your the amount of data that you
have increases the this Basin sort of
moves to the right and so you have to
make your model larger and larger and
larger and larger to stay in this space
and also as you get more and more data
and so
um it's still the case that we sort of
as we get more and more data we sort of
want larger and larger models you know
more and more parameters
um even even if we're in this case and
certainly if we're in that case
questions
uh so earlier you said that machine
learning models seemed biased against
finding this memorization based pattern
so why does that seem to happen here and
not in other circumstances like the car
example
yeah that's a great question so what's
happening here is we're sort of forcing
it here we've sort of put it in a
situation where it's just barely big
enough that it can fit the data if it
like devotes every parameter to
memorizing it exactly but it's not large
enough it doesn't have enough parameters
to to sort of represent anything you
know sort of more interesting than that
and so because of that you know one of
the restrictions right is you can think
about the machine learning as well it
tries to find this you know this simple
Basin but it's always going to find you
know some Basin which actually results
in good performance right if there's
nothing you know if if all of these sort
of possible simple things that it you
know that it can find that it has
available to it none of them actually do
a good job on the data well then it's
not going to find them right it's only
going to find things that actually in
fact do a good job and so we put into
this situation where it sort of he
doesn't want to find these sorts of
um you know things which are
um you know memorizing but uh We've sort
of forced it to because it's sort of the
only option in the space available to it
that's sort of the theory for for what's
going on there
I'm still not quite so sure what happens
when we extrapolate this even further so
we are cutting this graph off at a
hidden dimension of about 500. but
today's Transformers are much larger
rights
um
is there something on like a paper that
looks into what happens when we actually
drive it in higher and
I had regard also on observation here
steps the test loss
doesn't seem to dip below its initial
like below point is that something that
you can make do with by increasing the
heater size yeah if you if you keep
making it larger uh at least along like
it's a little bit unclear if you're just
looking at this one particular Dimension
but if you're just like making the model
overall larger yeah you will eventually
reach substantially better performance
than any other point by extending this
graph uh you know out in that direction
um I will say so in practice yes we do
do things that are much larger than this
we also do a lot more data than these
were trained on right so that that
changes the overall structure of the
graph
um
but yeah uh you know it is absolutely
the case that and and this is pretty
well studied um that you can sort of
replicate this um in many different
contexts
um though exactly where you'll find each
individual point will vary based on you
know the amount of data they're using
and stuff like that um but yeah um you
know if you keep going you will start to
get even better and better performance
um most of the time because
um you know the sort of compute
efficient Point often will end up being
relatively early in this graph that's
because you have huge amounts of data
that sort of push the whole thing out
and then we sort of find you know
something relatively early on that is
able to you know fit the data
you know both of these are sort of valid
strategies for sort of finding simple
algorithms either being really early or
being really late you just don't want to
be in the middle
um in both cases you know what you're
doing is you're sort of trying to find
something which is you know structurally
simple and able to fit the data and the
question is just sort of you know which
is the best place to be on this graph to
find something which is you know
structurally simple now you know the the
important point though you know the
takeaway sort of that I want here is
just that
um when the reason we build these larger
and larger models you know the the value
of them is that they are generally able
to help us find these more structurally
simple algorithms in both cases you know
either whether we're way over there or
whether we're over here in both cases
the larger models help you find simpler
things because as I have a larger and
larger model I can use you know more and
more data and you know find some simple
thing here or I could you know push it
way over there and find some you know
simple thing over there in either case
what we're doing is we're sort of by
having a really a much larger space of
possible algorithms to search from we
can find you know ones that fit an
add-on yet are
um you know are still sort of
structurally simple and meaningful way
question
there's a little counter-intuitive to me
um in some ways because in like
traditional machine learning
um larger models we sometimes think of
as finding a more complex solution was
here you're saying that larger models
are able to find on the simpler solution
and I'm wondering
um if there's something fundamentally
different about neural Nets or
Transformers or am I making the false
comparison between parameters and the
sense of the neural net and complexity
or large in the sense of a simpler
algorithm yes you're definitely so
totally valid and and this sort of
classical machine learning you know take
would just give you this part of the
graph this early one and would not give
you the later part and so the later part
is a little bit you know confusing and
it's sort of unique to machine learning
um but what's Happening Here is
fundamentally not that complex so
um what's happening is
that um
what we're doing is we're we sort of are
doing what's you know with the sort of
author's original you know double
descent paper we'll call this is
interpolation where the idea is that you
know in in sort of the standard you know
machine learning sort of in the standard
like uh you know classical Paradigm the
idea is well we just have some you know
space and we're just gonna we we're
gonna you know take the space and we're
just going to select those algorithms
which in fact do a good job on some data
and because of the finiteness of our
space that gives us some guarantees
about how good those algorithms have to
be on new data that's sort of not what
machine learning is doing right the
space is often so big that it's not the
case that we're sort of getting our
guarantees from the finiteness of the
space the reason that it does well is
because of this sort of you know basic
difference between you know some
algorithms being favored to be
implemented and some not right so some
basins being larger than others some
things being you know you know have many
possible parameterizations to implement
them
and so what you can think about this is
happening right is that
um what's really going on is that rather
than sort of you know just doing this
sort of finite you know space thing it
has you know a basic you know prior that
describes you know for each individual
type of algorithm you know how simple is
that algorithm how reasonable is that
how structurally useful you know is it
and what machine learning does but you
know what we've able to we've been able
to find these you know uh architectures
and these search processes which result
in being able to select out the you know
algorithm which is sort of does a good
job on the data while also having those
sort of properties
um that are sort of being selected for
by the structure right you know that are
in fact simple algorithms
um and because of that that's where we
get our guarantees right the the reason
that it generalizes is not because we
sort of forced it to by selecting from
some small space you know the the
algorithm that actually did had good
performance it generalizes because the
real world actually has the property
that it is simple right and so you can
think about this as you know the
guarantees of machine learning you know
the ability to sort of learn algorithms
which actually work in practice in the
real world what didn't work if you were
in a world where real world data wasn't
simple right if you were in a world
where like real world data was just like
totally randomly selected it would not
be the case that any you would see any
of this right you wouldn't be able to
you know machine learning wouldn't work
in the same way you wouldn't be able to
find you know this sort of property
where you can just sort of you know take
these sort of things and find the big
basement and that actually does a good
job
and so you know
um because of this it's not coming from
the sort of basic mathematical property
about like you know uh you know the sort
of these finite space properties it's
instead coming from the sort of property
of the world that like you know real
world data actually is simple and we're
able to find you know things that are
biased towards simple algorithms and
that's why it does a good job so in that
sense it's sort of it's sort of in a
different regime
um you know it called this interpolation
right where it's like we're not trying
we're trying to sort of take data points
and find the sort of simple thing that
goes between those data points right
rather than
um you know trying to just sort of like
you know uh fit fit in some sense um
yeah those are class scores
yeah
I believe last year there was a paper
called the chinchilla which showed that
modern language models were
unbetrained in terms of the data they
use and that should actually be smaller
but they're trying a lot more data is
this basically showing that they could
go further before they reach that first
optimal point on this graph
yes that's a good point so I I won't go
into
too much detail on the structure of the
scaling laws
um
but what chinchilla was showing was was
essentially that you know if you do your
training right because there was some
sort of problems with you know stuff
like the learning rate schedule and the
original scaling loss paper
um you can make it so that the sort of
where you where you find these sort of
optimum what the slopes are ends up
being different such that what they're
sort of trying to calculate is the sort
of compute optimal point so how many
data points how large of a model should
you pick to find the best performance
given that you have a budget in terms of
you know how many you know GPU Cycles
are you willing to spend
and that ends up being a little bit
tricky it's often very dependent on like
you know how we do sort of computation
um but one of the things that chinchilla
shows that is that they're sort of
through these relatively natural scaling
laws where you know the ratio of the
amount to which you should increase data
and the amount to which you should
increase the size of the model
um actually when you do it right ends up
being one to one you're sort of uh you
you sort of increase them uh you know
together uh in the sort of fixed ratio
and so
um this is an interesting fact that sort
of chinchilla has found
um but it's not super relevant here it's
just like okay if you were sort of to
think about how do you what happens when
you sort of scale these graphs up and
you look at the trends and sort of where
things you know where things and end up
you know what sort of Trends do you
generally see
um but the trends hide a lot of things
they hide a lot of you know okay what's
actually happening in practice in terms
of what algorithms we're implementing
you know one thing that's kind of fun is
that if you look at this at this graph
you can see there's this little bump
right here and at the time when this
graph was made I don't think anybody
really understood what this pump was but
I think I can say with relative
confidence that we Now understand
exactly what's happening here
um uh it's learning a particular type of
algorithm like the windows and top of
Wheels algorithm called an induction hat
so there's a paper that finds actually
when you do language training you almost
always see a bump around here uh that
describes the formation of a particular
type of algorithm so it's pretty
interesting
um you know what's what's sort of
happening when you sort of you know look
at the individual details when you zoom
out you know you can see these sort of
nice scaling laws that describe these
sort of General properties about you
know uh you know as we change the data
and the model size how are we changing
our ability to find simple algorithms
that fit the data in general and those
sort of properties tend to be relatively
uh continuous and predictable but the
actual algorithms that it ends up
implanting are oftentimes you know can
vary a lot
cool okay another uh nice example of
what's happening in this sort of
particular case of you know when you
have really large models is rocking so
this is a case where uh take a
arithmetic task so uh you just sort of
trained on on a particular uh you know
math mathematical task and
um we have the sort of training loss uh
uh in in red or this uh so or I guess
this is not this is accuracy not lost so
so higher on this graph is better where
um originally you know after after some
number of steps it learns exactly you
know to fit the data perfectly
but on the held out data it's learned
nothing it has no ability to find any
algorithm which is Meaningful to solve
this larger data because it's
effectively just memorized something but
if you keep training for long enough
eventually what you find is that it does
learn it the it does find the
structurally simple algorithm which is
able to solve the task
and so you know what's happening here is
this is another case where it's you know
as we train for longer as we have larger
models we're able to find these sort of
structurally simple algorithms and once
we've found them they're sort of these
really powerful you know basins these
attractors that uh you know we can stay
in and can really help us solve problems
but they take these you know oftentimes
really big models and a lot of search to
actually find things which successfully
are able to learn these sort of
structurally simple algorithms and so in
this case it you know it takes a really
long time and a lot of optimization
before it finally clicks and learns the
actual sort of structural
um you know arithmetic algorithm which
solves the task
and so that's what's happening
eventually here at the end
um but you know it can it can sometimes
you know take a lot of search around the
space until it finally finds this you
know this sort of basin which actually
has this sort of structurally simple
thing Yeah question
looking at the nose ER grounds but
foreign
accuracy here
yeah
if we're looking at the loss instead of
the accuracy would look look very
similar just uh you know inverted yeah
very similar
we're trying to optimize for training
loss in terms of our gradient descent
how is it that the AI doesn't just get
like stuck at a point where it never
learns anything how is it an eventually
actually does learn to generalize
despite not really having any more gains
to be made by doing so
yes that's a great
just optimizing right for performance
right it is in some sense sort of
looking for things which are simple
right and so the question is like well
how is it looking for things that are
simple and it's a little bit complex
there's multiple different factors so
one factor is just the fact that we do
do regularization right so we actually
have regularization is just the process
of we force it to learn simple things
right we sort of force it to find
algorithms which are you know have small
values for the weights so that they're
you know smooth and continuous and so
what that means is it just has to keep
searching a lot until it finds something
which actually is you know able to have
small you know simple weights and still
you know simple you know parameters uh
weights are just the parameters in the
parameterized function which in fact
result in good performance and so so
regularization is an important component
of this
um but also just you know the stuff like
Basin volume right where it's like okay
as it keeps searching around you know
simple structurally simpler algorithms
have this property that they're sort of
attractors in the space right that you
know once they tend to octave by larger
volumes they're sort of once you find
them you sort of tend to stay there
and so you know as we keep searching we
can find these sort of you know
structurally simple things
um you know because of those sort of
basic properties of of the space Yeah
question
is a structure the simple algorithms
have huge basins can you help me
understand why is it true
yeah yeah so great question so it's a
little bit unclear I think that you know
uh everything I'll say was just sort of
you know some speculation but I think
there's a couple of things that we can
point out so one thing um that I was
talking about a little bit previously is
this idea that
um when you have simple algorithms
there's more ways to implement a similar
algorithm because uh if I think about it
let's let's think about it this way so
let's say I'm only using half the
weights because all I have to do is
implement this like you know relatively
simple thing it only takes a couple of
computations and the rest of them
basically don't matter well if they
basically don't matter then that means
any possible value for them is all in
the Basin all of the values of those
remaining parameters all end up in the
same base sentence so because of that
um you know it ends up occupying very
large volumes that's one reason right
you know the sort of possibilities
redundant parameters
um also algorithms that have symmetry so
if I have if there's like two different
ways to implement the algorithm that's
sort of effectively symmetric with each
other
um then you know both of those will sort
of occupy the same Basin for that same
algorithm and so algorithms that are
sort of very symmetric that have the
property that there's multiple sort of
symmetries in how you implement them you
know something where it's like uh you
know if I sum a bunch of numbers I can
either start from the left side or I can
start with the right side you know
there's multiple different sort of same
similar ways you know you know symmetric
ways do the same algorithm those will
end up having you know multiple ways to
implement it in the same Basin and so um
you know those are some factors right
you know that there's there are multiple
ways implanted that can yield these sort
of simpler algorithms have larger basins
I mean also other factors like
um just the regularization right you
know we we force it to try to find uh
you know parameter uh you know
algorithms which have the you know low
parameter values we often do stuff like
Dropout where we actually will take some
parameters out randomly and force it to
still be able to work even if we remove
parameters and so that forces it to sort
of have to find things which you know
aren't sort of super brittle algorithms
and so we
um you know like one of the one of the
really important things here is that
it's not just that we like you know
Humanity just stumbled upon you know
some you know way of doing machine
learning you know which just magically
always results in like simple things
that's sort of true but in important
part of the picture is also what we
wanted it to find simple things right
like the reason that we do machine
learning in the way that we do it is
because there has been uh you know years
and years of progress on trying to find
the particular ways of parameterizing
functions which in fact have the
property that when we search over those
parameterizations we find simple
functions right because that's the thing
that actually gives machine learning
it's it's power its performance it's
what lets it you know find these
interesting functions they're actually
able to solve complex tasks
um and so that's an important part of
the story as well
intuitively it seems to be like Dropout
might lead to a more complex algorithm
like I'm imagining I've got a fourth
step process to solve a function but it
one of my steps might just stop working
10 of the time I feel like I would need
a more complex solution to actually get
to the right answer every time so why
does Dropout actually simplify these
algorithm instead of making them more
complex because then for redundancy in
the light
yeah that's a really good question and I
think not one I have
because I think the problem is you know
we don't actually really well understand
exactly what it what it does do you know
things that we do understand are
oftentimes it does seem to improve
performance and you know not just train
performance right but test performance
and so to the extent that we believe
that like real world data is actually
distributed according to you know some
something like Simplicity that means you
know by you know Occam's Razer you know
it has to be learning something simpler
but that's obviously not very satisfying
you know we really want to understand
well okay why how is learning you know
some simpler algorithm
um and it's a little bit unclear right
so I mean I describe something you know
in Broad Strokes that's like well you
know in some sense you know if it's
really brittle you know that's a really
complex algorithm because you know
simple algorithms should you know have
you know this property that they're sort
of continuous that they don't depend on
like tiny little pieces now you know is
is that true I mean you know I think one
of the ways to think about this right is
that when I say simple right what I mean
is not you know the same as what it
means right to humans right simple is
using big quotes here because simple is
just whatever the actual property is of
the world that results in you know
having you know algorithms which have
that property do well and whatever
property it is of neural networks which
in fact results in you know uh it being
you know uh generally finding algorithms
with that property and so you know we
can't always sort of interpret it in the
same way we would try to interpret sort
of you know our notion of Simplicity
um and especially in the case of Dropout
yeah I agree that the thing you're
saying makes intuitive sense and I wish
I had a really nice satisfying answer to
why like okay but actually the notion
Simplicity doesn't work that way
um but I don't I don't I think we just
actually don't really understand you
know super well exactly how the you know
these sorts of you know biases work
yeah
I I remember raising a paper from 2016
by
generalization and in this paper
it was kind of strange because they
showed that
you could train a collisional network to
fit noisy data entirely
and if you train the modern to classify
proper data bits with also partially
noisy data it first learned to properly
classify the real data and then over fit
to the noisy one which
seems to imply that
it nerds first the simple algorithm to
classify
the data well and then
have to rely on overfitting which seems
to be the opposite of what's going on
with working so do you think it's
because of uh
in the case of croaking modular division
it's harder to learn than learning the
Detector by herbs yes this is a really
good question so I think that what I
would say there is what's happening is
that if this is this graph here this
thing that we call Double descent where
you know when you're when you're early
on right when you're in this initial
regime then what you're doing is you
know you're you're sort of you know as
you keep making it larger as you keep
training you end up in a worse spot
because you sort of do this overfitting
where you end up uh you know learning
you know just sort of memorize a bunch
of data points but if you kept going you
probably would eventually see that you
know the second part with even larger
you know more training you would start
to do better again
um groking is just sort of a very
extreme example of that if you look
closely you can actually see that there
is a double descent right where you
actually you know you you do go up and
then go back down again and then
eventually you just have this very
extreme thing at the end where you find
the sort of exact simple algorithm
um what the shape of that sort of double
descent curve is and how it looks will
vary in the particular setting but yeah
I mean I think that basically that's
that's a phenomenon that's going on
there
okay uh so we'll talk a little bit about
what this looks like in practice so uh
you know we've talked a bunch about the
sort of relatively simple classification
problems cases where you just have you
know shapes in color or you know where
you're just sort of trying to solve some
relatively straightforward
classification problem like that
um in practice that's not always how we
use these sorts of systems so you know
one example of a thing that we will
often do is reinforcement learning
um I think you should not think of
reinforcement learning as really
fundamentally different than anything
we've talked about so far
um structurally it's doing something
very similar so what we are doing is
rather than searching for the function
which sort of has the best performance
on the data we're searching for the
function which when it acts in some
environment results in the best
performance in that environment so we're
sort of instead of searching for the
sort of you know simplest function which
does a good job at um you know doing you
know classification of images we're
searching for the simplest function that
when you give it a go board and it you
know produces a go move if you iterate
that many times results in good
performance at the game of Go and so
we're still fundamentally doing the same
thing where we're just searching over a
large space of parameters trying to find
those set of parameters which in fact
result in good performance on some data
um but now we sort of have this
interactive setting where the good what
good performance means means does it
have the ability to actually play and
win over you know many series of
interacting with an environments and we
use various algorithms for being able to
you know search over that space in a
sort of meaningful and effective way but
it's still essentially doing the same
thing we're still searching over this
large space of possible parametizations
trying to find some algorithm with which
has some particular property on some
data
okay and so you know we this is this is
often you know yields great results in
practice uh you know like stuff like
alphago
um uh but but structurally it's doing
something very similar uh where you know
we're still trying to find the symbolist
algorithm uh it's just you know now
we're sort of trying to find the
simplest policy the simplest sort of uh
you know way to um act in an environment
okay
uh now you know probably I think you
know if you're familiar with you know a
lot of the big advances of machine
learning you'll be familiar with stuff
like alphaco it'll probably be even more
familiar with uh language models large
language models you know that probably
are the most you know well known and
maybe probably the most powerful uh you
know the existing sorts of models that
exist today uh and they're really good I
think that if you have never interacted
with a large language model before or
haven't really interacted with one very
much I think there's really no
substitute to just doing it yourself I
think that uh the best way to sort of
get a handle on what it is that these
things are and like effectively how they
work and you know what they're capable
of is you know find you know one online
that you can interact with you know chat
gbt whatever and just sort of talk to it
ask it questions I think this is a
really valuable experience for anyone
who's interested in trying to understand
AI uh and how it looks right now to do
um but but long story short is they can
do a lot of really impressive stuff
right so what we did is you know we just
you know found the simplest algorithm
which fits the entire internet uh you
know effectively and it turns out the
simplest algorithm that fits the entire
internet uh you know can reason it can
write stories it can talk you know in a
human sensical way it could do a lot of
really crazy stuff
okay
um but fundamentally you know what what
it is that it's doing you know is um
it's hard to understand because now we
have a data set that is so large and so
complex that you know the simplest
algorithm is able to fit a data set that
complex and that large well it need not
be that simple of an algorithm anymore
right the out the you know once we put
that constraint that it has to be the
simplest algorithm that fits the whole
internet that's a strong constraint and
it means that the resulting simplest
algorithm might still be very complex uh
and very complicated and so you know
unfortunately our understanding of what
mechanistically large language models do
you know what happens when you take the
simplest algorithm that fits the whole
internet is uh is quite poor
and so we'll talk uh a little bit later
on in lecture series after a bunch of a
bunch of lectures we'll return to this
question we'll talk a little bit more
about what it is that large language
models might be doing mechanistically
but even then all I can promise is
speculation so we're going to speculate
a little bit later on about what it is
that these things might be doing but um
it's complicated and you know our
current our understanding of exactly
what mechanistically they are doing is
is not super great you know uh what we
have is just well there whatever the
simplest thing is you know that fits
fits the internet uh you know whatever
that is you know and even then it's like
well you know exactly what the inductive
minuses you know inductive biases are
just a name for uh whatever it is that
determines the sort of metric of
Simplicity in in neural networks and in
machine learning um whatever those
inductive biases are whatever that
metric of Simplicity is you know uh even
then we don't fully understand it right
you know it's not exactly the same as
human Simplicity it's some notion of
Simplicity that does in fact a good job
and whatever that notion of Simplicity
is you know there's some algorithm which
does a really really good job on that
that we find when we train these sort of
massive language models and in practice
what we see is that whatever that is it
does a really good job in practice but
what it actually is doing is is quite
difficult to understand Yeah question
what do you mean when you say training
an algorithm that fits the entire
internet uh can you cover the board on
the data set that's when we compose we
retrain these language models and how is
that hey I'm literally training though
the entire internet
so it's basically true right uh you know
it's there's there's a couple of you
know kawaiyats here uh in terms of you
know exactly how scraping is done
exactly how you know data set creation
uh happens but basically the sort of
standard state of the art currently for
training very large language models is
make a massive scrape of as much of the
internet as you can find you know trying
to do some amounts of filtering and you
know extract out the sort of you know uh
you know data points which are actually
going to be most useful and relevant uh
and then just train on them you know
find some simple algorithm which is able
to fit all of you know internet attacks
which is able to predict what the next
word will be in any situation that it
finds itself on in the internet
um
what
the Western internet no it's not just
the Western internet uh in many cases
you know scrapes will exist of you know
basically anything that is you know
publicly available
um it really will depend on exactly what
you know case you know sometimes there
will be cases where people will filter
to just English text
um that's not uncommon
um but it's not always what's done so
you know it it just sort of varies on
exactly how you want to train your model
and what you want it to be able to do um
if you're mostly interested in just
being able to interact in English then
you know it's going to be you know
wasteful to sort of train on a bunch of
things that are not English and so
oftentimes you can try to you know just
filter down English but that's not
always done
um so you know it really depends but
essentially it is you know take a really
large scrape of you know a bunch of
texts from the internet that is as high
quality as you can make it while still
having a huge amount and then just spend
millions of dollars training you know
some really really large parameterized
function that in fact results in good
performance on that data set
so the answer this is going to have to
be just speculative but
given what we've learned to before about
double percent and how eventually the
model reaches a point where the simplest
function it can understand that fits the
dog the best is just memorize everything
it seems that current language models
have not reached that point do you think
it's possible that we could end up with
like gbt4 or gbt5 or something that
actually does just memorize the entire
internet and then gbt6 would actually be
worse than gpd5 if they continue to
expand it
um so that's only possible if the sort
of people building these models sort of
you know made a mistake because
um everyone knows you know the graphs
that I was presenting right and so we we
sort of have actually pretty good
ability to be able to predict and
understand where the various basins are
going to end up on those graphs and so
you can extrapolate and pick exactly how
many tokens to train on how large of a
model to use uh for the purpose of
ending up in a spot where you know
you're you're sort of scaling predicts
it's going to have good performance
um so that's what people do in practice
and so
um you know if you sort of just did the
naive thing of just like take the exact
same model and make it bigger and bigger
and bigger and bigger uh well like
changing no other properties of it then
yes you would you would encounter that
but that's not what people do right they
actually change the amount of data and
another factor is as you sort of make
the model larger to you know prevent
that from happening and so um yeah if
you were sort of doing something naively
you would see this but because people
are not doing it naively you know you
you shouldn't see that
so basically you'll end up with this
point where
the model might need to either need more
data or a lot more compute to get better
performance but we would know that
instead of just spending millions of
dollars on training a model I'm like
whoopsie
yeah I mean so it is possible that like
some of these sorts of you know scaling
laws could break you know things could
happen differently than we predict but
um you know we do at least understand
this phenomenon enough that you know
it's unlikely that it would just be you
know a situation where you know we just
trained it into accidentally into the
sort of wrong regime
um there may be other cases where
performance decreases for other reasons
so there are examples of what's called
inverse scaling where as you sort of
scale up your language model you know
various different metrics that are of
like desirable properties can can go
down in certain cases oftentimes you
know one example of this is cases where
um there's sort of a you know misleading
answer to a question where the model
will sort of be you know larger models
are often better at being able to sort
of convince you know people of these
misleading answers and so you know it
can sort of you know do worse on a like
you know how good is the information
quality as the the model gets larger
sometimes but that's still it's not a
case of like you know oh you know we
just sort of like you know totally
screwed up and like trained the wrong
you know size model it's a case of okay
there's some sort of more complex thing
happening in what it means for a simple
function to fit this data right there's
something going on in how the sort of
function operates that causes it to sort
of have this particular Behavior change
as we get sort of simpler you know
functions that are better fitting
without
cool okay so we're going to sort of
shift gears now uh for the last part of
this talk so uh you know we've talked a
bunch about machine learning and we've
talked about you know this basic process
of you know taking these large data sets
you know Finding simple functions which
do a good job on the data set
um and you know the sort of fundamental
problem that we've encountered is that
we often can't know exactly what it is
that the sort of simplest function that
fits a data set is doing because we
don't fully understand what this notion
of Simplicity is how it operates and
sort of what it corresponds to in terms
of what the actual algorithms are that
these models end up implanting and so I
want to sort of take a step back and ask
a sort of basic question which is what
is the worst case scenario right what is
the worst possible thing that it could
be that these models are doing
um that we would be concerned about
um given that you know we don't know
right you know if we are in a situation
where we're building really powerful you
know systems and we don't understand you
know structurally what those systems are
doing
um I think it's you know prudent to take
a second and sort of try to understand
you know okay how could things go wrong
what are the worst possible scenarios uh
for what these models could be doing
okay so that's that's that's part two of
the first talk so uh what we're gonna be
talking about here is this notion of
instrumental convergence
um so I'm going to borrow an example
from Stuart Russell who's uh professor
of AI at UC Berkeley uh and likes to
talk about this sort of thing a lot
where he sort of uses uh a sort of
analogy to illustrate the sort of why we
might generally be concerned about the
situation where we have very powerful AI
systems in the world
um and situations where they start to
get more powerful than humans
uh he sort of calls this the gorilla
problem
so the question that we sort of that he
likes to ask here is well
um you know what is the second most
intelligent species on the planet
um you know I mean pick your favorite
great ape I'm not gonna claim it's
definitely the gorilla uh I'm not an
expert but uh whatever we'll suppose
that it's you know it's the gorilla you
know substitute chimpanzee or whatever
other you know animal you prefer it's
probably something like that is the most
the second most intelligent species on
the planet after after humans
um and so you know what what is their
fate right you know what determines the
ability of the gorilla you know as a
species and of individual gorillas for
them to be able to you know Thrive and
live good lives and and do the things
that they want to do
um and the answer is while it's mostly
not about the gorilla he mostly has very
little to do with what the gorillas do
and almost everything to do with what we
do right um you know the ability of
gorillas to continue to have a habitat
to continue to exist and not be
endangered to you know Thrive to be able
to you know find food and resources and
live their lives is dependent on our
ability to let them do that you know
it's dependent on things like
environmental measures on things like
global warming on things that like
environmental policy that the gorillas
probably don't even understand or
comprehend or could comprehend
um but that's what determines their fate
right the thing that is going to
eventually determine the fate of the
gorillas is the you know way that we uh
uh you know treat them in the general
policies that we build
has nothing to do with the gorillas and
so this is a sort of you know uh maybe
striking example of a case where um you
know it's not great to be the second
most intelligent species on a planet
um you know even though you know maybe
we you know people have tried to
communicate with the gorillas it's not
like they're you know totally you know
incapable of communication they clearly
have the ability to you know have you
know meaningful preferences in some
cases
um but their ability to act on those and
actually you know you know is is highly
constrained by whatever you know however
we treat them
and so if we are in the business of
Building Systems that we expect you know
maybe we'll eventually be uh more you
know more intelligent than us
um you know we should be concerned that
we may end up in a similar situation
with the roles reversed where you know
if we are building uh you know systems
which are more intelligent than us that
you know our fate May lie more in the
hands of those systems that than it does
in in ourselves
and that's a concern right you know if
we're talking about you know originally
what is the sort of whole point to this
but we want to avoid egg Central risk we
want to avoid situations where humanity
is disempowered where we no longer sort
of are able to control our own destiny
and so if we are in a situation where um
you know that control is rested away
from us in the same way it sort of rests
away from the gorillas
um we're concerned
okay so you know this doesn't say
necessarily that we should be concerned
you know maybe you know AIS will treat
us well or maybe you know uh you know we
don't know right we don't know what
they're doing but we want to understand
the worst case scenario so um we want to
understand
um you know okay if Our Fate is in the
hands of these sort of very powerful AI
systems
um you know we don't know what they're
doing but what things could they be
doing that would be bad for us
so I have here a Minecraft map
and I sort of want to consider a thought
experiment
so let's say we are at spawn we're at
the center of this Minecraft map and we
have some objective you know maybe we
want to build a house in you know the
top left quadrant we want to you know go
thousands of blocks away and build a
house
what do we do right you know what's the
best strategy for us in building that
house well you know what we're going to
have to do is you know we're gonna have
to you know build a boat you know chop
down some trees you know use those to
you know build Transportation we're
going to have to uh you know gather
resources start you know mining you know
gold and diamonds
um okay those are some of the things
right that we would have to do if we
were to try to you know build a house on
the top okay but let's say instead of
trying to build a house what we wanted
to do instead was you know blow up with
TNT the entire you know you know Chunk
in the bottom okay that that's a totally
different goal right it involves
completely different you know objective
uh but what's the best way to accomplish
it well we're still going to need to be
able to get ourselves down there where
we want to be able to blow things up so
we're still going to have to be able to
you know build a boat gather the
resources to be able to transport
ourselves and we're gonna have to get
all the TNT from somewhere so we're
still gonna have to mine things out and
you know kill a bunch of creepers and
gather a bunch of resources so we can
actually build the TNT that we need to
be able to blow the thing out so it
looks pretty similar right you know we
have these very disparate goals you know
in totally different places where we
want to do very different things
um but in this sort of setting they they
uh they require inputs resource inputs
um which are very similar they require
sort of you know same basic resources to
accomplish these very disparate uh uh
objectives
um and so there's sort of a basic
structural property of the world at play
here
um and an important uh yeah so important
thing to point out is that this sort of
structural property here that we're
talking about it's not a property of you
know the goals per se it's a property of
the world it's a property of the fact
that in in Minecraft there are resources
uh you know you know items you know
blocks you know things that you need to
be able to do almost anything in the
game
and because of that property of the
world almost anything that one might
want to do in Minecraft requires these
same resources and what that means if
you have multiple agents in the world
that might want to accomplish the same
goals or or might want to accomplish
completely different goals they're still
going to have competition for the same
basic resources
okay so I'm going to borrow another
example from Stuart Russell
who likes to call this problem the the
sort of problem of you can't fetch the
coffee if you're dead so the idea is you
know you have you know some model you
know some agent some system and it has
some some goal whatever it is you know
in this case it wants to fetch coffee
and the problem is well Fetch and coffee
has you know basic resources that you
need one of which is your own Survival
you know you need to not die right and
so in the Minecraft example you know in
all of the cases I'm going to probably
need to get some gear you know to
protect me right against enemies in this
case you know I I need to make sure I
don't die if I'm going to fetch coffee
um and so there's basic structural
resources basic things that are
necessary to do almost anything anything
that you might want in the world
requires the same sort of basic inputs
in many cases
um and of course you know this is not
just true of Minecraft it's true of the
real world as well so you know this
there there are basic resources in that
exist in the world you know uh materials
uh you know matter energy you know neck
entropy you know uh you know atoms of
various different varieties elements you
know all of these things you know that
exist in the world and one needs to do
almost anything right anything you want
to build anything you want to accomplish
that happens in the world requires the
same sort of basic resource inputs
almost regardless of what it is
um and uh this sort of structural
property of the world uh you know the
sort of basic property of the world that
you know different very disparate goals
can require the same resources is is the
property we call instrumental
convergence where
um the same sort of you know all of
these very different goals can Converge
on the same instrumental goals Where You
Know instrumental here is just a thing
which is useful for accomplishing
another thing right and so the things
which are useful for accomplishing other
things tend to be very convergent across
sort of many different possible things
one can do in the world
one thing I would point out is that you
know I did sort of make a hidden
assumption here which is like well why
would your AI even want to accomplish
anything in the world at all
um and that's an interesting question
it's one that we'll sort of going to be
talking a little bit more about in the
next two talks
um but uh the thing I will say right now
is well okay
um you know we don't know you know it
might it might not be um it's very
unclear you know whether whether we will
train systems that are trying to
accomplish things in the world but the
point is that if they are trying to
accomplish things in the world and we
don't fundamentally understand why they
would or would not be
um almost regardless of what it is that
they might be trying to accomplish we're
we're concerned because humans want
those resources too to be able to do the
sorts of things that we desire and if we
are in competition with AI systems for
those resources it could be um quite
tricky
okay uh question
here you go
what is my goal as Journey just to make
coffee within five minutes sure they it
would be easier to just make the culture
than to fight the people over the world
and steerable cripples resources
yeah so I think that's a good point I'm
definitely not claiming that like any
possible goal that you could have would
always yield your model you know trying
to take over the world
um I think that there's a couple of
things that we can say here so one is
like well okay we're trying to think
about the worst key scenario and we're
trying to think about okay how many
goals are there that result in bad
things and even in that case you know
there's still some resource competition
you know there's some resources that you
need you need to like exist for enough
time to get the coffee for five minutes
you need to be able to do enough things
in the world to accomplish that that
goal
and if you're really you know gung-ho
about that goal then you know you also
have to do other things like you know
you know protect yourself you know build
you know as many defenses at you know
against possible ways which you could
fail your five-minute task make sure
that you know you're as likely to
complete that task as possible and those
things could could yield increasing sort
of you know resource consumption
but the general point is not that
um anything that your AI system could
possibly be doing is going to result in
it wanting to you know kill all humans I
think that's that's not true I think the
point though is that
um we don't know what it is that they
are going to be doing structurally you
know all we know is something like okay
it's going to be something like the
simplest algorithm which you know fits
some massive data we don't exactly
understand what that algorithm might
look like or what it might be trying to
do in in you know in whatever sense
trying means
um and it's conceivably you know
possible that it could be trying to do a
very large variety of things many of
which if the things that it is trying to
do involve accomplishing something in
the world results in uh you know
outcomes that that we might not like
right result in outcomes where they are
competing for resources with humans
resources that we would we would like to
use um you know to make our lives better
um instead
and so you know there are many cases
where that resource competition is maybe
okay right so maybe we want you know if
we want if our AIS are competing
resources on our behalf they are trying
to gather resources for us in the way
that we would want them to be doing so
then maybe we're okay right you know but
there's a lot of possible things they
might be trying to accomplish almost
anything you know any sort of randomly
selected thing in the world you might be
trying to do
um where um you know if that's not you
know exactly you know the sorts of
things that we want
um you know that humans might end up in
not so good position
and I think that I'll point out about
this is if you're interested in this
basic phenomenon of instrumental
convergence
um it's something that you show that
shows up not just in you know Minecraft
not just in the real world it's a very
basic structural property of almost any
uh environment so even a randomly chosen
environment where you just randomly
choose what the states are in that
environment and what the transitions are
between different states has the
property that optimal policies in that
environment the sort of best way of
solving that environment for random
goals random desired States will result
in a property where the optimal policies
will tend to seek power where states
that are powerful in the sense that if
you go to that state you have the
ability to sort of unlock a large
variety of different new states that you
could go to from that point will be sort
of convergent for almost any way of sort
of uh you know accomplishing any random
goal in a sort of most optimal way
uh and so this is you know uh we can
sort of prove and you know study this
mathematically and so if you're
interested you can you can take a look
at the often policies tendency power
paper
um but the basic you know sort of
structure here is sort of very
fundamental to almost any of these sorts
of environments it's just the case that
you know in any case in any situation
where there are sort of you know General
properties States you know resources
which are useful uh for like doing
things then those resources are going to
be in competition and they're going to
be you know useful for a wide variety of
different possible uh you know things
that one might want to accomplish
and that's the core concern here
further being research a couple years
ago around satisfy service rather than
maximizers where they're just trying to
get like a specific threshold maybe
capping their instrumental weakened
burden goals
um I'm also kind of drawing up vanilla
here to wanting to create a system that
eventually is Happy turning left rather
than going right and kind of staying
there
um is there also kind of an element here
of the tragedy of the commons problem or
even if we knew how to correctly and
successfully do that
systems that go right or simpler in some
sense drawing back to 10 of this
parallel machine learning producing
silver systems and so there's this
really heavy pressure impetus towards at
least some of the systems always
perpetually dealing right and their core
kind of failing a tragedy that comes
from
yeah okay a bunch of questions there so
I'll start with the uh satisfizers so I
think satisfy is a really interesting
idea for people who are not familiar the
basic idea is well what if you try to
accomplish a thing right and then stop
sort of similar to The Proposal uh you
know that was that was talked about
earlier about you know what if you you
know get the coffee in five minutes and
then you're done right you sort of have
some level at which you want to achieve
your goal and then you stop
um it's a really interesting idea I
think the main you know most fundamental
problem is you know what if that's not
the simplest way right to fit the data
right we're sort of in this setting
where we don't get to pick exactly what
the structure is of the algorithm that
we find when we do machine learning
right we get some control over it right
we get a lot of control over the data we
get a lot of control over the
architecture but how the architecture
and the data combine to find the sort of
simplest algorithm that fits that data
is is a is a complex process that we
don't get a lot of control over right it
depends on these basic structural
properties about what algorithms are
simple
and so it may be the case that we can
find mechanisms for consistently
producing satisficers rather than
optimizers and it may be the case that
we can't I think currently I don't know
of a way that you could consistently
train models and believe that they're
definitely going to be satisfizers I
think that's a really tricky thing to be
able to do you can you can certainly do
things where you sort of train models on
fixed reward schedules where they you
know they can't ever you know get more
than some level of reward but even in
that case you cannot end up in
situations where your model wants to
maximize the probability of it achieving
that level of reward right where it
wants to you know maximize the
probability that it's able to you know
get at least that much of whatever you
know it is that's trying to accomplish
and so you know it can get very tricky
in terms of you know actually getting a
thing which is well described as
satisficer and even then if I'm training
on a reward which is just like you know
whatever the loss that I'm trading it on
I don't have a guarantee that my model
in some sense cares about the thing I'm
training on all I know is that it is
some simple algorithm that fits the data
well and that could fundamentally be
doing doing almost anything
and so I think satisfy is an interesting
idea but but it's how to actually get
them to work in practice is is is
unclear
um
and then let's see uh
going on the other questions about the
tragedy of the commons you know what
this might look like sort of uh you know
in terms of
um
you know in practice sort of how this
plays out I think that
it's really sort of unclear I certainly
think that you can sort of think about
what's happening here as a sort of
tragedy of the common scenario where
it's like okay you have a bunch of
resource competition it would be really
nice if we had some ability to you know
be able to sort of you know manage those
resources in a way that wasn't just you
know whatever the most powerful you know
intelligent species is is able to just
you know control over them
um but of course you know we we don't
have you know some mechanism that we can
use to just sort of constrain any you
know possible agents you know uh you
know certainly you know we have
mechanisms that exist in the world today
that we can use to constrain humans uh
you know we have things like laws and
governmental institutions you know that
we use to create a society that
constrains humans in particular ways but
we don't necessarily have the ability to
constrain some arbitrary intelligence
right you can imagine you know certainly
there are rule you know there are sort
of you know in some sense rules you know
maybe among gorillas right you know
where there are things that you know
they'll you know they punish you know
defectors against their you know you
know what what you know if you try to do
something in the you know gorilla
Society you know you uh you know maybe
they'll you know cast someone out of the
group or whatever but um you know those
those have almost no bearing on us right
you know we don't care you know that
there are you know particular you know
ways that gorillas punish other gorillas
you know that has no bearing on us
because we don't have to care you know
the we we are so much more powerful and
intelligent than the gorillas that
um you know it doesn't matter
um whatever structures they have and so
it's possible we could build systems in
ways that are able to constrain you know
a society of you know humans and AIS you
know having very powerful intelligent
you know AIS in the world um but It's
Tricky right you know we can't just do
it via the same you know mechanisms that
we use to constrain humans because you
know those mechanisms are limited by you
know humans actually being willing you
know to cooperate and uh for us to be
able to have the ability to you know
make them do what we want when they
refuse to cooperate and it may be very
difficult to do that if you have a you
know system that is substantially
smarter than you that can you know
outfit you in any situation
and um so that's potentially a concern
okay so if there's any last questions at
the end we can sort of I can take a
couple of last questions but that is uh
it for lecture one uh we're gonna you
know there are still a bunch more to
talk about and we'll get to those uh
later but that is the end of the first
uh the first part
[Music]
[Applause]
okay final questions
so uh you mentioned at the very
beginning of this talk that a lot of the
assumptions would be made in was that
prosaic AGI
but the Mosaic systems could lead to AGI
How likely is this actually to be the
case
yeah that's a great question so I think
it's very unclear uh you know predicting
the future is always you know a very
tricky thing to do and you know in many
cases you know that's sort of the
business that that we're in right now
we're talking about this sort of thing
there's a couple of things that I'll say
so the first thing I'll say is
um
you know
in the case of like you know prosaic AGI
I think that
um our sort of best guess should be it's
probably going to be something like what
we have right now right you know because
it's so difficult to try to predict and
understand exactly you know what future
developments might exist
um by default we should guess well okay
you know it's going to it's not going to
be the same as how things are work
currently but our best guess should be
how things work currently because we
don't we can't predict exactly in which
direction is going to vary from where we
are right now
um and so I think that you know trying
to start from if it was like you know
current systems how would we be able to
understand uh you know what to do in
that situation
um is a really good starting point and
as you know new Advanced is common we
sort of find new ways in which you know
uh you know AI is developed we can
change you know that understanding just
to suit those new directions but you
know by default we should expect those
new directions are probably going to be
something similar to what exists right
now and so understanding what exists
right now is sort of a key
um you know uh input into being able to
understand things in the future
and then the other thing I'll say is
that you know it is very difficult right
to predict exactly when things are going
to happen but you know I think that
right you know because because all of
this is you know a prediction but I
think that in many cases there are some
things that you can predict right uh
predicting exactly when something is
going to occur is very difficult
predicting that something will occur is
much easier right so we can understand
okay AI continues to keep getting better
and continues to keep getting more
intelligent
um as long as that Trend continues at
some point it will surpass humans at
some point it will become the most
intelligent thing and we'll sort of have
to deal with the consequences of that
exactly when that will happen you know
where do those graphs meet you know how
what is the slope how do they you know
change that is very tricky to predict
but the fact that it will happen at some
point you know barring some other
essential catastrophe that wipes out
humans seems quite clear you know
there's no fundamental barrier to you
know us can you know being able to build
systems that are smarter than us and
even you know a very large number of
systems that were the same intelligence
level as humans would also be you know
equivalently existentially dangerous and
so um
you know at some point it has to happen
and you know exactly when I don't know
and I'm not going to try to speculate a
bunch on on when because I think it's
just such a tricky thing to do uh and I
think this sort of involves you know a
lot of you know you know analysis and
also just a lot of you know opinions
um and it's very difficult and so you
know I I think that speculating on when
is hard but I think that we can talk
about these sort of General properties
and we can try to sort of you know
approach the uh task of being dealt you
know with this problem of how do we deal
with you know the problem you know the
problem that in the future these very
powerful systems might exist it might be
dangerous you know in as effective a way
as we can which is well we we have to do
something because you know the problem
is concerning and we want to try to
address it
um and so you know we're forced to try
to make whatever predictions we can that
are as good as possible
and you know so so we can try to you
know build upon what we have right now
try to understand how things work and
you know basically chart a course you
know for for being able to understand
future things
um but it's hard and so you know one of
the things we'll talk about sort of
later on in some of the other lectures
is telling multiple stories right where
like we don't know exactly how things
are going to play out and so we can
imagine you know multiple different ways
in which you know sort of things go so
for example you know one thing that
we're we talked about a bunch here is
we're really
um you know uncertain about exactly what
the inductive biases are right exactly
what the measure of Simplicity is that
models use but we can sort of Imagine
well what if we think about multiple
different possible ways that it could be
and then under different scenarios how
would things play out you know
um and so we'll sort of use that as a
mechanism later on for trying to
understand particular scenarios because
that's a way to sort of deal with our
uncertainty you know we don't exactly
understand what it is that is true but
if we think about multiple different
possible options we can see what's
convergent between those options and
that sort of can tell us what things are
really you know likely in practice
you mentioned one of the things you
encourage people to do is to pick up a
large language bottle at Chachi Kadin
and play with it directly just to get a
sense that it's capability that's
outfits it seems like you're at the talk
the actual process of understanding
basins and it will kind of pathway
through basins this ticket is also very
important to build it into a shop how do
you recommend people build that kind of
intuition through maybe toy models or
for more kind of theoretical
understanding of what's going on like
what's the best way to build an
intuition of that
yeah that's a really good question I
think part of the problem is that it's
just not super well understood what
exactly it is that
um
that that's sort of happening with those
with those basins
um
and so I think that
um you know maybe the best way to sort
of get an understanding there is just
um you know looking at what some of the
models are that actually exist and how
they work and what they do
um and sort of trying to understand what
is this what is this metric of
Simplicity right it's determining you
know what these models are and how
they're functioning certainly playing
around with different models and just
understanding in a lot of different
cases when we train on various different
scenarios you know what things does that
result in can sort of help
um but yeah it's just really tricky
okay uh if that's it we can we can call
it there and uh we'll have another
another lecture uh hopefully uh coming
up pretty soon
I can't do tomorrow I don't think we'll
talk about it