4:How Do We Become Confident in the Safety of an ML System?: Evan Hubinger 2023

today we'll be talking about something a
little bit different but you know at
least a little bit building off of this
sorts of discussion of various risks
that we've been talking about previously
um today I want to talk about this sort
of question how do we actually become
confident in the safety of some system
you know previously we've talked a bunch
about you know what sort of problems
might arise when you're training various
different systems
and today I want to move into this
question of evaluation right we have we
have some understanding of when to say
no right when to believe that there
might be a problem with some system you
know what sort of analysis might lead us
to believe that some particular training
regime is going to lead to an algorithm
that is that is dangerous in some way
but what we don't yet have is some
understanding of how to say yes some
understanding of what makes a particular
proposal a particular you know approach
to trying to build some sort of advanced
aligned uh you know AI system uh you
know one that we should trust one that
is good uh you know one that we can
believe in right what are the sort of
factors that might lead us to actually
have confidence that a system is going
to do the right thing
um that's sort of the thing that we want
to address today
okay
so we got to start with this sort of
basic question uh you know that we were
talking about a bunch throughout the
series which is
um you know when we train an AI system
or we use machine learning to actually
produce some algorithm which has some
particular property what do we actually
know about that algorithm right and
fundamentally and you know we sort of
stressed this a bunch all we really know
is we know that it is some algorithm
that performs well on the data that we
gave it
um and it's some algorithm that you know
was selected by the inductive biases of
the training process to be structurally
simple to be you know the sort of
algorithm that you know has a large base
in all of these sorts of basic
properties you know we talked last time
about all the various different ways in
which the inductive biases could go and
all of this sort of uncertainty that we
can have you know surrounding this but
the basic thing that we always know you
know is just well it's some algorithm it
performs well on the training data and
uh you know it was selected by the
inductive biases
okay uh and so the you know the basic
problem that we have is that if we want
to be confident in the safety of a
system we need to not just know its
training Behavior we need to know its
generalization Behavior we need to know
what will happen when you take that
model and you deploy it in some new
environment when you give it some new
task you know what actually happens what
sort of you know Behavior does it have
uh you know what sort of what sort of
basic structural algorithm do we
actually learn
um and so we need to know more than just
this basic fact about what it does on a
training uh training data
okay
uh yeah question
so you mentioned before about like
adversarial training and holding out
certain data points is it the case that
did you have something that calls well
on trading and then it performs well on
some held out data are we that confident
will perform well on all held out data
or is it really more of a domain
specific problem about a distribution
uh I think that the answer is well you
know something right you know if you
have you know some you know some case
where you're like well we don't know
whether it's going to have good
generalization behavior in this new
individual you know test case and then
we test it there and it has good
behavior you have learned something
about the basic structure you know the
basic you know thing that you infected
when you produce that algorithm right
you produce an algorithm which actually
had a particular generalization property
you know if I train on you know going
back to you know original example from
the first talk could we train on
something like you know classifying the
Pac-Man from the you know the little
triangle with a block you know they're
different colors versus shapes
um if I then try the generalization task
and I learned that well I always learn
to classify based on color I have in
fact learned some useful facts about my
inductive biases and about what
algorithm that I learned but the point
uh you know the sort of concerning thing
is that there's going to be a lot of
situations where we sort of don't expect
to always be able to tell uh you know
the sorts of facts that we really need
to know about our model in terms of its
you know safety just by that sort of
process alone so we talked about this a
bunch uh in the last two talks about
deceptive alignment where with deception
specifically it's a case where it's very
very difficult to actually gain any
confidence in whether your system is or
is not deceptive uh if the only thing
that you know is you know these sorts of
properties where well I did a bunch of
adversal training because as we talked
about previously there are multiple
different model classes which are
compatible with the sort of limit of
adversarial training some of which are
aligned and some of which are not
aligned
um and so we need to know some
additional facts other than just that
now that being said and we'll talk about
this a little bit later on today but
there are some ways that you could sort
of use this extrapolation to gain sort
of more evidence even about deceptive
alignment so you know for example we
might believe that well because we have
seen many times if we train in this
particular way we're very likely to get
a model that has these properties and
those properties are such that they
preclude deception so for example you
know if we train in this way we always
get a model uh that is you know not
doing long-term reasoning or something
then we're like okay if we expect if we
can you know believe that that Trend
will continue to hold then you can
believe that particular training
procedure is unlikely to lead to
deception even if you don't directly
have evidence that the model itself is
or is not deceptive you have some
evidence from sort of extrapolating
um but then the question of course
becomes how much evidence is that so so
we'll talk about these sorts of various
ways to gain evidence
I think the thing the place that I want
to start right now is just with this
question of uh you know what do we need
right what sorts uh what sort of
structure should we be looking at if we
want to understand you know why a
particular training procedure is safe
why it's sort of you know we might
believe it's going to yield uh a sort of
algorithm that we trust
so the basic thing that we I'm going to
say we sort of you know want is we want
you know first some theory of what
mechanistically the model is doing so if
you know you have some particular
training procedure and you want to argue
this training procedure is safe the
first thing that we need to know is well
what sort of algorithm is it actually
trying to produce right uh you know what
is the you know structurally simple you
know algorithm the basic thing that we
actually wanted to be doing at the end
of the day
um why do we need to know this well I
think it's really important for a couple
of reasons so to start with if we don't
know this it's really hard to have an
understanding of you know arguing
whether it is going to be safe or not
because the space of possible algorithms
is really really big
um and
if we don't know which what sort of
Target within that space we're trying to
hit and what sort of Target we you know
we believe we might have evidence we're
going to hit and we're just sort of
making very general arguments about you
know is this sort of in general like to
be pushed in a safe or unsafe Direction
I think it's very hard to sort of be
specific about okay what are the actual
alignment properties of the system do we
actually believe it's going to do the
right thing
um and so I think I really want to push
towards what we're going to be focusing
on is these very specific stories where
you know uh we have some specific story
for this training process produces a
system that you know in fact is
implementing this sort of algorithm and
then we want to ask you know do we
actually believe that is that algorithm
actually safe
um and of course that's the second thing
that we need you know given you have
some particular model for what you think
your what sort of algorithm you think
you're going to learn you also need some
reason to believe that you're actually
going to get that algorithm some reason
to believe that that you know the thing
that you the sort of algorithm you want
the model to be implementing uh is in
fact the you know the algorithm you're
actually going to get when you run your
machine learning process
so these are the two sort of most basic
components that I think we need when
we're sort of trying to understand you
know a story for why a system is safe
um so we're going to give these things
names we're going to say that the first
one is a training goal it this is sort
of the goal of uh you know our training
process we want to produce the system
that you know is well described in this
way
and then the second thing is sort of our
training rationale it's like why we
believe that our particular training
process is actually going to produce a
thing that you know uh falls into that
that class that is that actually sort of
satisfies that goal so what is the what
is the goal sort of mechanistically
structurally what sort of algorithms we
want to get uh and then why do we
believe that our training process will
produce that algorithm
okay so uh together we're sort of gonna
call these two things a training story
so we're going to say you know it a sort
of a combination of a goal this is what
we think our machine learning process is
going to produce and a rationale this is
why we think it's going to do that uh is
a story for why you think your thing is
safe right if you want to if you come to
me you know with some particular machine
Learning System and you want to say I
think this machine Learning System is
safe well then you need to say you know
we're going to say you need to say what
is it what sort of algorithm do you
think it's going to learn uh and then
you know why is it actually going to
learn algorithm right
okay
so let's sort of just give a very simple
example Yeah question
uh I might be over reading into the
notation on the previous slide but the
previous notation well it's like
training goal plus training rationality
implies training story and simple equals
training story is that a specific
difference that's what we're finding out
just an arrow I'm just like uh if you
have both of those two things you have a
trading story this is this like the
components of a training story
um yeah I don't mean to to to give a
sort of implies relationship there
okay so let's do a really simple example
so uh you know here's something that is
in fact probably safe you know I want to
train a thing to classify cats and dogs
um and you know even then we still want
to sort of want to work through what is
the training story right so we're going
to say well what is the training goal
well in this case we want a model that
does cat you know dog classification but
it's worth pointing out that there are
some models that would in fact do a good
job on cat dog classification some
algorithms you know that one could be in
planting that would do a good job on
this task uh that we don't want so an
example of an algorithm that we don't
want they would do a good job on cat dog
classification is you know uh agent
trying to maximize paper clips right you
know as we talked about previously if
you have some agent and it has some
long-term goal about the world then um
it is in fact going to you know do a
good job in whatever training
environment you put it in because it
wants to you know pretend to be doing a
good job so that it can eventually you
know do whatever thing it wants in the
world we obviously don't want that and
so we're gonna you know we're gonna say
okay the thing that we want is is not
just like anything which doesn't a good
job on cat on classification when we're
training a Canton classifier we really
specifically want a thing which is
implementing you know you know simple
human-like heuristics for cat dog
classification uh you know we don't want
uh you know it to be doing some crazy
sort of uh you know deceptive thing we
just want to be doing the basic thing of
okay humans in fact use some basic
visual heuristics to understand whether
a particular thing is a cat or a dog we
just want to capture those heuristics
and Implement them in our model if we
believe that it's in fact doing that
then we think it's going to be safe
um and then why do we think it's going
to be doing that well
uh we have some belief that you know
when we train a convolutional neural
network on a task that's relatively
simple
um
uh that you know the simplest source of
heuristics are going to be
um you know the same sorts of human
image heuristics you know when we look
at image we sort of have some idea of
how to distinguish between cat and a dog
there are basic structural facts and and
you know you know about that image that
you know let us do this distinguishing
and you know we have some belief that
those are the sorts of simplest
heuristics
um you know that one can Implement in a
in a sort of convolutional neural
network uh to solve that task and so in
that sense to the extent that we believe
that convolutional neural networks are
in fact Implement these sorts of
structurally simple algorithms and we
believe that you know in this particular
task this sort of structurally simplest
way to do it is just these sorts of
human-like image classification
heuristics then we think we're in fact
we're going to be learning uh an
algorithm which satisfies this training
goal
yeah
you say that we believe that the
simplest parameterization will Implement
human-like cat detection heuristics is
that because of the implication of just
Simplicity in general or is that because
of The Preserve these things similar to
our windows and wheels examples from a
few talks before
yes that's a really good question so I
think there's a couple of things so the
first thing is um you know in fact we
could sort of think about this as like a
pre-theoretic you know if we didn't
actually know what happens when you try
to train a cat dog classifier what would
the sort of arguments would you know
what sort of arguments would we have for
why it would turn out one way or the
other
um but then you could also be like okay
but we have built this thing right you
know we've in fact trained you know you
know across the world you know the
number of times that somebody has
trained a cat dog classifier is very
large there's a lot of these that have
been trained and you know we in fact
observed that you know especially when
we do transparency we look at them they
really don't seem to be doing anything
other than these sorts of basic image
classification heuristics
um you know stuff like you know uh
windows on top of Wheels
and so we're like okay we have that sort
of piece of evidence as well one thing
that you know we're talking about this
more sort of these various different
types of pieces of evidence one is sort
of really important difference between
you know the sort of well we looked at
it after the fact type of evidence and
the well we have some reason to believe
a priority type of evidence is
eventually we're going to have to deal
with situations where we have to
determine whether these sorts of we buy
these stories and whether we think
they're actually correct in situations
where we don't know where how they turn
out right so if we were you know didn't
actually know what was going to happen
you know when we trained a cat dog
classifier and we were just looking at
this sort of basic story we'd have to be
you know make some determination about
how confident we really felt in it and
in this case you know it obviously it
ends up being fine but I think it's
worth pointing out that the theoretical
arguments you know that we can make for
you know how likely it is for this to
actually turn out fine sort of a priori
are a little bit sketchy you know I
think that most of our confidence right
now does really come from the fact that
well we've done it a bunch you know and
we looked at it and it doesn't look like
it's doing the sort of you know the bad
thing
um but it's you know very difficult to
sort of tell really strong stories about
why we would have known that in advance
you know what sorts of properties you
know what sort of things could in fact
have told us uh you know in advance that
the model would be doing this sort of
you know structurally simple heuristics
that we wanted to be doing
Yeah question
I mean in the last two events through
the way can developed deception and it
mean it was that it already develops a
good word model when it understands that
it needs to take instruments and
understand the situation well enough
that they'd understand that if it
pretends to do that later it can take
over the world I think we can a priori
suspect that if we just gave it all the
training data of cats and dogs and only
pictures of cats and dogs and all no
text and no internet anything that it
will be really unlikely to figure out
its situation and
um and instrumental planning and so on
yeah so I think this is basically
exactly where we're going so you know
that the thing that you just described
you know is the sort of training story
that we want right is a you know
concrete reason to believe that there is
a particular thing that would have to be
present to you know get some particular
failure mode and we think that that
thing is not present in our training
process and therefore we think it's not
going to yield you know some particular
thing as instead going to yield the
training goal that we want now we want
to be a little bit more strict than that
right we don't want this you know this
is not just a it won't produce deception
we also want it to be well it is going
to produce the thing that we do want
right
um and so we also need to have some
argument for why you know it's in this
case it's you know it's actually going
to produce the sort of classifier that
we want
um but yeah I think basically the thing
you're saying is correct you know that's
the sort of evidence that we're looking
for right these sorts of piece of
evidence that can be really strong you
know strongly convince us that we're
going to get a model that has these
sorts of particular properties uh you
know and it's not doing not not
deceptive you know because we have some
good reason to think that
okay so these are the sorts of stories
that we want to be telling
um okay so some you know some some sort
of you know facts about these trading
stories I think are important so the
first thing you know that's worth
pointing out is that these sorts of
stories should be sufficient for safety
so uh if the model sort of conforms to
the training goal
um then it should be safe so in this
case uh you know we have some argument
as part of the training story as to why
the training goal is safe in this case
we think that you know human-like cat
detection heuristics are safe because
you know they're just the same sort of
heuristics that humans use uh we sort of
we know that they're in fact safe they
don't yield anything bad it would be
really hard you know for them to be
influencing some sort of dangerous
optimization procedure we believe that a
lot of the sorts of really dangerous
things that models produce require them
to be implementing some sort of
optimization procedure given that we
think that it's not going to be doing
that you know or given that you know we
have a training goal we say it's not
going to be doing that given that it
actually satisfies the training goal and
is in fact not doing that you know it
should be safe
and then you know if the training
rationale holds then you know we believe
that in fact the resulting model will
confirm conform to the training goal and
so we have a reason to believe that it
you know we're actually going to get an
algorithm and that algorithm is in fact
going to be a safe one
okay so in this case you know again the
reason you know that we're sort of
giving is we're like well okay if the
simplest parametization the classifies
casts from dogs you know uses these
sorts of human-like heuristics then um
you know uh and and we believe that
gradient descent finds these sorts of
simple parameterizations then we think
you know we should conform to the
training goal
okay uh and you know these sorts of
training stories are falsifiable so you
know I think one you know thing that's
maybe worth pointing out is you know the
training story I was just giving is not
clearly true so you know uh it's
actually not clearly the case that
actually we do learn you know only these
sorts of human like heuristics when we
do cat dog classification I think
there's some evidence that um and this
is from uh adversarial examples uh our
features not bugs where there's some
evidence that actually sometimes we
learn
um
sorts of you know these sorts of
features that are not human-like at all
they are these sorts of pixel level
features that are real features in the
data that do really correspond to things
that are related to you know whether an
image is a cat or a dog
um but they're not the sort the same
sorts of high level structural features
that humans often use when they're
distinguishing various different uh you
know shapes
um and so in that context we can sort of
believe that well we have you know this
sort of very basic story you know we
might hope that even in the sort of in
the case of something very basic and
structural you know simple as cat dog
classification we can tell a really
precise story where we're just like okay
we want we're going to get exactly this
type of algorithm we have these reasons
to believe that this training process is
going to produce it um but even then
it's very tricky because uh there are a
lot of sort of very a bunch of
subtleties to what sort of algorithm we
might be learning what sort of features
might be paying attention to
um and in this case you know
um we're not super concerned about it we
have some understanding of what these
sorts of pixel level-like features are
and we don't think that they're you know
doing some sort of crazy optimization
that is existentially dangerous but it's
still worth pointing out that they're
doing a you know type of uh you know
algorithm which is not the sort of
algorithm that we maybe wanted when we
originally were like okay we think CNN's
may be similar to how human you know uh
you know image you know regions of human
brain work and so we're going to try to
train something similar to human image
classification in fact we often don't
get that we still get an algorithm that
actually does do a really good job at
the task but it's one that is maybe a
little bit different than the algorithm
we were hoping to get and it's hard you
know in advance to really be able to
predict what sort of algorithm I'm going
to get it's very difficult to understand
are we going to get these sorts of you
know weird pixel level features are we
going to get the sort of human-like
features in this case we you know we do
get a combination of the two
um
but making these sorts of you know
predictions getting the evidence
required to be able to know are we going
to end up with this algorithm or that
algorithm is the sort of whole name of
the game here
Yeah question
are these training stories a standard
method used in AI research and if not
why not
uh I don't think you'll hear many people
using these same sorts of terminology I
really like them
um I think we'll talk a little bit later
about you know some of the sorts of
other terminology that is often more
common
um oftentimes you'll hear you know inner
and outer alignment used to describe
some of the same sorts of things we're
going to be talking about later we of
course talked about inner and outer
alignment previously and in slightly
different context and I'll explain a
little bit later on you know what sort
of what they sort of look like in this
more General context
um but I think that these are useful
terms I think that if you're trying to
understand you know how to think about
inter you know how to structure your own
thoughts in terms of you know if you're
given some system is the system you know
is this machine learning process going
to be safe I think this is the right way
to think about it regardless of what
terminology you use the right way to
think about it is you know have some
understanding of what algorithm you
think it's going to produce and why you
know what reasons we have to believe
what pieces of evidence we have to
believe that it's going to go there
okay
okay
um and then you know again I sort of
want to point out that I think these
training stories also are very general
right so we can use them for cat
classification but as we'll see later
you know we can also use them for very
complex alignment proposals uh you know
any situation really where you want to
rely on some AI system and that system
is trained via machine learning process
you have to have some reason to believe
that among the space of all possible
algorithms that that machine learning
process could learn you know why did you
learn the one that you were actually
trying to get
um okay great so this sort of idea is
anytime you're training machine Learning
System you sort of theoretically sort of
should can have a training story in the
back of your mind you know when I'm
training this thing I'm training on this
task what am I trying to get what sort
of algorithm do I want to be you know uh
implementing and why do I at all believe
that among all the possible algorithms I
could find you know that's the one that
I'm going to get
okay
so uh breaking things down a little bit
more so we've sort of been glossing over
this but with within each class of the
training goal and the training rationale
there's a couple of things that you sort
of always need to have so in the trading
goal we always want to have a
specification we want to know you know
exactly what it is uh you know
mechanistically the sort of algorithm
that we want and we sort of need this
desirability we also need to have some
reason to believe that whatever that
mechanistic algorithm is we actually
would like it you know it's actually
going to do good things if we got a
thing that actually satisfies our
training goal
um uh you know so these are sort of both
necessary
and then for the rationale we sort of
again have two things we have you know
these sorts of constraints we have
things that we know are true sort of
hard facts about the structure of our
training process and then we have sort
of what I'm going to call nudges which
are the various different sort of
individual slight bits of evidence that
push us in one direction or the other so
you know the basic constraints are we
know that is in fact going to be some
algorithm that fits the training data
right this is the basic constraint of
the structure of the training process
um but then we also have a bunch of sort
of uh you know things that are not those
sort of hard constraints things that uh
you know give us some piece of evidence
in One Direction or the other you know
okay we think maybe this algorithm is
slightly simpler than this other
algorithm
um but it's not this it's not sort of
Hard Evidence it's it's uh and so we're
sort of gonna you know try to isolate
these things
um together so we're sort of thinking
about what are the sort of classes of
models that we could find and within
those class of models you know why do we
believe we might sort of be pushed in
One Direction or the other
okay
all right
um I think one sort of important sort of
thing to sort of touch on is how
mechanistic do we really sort of need
these descriptions to be I think this is
a really tricky question and sort of a
lot of the complexity of you know what
I'm talking about right now comes in
this question of uh when we're thinking
about you know specifying a particular
training goal we're thinking about
specifying you know what is it what
algorithm is it that I want my model to
be implanting how specific do we need to
be and I think that one thing that's
sort of worth pointing out there is that
um
yeah so if you uh you sort of don't want
to if you specify too little then you're
in a situation where um you know you're
not actually constraining from a safety
perspective right if I just specify you
know as we talked about the very
beginning if the only thing I say is
well I want some model that in fact does
a good job on cat dog classification
that's not enough facts you know in
theory to know that your model is
actually aligned you need to know
something about its generalization
Behavior or something about what is
actually doing right it is in fact doing
something that mechanistically uses the
sorts of heuristics that we want
um you know is sort of specific enough
in that case to sort of imply safety but
you need something that's specific
enough to imply safety but at the same
time if you specify too much there's
sort of this question of well why are we
even doing machine learning in the first
place if we already know what algorithm
we want the model to implement right so
you know if I know exactly what
algorithm I want I know exactly how it's
going to work I know you know what the
structure is well I can just write that
algorithm right the reason that we do
machine learning at all is because we
believe that well we don't know how to
write the algorithm but we know how to
search for the outcome right we believe
that we understand a space of possible
algorithms uh you know and in that space
we think that there are desirable
algorithms and you know we have some
reason to believe we can distinguish
between the desirable undesirable
algorithms and find one that can want
the right thing even if we don't
ourselves know how to write that
algorithm directly
and so you know if we want to continue
doing machine learning at all you know
we can't throw away you know that basic
thing that is the reason we do machine
learning which is what we don't know how
to do what what the algorithm that we
want is and so we need to have some
description that is on a high enough
level that it actually still allows us
to do machine learning at all and so
this is sort of this sort of tricky um
uh you know sort of dichotomy that we
have to deal with between you know we
want to be as specific as possible
because we want to sort of have some
understanding that is clear enough that
it precludes things that would be unsafe
um but you know if we're too specific
then we've gotten into a situation where
we've sort of eroded the advantages of
doing machine learning at all
okay so we want sort of as much
specificity as possible while still you
know being able to uh you know not have
to describe exactly the algorithm and
sort of stay competitive with other
approaches
okay so how much specificity is enough
so I here's like you know an example I
think of you know what sort of
specificity I think do we really want in
a training goal so uh here's an example
is courage ability right so courage
ability uh what is a sort of cordial
system well uh you know the corrigible
system here you know Paul has a bunch of
examples of the things that a cordial
system would do behaviorally right where
like figure out whether I built the
right Ai and correct any mistakes I made
uh remain informed about the ai's
behavior and avoid unpleasant surprises
and a bunch of these you know basic you
know facts about what we want the
behavior to be I think this is
insufficient so why do I think this is
insufficient well uh what's sort of
Happening Here is we've it's sort of
very similar to the thing we were
talking about at the beginning where we
just described the behavior that we want
from the model but we haven't given any
information about what mechanistically
Algae the algorithm it is that it might
be implanting and so I think this is
sort of a problem because it puts us in
a situation where if this is the if this
is the sort of basic Target that we have
it's very difficult to understand you
know what would it actually look like to
find an algorithm which has these
properties what sort of algorithms do
have these properties right these are I
think that this is sort of useful as a
list of desiderata in terms of what
properties we want and to be clear it's
not the case that Paul is sort of
intending this to be a training you know
goal here but I think this is useful as
a list of design but it's not sufficient
as a training goal because it doesn't
tell us you know given that we know a
model does some of these things in the
training uh you know uh do we have any
reason to actually believe it's going to
do the right thing later so we need
something a little bit more specific
than this so here's maybe a better
example uh and this is also from Paul
so here at the you know Paul has a more
specific case of what sort of uh model
you know he's looking for Yeah question
a little confused about your example
without encourage abilities because it
says here that training goals need
specification and desirability so
whatever we want and why we want it was
the training rationality with the
training rationale gives the constraints
and the nudges it seems to me that
Paul's description of Courage ability is
a training goal but not a training
rationale so what essentially is it that
it's missing from the training goal side
of things
yeah so I think that you could sort of
use it could be useful as a way to sort
of have training goal desirability right
it's a reason that you might like a
trading goal is that that training goal
is courageable but I don't think it is a
training goal itself because it doesn't
give us a mechanistic description of
what the model is doing right if I know
if you say ah the model's corrigible
that's a property of the model's
Behavior right it is this thing that is
saying this model will listen to us and
you know change based on the things that
we say right it'll try to correct you
know mistakes that's a that's a property
of how the model acts but it's not a
property of what the model is doing
right like what is it the model might be
doing that caused it to have that
behavior so maybe a more specific thing
right you know that you might have would
be we talked previously about this idea
of right corrigible alignment you know
the Martin Luther models right so you
could have some description that's like
I want a Martin Luther model and then
the reason I think a Martin Luther model
is going to be good is because I think
it's going to have these behavioral
properties of Courage ability right but
you need that additional step you need
that reason you need that sort of claim
of what algorithm is actually
implementing right if the only thing
that you have is ah it is you know going
I'm going to get a model that
behaviorally is trying to act corrigible
as we sort of talked about a bunch last
time that behavioral analysis is just
insufficient to be able to know whether
the model is safe right we really need
to have some additional facts about what
is doing internally to know anything
about its generalization because in
theory you know if the only things I
know are ah it looks like it's doing X
it could be pseudo-aligned it could be
you know deceptively aligned there's all
these sorts of various different things
in the model could be doing that are
consistent with any sort of behavior and
so we want to really be able to you know
in this case have reasons to believe
that among that set of models that are
consistent with the behavior why are we
getting what sort of particular one do
we want right and so this is why I think
the sort of basic op we want a cordial
model insufficient here we really want
something more specific than that
um so here's an example of something
that's more specific
so here Paul's describing uh this sort
of you know we want a uh model that has
some model of the world and some way to
translate between that model of the
world and natural language and so it
does the sort of direct translation
between the models World model and the
questions that we ask it I think this is
substantially more specific and sort of
much more direct about describing
internally what we sort of what we want
the model to be doing it is saying we
want it to have some model of the world
that just sort of is sort of a plain
model of the world just describes how
the model works and then some way that
directly and truthfully translates uh
between the sort of Concepts in the
models model of the world and the sort
of questions that we ask it
um that sort of direct translator uh is
is I think a very nice mechanistic
description of what we might want right
we're like okay if we have a model and
the sort of algorithms influencing is it
has the model of the world and then
directly translates facts from the model
of the world uh into natural language
and it tells us those facts then we have
some reason to believe that ah this
isn't just an algorithm that looks like
it's doing the right thing it is really
structurally doing the right thing it is
actually in fact honestly reporting its
true beliefs about the world and so now
we have a reason to believe that's doing
the right thing now importantly this is
still sort of a very high level
description right it's not the case that
it specifies exactly how you would learn
a world model and all of the facts that
would be involved in that world model
right we don't know all of those we
can't write them down we're doing
machine learning to try to learn them
but we still want to get a model that
actually has this basic structural
property that it does in fact have some
model of the world that it truthfully
translates into its into its outputs
right and so we're like if we believe we
have that mechanistic property then we
sort of think we can be safe and so this
is the sort of level of specificity that
I think we need at the very sort of at a
minimum right
um because if we have this level of
specificity about what sort of algorithm
it is that we're looking for then we can
start to reason about why we think that
you know among all the possible
algorithms is actually going to you know
be doing something safe
okay so this is sort of what I'm
imagining right we don't want something
that is just this high level behavioral
gloss like courage ability I want
something that is a little bit more
specific in terms of describing
mechanistically what it's doing but not
so specific that we're describing you
know exactly all of the you know
knowledge that it has
okay
okay so that's the sort of basic high
level you know how to think about these
sorts of training stories
um and now let's we're going to sort of
try to get a little bit into the
nitty-gritty you know in terms of
applying these and thinking about
concretely
um what some you know uh proposals uh
for you know how to build safe advance
that I might look like and how to
evaluate them sort of in this framework
okay so I want to sort of go back to
something I was saying earlier which is
well okay what are sort of other
terminology that you'll hear in this
context uh you know previously we
introduced this sort of inner and outer
alignment dichotomy and and we talked
about them uh previously in these sort
of strict contexts so we said you know a
system is outer aligned uh if it you
know has uh you know some loss function
where if the model were optimizing for
that loss function we would be happy and
a system is sort of interaligned if um
it has some mace objective and that mace
objective is sort of aligned with it's
the same as the sort of loss function
you're trying to get it to optimize
um and I think that in this context
these definitions are a little bit too
strict we sort of want to relax them so
so what's sort of the problem with these
definitions
um and and we can sort of think about
them in a training stories context so uh
the sort of strict version of outer
alignment and the strict version of
inner alignment are both presupposing a
particular training goal they're saying
suppose that your training goal was you
want a mace Optimizer you want some
model which is implementing an
optimization procedure and you want that
model to specifically be optimizing for
these sort of uh thing that you
specified in your loss function in your
you know in your reward function
whatever the thing that you specified
you were trying to get to do when you
were training
um if that is your training goal right
if the thing you want to get is a model
which is actually doing
um you know optimization and it's doing
it for this thing that you wrote down
then these sorts of concepts are exactly
correct they're the you know the things
that you want they are you know do we
believe you know the training goal
desirability do we believe that if a
model we're actually optimizing for that
thing that I wrote down it would be good
and then the you know training goal uh
the training rationale you know do I
actually believe and I'm going to get a
mace Optimizer that is actually has that
goal you know among the possible
algorithms that I could find
but the thing that is worth pointing out
is that well
um this is not the only training goal
that we can have right so we might not
want to build a model which is directly
optimizing for the uh you know some
particular objective we also might want
to build a model that is optimizing for
some particular thing but the thing
that's optimizing for is different than
the thing that we wrote down it's not
optimizing for our loss functions
optimizing for something else
um so you know an example of this might
be I want to train a model this sort of
um acts in a similar way to humans and
the way that I'm going to do that is I'm
going to reward it for
um you know cooperating in some
environment with other agents and we're
like okay well we don't directly want a
model that optimizes for cooperation but
we believe that the sorts of models
which do a good job in this sort of
cooperation environment
um have good properties right the sorts
of models that you know generally learn
things like you know be you know
Cooperative work with other agents you
know be nice whatever
um that's a training story that sort of
is trying to train a model which is
doing something different than the
direct thing that we're trying to get to
do right we have some loss function you
know we're directly training on
cooperation we're trying to get a model
that isn't just like optimizing for
cooperation it's trying to we're trying
to get a model that's doing something
different
and so in that context you can imagine
if that sort of approach succeeded then
you would have a model that is aligned
in the sense that okay it is doing
something good right it's doing you know
it's it's nice it's trying to cooperate
with other agents it's doing things in a
good way but it wouldn't be strictly
outer lined or Inner Line because it
would not be the case that if it were
just optimizing for this raw cooperation
objective we would be happy with it and
it's also not the case that it is in
fact directly optimizing for that strict
cooperation objective right it's in fact
doing something different than what we
sort of directly specified in the loss
function but it's doing something
different in a way that we want it's
doing the sort of correct thing instead
okay so we need we need slightly broader
terms right we need to say in the more
General case where we have any possible
you know approach that somebody might
have or thinking about how to build a
sort of safe system we need a way of
thinking about it that is a little bit
more General than just assuming that the
training goal is you know a model which
optimizes for for loss
okay so what are the more general
concepts how do we how do we sort of use
this in a more General case
so uh here is our sort of training
stories based evaluation framework
so we have this sort of General notion
of outer alignment you know which is
whether the training goal is good for
the world right if we got a model and it
satisfied the training goal would we be
happy with it right this is our training
goal desirability and in this context we
can sort of think about this as a more
General generalized version of outer
alignment it isn't saying okay the
specific training goal of optimize for
the loss function is that good we're
saying okay that might not be your
training goal whatever your training
goal is you need to have some reason
that if you actually got a model that
had that property you would be happy
with it
um and then we need a sort of notion of
competitiveness we need to believe that
if we actually did get that training
goal
um it'd be powerful enough to compete
with other AI systems so in this case
and this is sort of where it slice it
gets a little bit different from the
more General training stories framework
in this case we're thinking about you
know powerful Advanced AI systems right
these are you know how we evaluate
proposals for building uh systems that
are you know arbitrarily powerful for
aligning you know the most powerful AI
systems for making sure they are sort of
good for the world overall
um and if your proposal is something
like you know well just do cat dog
classification right well well that that
that shouldn't be okay right that's not
that doesn't actually solve the problem
the problem was you know people actually
want to do things that are not just cat
talk classification you know they want
to do all sorts of other tasks um and
they want to use machine learning for
them and so you know we as you know
safety researchers right you know need
to find some way to figure out how to
let the people do those things in a safe
way and so that means you know just do
cat dog classification isn't an answer
right we need to have some reason to
believe that whatever approach we have
here would act actually be able to
accomplish the various tasks that people
want to use the machine learning system
for
okay so this is this performance
competitiveness question
and then again we sort of have these
training rationale components so we have
this sort of more generalized version of
inner alignment we're saying uh you know
is it in fact the case that your
training rationale is correct right is
it actually going to hold you know I
have some reason to believe some piece
of evidence to believe that I'm going to
get this particular type of algorithm uh
is that true right do we actually
believe that I'm going to get that
algorithm among all the possible
algorithms that I could find right
um you know and again the sort of same
reason we talked about previously about
why this might not happen you know all
the reasons you might get a
pseudo-aligned Model A deceptively line
model these are reasons that your sort
of training rationale might not hold you
might get a different model than the one
you intend
and then we also need another sort of
competitiveness portion here so we need
a sort of implementation kind of address
we need some reason to believe that your
training rationale the sort of way
you're trying to build your model is
actually implementable right so in the
same way as you know train a cat dog
class with higher isn't a solution
um you know another thing that isn't
also not a solution is uh you know just
build a you know simulation of billions
of humans you're like okay if we just
have enough humans and we simulate them
in perfect Fidelity and we get them to
think about the problem enough then they
can solve it and we're like okay that
may be true but we can't build that
right we have no ability to actually in
fact you know upload billions of humans
and get them to think about the problem
or whatever and so we're like okay this
is also not a solution you know if we
want something which actually sort of
addresses the general problem with AI
existential risk we need some reason to
believe that you know Not only would it
be able to sort of solve the problems
that people want machine learning for it
also needs to be able to you know
practically do so right it means it's
not just in theory be capable of doing
it it means also in practice be
something we could actually do
okay so we need these sorts of basic
components right we need these inner and
outer alignment you know the generalized
forms you know reason to believe that
the thing we're trying to get is good
and that we're actually going to get it
and then a reason to believe that we can
do so in a competitive way that it'll
actually be able to solve the problems
people want AI for and that will
actually you know be able to do so in a
practical way
okay
all right so we're going to now look at
a case study a particular uh actual
proposal uh for addressing you know the
general problem of AI essential risk
this is maybe this is the first sort of
concrete proposal we're going to look at
uh in in you know this this series of
hard we'll look at some more later but
this is going to be the the one we have
for this talk right now
um and it's a particularly interesting
example because it's one that is
oftentimes very difficult to sort of
analyze under a more conventional
framework so um but I think it's a
really interesting proposal and so I
think it's a good place to start so what
is microscope AI so here's the approach
so the first thing that we do is we
train some predictive model on some data
we want to get our model that is trying
to sort of understand uh and sort of
predict some particular distribution
and then we're going to try to use
transparency tools to understand what
that model learned about the data
um and extract that understanding
directly and use it to guide human
decision making and so we're saying okay
in the same way you know we could think
about you know this sort of Windows on
top of Wheels example well we learned
something about the model you know what
the model was doing it was looking for
Windows on top of Wheels but if we
didn't know things about cars beforehand
we also would have learned something
about cars right we would have learned
that cars have the property that they
are detectable by looking for Windows
and wheels now we happen to have already
known that but we might not have already
known that right there we have the
ability to actually gain a ton of
information and useful information that
we can use for building our own systems
and you know doing things directly that
don't have to go through machine
learning by using machine learning right
the thing that machine learning does is
it produces a model which has really
good understanding of some data we could
maybe directly extract that
understanding and use it ourselves
Yeah question
I would actually know the
SE things will be
[Music]
we know something about wheel spin is
somewhat in need to detect but if it
uses some alien concepts with which you
can 100 detect what a car is
um how do we know that there is this
this sort of information to be gained
and also
yeah no perhaps give it that
yes this is a really great question and
we're gonna we're gonna talk about this
very soon I think the answer is
basically we don't we absolutely do not
know that uh and so you know if we're
trying to ask you know is this you know
a actual solution you know which of
these sorts of criteria does it succeed
on I think the answer is well it
definitely doesn't we definitely don't
have a good reason to believe that it
satisfies these criteria that we just
talked about right
um it's it's not a solution right it may
eventually become a solution but it you
know we can um the thing that we're
hoping to sort of understand by
analyzing in this way is what are the
things that would be we would need to
have to believe that it is a solution
right to believe that it actually does
help with this problem that this is an
approach that we could use you know to
solve the sorts of problems that people
want AI to solve to do so in a way where
we actually believe we're going to get
the sort of model that we want
um you know what are the sorts of things
we would need to know uh to to believe
that right and so that's sort of what we
want to understand here
um does it actually succeed I think the
answer is well we have no reason to
believe really that it would right now
but maybe we could eventually get there
if we have you know really good
transparency for example right some
ability to actually believe we could
extract you know the sorts of Concepts
in a human relevant way then maybe you
know this would become a viable approach
I think that right now it's not a viable
approach but I think it's a really
useful and instructive approach because
it's going to help us understand how to
think about these sorts of approaches in
general right
um yeah another question
how would this score under
competitiveness like let's say that we
do have transparency toable that we can
use those that understand heuristics of
the model I've learned isn't that still
significantly less effective than just
running the model and letting it tell us
things
yeah so really liking these questions
because they're exactly the sort of
thing I'm trying to get you to do right
which is take an approach and then try
to analyze it you know under these
questions that we've been asking right
you know do they actually have these
sorts of competitive properties that we
want do they actually have these
alignment properties that we want I
don't know I I don't know the answer to
that question
um I think that um in this case like I
was saying at least it seems like there
are at least some difficulties uh and
you know like you were saying you know
it seems it might be hard to actually
get a system which can you know do all
the sorts of things that we want I'm
going to talk in a little bit about you
know going through those difficulties
but but yes I think the thing you're
saying is exactly correct which is that
there are competitiveness difficulties
here
um that seem you know quite difficult
we'll talk about you know what they
might look like but but yeah absolutely
Yeah question so
for example now in chess and Google
these AI is very much be test and still
go players did improve like after seeing
even without good interpretability after
seeing all of a ghost moves they did
develop new strategies and they
developed but still they are very far
behind or they're actually AI do you
find it losable that it could
interferability we could understand the
strategies of the AI is so bad that
human players could just learn that and
beat the AI
uh so I think the chess and go is a
particularly interesting example so I
think one thing I will say is that well
okay we don't just have to do it
exclusively with human brains we can
still implement it in our own algorithms
right so if you look at something like
stockfish which is the you know a human
engine that uses substantially less AI
than um alphago uh or sorry in in like
chess for example uh you know uses
substantially less computation than
Alpha zero
um I believe that like current versions
of stockfish are are are better than
Alpha zero was at the time when Alpha
zero beat stockfish
um and that has done you know that's
partially through AI but it's also a lot
through understanding some of the things
that Alpha zero was doing and trying to
you know actually write them in
algorithms ourself so there is some
extent to which you know uh yes I think
the thing you're saying is correct it
does seem like there's a large extent to
which you know
um it's it's gonna be really hard to get
humans to the level of like you know
these sorts of systems but also they do
teach us things right they have improved
Human Performance and they've also
improved our ability to write other
outcomes which do these sorts of things
so you know could this approach
eventually succeed I think maybe but I
agree that there's sort of some pretty
major obstacles along these lines
okay so just you know talking about this
a little bit more Chris Ola who's the
head of interpretability and impropic
also is is the person who uh you know
created this approach you know I think
you know one good way of thinking about
it is sort of what he talks about here
so the idea is that um you know when
you're doing interpretability these
sorts of visualizations are a bit like
looking through a telescope so just like
a telescope transforms the sky into
something we can see the neural network
transforms the data into a more
accessible form one learns about the
telescope by observing how it magnifies
the night sky but the really remarkable
thing is but one learns about the stars
and so the idea is you know uh
visualizing representations teaches us
about neural networks but it teaches us
just as much potentially about the data
itself so the idea you know this is the
sort of basic idea behind this approach
is well you know we are learning
something about the network when we do
interpretability maybe we're also
learning something about what we were
trying to understand
um
and so you know we can sort of use
better AI systems to improve you know
human decision making
um and then you know use that to sort of
build better uh better systems
okay so this is an idea it's an
unapproach that we can sort of think
about as a way to okay you know we need
some mechanism for being able to build
systems which can solve these sorts of
advanced tasks and we need some
mechanism for doing so in a way we
believe it's going to be safe and this
is a mechanism and so we want to
understand is this is this a good one
okay
so what's the training story here so
um the sort of training goal right is a
model that predicts the data given to it
um and importantly you know we want it
to be sort of uh you know not optimizing
something over the world it's sort of
just a world model you know just doing
some basic prediction tasks and we
wanted to sort of be doing that in a way
this human understandable we want to be
using Concepts that once we understand
and extract those Concepts we'll be able
to understand what they're doing so we
want it to be sort of just understanding
the world and doing so using human
understandable Concepts
and you know the reason we think we're
gonna get this well we think that you
know if you just train on a prediction
task on some really really large
prediction Set uh you know set of data
you're going to learn a you know just a
pure predictor that's doing you know
just prediction why do we think this
um well you know one reason you might
think this
um you know it's it's unclear whether
this is true but one reason you might
think this is you know we were talking
previously about this sort of you know
why would you get deception and one of
the things we sort of mentioned a lot
was if the thing you're trying to train
on the actual objective you're trying to
get to do is very very simple then the
case for why you would get something
like deception is much less because uh
you know internal alignment is actually
really easy to describe it's not like it
takes you know this really long and
difficult path and so maybe it's more
likely you would get that so we're
thinking well okay if you're trying to
train something that's just doing
prediction maybe we believe we're
actually going to get something that
really is just doing prediction we have
some reason to believe that
um and so we think that you know we can
get something that's doing the right
thing there's also separately this
question of why we think we're using
human understandable Concepts uh which
is a really big and tricky question
um I think maybe one way to think about
this uh
is um
uh yeah well we'll talk about this in
just a second actually uh in terms of
how why you might actually be getting
human understandable Concepts I think
that you know the most basic case is
that you might get human understandable
concepts for a while and so you know you
can use this as you sort of scaling up
the power of your systems but eventually
you probably will stop getting human
understandable Concepts
um yeah questions
even now with current
um image mode up image recognition
models it's not that clear or not to be
do you guys human understandable because
I mean sometimes we do like these videos
and rainbows this was a good example but
my impression is that sometimes we don't
and these are just very simple things
working on it's not very under yes this
is a really good question really good
good point because you know I I made the
exact same point earlier in the talk
right you were talking about the cat dog
classifier it actually often doesn't
learn just human understandable Concepts
in just a second I'm going to put up a
graph that is sort of you know gonna
gonna give a rationale for why you might
expect this to work when that doesn't
work but I agree that it's a serious
sort of you know concern uh and problem
for this style of approach
okay um but yeah so let's talk about you
know why you might expect this to work
so in terms of the training goal you
know we need this outer alignment so you
know whether the training goal is
actually good for the world right so is
a pure predictor actually safe if you
have this pure prediction is it actually
going to be okay you know you're going
to be happy with it
um and this has a bunch of different
questions so one is like are we actually
going to be able to make use of it right
is the knowledge you know actually you
know the sorts of knowledge which helps
humans which makes humans actually like
you know do good things or does it make
them do bad things we also have sort of
tricky questions about whether
predictors themselves are safe and this
is something we're going to be returning
to in a later talk but we have things
like self-fulfilling prophecies so if I
have a predictor and that predictor is
trying to produce predictions such that
those predictions are very likely to
come true then you know I can have some
system that is like well if it says the
stock market goes up then people trade
on it and the stock market goes up and
if it says the stock market goes down
the people trade on that and the stock
market goes down and so any prediction
the system makes might be true and so
you know we might be a little bit
concerned that it's actually going to
produce the correct the sort of
prediction that we want because you know
any of them are equally valid this sort
of thing can get a little bit tricky
again like I said we're gonna we're
gonna come back to these sorts of
difficulties dealing with predictors
later
um but but suffice to say there are some
sort of issues that you might be
concerned about even if your model is
just doing a prediction task
okay and then again you know we talked a
bunch about this we have this
performance competitiveness issue right
you know is it actually sufficient uh to
accomplish all the tasks you might want
with the AIS just to be able to do this
you know information extraction in terms
of you know extracting the the knowledge
that the model has learned
um the answer is maybe I think that my
best guess would be it is possible for
some tasks and not for others so
especially for tasks that involve you
know a really large deployment or a
really you know Fast Response interval
so something like a dialogue agent or
like a you know Factory agent you know a
lot of things where you might you need
an AI to you be doing something directly
in the world and be doing it a lot of
different cases it seems very difficult
to make something you know using this
approach
um but there might be other situations
where you might you might theoretically
have otherwise wanted an AI system that
this can sort of cover for so maybe you
wanted to build AI systems to you know
as you know managers in a company uh
well you know uh maybe this sort of can
give you an alternative to that which is
well maybe it's actually better to sort
of use the AI system as a way to extract
really useful information about how to
do management and then give it to humans
and so it's certainly a situation where
you could imagine a lot of possible use
cases sort of being solved by this but I
I think my sense would be it does seem
like there's some use cases where you're
going to really want AI systems that
this sort of doesn't address and so
you're like well if this is our only
proposal for being able to sort of make
Safe Systems then you might be very
concerned that uh you know there's still
going to be a bunch of possible ways
that you can build AIS that you know
this sort of doesn't doesn't address
okay
um and then so for the training
rationale we have this sort of inner
alignment question you know um is this
sort of training rationale I'd like you
to hold are we actually likely to get uh
sort of human understandable concepts by
default we actually like to get a
predictor so I talked about this sort of
Are We likely to get a predictor
question you know this sort of you know
riding on the Simplicity of the
predicting prediction objective
um I promised a graph about uh How
likely it is that you would get a human
understandable Concepts so here's that
graph so the idea is this is sort of a
hypothesis it's unclear whether this is
true but the idea would be well maybe uh
as we sort of increase the strength of
our models the way in which we get the
sort of Concepts those models learn
change like this we think that you know
as we have really simple models maybe
they're really hard to understand uh
because they're doing something you know
so uh or I guess sorry as we have really
simple models they're very easy to
understand we have something like you
know um linear regression it's very easy
you just look at the you know the
weights it's got a slope and an
intercept and then we have you know more
slightly more complex models and they
get substantially more uh you know
confusing because they learn these
really weird Concepts but then you get
more powerful models and then they start
to learn these really simple
abstractions you know these really basic
structurally simple Concepts those sorts
of Concepts humans use
but then you know as you keep pushing
further you get you get reader concept
Concepts that are like better than the
ones the humans use and ones that you
know maybe we don't exactly understand
and so it starts to get worse this is a
hypothesis um I think that we have maybe
some data to support this you know we
have looked at you know how hard is it
to interpret models over time
um I think that we have seen this to
some extent so there are things where
like you know very very simple models
are very easy to interpret and then you
have you know more complex early so no
vision models stuff like uh you know
alexnet that are often very hard to
interpret and then with later Vision
models they often get easier so
especially you know once they have stuff
like residual connections
um they start to become you know use
Concepts that are easier to understand
so there's maybe some evidence to
believe this but
um you know we have no reason to believe
that we would get you know necessarily
this exact sort of shape around here so
you know maybe this would work
um but it's unclear but this would be
the sort of thing that you would be
relying on if you were if you were sort
of is this you know going to work you
know we actually think you know we're
going to sort of rely on this proposal
we we need to sort of be able to rely on
you know something like this graph
holding
okay yeah question
I may have missed this but has this been
observed to happen up to like a certain
point already like how we started to see
this curve go down and then up again
uh yeah so I mean I think we have seen
this at least with early Vision models
so like really really simple Vision
models you know where it's just like
doing you know hard-coded Edge detection
you know we understand it you know early
cnns are very hard to understand but
later CNN's are often easier
um so so
um at least with simple Vision models we
have I think we do see this sort of this
this you here but even that is a little
bit subjective you know it's just based
on well you know people have looked at
these and you know how easy is it
generally to find Concepts in them but I
think that we sort of see something like
this you happening for
um simple Vision models though it's it's
very unclear question
this diagram looks pretty similar to
deep double descent for lecture one so
is there any connection to that
I think that uh
yeah I think that it's not doing that I
also I I would say that in some sense
this is sort of inverted of what the
double descent graph looks like because
uh like with the double descent graph
the idea is that performance sort of
goes goes down here and then goes goes
up whereas here we sort of you know uh I
guess if we're thinking about
interpretability as performance
um though of course we're imagining that
the reason it learns these alien
abstractions is because they improve
performance they're better than the
human abstractions and so
um
yeah I guess you could maybe think of
like this point here where it learns
these really confused difficult to
understand abstractions is like the
worst part of the double descent you
know you start with these sort of you
know relatively you know you get simple
things then you go up and you get these
sort of dumb memorized things and then
over time you get you know simpler and
simpler things and some of those early
simple things maybe you're human-like
but then eventually you get these sort
of simple things that are non-human like
so if you wanted to sort of superimpose
it it would the sort of you would look
you know more like uh with the the hump
of the second descent there probably but
but I don't think there's necessarily a
clear relationship but in terms of you
know trying to to analogize and
understand yeah another question
so I'm not sure if this is strict word
answer but um what's the force that's
pushing the model to go towards
increasing alien abstractions if you're
trying to predict data that's been
generated by humans
um and black humans have human
abstractions it seems like that's the
useful level of abstraction to reason
about predicting the data in since
that's how it was generated so is there
really a reason to push beyond that when
like once you get there you probably
have a good ability to start predicting
that data
plausible I think the thing you're
saying is absolutely plausible
um but I think there is an alternative
case the alternative case would be well
you know a lot of the data is just the
world right it is just like how do
things in fact function in the world how
does the world work what sorts of things
happen in it and how do those things
relate to each other we as humans
understand some of those things right
you know we have the ability to
understand and predict various things
that happen in the world to some degree
but we're not like experts at it right
there's all sorts of things about the
world and Concepts and relationships in
the world that we don't fully understand
that we don't have great concepts for
working with and so you can imagine that
you know there's room for improvement
right there's no reason to believe that
just in terms of basic concepts for
understanding what happens in the world
the humans are you know at the absolute
Forefront and so you could you know
imagine getting substantially better
Concepts than the ones the humans have
even in just the sorts of understanding
humans right you know if you're just
thinking about good concepts for
psychology right for understanding human
behavior there's no reason to believe
that human concepts for understanding
human behavior are at the limit right
you know there could be better concepts
for understanding how to think about you
know even how humans work right you know
there's a whole field you know of
psychology right that tries to produce
better concepts for understanding how
people humans work there's no reason to
believe that you know we are at the
limit of psychology or something
Yeah question
uh to go back to the chess Excel thought
we have a neural network what says uh
would you think that the current systems
are already in the increasing me and
infection domain or which is say that we
try interpretability of those we will
get Chris abstraction with the arrival
did yeah that's a really good question I
think the answer is maybe there's some
interpretability work that has been done
on like alphago you know where
um some uh you know some some very good
you know chass and go players have tried
to look at some of the things that the
model you know is paying attention to
and understand what it's doing and
sometimes we understand and sometimes we
don't
um you know it often you know there was
some work that found that a deep mine
paper they found there was like a large
amount of correlates between you know
the sorts of things the Stockbridge pays
attention to and the sorts of things
that you can sort of probe out of alpha
zero
um but but even then you know it's not
perfect there's a lot of things that
it's clearly paying attention to they're
different than the things that we know
how to pay attention to so it's unclear
I think the answer is you know maybe
okay
and then yeah so implementation
competitiveness how hard is this
actually influence
um there's some you know there are some
questions you know how hard is it to
actually build uh you know a system
which is doing
um
uh you know prediction you know
hopefully this is not that hard because
it's like one of the most basic things
that we often do in machine learning um
but there's also a question which is how
hard is it to actually use the
transparency tools that might be
extremely difficult right it may be the
case that actually even if we can
extract the concepts it's very
computationally intensive or you know
human labor intensive to actually be
able to understand and interpret in
which case that could you know be
another thing that's a real uh you know
a very serious problem here
okay so we have some understanding now
of what it sort of looks like you know
to to sort of understand this this
proposal we have these ideas you know
how to think about you know what the
competitiveness is how to think about
you know the alignment overall you know
I think that there's there's some
reasons to like this approach and some
reason it's not like it right places
where it might be helpful places where
it might not be helpful the idea of
doing this sort of analysis to
understand you know when can we trust it
when can we believe that it's going to
solve various problems that we have
um uh you know so that we can we can
figure out how to make use of these
approaches right so we're not just ah
you know here's an interesting approach
we have some understanding of how it
actually fits in to you know the broader
picture of you know how we can use
various different approaches to sort of
you know eventually you know uh you know
as Humanity have ways to be able to
address all the problems that we need to
be able to address
okay so uh Now sort of for the sort of
the last bit of this talk I want to sort
of take a step back and talk a little
bit more generally about you know what
are the various different sorts of
training stories that we might tell in
general where do we get evidence right
what are the sorts of training goals we
might imagine uh and what sorts of
pieces of evidence might we find uh you
know and ways in which we can have to
generate evidence to believe that you
know some particular you know training
rationale would hold
okay so what are some training goals so
you know one example of course is the
example that we talked about previously
in this sort of strict inner outer
alignment sense right which is you know
maybe the training goal we want is we
want a model that is just directly
optimizing for that you know loss or a
word function that we specified we want
a model that's just directly doing the
thing that we wrote down this is one
thing we might want right it is
absolutely a thing that you might be
trying to get in various different
circumstances right now it's not always
the thing that we want right so in
microscope AI it's definitely not the
thing we want right we're not trying to
get a model that is like minimizing its
you know its accuracy on on something
we're just trying to get a model which
understands the world right uh in some
very you know General sense and so it's
not always the thing that we want but it
sometimes is right you know sometimes
sometimes maybe that is the sort of goal
you're trying to get you know you
actually believe if you have written
down an objective that captures
everything that you care about and you
want a model that just is trying to
optimize for that objective
um so you know the idea would be that
you know in this case we sort of have a
model that is uh you know instead of
optimizing for some direct thing it's
sort of just optimizing for the reward
now the problem with this training goal
right is that it's sort of a little bit
dangerous right because we have a
situation where uh you know we sort of
get reward hacking potentially which is
this idea of well you know it's very
very difficult to have a reward function
right or a loss function that actually
fully specifies all the things that you
care about and so there may be
situations where the model can sort of
get something that satisfies the
technicalities of the loss but doesn't
actually do the thing we really
intending it to do
um you know classic examples of this are
situations where you know there's this
sort of classic boat uh game where
opening is trying to get the model to
you know uh you know get a boat to sort
of go around in circles uh and it
discovers that you know it is actually
trained on with sort of getting these
gold coins in the in the race that are
sort of spaced around and it finds they
can just sort of run in circles hitting
up against the wall to get this one
respawning coin over and over again and
so the idea is while this was
technically a thing that satisfied the
law loss function but it wasn't really
the thing that we intended and so this
is sort of a thing that you can be
really concerned about with this type of
training goal so we really want
alternatives
okay so here's another thing that we
might want maybe we want an agent that
is sort of myopic so we talked about in
the case of deception you know one thing
that we're really concerned about is
Agents they're sort of optimizing for
things over a long time Horizon right
they care about something in the world
they want to get something in the world
over the long term and that's sort of
the reason that they try to deceive us
so maybe the thing you want is you want
an agent which isn't trying to optimize
anything in the long term it just has
some sort of short-term goal
um this is like another thing that you
might want this is another sort of type
of training goal that you could have
um but there are problems here too so
you know issues that sort of arise in
you know trying to make these sorts of
training goals desirable
um I'll relate to these sorts of issues
we were talking about about
self-fulfilling prophecies and sort of
issues we're going to talk about later
about you know in case we have these you
know agents that are just you know
trying to optimize one thing in the next
time step
um there are cases where that can sort
of break where that's sort of a very
brittle abstraction so you know a simple
example might be
um if I am just sort of just optimizing
for my one time step but I know that I
have the ability to sort of cooperate or
not with a bunch of other agents that
are also optimizing for their only for
their one time step then we can all sort
of agree well if we do this thing then
it'll be best for all of us overall
um and so they can all sort of cooperate
and coordinate on doing one individual
thing that is best for all of them and
that can in practice look like
optimizing over a very long time Horizon
because they're sort of coordinating
together over time even though each one
individually only cares about one sort
of time step so so you can have examples
like this where it sort of gets very
tricky but you know this is another
thing you might desire right we want
something which is really just
optimizing over the short term
okay you know another thing we might
want is a sort of truthful question
answer so this is you know similar to
the thing we were talking about uh you
know at the beginning right you know
Paul's example where we have you know a
model and it just sort of truthfully
translates between its model of the
world and the questions that we ask it's
this sort of direct truthful translator
you know honest question answer you know
there are various ways which we might
try to get this you know uh things like
debate the applications we'll talk about
later
um but you know something which is just
sort of directly trying to answer you
know questions truthfully
okay uh maybe we just want you know one
thing we might want is we just wanted to
learn human values right you know it is
totally you know on a possible training
goal you know is the sort of most fully
ambitious training goal right we're just
like we want a thing that is just
directly maximizing for you know the
things that humans care about right this
is maybe you know the most original most
basic thing that oftentimes people who
thought about AGI have sort of wanted
but you know it's only one of many
possible things that we might be trying
to get in various different
circumstances it is one thing that we
might try to train a system for it's
worth pointing out it's maybe probably
the hardest thing of all of these right
it's a really complex goal it's one
that's very difficult to specify right
so it's one that can be can be very
difficult to get but it might be your
goal right in some cases eventually
maybe we might want to build these sorts
of systems though I would say we
probably don't want to build these sorts
of systems uh in the short term at least
okay uh you know we talked about
corrigibility right so I was saying
courage ability you know in this sort of
basic behavioral sense is not you know
good enough as a uh adjective but there
might be other senses right so we talked
about the corrigible in line models
right like the Martin Luther's might be
another sort of thing that you know you
might be trying to get Yeah question
sorry I just realized that I don't
really understand how those minimizing
models would fit into this list like
why why would that be safe like what
would be the loss function for which it
would be safe isn't it before the
aligned already
uh yeah I mean so I guess that there are
some cases where it might not be the
case that the loss function specifies
everything about human values but it
still specifies enough that you're sort
of okay that it's not going to like
destroy the world right so maybe your
loss function specifies something about
impact regularization so it specifies
you know I want you to uh you know just
fetch the coffee without doing anything
else crazy in the world right and maybe
if you actually believe you've really
nailed down what it means to not do
anything else crazy in the world than a
model that is really just optimizing for
that would be okay right and so there
are cases where you could have a loss
function right and a model that was just
directly optimizing for that and you'd
be okay with that even though that loss
function wouldn't like you know imply
that it's doing the right thing in all
situations it at least wouldn't be doing
a sort of sufficiently wrong thing that
you would be really concerned about that
you know they'll be bad for the world
okay
um and so great so courage ability so
you know we don't want this sort of
Behavioral description but maybe we
could get something more mechanistic
like this sort of Martin Luther models
right
um okay uh you know we've talked about
this sort of predictive models you know
models like in the microscope AI case we
were just trying to train a sort of
predictive uh model that is just trying
to do some uh you know sort of trying to
understand the world it has some World
models to try and do some prediction
task we're going to return to this
specific case more later uh because
there's a lot sort of more to talk about
in terms of what it might look like in
this sort of specific case I think this
is often what a lot of people want with
large language models
um and so this is sort of a pretty
common training goal and I think one has
a bunch of really interesting properties
uh it's another thing that you might
want people often refer to these as sort
of generative models or simulators the
idea is you know a model this really
just has some World model is trying to
do a basic prediction task based on that
okay
um oh an important thing to think about
uh is the sort of difference between the
predictive models and the truthful
question answers so the truthful
question answer also has a world model
but it answers questions truthfully so
if you ask it you know
um some question about the world it will
tell you exactly what is true about the
world whereas the predictive model will
only tell you you know what it would
what you would observe right so the
predictive model is maybe predicting you
know what would show up on the internet
in some particular case you know what's
what is most likely what sort of tokens
are most likely to occur on the internet
what sort of tokens are most likely to
occur uh in some particular situation
and that's not necessarily truthful so
the predictive model is not always
truthful whereas the the truthful
question answer is always truthful Yeah
question
so does that mean like theoretically you
could train either one of those agents
you would always want to train the
truthful question answer and there's no
Advantage a particular model has
a gist of the predictive model is easier
to train with our current method
yes I think that's basically right so
you if you if you could theoretically
train either one it would be equally
easy you would want to train the
truthful thing but you know maybe you
think that it's too hard to train the
truthful thing but it's easier and
sufficient to just train a thing which
is making you know more general just
like predictions
okay
um
okay so you know another possible thing
is sort of narrow Asians so you know
maybe you want an agent to sort of
relate to the impact or utilization
thing they were talking about previously
that you know just solve some particular
narrow task
um but doesn't do you know some very
general uh you know saw you know
optimized for human values in some very
general cases just you know trying to
you know get the coffee and you're
trying to train in a case where it's not
going to be doing any other things right
um this is like you know another sort of
thing you know gold you might want you
know another training goal is the sort
of you know concept I think that exactly
how this thing would work
mechanistically that was a little bit
poorly understood right what actually
would it look like for an agent to be
doing something like this there are some
you know possible hypotheses or
something like quantization would be an
option where the idea is you know it has
some objective but then rather than
optimizing for that objective it only
sort of uh takes the top 10 of outcomes
then uniformly randomly picks from uh
from from so maybe that is a lot safer
because it's not sort of directly
optimizing it's just sort of doing some
narrow task and only you know so well
okay uh and I I definitely you know one
thing I really want to point out is this
is not an exhaustive list so the idea of
this sort of thinking about you know AI
in this way uh is not to sort of be like
okay these are the possible options you
know pick one the idea is to sort of
really thinking about okay what are all
of the different possible things that
you could have in machine learning you
know process uh you know try to find
what are the sort of algorithms we might
want uh you know so that we can have you
know you know a large possible space of
you know what are the things we might
might be trying to get so we can think
about which things do we want in
different circumstances which things are
going to be safest which things can be
easiest to get uh etc etc
okay okay so we might also want to do
the same thing with rationales so you
know we've talked about you know this is
you know some you know these are sort of
possible goals you might have we also
want to understand what are the possible
ways of gathering evidence about these
uh you know particular types of training
goals what things might convince us that
you would actually have a machine
learning process which would get a model
which is well described in that way
um so what might that be so you know one
thing is inductive bias analysis so
we've talked this is sort of the thing
that we spent a bunch of time doing in
the previous lecture you know really
trying to understand okay in various
different versions of the inductive
biases of you know the the you know
machine learning system and how we can
think about them How likely is deceptive
alignment right and this gives us some
information right it tells us some
things that you know we can predict
About You Know How likely is you know
something like deceptive alignment in
various circumstances but it's very
difficult piece of information to work
with because it's really uncertain you
know we don't know that much about the
inductive biases right now
um and so it can be hard to sort of
really reason about this rigorously
there are some examples of sort of doing
this a little bit more rigorously so I
have one over here that is a little bit
more rigorous example of inductive bias
analysis but in a little being a little
bit more rigorous it's also much more
narrow it's sort of focusing on a much
sort of uh you know narrower uh task
um but but you can try to do this you
can try to sort of work through these
things in theory and try to understand
you know based on various different
versions of the inductive biases how
would things go this is also using a
much more sort of a very theoretical
sort of notion inductor biases that is
not maybe very you know well grounded
and so you know there's there's often
here a lot of tension between Type you
know inductive is analysis that we can
do in theory and actual practical
inductive biases that would exist in
practice and so you know we saw a lot of
this tension previously where you know
in the last talk we had to use these
entirely two different Notions Dr biases
to even have any reason to believe that
you know we're going to get some
conversion result at the end
okay so that's one way that we can do it
and one advantage of the inductive bias
analysis of course is that it you know
it's something that we can do sort of in
theory even before we built the system
um transparency durbability is sort of
the opposite this is something that we
can do sort of often it can give us a
lot of information you know if we just
look at the model and we see ah it's
doing Windows type of Wheels we can get
a ton of information about what it's
doing but it's something that is that is
sort of very difficult to do in advance
right so we can maybe do some
transparation durability on early models
we can maybe see you know how it's being
trained to do transparent
interpretability along the way sort of
see how it's developing but this is you
know it's very difficult to get
information about the model uh in
advance by doing transparency because we
often have to sort of build the model
and then figure out what it's doing and
that can oftentimes be sufficient you
know it can be okay to just build the
model and then look at and see if it's
okay
um but it also can be a little bit
tricky if you sort of think that just
building the model might itself be
dangerous which at least for
sufficiently powerful models is
potentially a possibility
um but I I another thing I would say
here is I also think this is maybe the
strongest piece of evidence on this list
right uh you know what is the thing that
when most convince you that the model is
definitely doing the thing that you
wanted well we looked at it we
understood exactly the internal
structure and it had this internal
structure right it's doing exactly this
thing and so maybe you know I think in
some sense this is maybe one of the
strongest piece of evidence if you can
get it though it's very difficult to get
because you have to actually do this
transparency and understand the concepts
that it's using which can often be
really tricky
um okay so you know another thing maybe
is the sort of game theoretic analysis
so if you're trying to think about
something like the you know train for
cooperation right that we were talking
about previously well why would you
believe the train for cooperation would
work well you know the training goal
there right wouldn't be you know just
directly optimized for cooperation the
idea would be well maybe the sort of
equilibrium of you know various agents
playing in this environment is the sort
of equilibrium that we want so you know
and sort of example of this uh you know
approach is sort of thinking about well
maybe you know you know humans in fact
learn some particular values and we
learn those values you know based on
them being you know good in some
ancestral environments so you're like
okay you know maybe we can have some
environment where the equilibrium
solution you know the agents that you
would be most likely to learn would be
the sorts of agents that um you know
would have the sorts of values that we
want so you could try to you know get
some evidence by this you know doing
some sort of theoretical analysis or or
empirical analysis of you know in some
environment what are the sorts of agents
that are likely to do well in the
environment uh you know they're likely
to sort of cooperate and coordinate with
other agents that are likely to sort of
survive thrive uh you know past multiple
rounds of natural section etc etc or in
this case artificial selection
okay
um another thing that's maybe worth
pointing out here is
um capability limitations so you can
have situations where uh you know the
reason that you believe your model is
you know going to be you know
implementing some particular thing is
just because you didn't give it a bunch
of information or you just didn't give
it the ability to implement some other
thing right so if you think about
something like the cat dog
classification example maybe the reason
that you believe it's safe is just that
well it's such a simple task that it
can't possibly learn you know these
really complex things that would be
necessary for it to do something
dangerous so this is like another piece
of evidence you know important piece of
evidence that you can use here you know
one example of this is you know trying
to sort of maybe restrict modeling
humans because maybe you believe that
being able to understand humans is
really important for being able to
deceive humans and so maybe you just
don't try to give it understanding about
humans then you could you know be more
confident in it of course that's hard
because we often want to be able to
understand humans so this can get very
tricky but it's another you know piece
of evidence that you can use
okay uh you know having oversight right
so another piece another way you can get
evidence is well throughout the entire
process of building the model I've been
able to look at it and understand what
it's doing
um we're going to talk later on about
you know what these sorts of oversight
processes might look like amplification
debate
um but the idea is well okay if I have
some way of sort of continuously
overseeing the model's Behavior then
that gives me some information to sort
of predict well okay you know I have
some reason to believe that it's going
to be doing the right thing because I've
been looking at it as I've been training
it
um another thing that I sort of uh like
to talk about the sort of AI cognitive
science idea so you know maybe just by
looking at the model's Behavior we can
build hypotheses about what sort of
mechanistic you know things might be
doing internally and then put the model
in various different situations to test
those hypotheses and then extrapolate
forward so you know if we're thinking
about okay we have two different
training regimes we want to understand
How likely is that training regime to
yield a model that is you know
optimizing something over the long term
or not we can build you know hypotheses
about you know what would it look like
what sort of consequences would happen
from a model that's doing that versus
not doing that and then we can you know
figure out okay does this training
procedure yield that sort of model or
not and then we can predict okay if we
keep doing with this training procedure
it's likely to yield you know things
which are consequentialist and in this
training procedure it's not and so we
can sort of use that as a way to predict
you know what's sorts of behaviors are
we going to get later on you know one
thing that's very tricky with this sort
of approach is that it's very behavioral
right we're just focusing on you know
okay making inferences based on things
that we observe about the model's
Behavior which of course you know can
can get us into problems with with
deception like we were talking about
previously right where you can't always
be fully confident meant that the model
is not deceptive just by looking at its
Behavior because it could be trying to
trick you
okay so one thing that's sort of very
related is the sort of precursor
checking which is well instead of you
know if we if we think that behavioral
checking is always sort of going to be
potentially running into issues well
maybe instead of looking for something
like deceptive alignment directly since
if you look for deceptive alignment
directly then the model can always sort
of be trying to trick you you look for
some precursors things that are
necessary for deceptive alignment to
arise but not deceptive lineman itself
right so is the model you know thinking
about something you know uh thinking
about the training process is it
optimizing a long-term goal right you're
like okay what if we just look for those
sort of precursors then maybe you can
have more confidence right even only
doing behavioral analysis
um another important sort of component
of something like this uh
um so yeah another thing you could do
would be like lost landscape analysis so
this is sort of related to the inductive
bias analysis but maybe sort of on the
more High path dependent side so with
the inductive bias analysis we were
thinking about you know something more
low path dependence where we're just
sort of thinking about you know
Simplicity and you know and speed but
you can also think about things directly
by looking at the you know the basins
trying to understand How likely would
various different you know paths through
model space p
um so it's worth bringing out that
there's you know both low and high path
dependence versions of this
and then finally another thing is sort
of this is related to the precursor
checking idea is you could do sort of
scaling laws where you try to understand
you know as I you know vary various
properties about my models how does it
change various alignment properties
right and I can use that to sort of
predict you know as if I'm training in
this particular way I generally get
models that have you know long-term
goals and I train in this way I
generally get models that don't and I
see the long-term goals going up here
then you can predict that mail that it's
going to continue going up and you know
you're going to eventually end up
potentially sort of in in the regime
where you can get some long deceptions
you can use these sorts of ability to
understand earlier models to you know
give some information about what future
models might do
okay uh and again you know this isn't uh
you know an exhaustive list there's lots
and lots of other ways that you can get
pieces of evidence and information uh
about you know what sort of algorithm
your model is going to be learning but
the basic idea is this is the this is
the sort of business we want to be in we
want to be you know uh trying to gather
information this way to help us
distinguish between possible algorithms
the model could be could be wrong
okay and then one very final thing that
I want to talk about is you know once
you have this training story you have
some you know basic understanding of
what mechanistic algorithm you want to
be doing you know why you believe it's
going to be doing that you also sort of
you know want to then you know really
have some ability to understand okay you
know how robust is this understanding
right we have some reason to believe
that we think maybe this training
process is going to go through uh you
know this training story is is going to
be correct
but then you also sort of really want to
be able to put in a bunch of analysis
and be able to get you know
probabilities out for people to
understand okay you know do we actually
trust this right because there's a
really big difference between okay this
training story sort of makes sense I
maybe understand you know why I would go
through and like we're super confident
you know we're willing to you know stake
the world on you know this being true we
have like very good you know rigorous
analysis right and so we sort of want to
be able to understand okay how can we
get to that point right like what are
the sorts of things that you could do to
really test and you know probe at your
training story so the idea is this sort
of sensitivity analysis so you know you
can analyze how robust your training
story is uh by looking at sort of how
sensitive it is right so you can be you
know uh we can look at other you know
similar smaller models you know see how
they fail you know extrapolate those
failures right if we have a bunch of
other training stories you know and
we've seen how well those training
stories do in general we can have an
understanding of How likely is any given
training story to succeed right so we
think about something like the cat dog
classification example we can be like
okay you know maybe a priori we expected
it to do this particular thing we
expected it to do uh you know human you
know Concepts and then we found out that
it didn't right and so we're like okay
in general when we build training
stories how often do those training
stories actually match on to what we
find and so you know how often are we
wrong right and so we can use that as
some understanding of okay as we start
building more and more complex training
stories they're more and more difficult
for us to really predict in advance you
know how often are we actually able to
get them right
um you know we can also just sort of
characterize the space of possible
models right so you know we did this you
know previously we're like okay you know
some of the possible options you could
get are things like you know the Martin
Luther models uh you know there's
definitely align models you know the uh
you know the internal line models all
these different types of of models and
that gives us some understanding of at
least what the options are so we can
understand okay how bad were the
possible failure modes be can we at
least rule out you know some of the
worst possible failure modes
um and you know we can also sort of look
at uh you know even direct sort of
perturbations we can be like okay once
we have training stories which are as
mechanistic really mechanistic and
really clear uh you know okay we think
we're gonna get exactly the sorts of
algorithms we can start to understand
you know what happens if individual
Parts on that path go differently right
so we were talking about you know in the
high path dependence case where we're
thinking about you know this is like the
path to the query line the Martin Luther
model you know first you you know you
learn these sort of world modeling and
um you know proxy objective
simultaneously but then you know uh and
then you sort of eventually you know
gets replaced with a pointer but the
deceptively aligned question you know we
can sort of ask well what if there was a
small perturbation right what if instead
you learned you know um you know first
you learned you know how to understand
the training process and you use that
right and so we can sort of think about
these paths and think about well okay
what if you know various individual
things in our understanding of how the
sort of model is going to you know be uh
be trained we're slightly different we
can sort of you know reason from these
perturbations to maybe sort of start to
have some more confidence that even if
our model was slightly wrong things
would sort of still go the way we want
okay this is again not an exhaustive
list but the idea is you know some ways
to sort of you know once we have these
training stories we don't want to just
be done with like okay here's a maybe
plausible case for why we think some
machine learning training process would
yield a particular algorithm we really
want to be as robust and as confident as
possible that this training process is
actually going to yield the algorithm
that we want
okay so that's the end of the talk uh
hopefully this was a little bit helpful
for starting to get into understanding
you know evaluation right starting to
get an understanding okay we have some
idea of the problems you know how do we
actually uh figure out how to evaluate
various different proposals
[Applause]
foreign
okay any last questions
yes
um you've said World models a couple
times in this talk on the last talk um
could you tell us exactly what you mean
by World model
oh that's a tricky question yeah I mean
I think that
um and this is maybe something also I
wanna we'll sort of touch on a little
bit when we talk about prediction but
it's It's Tricky I mean I think that the
basic idea is we're like okay it's some
understanding of the world that is not
necessarily sort of that is just
separate from how you act on that
understanding right so you have some way
of understanding facts about the world
think patterns in the world things that
you know about how the world works and
that's distinct from okay given that I
know these facts about the world you
know and I'm trying to achieve you know
goal X I choose to Output you know
action a right and so we have some
understanding that is like okay
we really want to separate those things
we want to separate your understanding
from the way you use that understanding
to produce outputs and so we sort of
want to call the understanding part the
world model now I think that that
separation is a little bit fraught it's
not clear that it's always even possible
um but oftentimes it is a really useful
piece of analysis and so we'd like to be
able to do it
um
but it is I think it is really tricky
okay well uh we'll call it there so uh
yeah hopefully that was that was good