248  Eliciting Latent Knowledge 2

hello and welcome to session 248
in the ai safety.com reading group
tonight we'll be continuing with
eliciting latent knowledge how to tell
if your eyes deceive you
this is still the first work of the
alignment research center by paul
cruziano ayakotra and maksu
and we are at this part two probably out
of four
i'll start with a brief summary of what
we talked about in the previous session
this is the smart mould a vault designed
to
keep a diamond safe which is uh
controlled by a number of
actuators that are controlled in turn by
a an ai that we have trained for the
purpose of keeping the diamond safe
we as humans can only look at the
diamond through some sensors and there
are potential evil robbers that want to
steal this diamond from us
we can't
understand
the small smart world entirely but we
can look at the observations and then
give adjustment
about whether it's good or bad and
whether we can see the diamond
and the problem is that in this case
everything is good the the diamond is in
fact in this place but there are also
attacks that
cheat the sensors in a way so that we
see
that diamond in this place but actually
the diamond has been stolen
the way we want to
avoid this problem is since the
knowledge the diamond has been stolen is
predicted by the ai the ai must have the
knowledge somewhere inside itself um and
this part of the ai
that controls the smart vault that knows
what is going on that's the part we want
to hook into in order to get um
some kind of uh reporter that we can
later ask questions like is the diamond
really there instead of just looking at
the video feed that might be corrupted
we imagine that the ai is internally
using some kind of bayesian network
with
the actions that it's taking the input
video sequence and the output video
sequence
and then uh the actions that it's taking
um
yeah down here
and then somewhere inside this network
is the knowledge that the diamond has in
fact been stolen since if
we're in the case where the diamond has
been stolen and we're predicting that
we'll see this but the diamond has been
stolen
um
that came out wrong
like this knowledge here that the
diamond is being stolen and we're seeing
a um
a false image since that the false image
is the one that's predicted this must
this knowledge must be present in some
of these notes
now a human analyzing this
issue will not really understand what
this complex action sequence does and so
the human when
looking at this
has a model inside the brain that says
does the the brother take the diamond
and this part will answer no
the diamond is still there because the
humans network doesn't contain the
knowledge that the sensors are being
fooled
what we would really like is some kind
of honest reporter a direct reporter
that takes the knowledge that the ai has
and transforms it somehow into the same
notes that the humans are using
so the humans can understand it so the
humans if asked
this is the part of the human
network that deals with where is the
actual diamond
and in this case does the robot take the
diamond
the answer will be yes because this
knowledge that the screen up here is
showing something false that was present
somewhere in here and that means that
this
direct reporter
moves the information into the
human
understandable patient network and
allows the human to conclude that in
this case yes the rubber has indeed
taken the diamond
the problem is that if we start to train
uh this
you know in
in the ai in in the simple way where
we're just trying to uh look at the
cases where the human knows what is
going on the human can distinguish well
if we're looking at what the humans know
then the ai has the option to simulate
the human and ask its simulation what is
the status of the diamond
so this is called the human simulator
where the ai has its own network and it
also in addition instantiates a network
with what does the um
the human know
and in this case the reporter will then
ask this
this part of the network which knows the
same thing as the original human does
and so doesn't know that this
output here is indeed something that has
been faked
so that is the uh the problem with the
most naive way to try to keep the
diamond safe
before we look into some more complex
ways uh the the authors have a
digression where they talk about the
research methodology and why this seems
how this kind of argument is structured
on a meter level so let's dive into this
first we'll notice that the argument is
for the worst case in the sense the
in particular in the sense of high
capabilities
um and why do they do that well if there
is a case where it fails then they want
to continue because most likely as the
ai gets more and more capable it will
just fail worse and worse
and
by
having these um
by quickly being able to find holes in
prospective solutions they hope to be
able to israel
um
in particular because many of these
strategies that many naive or i would
say most naive alignment strategies fail
for ai that is that is substantially
more smart than humans and if these uh
strategies fail then we have literally
no time to react and that's why the
focus on the worst case will find fewer
solutions but hopefully some that don't
fail catastrophically and immediately
once we get ar that is smarter than
human
one thing that the research methodology
doesn't really make entirely clear is
to what extent the worst case just means
um very high levels of capability we saw
with the previous example where the
worst case was kind of more that the um
the ai found a human simulator that was
really bad rather than having a
particular high
level of capability so those two things
are almost the same but not quite the
same and the um the article doesn't
really get into this at least in the
part we have seen
so let's zoom out a a bit and see this
methodology as a game between two
players a builder and a breaker
so it starts with a builder proposing a
training strategy in the first simple
example the training strategy was to
just look at the places where humans can
tell if the diamond was safe or not and
and then hope that generalizes
uh that is passed to the breaker who
gives uh a
predictor and a world a world in this
case is like something that would be um
a scenario where this would go wrong i
don't think this is used in any of the
cases we saw either uh we will see today
or so last time but in this case the
predictor is one um where which makes um
predictions with a different base in
that than than the humans
the next step of the game is that the
builders
specifies what kind of reporter he would
like and in this case we want something
that uh translates from the predicted
space net to the human space net
and
unfortunately the breaker says
that the
reporter that
will be trained using this training
strategy is
it could be one that is doing inference
on the human base net instead of
the desired reporter
after this there should be a dialogue
about possibility the uh
like the translation must exist the
builder can say since it has it can map
to reality like english seems like a
reasonably intermediate
um but uh the player can argue that
since it's the humans who make this the
answers and they can potentially make
errors and things like that then um that
is in fact the one that will achieve the
lowest loss and that's why you're gonna
get the bad reporter and not the desired
reporter
um
informal steps
so the idea also with this chain
strategy is that we start with a very
high level idea about uh what we will um
uh
find a counter example for using this
game and then if we can't find something
that really totally defeats
a training strategy in all cases we'll
try to dig in a bit
and specify more things about the
training strategy and see as it becomes
more specified if there are any problems
with this and um
we probably will won't really get into
the part where it's totally specified
but the idea is that we want to to have
a counter example on a very broad level
and then if we can't find that we get a
smaller and smaller and smaller one and
hopefully the end is that we have one
that is so small that we can't find any
errors in it
in some kind of in one training strategy
and at that point we want to you know
test it in reality see if it works
what counts as a counter exam
for eliciting latent knowledge
um
they describe this as a narrow existing
latent knowledge i don't know if there
will be something more uh general later
but this is
not a thing like
uh
like when we're talking about stealing
the diamonds we're not trying to do
something philosophically complex um
like replacing the diamonds the athens
and diamond slowly or this kind of thing
we're just talking about something
that's really unambiguous there are
three
um requirements in total
um oh yeah um and this is actually
something that could potentially be
problematic um some work done
on things like under indescribable hell
worlds seems like something that could
potentially be a problem for this this
an ambulation
um
the three requirements are that the
model must understand the question the
model's answer is unambiguously wrong
and the model knows that the answer is
wrong
and the example of the human simulator
satisfies all three of these
so
now we'll look at three training
strategies that hopefully will do better
than the naive one we did
and hopefully we'll leave the diamond in
a better
position than this screenshot from
computer game
so the first question is the question
that is motivating all these three
strategies is we want to have a better
data set because what we
really want is the correct data set is
the diamond safe or is it not and we
started out with something that was
really poor so i think does this look
correct to a human and we want to get
something better
um
because the obvious problem is
if you're looking at something where the
humans can tell what's going on and the
ai is much smarter then you're really
only looking at a very very small set of
uh possible action c
sequences this is the case of all the
the space of all action sequences and we
assume kind of that the the space that
the ai can suggest is like almost as big
as this right
and then we have a very small area where
the humans can tell that the diamond is
safe and are very small where you can
tell that the diamond is not safe and we
want to expand these we would prefer
that it covers everything but
we'll see how far we can go
and
of course since um
yeah the human simulator the reason it
was feasible um is because we were just
looking at where what can the human tell
um
but if we had just another
kind of small advantage like imagine an
oracle that once you know 100 gives us
the correct answer then that would be
probably enough so that the last uh
function the loss would be better at uh
the direct reporter right rather than
the human simulator so we want some kind
of advantage
um
and in the next um in the rest of this
session we'll be discussing three ways
to get a bit more of
advantage the first strategy is not just
having the human um observe the results
but having the human actually operate
the smartphone
so
in in this basic scenario it was of
course the ai that operated the smart
walls and if we have another human that
is somehow smarter
then it's possible that
that human can make something that um
that fools the the observer like you
imagine that there are
a large number of possible
vulnerabilities and then the um
the
operator looks at uh tries to figure out
what kind of vulnerabilities are
feasible and then the uh the observer
will later think about what uh
vulnerabilities are possible and if they
don't match
um then there's a good chance the
operator has found a vulnerability that
the observer cannot easily find
and if we can get this kind of
situations then we can label more
situations than just what the human
uh
the human observer could evaluate
you can see it here in the picture we're
now
labeling as a larger green area and a
larger red area
and one of the things that they don't
say
is that um
action sequences
the kind of action sequences that we're
looking at are explicitly the ones that
are designed to fool so you imagine the
screen put in front of the camera uh and
that's of course the kind of things that
we're really really interested in so
it's not just that we are looking at a
larger set of
powerful space but we are also looking
at a far more interesting part of the
space
so
how will this help well um
in this case just if we try to learn if
if the ai tries to learn the um the
human simulator then if if the knowledge
of this exploit that's being used is in
the operator snip and not in the
of service net then making uh
like the information is not present in
the observers so it will have to learn
the uh
uh the direct report
and we assume
that
this works because as i said before that
um
finding exploits and ways to trick this
camera is probably a lot easier than for
uh
for the observer to determine whether
the camera is working correctly
so in this case we have the same network
as we saw before but in this case the
attacker is thinking really really hard
about something
like how can you put up a screen um
and then picking uh some action that
includes uh um
putting on putting on the screen and
in this case once the uh the human
simulator tries to think about this then
the human simulator has instantiated the
observers
network and not the operators network
and inside this there
they the observer hasn't thought
especially how about this node so when
when you query this area
does the robber take the diamond then it
will just say no the ai because the
observer doesn't know this but in fact
the operator does know this
because he has thought really hard about
it so he can
he can in this way train the ai to give
better answers
um so in this case the uh the human
simulator will perform worse than the
direct translator and um hopefully that
will mean that the ai will then not
learn the human simulator but we'll
learn the diagrams later
there's some speculation about what will
happen if the human outsmarts the ai
in this case i believe it will be a tie
that's not really important because what
we care about of course is the case
where the ai is smarter than the human
so that was a strategy but unfortunately
there is a counter example and the
country example is that the ai has
another option that will
allow it to get a lower loss by having
the human simulator and that is by doing
better inference in the human base net
so
if we assume that the human observer and
the human operator have the same
bayesian network that kind of seems like
a reasonable assumption that there is no
really big fundamental difference
between
these two in particular
if the operator explained the
the plans to the observer then he could
follow how it has how that worked um in
this case
the uh the advantage of thinking hard
about uh
part of the network disappears because
the um
uh the reporter um the um
um the ai if it does uh
perfect
like uh uh maximal human that's my word
inference so it thinks about all the
nodes in the bayesian network as much as
a human that thought for as long as he
could would do
in this case the the advantage of the ai
um that
of the operator who thought really hard
about a small part disappears because
the ai can just
think hard about everything that the uh
operator could think about
and in this case then the reporter would
in fact be able to answer the training
question correctly by just having a
smarter human simulator um and in this
case
that would answer correctly if
everywhere the operator could also
understand it and it would also run in
the other cases
next
we have uh
this
this
these images here were for the um the
simple things that humans could
understand and this is for the action
sequences uh performed by the smart
vault that humans can't understand and
the way we would imagine we could
get something better would be to have
the ai help the human figure out what
does this action sequence actually do
um
so uh the uh the limitations of the
human base network is probably um
something that can be
pushed against by
basically education uh like humans have
individual individual education where we
just look at arguments and things like
that and humanity as a whole
can also uh
be improved we can think about doing
experiments and things like that um
this is uh something that paul
christiano calls doing science
and i'm entirely sure i think that's a
great um a great way to put it um
because um
but we when you think about doing
science you think about
yeah formulating hypotheses and running
experiments and such things but really
we are looking at a very very specific
small part of reality which is the smart
goals
and i think doing science sounds like
something that is way bigger than uh
what we're actually talking about
and traditional science of course we
should expect the ai will uh be much
more
efficient in particular because
um the uh
when you talk about traditional science
you're generally not talking about uh
something that is looking into one
extremely specific security arrangement
most of traditional science is working
on with you know something else
ai assisted science a feasible thing
well
possibly we could imagine
ways to do this um
i think in general uh
as ai becomes smarter
english and the way human brains work i
mean human brains are probably optimized
for something like
survival thriving on the savannah
and that's unlikely to be um
the same kind of concepts and structures
that an optimal bayesian agent would use
um
so uh this creates a new problem
unfortunately
that is that the
the science assistance ai could be
misaligned
and that's a substantial problem and
work has been done on this
um and the general way we try to make
ai assistive science safer is by having
tasks that can be broken down into
something that humans can test there are
uh at least three ways to do this being
under 50 by a debate it's rated
amplification and distillation and
recursive reward modeling
and uh
whether these work is a very open
question but for the sake of this
argument we are just going to assume
that this is something that can be done
it is possible to have a nai assist us
in understanding this without
any alignment problems and
we will show that even in this
it will be insufficient the air will
still be able to outsmart
uh us here we see an example of how this
training could work the ai which has a
very uh
sophisticated model of the of the world
uh points out some security
vulnerabilities that that humans then
add to their own network so then the
humans know a bit more and hopefully
they'll be able to um
to spot some uh
uh
uh some cases where there is um
where the diamond is being stolen
so now we are looking at potentially a
an even larger area where we can uh
tell where where an operator can tell
that diamond is being stolen
with the help of an ai
i
i would
question whether the green area would in
fact be this large because telling that
the diamond is safe in general
is asymmetric from telling that it's not
safe
it's usually a lot easier to see that
there is a problem uh some kind of
security vulnerability in something
compared to saying seeing proving that
there is no vulnerability in some kind
of setup
okay let's go back uh and see how this
ai teacher willa is
was still fall for the same problem
and
which was a maximum inference
in the human beings
and here i'm making a description that
doesn't follow the paper very closely
so all errors are mine of course
so previously we used just the observers
need to answer questions and now the
human operator
he had before the advantage of just
thinking heart and now he will also
learn these perfect vulnerabilities from
the ai will that help him
well if we uh make some kind of venn
diagram here's the original instrument
observe what he can think of and the
original
thinking thought about specific things
and now we have an enhanced human
operator
by the human simulator is doing
insurance into original human observers
neural network and by doing ideal
inference he can the the human simulator
can expand to also cover the original
human operator but he can't necessarily
get to the enhanced showman operator so
the enhanced children operator that
knows about these kind of security
vulnerabilities can make plans that the
unenhanced could never think about and
in this case the human simulator would
be unable to think about these
so the human simulator would um
would perform poorly whereas a direct
reporter if we're writing that then the
diet reporter obviously could think of
the same
things as an enhanced human operator
would be capable of
and that hopefully should privilege the
direct reporter and cause us to learn
that and not the human simulator
unfortunately there is a new counter
example and that is that breaking
descent is probably more efficient than
science
um
gradient descent is the algorithm used
by the ai reporter to predict
what would be going on in the training
set and
the uh
the teacher ai the ai assisting in
science is expanding the human base in
that one
these are two very different uh
processes and probably they take
not the same amount of time in
particular we would expect that teaching
a human is often hugely slower
than um that teaching a um
nai
and also
they might learn different things
paul krishan says that this argument
might be safe if a builder could give an
argument that anything learnable by
gradient descent can be learned by human
science
um now we don't i think paul christian
is actually um uh requiring too much we
don't really need to be able to learn
anything just a
a very small subset of things need to be
learnable but i still
think that it's hard to make such an
argument
and i would also add that there are two
different things here well what things
can a human learn and how long time does
it take to learn
well we should certainly expect that
human science is a lot less computer
efficient and and it is an amazing
knowledge part of the promise of this
strategy was to find a competitive way
to gain this knowledge
and
one thing that i would add in particular
is a problem with this strategy is that
if gradient descent can find something
that this ais to science cannot find or
doesn't find in time then we won't know
about it until it's too late and uh
until the diamond is actually gone
so let's try to do something even
smarter
we will have humans adopt the optimal
base net
there's a technique for this called
immediate imitative generalization
and uh the way you do that is you have
an ai
that
looks at all
different ways that humans could change
their minds could learn more things and
then
from this area it looks at what would be
the implications if the humans learned
this fact what would be the implications
if the humans learn this fact or
whatever
and then
try to figure out what is the best thing
to teach the human this is what they
call human plus plus
even though plus plus isn't the correct
notation it should be plus plus human
but that's brilliant
so the idea with imitative
generalization is we look at the things
that are best
that if you tell humans cause them to
make the best predictions
um and then you tell the humans that um
well this is possible in practice is
really speculative um but uh again we
assume that the technical difficulties
can somehow be overcome
and this defeats the previous counter
example because
now we can find the best we assume that
this technique can find the best patient
that can be that is possible to find
within this computer budget this is a
really really really strong assumption
but if we have this assumption then uh
then this then
gradient descent is not faster
and in this case we can
label cases uh
that would fail even with uh
the ai uh making a human simulator that
not just looked at the best possible
humans
at
the best possible inference in a human
network but the best perf
the best
inference in the best network that
humans could reach through science
and that means we get
something that is indeed very very
strong
and here you can see some of the changes
this is the reporter again and we make
some changes to both how the humans
predict and how they report and then we
ask questions to this
reporting module
and hopefully they will be good
but unfortunately that
even stronger strategy
might also fail
and the way it can fail is that um
with a antology mismatch that the
entities cease
matching to the things that we were
talking about like when we in our brain
have something that maps to like this is
a diamond this is a rubber then the
optimal network might throw this away
and and build some different ontology
that where different things matters and
are categorized in a different way
and it could do even worse it could
change the meaning of the of the uh
the point that we have in our brains
towards a diamond and in this case we
wouldn't even notice that we don't have
anything corresponding to the diameter
any longer
so we could see something where we
through this generat
this process gets a
a much better bayesian network inside
our brain that can make much greater
predictions but previously we had the
diamond the location of the diamond
within these two nodes that was the
reporter but in this new network the
diamond has been lost in some sense we
can't find it in
in any way
i'm not entirely sure this could not be
rescued because uh we want a
reporter that answers the questions of
where is the diamond in the same way as
the previous ones so that it seems like
some kind of side constraint
would be possible to add to this
imitative generalization
algorithm
well that would be feasible is of course
clear since we made some really strong
assumptions about that
so that seemed like the best thing we
could do like
trying mahara and hara to get some more
information and we basically got stuck
because if we do
try to get something that's better and
better it will cease to be good enough
until we get to the point where
we might require a new ontology in this
case we can't find the diamond any
longer
we can do something still
because we want to find the direct
report and we do in fact not have we
don't really have to ask the ai what
corresponds to this diamond we could do
something instead we could do use
interpretability network
interpretability techniques to look at
the network and figure out this node
corresponds to the location the
x-coordinate of the diamond or something
like that and then we can use that um
to figure out how uh it's making
predictions
um
and then once we have the um
the true correspondence between the the
neural net of this ai and reality then
we can define the difference between
these two uh that's basically a loss
function we can use for for the diamond
component so
we can find the uh
if we can find the the
the actual location of the diamond um
then we can just uh
uh get the report by seeing uh does it
answer questions about this curriculum
so that's at least a strategy but
unfortunately this brings us effectively
back to square one
and there they are trying in this last
part of the table they're trying to use
some kind of analogy i'm not entirely
sure that this analogy really holds uh
i'm gonna try because they don't make it
explicit at all uh so i'm gonna try to
make it explicit and we'll see how good
how well it goes
so this is my interpretation so uh yeah
buyer beware um
the analogy is written as
follows and then we'll see it for smart
vault and for interpretability
the analogy is that a human with ai
resistance is standing at a base method
which makes great glitchy and we want to
figure out how to use to answer
questions about the things we care about
like the diamond being in the room
we think there should be some correct
reporter but if we search for it in the
naive way we might get a human simulator
so in the case of the smart bolt
we have we're staring at a base network
that makes great predictions in this
case it makes predictions about camera
outputs
and we want to figure out how to get to
answer questions and in this case with
the smart vault we literally ask
questions to the reporting module and we
get answers
there should be some correct report in
the operating system the smart wall then
the knowledge about what is going on is
in fact latent
and if we search for it in the naive way
we might get a human simulator now
that's true in this case because the
human simulator gets
low and perhaps the lowest loss how
would this look for interpretability
well
that was no longer predictions about the
under uh about what does the camera
output but how does the
the the parts of the behavior network
correspond to underlying reality
and we no longer get the information by
asking questions we directly read it off
from the values of the network
the knowledge is still latent in that it
must um
it must correspond to reality somewhere
deep inside
um and
searching for a reporter can we get a
human simulator the obvious question for
instant probability research for
interoperability is no why would we get
that but i think we might actually get
one i think the analogy could hold in
integrability tools as they get more
complex we actually i could see us
starting to in fact run into the same
problem where our interoperability tools
will in fact
stimulate a human and then we're no
longer then then we don't
get
get anywhere but this is
my interpretability
in the interpretation of
this analogy and i'm not at all sure
it's correct
that is all for today thank you and see
you next week


256  Where I agree and Disagree with Eliezer 2

hello and welcome to session
256 in the aisafety.com reading group
tonight we will be discussing the second
part of where i agree and disagree with
eliezer by paul christiano
paul cristiano probably needs no
introduction at this point
and we'll be going through part 11
through 19 of the list of disagreements
so the first thing that
po christiano asks is where does eliezer
yudkowski
find the evidence that he uses for
talking about how difficult the
alignment problem is
well
the answer according to paul christiano
is probably in his own experience on
working with solving the alignment
problem
like on the face of it that seems a bit
uncharitable in the sense that elias
kowski has seen a lot of other people
try to solve alignment um but
uh knowing from what i know about elise
yotkowski i think
i would not be very surprised if he is
indeed basing most of it on his own
personal experience
then another thing that paul cosgiano
asks is society has apparently spent
very little total effort on solving the
alignment problem so based on that we
can't really say very much about whether
it's a hard problem or not
i'm not entirely sure i like that
framing because um
how much
your society could do a lot if we like
spend all our resources towards that but
that's not gonna happen i would agree i
would think that paul cristiano also
don't think we're gonna get a fire alarm
or anything like that so we're not gonna
devote all that effort so comparing it
to the total amount of effort we could
spend is probably not relevant
i prefer uh nick bostrom's concept of
recalcitrance in how much progress do
you make on the uh
on the problem uh given a certain amount
of extra input
there's a comparison with existing
research fields and the problems they
they routinely solve and it seems like
they're working in a different way and
solving different things uh compared to
miri so we shouldn't update too much on
the failures of miri
i think at some point
i think we are comparing apples and
oranges here because the
thing the fields that solve problems
routinely look very different from the
pre-paradigmatic field of ai alignment
i don't think these can be compared in
any particular way most existing
research fields would also be quite
incapable of dealing with
pre-paradigmatic
research
so miri tried and failed to find a
coherent formula for correct ability
does that mean that
corridability is unworkable
no paul kushner says because we don't
we don't have
miri is just not competent enough for us
to conclude this
um so a small nitpick here is that the
quote from
the way that paul christian quotes elias
yatkowski is technically not correct
he's splicing two quotes together
but that's just a nitpick i think the
the real answer to this would be that
miri
spent a good amount of effort on this
they had something like five to ten
research papers um
published on this and they gave up of
course and then after that for five
years no one was able to um
to improve on what miri did and i think
that is a substantial amount of evidence
that
corrugability is unworkable
as for the total
difficulty of the problem
paul
states that eliasa is
certain about the how difficult the
problem is
and uh to some extent that flies in the
face of the uh the introduction where
elijah in his
list of lethality's post
says the problem may in fact not be that
hard in fact he is
giving the example of a textbook from
the future that contains just simple
solutions that work so the problem may
even be quite simple
it's just that we are
we're not seeing and we have a hard time
determining which solutions are going to
work or not
right now we mostly don't know how hard
the problem is and that is uh very true
uh i do think that we have a significant
lower bound in that the problem is
almost certainly not easy i think enough
people have worked on solving the
alignment problem that we can rule out
most easy solutions
disagreement number 12 is about elias's
understanding of science
some of the things elias erdkowski say
seem to have a relevance towards uh
science in general for instance the the
pattern of bright-eyed young scientists
they ignore warnings from cynical
veterans
um that is something that paul
christiano thinks is probably ungrounded
um
three things that paul christina thinks
would ground this would be um
knowing more about the history of
science being knowing how functional
academic disciplines uh work and having
research experience
of the three i think elias utkowski
probably knows quite a bit about the
history of science that's at least the
um
like precisely how much is a good is
obviously not a real scholar of this but
i think he's
very well read um
i think uh the dynamics of modern
functional academic disciplines is
mostly irrelevant in the sense that
alignment is a dysfunctional academic
discipline
research experience you could say that
he has been like a researcher
most of his life
so some research experience could also
be argued but in general i think that um
this is
like the paul christian's criticism is
to some extent correct i think
the place where elias witkowski has in
fact experienced and is able to to talk
about this with some uh
degree of certainty is within alignment
and
like nanotechnology and these kind of
pre-paradigmatic fields which is kind of
his specialty so i think the patterns
that he is uh saying i would expect them
not to hold in
functional academic disciplines but i
think they might indeed be um
be valid in less functional academic
disciplines
as evidence for this poor christian
pulls out a bet that elias yudkowski
made back in 2009 where he bet 25 that
the large hadron collider would not find
the higgs boson um and that turned out
obviously to be wrong
now when i first saw this evidence i was
really meh in the sense that obviously
betting and even odds is something you
do if you're like 75 certain of
something and having a prediction that
you were 75 percent on uh
like so many years ago that seems like
really weak evidence to me but then i
looked into it some more and to see
elijah's
reasons for this and the reason he's
giving is
he doesn't think the modern field of
physics has its act sufficiently
together
and
that is in fact quite relevant if we
take elias's position on this being that
he doesn't think the modern field of
alignment has its act sufficiently
together um
so
like obviously alias kowski should have
updated on this back in 2009 but in
general it's really really hard for
people to update sufficiently on on
evidence so i do in fact think that uh
paul cristiano's point here is correct
also i could you might see here in the
background uh this is uh from the uh
comic uh the simpsons back in 1998 where
up here you'll see uh homer simpson
predicting the mass of the higgs boson
back in 1998. so at least some people
have um understood uh
understood this for a while another
thing that paul christiano um
points out is that um
there's a prediction by elias swijkowski
back in 2010 where will we see real ai
that's probably what we would call agi
now
and that's more likely to come from a
small group rather than large industry
that's given as uh just
straightforwardly by paul christina as
an example of almost a failed prediction
um
i don't think i would count this as a
fail prediction in the sense that if you
mean something like agi then the jury is
still out where will it come from
and the question then is if it comes out
from something like deep mind or
open ai is that like
an entire industry that is creating it
or is it um
like a small group to some extent the
number of key researchers in deep mind
and open ai is not that large i think
if uh
i think you could really argue that if
openmai or deepmind were the ones who
create agi for the first time
that would be a small group i i don't
think this prediction is horrible uh but
i do believe that um
elias elkowski would probably frame this
in a different way now than he did 12
years ago
disagreement 13 is about
interpretability
where elizakowski in his list of
lethalities have some central
difficulties of sufficiently good and
useful
interoperability um
and this point in particular is one that
poor christiano is not super happy about
and that is because the
uh if you find a number of um of
problems then the uh the future
researchers uh they can just choose uh
among the different projects that they
have in front of them the ones that
don't suffer from from this problem
and that's a
an important thing like just because you
can rule out 99
of the solutions doesn't mean that it's
doomed at all because the researchers
can just choose among the one percent
that don't suffer from the problem
having said that
what little interoperability research i
have seen seems strongly like it does
not in fact um consider
the lethalities that eliza utkowski is
pointing out most of it seemed like it's
only trying to get to sufficiently good
and useful
interpretability and that seems to be
really hard um but on the other hand
like
you have to
you have to solve interoperability to
the point where you can say something
like is this ai trying to deceive us
before you can
try to figure out what you want to do
with that so to some extent i think it's
fair enough to have that as a
prerequisite
um and
um
eleazar yutkowski is not really
according to pro christian engaging with
the best version of interoperability the
best strategies based on
interpretability
and i think
i think it's quite obvious that the uh
the strategy that paul christiano is
thinking about is his own uh eliciting
latent knowledge
uh strategy and i think elias you're
casting what little i've seen him engage
with that has mostly been very
derisively um so i think definitely he
should engage much more with that um
paul cristiano gives an example of um
five things uh from the
listing latent knowledge
report that
overlaps with this list of um
of difficulties
i think it's good that there is some
overlap but i think in fact that
five points is not that much
um and i think that this list is
in fact something that um
i think it's more like that the
eliciting latent knowledge should engage
with the uh
the list of difficulties that elias
yukowski is pointing out
a
an issue with with uh trying to predict
the future of interoperability research
is that you're going to learn a lot more
details in the future and does this
overcome the objections um
procrastina is framing that as a
rhetorical question and i think in fact
when i read this um the section b3 with
these uh central difficulties seem like
the kind of things where learning
details will not fix them of course it's
possible we will learn things like that
like there is a
simple canonical human understand and
double version of all concepts and that
would be huge for interpretability
research
but that's not really what we would call
a detail
i think the the objections will not uh
be solved by mere details um
that's also probably what elias utkowski
um
that's why he calls them central
difficulties
but paul christian is not really
concrete enough that we can't that i can
say this for certain
so
future ai systems
um
considers like the existential
qualifiers that them if there exists one
ai systems in the world which has one
way of taking over the world then that's
very bad um so that's a very broad
classification because there are many
ways of taking over the world there are
many ai systems potentially in the
future
but
humans are also manifold and
the existential qualifiers there that if
there exists one human who has this good
idea or one research project that ends
up with um
with getting a good grip a solution to
the alignment problem um
that's something that gives paul
christiano optimism that there are this
number of research directions um and
elias yorkowski is according to him not
taking that into account
uh i think um
the way i would uh state this using uh
uh quantifiers is the for all quantifier
saying the uh that um
if you realize you ask you have a list
of 43 different lethalities then any uh
project need to
like pass all these bars uh of course
it's not literally 43 bars
whereas an ai only needs to find a
single way to take over the world um
that's one way to look at it like that
it's actually not an existential
qualifier um
for for the alignment uh proposals uh to
pass but it has to
pass through all the uh the lethalities
point fourteen how much does elijkowski
know about this the state of the art of
interpretability
um and paul crocin's point is he doesn't
know very much so obviously
he can't really be that confident in his
prediction that interoperability is not
going to be successful
um
i
would reserve judgment on this case um i
don't know enough about the state of the
art of modern interoperability
so i can't really tell that one thing i
will say however is that it's not
a priori impossible to have a strong
opinion about the feasibility of a
research field even if you don't know
almost all the details
an example would be astrology
where i'm i have no clue about
what astrology what astrologers do in
practice but i'm quite certain that it's
all bunk
that's of course not to say that
interpretability is the same thing it's
obviously not
but it's i think it's possible to have
strong criticism of a field even if
you're totally ignorant of the state of
the art
idiozytkowski has this quote there's no
known case where you can entrain a safe
level of ability on a safe environment
where he can cheaply do millions of run
and deploy that capability to save the
world
so three requirements for melee
associate a safe level of ability a safe
environment and must be cheap in order
to get something that's capable of doing
a pivotal act
according to paul cristiano this is not
engaging with the kind of system that is
likely to be built
i have an answer to this that is kind of
snarky and maybe not really in good
faith and the reason for this is that in
fact the the systems that are likely to
be built are going to be
systems that like i don't know try to
cure cancer or they try to um
make a hollywood video or something like
that from a transformer or something
like that and
the things they will not be doing
will be to try to do a pivotal act and
they will not be designed for safety so
in that obvious sense paul christiano is
right he's not engaging with the kind of
systems that's likely to be built
because they're going to
optimize for the number of
views on youtube videos or something
like that
paul cristiano talks about early
transformative ai systems doing
impressive projects
um and
has some uh thought about whether that
is possible with like smaller more
myopic ais
that are composed i think the framing of
transformative ai and impressive
technological projects is
uh suspects in the sense that sure if
you have something build something that
can cure cancer that's a
transformative thing to do and it's a
very impressive technological project to
come up with to have
an ai that comes up with a cure for
cancer but it's not a pivotal act and in
that sense it is probably irrelevant
because if you do these uh beautiful
technological things um but you don't
flip the the game board then the next ai
system that comes along will then be
be the one to destroy the world um so in
that way for that reason impressive
things are quite irrelevant the things
that are
truly transformative in the more pivotal
sense is doing things like alignment
research
so can you in fact do alignment research
by uh being trained on small smaller
tasks with short feedback loops and then
composing that
um that's a good question
my intuition would be that that is not
something you can get without
having a very general ai that is capable
of generalizing very far
but it is an open question whether
alignment can be thus decomposed
i i don't think
it's obvious from elias you'd casca's
point uh here that
it cannot be done but i do think that uh
if i have to give my probabilities it
would it seems really hard to
have very very narrow and limited and
myopic ai solve the alignment problem
disagreement 16 is about when problems
appear
elias zukowski makes the statement that
many alignment problems will not
naturally appear at earlier passively
safe levels of capability
paul cristiano
answer to this is that
we can we can study the deceptive
alignment and these kind of problems
before superintelligence and that is
true but it's also not really answering
um
eleazarkowski's point which is when do
these things naturally occur and not
whether it's possible to study them
before super intelligence
eliezerkowski is claiming that there
might indeed be a lot of such problems
like in alignment and what have you and
all of them need to be theorized
recreated and mitigated in advance
percussiano's
answer probably to this is that
if we fail to solve the problem of
deceptive alignment then we won't notice
other problems such as in alignment but
that doesn't really affect the
probability of solving the problem um
because we have to solve them
sequentially um and i think uh
uh paul christiano and elias yokowski
are talking past each other in the sense
that paul christiano is talking about
first we solve this problem and then we
solve this problem and then we can see
this problem after we have
a an ai that's strong enough to do um
deception whereas
kowski is thinking about doing this in
advance
before we get to unsafe levels of
capability
uh
disagreement 17 is more about engaging
with uh alignment work even though we
did have that in i believe disagreement
13. and
the
the statement here again is that elias
is not really engaging with the most
serious alignment work and most serious
is of course a
a difficult
term
so
is there in fact any any alignment work
that
explicitly considers all 43 lethalities
as far as i can see
there is not every single alignment
problem
uh
solution to the alignment problem that i
have seen seems to disregard some of
these lethalities
so um
in in fact there is a a a meaningful
engagement verb that is very possible
between the list of lethalities and
just about every single proposal
and also i must say that i
dislike um the way paul chris channel is
talking about the most serious alignment
work when um elias kowski in his rants
starts with saying that everybody uh in
alignment have their own thing that they
consider like the key thing and the most
serious thing and um
uh and nobody agrees what this is and
it's impossible for elias kowski to like
answer all of them and in this case when
paul is just saying uh
we should talk about the most serious
things but not really cashing that out
to uh like a concrete uh statement like
saying something like uh he's not
engaging meaningfully with uh enlisting
latent knowledge um that it's just um
to some extent answering uh like elias
yutkowski has already in his post said
why
you need to uh to cast this out and say
explicitly this is the problem you can't
just assume that everybody knows what is
the most serious alignment work because
there is in fact
uh
a lot of uncertainty about this
so why do you want to engage it's
important to engage with the the best
work if you want to
one assess the probability of doom
do you actually need that
i think in order to
assess the probability of doom
it's it requires a lot of data and
requires a lot of input and it requires
to some extent both some
a clear understanding of the problem and
why it's legal
and an understanding of the alignment
work so i don't think you can assess the
probability of doom
without both of these things
the second thing is that it's important
to engage meaningfully with this most
serious alignment work if you want to
contribute meaningfully to solving the
problem
i think that in fact a list of
lethalities is a very substantial
contribution in the sense that there is
a number of gates you need to pass in
order to
have some alignment work that doesn't
fail for uh like reason number 27 or
something like that there's a number of
people in the comments uh to the post
that disagrees with this
one way i guess you could think about
this is um consider the classic problem
of whether p equals np
um
one of the reasons why we strongly
believe that p does not equal in p is
that we have
managed to come up with a real good list
of lethalities
for any algorithm that prepares to have
p equals np
like there are a lot of gates such an
algorithm would need to jump through and
these gates look really really difficult
and for that reason we are pretty
convinced that p does in fact not equal
np
i don't know if that analogy made sense
and finally you have to engage
meaningfully with the most serious
alignment work if you want to be able to
complain about other people not making a
similar list of lethalities
i think that's just plainly a
non-secondary as far as i can see that
doesn't make sense at all you can
totally complain about other people not
producing similar lists without engaging
meaningfully with the most serious
alignment work
disagreement number 18 is about
evolution
and human evolution and whether natural
selection is a good analogy for
training machine learning models
paul cruciano says it's a weak analogy
and in order to
go into details here it would be really
nice to have some kind of pointer to
which
way like idiosyncratic uses this analogy
in a number of places and which of those
are
procrastinator referring to
so i looked through there were like
three main uh places where earlier
zuckowski uses human evolution as an
analogy for
something like
ml training
the one that is most central as far as i
can say is this agreement is
lethality number 15
which is that the history of
humans
was one where we had a
relatively stable level of capability
and then a sudden spur uh
sprint of uh burst of uh capability
growth and three things happened at
roughly the same time we got
self-reflection we got contraception
and we started having being primarily
programmed by cultural rather than
genetic factors and all these three
things to some extent broke the inner
alignment that we previously had with
evolution where we were just trying to
spread out genes as most and get as many
ancestors as possible
descendants
and
the the reason paul christiana says this
analogy does not hold is because we can
deliberately shape
ml training
and he uses the example as of animal
breeding as an a better analogy
it seems to me that paul christiano is
really not answering uh eliza caskey
here animal breeding seems like a bad
example in the sense that
the animals like if you have something
like the the evolution of dogs or
something like that dogs are not um
getting a a grow
a sprint of capability growth and dogs
don't have self-reflection they don't
have contraception they don't have any
particular noteworthy culture
so there is no break with uh inner
alignment no inner misalignment
happening in animal breathing at all as
far as feisty can tell
in this
lethality uh number 15 elias kowski has
this particular quote that i'm going to
read in full
my model of this variety of reader has
an inside view which they will label an
outside view that assigns great
relevance to some other data points that
are not observed cases of an outer
optimization loop producing an inner
general intelligence
so when i read this this seemed really
prescient in the sense that um
this is basically uh elias kowski
answering uh paul christiano's objection
in advance
um like this is not uh
answering afterwards it is a text that
um pokes gianno had access to when he
gave the example of animal breeding and
i think in fact um
paul cristiano probably has an inside
view and he's certainly labeling this as
an outside view analogy with animal
breeding and it is
not an observed case of an auto
optimization loop producing an inner
general intelligence
um
so in in that case um in this sense it
seems like elias rutkowski answered paul
corsano's objection in advance
so that was my first thought my second
thought was actually
i've previously complained that a lot of
the
theoretical foundation on
inner alignment rests on like a single
example of from human evolution and we
don't really have other examples so in
this uh
in this case um like if we don't have
more than one uh observed example of in
a misalignment um
then
um it's a simple easy prediction for
eleazar to make that people who will
admit to this with analogies will have
poor analogies like if there are no
other analogies
i
when i look back on this i am somewhat
in doubt because
there are a number of other places where
elias yikowski makes reference to human
evolution
in other lethalities
and it could be that when paul
christiano is saying that
human evolution is a bad
bad analogy
he could be referring to those i'll
briefly go through them one of them is
about whether capabilities generalize um
and how much they generalize and it's
possible that
we can
shape
ml train to some extent so alignment
generalizes more than capabilities uh
i think
pro christiano uh he the example he
gives is um to consider breeding animals
for friendliness and courage ability to
what extent that will work
i think just breathing them for for
friendliness and not also
for capabilities
that's a strange example to pull out
here
because obviously paul if paul cristiano
here is referring to this statement by
eliza ykowski then um here just bringing
them from one thing obviously um is not
very relevant to here where we are
talking about both getting more
capabilities and getting more alignments
getting better alignment better
credibility
so i was thinking whether it would be
possible to
change uh paul cristiano's example here
to not just breed for friendliness but
also for intelligence well you could do
both of that um
could you for instance uh like
uh if you get got something that was as
smart as a human could you get something
that was uh that's strongly identified
with like an outer optimization really
cared more about this fitness
um i
am most confused about whether this is
possible it seems to me that
the the genes that make up humans
basically seem to be selected 100 for
fitness and zero percent for other
things like courageability and with uh
but i'm
i'm fundamentally on unsure about what
this line of argument refers to and for
that reason i don't think this is the
thing that pokes genre is referring to
there's also a third place where elias
yutkowski
talks about
ml training as an analogy for uh human
evolution or human evolution as an
analogy for ml training i don't think
that makes
that third place makes a lot of sense
poker general has an interesting
analogy about breeding humans for
credibility if humans were bred for
corrugability actively
quite likely he expects we would be
courageable up through the current
distribution of human behavior
that's an interesting case uh
i one of the things i wasn't doubt about
this is he explicitly writes the current
distribution of human behavior as some
people may notice human behavior is
not very critical not very friendly
um in fact humans go to war with each
other and uh like uh there are many
reasons why the human distribution does
not uh reflect max
the way we behave does not reflect uh
perfect friendliness
so instead the way i mostly read this is
that instead of the human behavior then
i must think about like the human level
of correlability could you make us
courageable even while making our
intelligence as large as it is now
so if with that breeding process
continuously was run by the smartest of
the current friendly humans that seems
like it would
uh
like
not break down at the current level of
capability and perhaps even um
much far further than that
so um
the way i
imagine corridability is that you are
deferring to someone more stupid who
doesn't really share your values and
that's the way credibility for an ai
that is super intelligent would probably
look
um
so can
humans in fact be bred for something
like that
uh
one example i could come up with was
like uh slavery has been a big thing in
the past where uh people were
people who were slaves and revolted
against the masters generally did not
reproduce um so there have been some
negative selection
did that work my god is not really but i
don't really strongly know um mostly i'm
confused about this analogy
the overall question like our values
like how friendly we are how critical
are we uh is that something we get
mostly from our genes like something
that a breeding program could influence
or is it something that we get mostly
from culture uh my guess is that it's
mostly for culture and the way i would
explain this is
um
by assuming by changing this analogy
here so it's not actually the smartest
of the current uh
friendly humans that are in charge of
this breeding pro program but um
but aliens and so the credibility is not
towards other humans but towards aliens
so if we if somehow humans human
reproduction was interfered by aliens um
and selecting for credibility to the
aliens
and the culture that's the different
thing um that that is something that is
only left to the humans this seems more
like um more relevant to having a uh um
an ai that is a number of um
uh
perhaps a number of agents or a single
agent and there's a level on top that
we're trying to get in alignment and uh
courage ability from the agents towards
the uh the masters to some extent
um would humans be able to uh break free
of the journey of
corruptible genes i believe we would
have done that right now but i am mostly
really confused about this analogy
the final disagreement
sure
um uh pakistan is using uh the two words
credibility and friendliness
in the same um
in the same uh sense
so correlability in this case would be
that the um the ai is um
seeing
itself
on like on the inside in the way that
the humans see it on the outside
something like when we see the ai our
side
we know that the ai
is not um it doesn't understand of true
values it needs to be uh uncertain about
what we truly want needs to defer to us
even if it is really sure that we want
to be turned into paper clips
and
that's the kind of credibility that the
ai needs to internalize
yeah so
yeah um
like is uh to me uh part of the reason
why i like the analogy better when it's
like aliens doing this is to split out
the two parts whether it's like the
cultural thing and the genetic thing
where i feel
the genetic thing doesn't really have a
very strong influence on our value
whereas culture has a huge influence on
our values
okay um the final disagreement is on
whether it is possible to verify pivotal
acts
paul cristiano makes the statement that
that is indeed something that is
probably possible
and part of the
argument you use for this is that
in almost any
research and domain
research and development domain we see
that verification is much much easier
than generation
they much much is
empathized by paul christiano
and i think that is
strongly wrong in the sense that
verification of something that is
uh like you can sure you can build a car
and see whether it can drive but that's
very different from uh trying to
detect something that has been built by
an adversary and unaligned ai trying to
figure out whether that contains an
error or deliberate sabotage somewhere
inside
adversaries make this a lot more
difficult in
software would be an example where um
it's really really hard to see whether
someone has
deliberately put in some kind of
obfuscated um
bug somewhere
my favorite example of this would be the
underhanded c context where which is an
example of uh people trying to write
code that looks really really really
innocent and has some really really
scary and exploitable box uh in them and
that seems uh much much harder to prove
the absence of this kind of um
of attack
um
i think in fact
very very often in almost all domains if
you have an attacker in something like
lock picking or what have you then um
figuring out whether it is impossible to
pick a lock is really really
difficult
and i'm surprised how uh
this point the i think that's there must
be some kind of miscommunication because
this seems like a really basic point
so the example that paul channel is
thinking about here is contributions to
alignment research
so
to what extent can we prove the accents
or convince ourselves of the absence of
adversarial prop
inserted problems in alignment research
can we truly verify alignment research
if it's something that's generated by
someone who is stronger
substantially more cognitively powerful
than us
then my strong expectation would be no
it is not possible if we have an
underlying super intelligence and we ask
that for a solution to the alignment
problem that's going to end really
really poorly
okay but paul cristiano's suggestion is
mostly that we are getting this before
we have have a full super intelligence
um how about if it's something
at the level of a
human can that insert um
errors um
i think um
my expectation would be that it is in
fact possible it's really hard to
evaluate whether alignment research does
in fact solve the problem
i think it's
very easy to come up with uh
like
explicit problems with with research but
if you have if if you get some code from
um from the ai and says run this this
will be an aligned super intelligence
then i would certainly not trust this
uh
when i'm thinking about this kind of
thing i imagine that like in a red team
i'm trying to do um alignment research
as a saboteur and trying to insert
errors or something like that and i
think that seems quite feasible
i could easily imagine ways to do
alignment research that looks benign but
is actually going down a um
a wrong path to some extent
um
paul cristiano has a
suggestion for a way around this this is
in the comments and that is to require a
the contributions to come with
convincing justification
and as we are this is not being done by
a particular powerful ai the way we are
going to do that the only way that would
be feasible would be optimized for
convincing
justifications
and trying to
do that seems to me straightforwardly
really really dangerous
in the comments further paul christiano
is called on this and he gives an
example of how verification is easier
than um than generation
creating a machine learning hardware and
i think that's a clearly um
not an example of where you would have
like this kind of adversarial attacks
even though that's totally possible to
have um and i think it seems to me when
i read the comments that paul christiano
is somewhat touching the question of
what do you do if it's not just
verification for box but verification of
things that are adversarially entered
um but again i am unsure it seems like a
basic point and it's possible of course
that it is just me who have
misunderstood paul christiano at some
point
that is all for today thank you and see
you next week


182. The Offence-Defence Balance of Scientific Knowledge

okay so this paper is called the offense
defense balance of scientific knowledge
does publishing AI research reduce
misuse and the paper is by I see their
strange imperfection here but the
authors are Toby Shevlin and Alan Defoe
and they presented this paper at a
conference held in New York in February
this year okay one moment good these are
the authors and I am I apologize that
I've got some alignments wrong these
slides I cannot account for that they
were too perfect just now but anyway the
first author is is chef Lane at the top
there and he is that both the authors
are at the future of humanity Institute
units word and I think that ship lanes
specialization is in law he's the guy at
the top and the other fella
Alan Defoe is the director of the center
for the governance of AI at Shi
now excuse me I'm just going to have to
adjust that you can probably hear that
we're applauding the key workers in the
background we had noise so this paper is
concerned with the balance of offense
and defense the balance between two
possible effects of disclosing technical
technological knowledge either
facilitating misuse of the technology or
contributing to protections against
misuse they say that the balance between
the good and bad results of publishing
will vary across scientific fields they
say that the existing conversation
within AI specifically has imported
concepts and conclusions from prior
debates within computer security
specifically with the disclosure of
software vulnerabilities and in that
field
they say that publication of information
generally favors defense against misuse
and
that it's not necessarily appropriate to
carry ideas from software security into
AI research and that the AI research
community should consider concepts and
policies from a broad set of adjacent
fields and they say that the field of AI
is in the midst of a discussion about
this topic nowadays and that everybody
is concerned about a range of potential
misuses including I know this familiar
familiar a list of possible abuses using
facial recognition for targeting
vulnerable populations using synthetic
language and video to impersonate human
beings and to impersonate them
deceptively using algorithmic
decision-making which can amplify biases
and unfairness using AI in drones to
well to disrupt air traffic to launch
attacks so those are familiar then they
talk about the example of Tex generation
and next generation is an area of AI and
they say that the discussion of this has
been influenced by the discussion of
software vulnerabilities in general now
recently open AI released there
gbt to model of Tex generation caused
great stir and they adopted a policy of
staged release because of concerns about
its potential misuse to mass-produce
plausibly human written text
this staged release has long been
practiced in computer security if
somebody spots the floor in a piece of
software whether it's the producer of
the software or whether it's some
outside user this is generally released
to the public but after some delay when
it's first made available to the
producer so the producer can fix it and
then the public at large gets to know
about it
so returning to open I and they release
release of GPT two people criticized
open I I using such arguments as that
malicious actors would find it easy to
build their own model anyway so that
there was no point in delaying the
release of the curve that some knowledge
of the possibilities for attack is
useful for preparing defenses so that
there should have been full disclosure
anyway and that's quote security through
obscurity unquote is ineffective and I
take it that in this context security
through obscurity just means
um well if you don't talk about if you
just keep your software to yourself you
keep the code to yourself then nobody
else will want to copy it and release it
they say that that wouldn't make much
sense
another piece of software I came up
quite recently is called Grover and that
was designed to generate disinformation
or fake news this was published
alongside code for building the model
and but and constructing or
reconstructing its dataset they don't
have mean presumably the dataset had
been trained on and the researchers
would actually make further information
available to anybody who chose to
contact one and finally the code and
dataset or open sourced
intention behind doing this of course
was to increase our understanding of AI
generation of think news so that we
could build tools that could guard
against the harm it could do now talking
about offense defense amongst
technologies generally they say that the
field international relations has an
existing body of literature on such a
balance
they actually cite very little but
excuse me but one example they cite is
by Shapiro and Segal and they analyze
when states should publish information
even though it might be useful to
terrorists for example a government
might wonder whether to publish
witnesses in commercial nuclear power
plants and those authors Shapiro and C
will find that it's safer to disclose
such information if you think that
attackers already hold such knowledge
anyway so they don't make not making a
difference and it's also safer to
disclose it if the government that's
disclosing exits always got Munson in
the case of nuclear technology if they
can
use this openness to find and fix
weaknesses so I mean there's a to
intuitively obvious considerations about
when you it might be wise to release
such information so our authors chirlane
and Defoe want to produce a theoretical
framework that covers technologies in
general and I suppose other areas of
expertise but some technologies in
general and they're going to apply it
way and I in specifically a bit later
and they say
the net effect of disclosure on misuse
depends on which of the two effects of
is stronger the effect of aiding actress
wants to cause harm and aiding actors
who want to protect others from harm and
the balance between those two will be
different for different types of
disclosure and just before launching
into it they say that their analysis is
limited in scope only deals with certain
consequences of disclosure and they're
only talking about harms that are a
direct intentional use of the technology
they're not talking about cases where
technology is used in cautiously or
contains a none safety thing so then I'm
talking about technological accidents
they're not talking about structural
risks from technological proliferation
such as military and stability or
unemployment all those important topics
but they are only talking about these
deliberate intentional malicious users
and their framework does not weigh the
benefits of polishing scientific results
other than how they relate to security
they have a bit more to say about this
at the end however so this balance and
I'm sorry it you don't look as if you
can't see the very last line here but
what that last line is saying is
therefore our framework should not be
the basis for an assessment of
publication norms but only one input to
such an assessment in other words there
is more to a question of polishing than
just questions of risk and security
um of course if you think that with
which with my existential risks you
might not agree with that you might
think that security is be-all and
end-all but they I don't they don't use
the phrase existential risk at all in
their paper okay some fact now you will
what follows begins by being extremely
abstract there this is an abstract
analysis of benefits and harms and
different types of cause and effect but
there are a few concrete examples to
lighten things up as we go on so don't
okay
factors where disclosure affects the
attackers a capacity to cause harm the
first is called counterfactual
possession that is a jargon phrase that
I thoroughly disliked it just means
possession of some piece of knowledge
that's achieved
independently of being disclosed in
other words you think that some bad
person out there is going to get this
knowledge anyway so you might as well
publish so as they say the more likely
it is that this will be achieved the
less impact the publication won't have
another factor that affects the
attackers capacity to cause harm is
whether potential attacker said well is
the fact that potential attackers have
three main avenues to achieve so-called
counterfactual possession
in other words independent possession
independent discovery or sharing
information amongst themselves and
so-called counterfactual publication
that is someone else will publish soon
so there's no point in holding back and
the authors say that we believe these
considerations should be excluded from
the decision if you're thinking about
whether to disclose a piece of knowledge
in a I'm set
you should discount these factors above
instead of considering the impact of an
individual decision to publish the
researchers should ask what decision
would I want rolled out across the whole
field in other words they should do what
they would be happy for everybody to do
in similar situations this is very candy
and if you know your Immanuel Kant
more factors are affecting an attackers
capacity and that is the ability of the
attackers to absorb and apply the
research and of course this has got a
person considering whether to disclose
some research it's got to make their own
judgments as to what they think
potential attackers be able to do with
it
the attackers attentiveness in
convention you know is that it is
anybody paying attention nice I assume
that in the end nothing nothing goes
unnoticed so we've got to assume that
everything will be in will be noticed
but anyway they say will the research be
read and understood by potential
attackers this wall so depend on the
disclosure itself how much knowledge is
disclosed through what channel how has
it presented and then answer the
question of it is what you presenting
sufficient to be used for harm does it
contain all that is needed to carry the
behavior at the other end of the
spectrum adopting the research will
involve
a large investment in resources and
complimentary knowledge in other ways
this is a case where it would need a
huge effort
like for example knowledge to building a
nuclear weapon might involve a large
investment in resources and
complementary knowledge and so you might
think that it's safe to publish if you
think that that is out of somebody's
reach know there's a line missing at the
bottom so I'll just look through and see
that might be
oh yes the last factor which I'm sorry
you can't see there my poor skills in
making slides the last factor is
transferability the ability of knowledge
that promotes good ends to be
transferred to bad ends
moving on there are factors affecting
the defenders ability defenders means
yeah any anybody who seeking to combat
some weakness in technology or some
misuse of technology so it could be
somebody inside the organization
producing the AI material or it could be
outsiders users hackers spies
and disclosure could aid defenders by
disseminating ideas that are useful for
finding solutions Orca simply sound an
alarm or see some examples of that in a
moment
success depends on a number of factors
once again counterfactual position comes
up with once again with the defenders
independently to discover or otherwise
obtained the knowledge how easily could
the defenders have got to that insight
themselves with the defenders have
already been aware of the problem you
don't want to announce a problem which
nobody else knows about as a sera me at
once again the ability to absorb and
apply the information as with the case
of bad actors and then if you give the
disclosure make the disclosure how many
additional individuals or organizations
will work on finding a solution you
think you can mobilize a lot of people
that's an ikemen in favor of disclosing
the positive effects of disclosure
depend on the potential for a good
defense against the misuse is the
weakness agent of the fundamental
southern system
what could a relatively superficial
change removing it
is the attack that should really say is
an attack detective law and its
detection sufficient to defend against
it or to deter it is detector powerful
but it will overwhelm any defenses so
even where a solution to some problem
with your disclosure exists it might be
difficult or costly to propagate that
solution that is a factor you have to
take into account
okay so we're still continuing so at
this very high abstract level we'll get
more concrete I promise you what sorry
so they just talked about the offense
defense balance for misuse risks that
have a higher potential harm
for example of ulnar ability in facebook
could be very harmful whereas well
virgin a much smaller website would have
less consequence so in the case of the
higher potential harmless security
consequences of disclosure will be
amplified and AI researchers confuse to
publish more or less of their basic
results and their insights or their
detailed results their code their data
says the train networks they can choose
to publish will not publish tools
easy-to-use tools that will assist
people outside these are things within
their control different outputs will
differentially benefit different actors
so a publication without practical tools
or code will be more difficult for low
capability attackers and defenders to
apply
researchers could could attempt to play
safe to give their release a defensive
bias in other words being cautious like
eventually publishing defensive tools
and best practice as opposed to no
offensive controls and I suppose less
than best practice and they can also
possibly the most things is that they
can attempt to circulate certain oral
tools exclusively among the scientific
community in other words trying to have
a privileged circle with wind with which
amongst young you circulate certain
knowledge interesting question as to how
long that would last or maybe it will
just give you enough time to develop
some sort of defense against what other
people would
there now contrast different fields are
interested in how a I might different
from mother might differ from other
fields then they say that that their
framework helps to explain why the
disclosure of software vulnerabilities
will often be beneficial for computer
security in other words why disclosure
has become the norm with software
vulnerabilities one factor is that
patches to software are often easy to
create and nothing can be made in a
matter of weeks so if so fixes are easy
to mange generally these patches are
fully resolved the vulnerability they
can be completely efficient the patch
can be easily propagated and independent
discovery of the vulnerability is likely
so if that's likely then you might as
well disclose these factors combined to
make their reasonable arguments in favor
of public disclosure of software
vulnerabilities at least after the
vendor has me given time to prepare a
patch contrast this with other fields
for example biological research into
pathogens if you create a new virus and
if you release information about its
general same reason samples it's
difficult to find vaccines against new
viruses or treatments against their
effects it's it's laborious and success
is not guaranteed
very contemporary so this lowers the
defensive benefit of publication it
weakens this argument the public that's
you know that knowledge is good public
knowledge is good this contrasts with
the case where an effective treatment
can be developed within a reasonable
time period which could weigh in favor
of publication
they give a couple of examples of
vulnerabilities involving hardware they
mentioned drones drones are now doing
very widespread they used a lot they're
sometimes used maliciously they have
been used in attacks in the Middle East
and I don't know of any physical attacks
you know that's an actual war locations
but anyway they they presently lack a
cheap effective solution according to
the authors obviously you can shoot them
down but they're very cheap easy to
reduce in large numbers and you can't
guarantee almost to hit them before they
hit their target they also give another
analogy which they is sort of them using
and interesting in the hardware field it
seems a lot of hotels apartment
buildings offices and so on still use
physical key systems keys individual
keys for individual rooms and a master
key no planning room I thought everybody
was using electronic swipe cards
nowadays but apparently not so already
stop back in 2003 maybe in 2003 some
some
ingenious fella published a system in
which made it easy for someone to create
a master key from a single example of
one non master key one room key say he
showed how it was possible to make a
master key from there and he was kind
enough to publish for details everywhere
and locksmiths and people who ran large
buildings were not used to this at all
because is somebody
explanted of this information then
really only you can do is replace your
whole key system with a with a non
master key system so your very expensive
I don't know if there's been any
progress since hand on my combating
matter where they just had to give way
to it whether everybody's just switch to
swipe cards another example of hardware
vulnerabilities is the question of
whether engineering research nuclear
engineering research such as uranium
enrichment which is only just one one
part of the whole process of making
bombs whether that should have been
published I don't know if anybody was in
fact arguing for publishing it but the
reasons that militate against publishing
it simply that you increase the ability
of a an opponent to make bombs and
destroy your own city's nuclear bombs
are a technology against which there's
no effective defense and that I mean the
best-known defense is deterrence
those technologies of Defense that do
exist would benefit very little from
knowing about one piece of the offensive
technology such as uranium enrichment
for example in other words a blueprint
for the design of the centrifuge does
not have one build a better defensive
bunker so their points here I think is
they're saying this is another case
where publishing some some information
about how to build Center uses
centrifuges that's not going to help you
build up your own defenses in any way
you're not going to get a bunch of
hackers coming back and saying okay now
this is how this is how you can improve
your defense in my only better defensive
bump is this recession no connection so
this is so that is a consideration
against publishing
and say they say that if both the
potential defender and a potential
attacker are given knowledge that helps
them build nuclear weapons that
knowledge is more useful for making an
attack than for protecting against an
attack the knowledge is but jargon is
offense biased as just to introduce
their offense biased although of course
there is still the question that you
okay so it helps you build it helps the
defender build but the talent which is
one form of defense okay excuse me
right
disclosure ones what they're working
towards is you know sort of an ethics or
codes of practice norms for disclosing
or not disclosing in pieces or
information and they are saying there
are loads of diversions that vary
between different fields I'll hurry
along a little bit here but they
mentioned obviously the Manhattan
Project or is very secretive more so
than the locksmiths perhaps with some
more secretive nowadays more secretive
than influenza researchers because
they've learned senator sessions tend to
to share their their knowledge because
currently I guess they're not too
concerned about people using their
knowledge for biowarfare maybe maybe
maybe that will change in the near
future anyway these classes of
researcher are more secretive than those
who find abilities in software they said
there was a culture clash between the
researcher who published the floor in
the master key system and there was the
locksmith so huesemann of being
irresponsible the different disclosure
cultures exist in the form of default
practices they're different places in
different areas but also in what they
call common refrains by which they mean
standard phrases or cliches or tropes
for example language about the virtues
of studying a problem or the value of
users being empowered by disclosures to
make decisions for themselves they're
saying that this sort of language comes
a little too easily and we really need
to think about it in a specific context
such language embeds implicit answers to
the questions we raised caution should
be exercised when importing concepts and
language from other fields
okay now they come to discuss AI
specifically to the extent that
protections against AI misuse required
interventions in social systems then
publishing AI research will have a more
limited defensive benefit oh come on
they they have more same of this AI is
especially prone to interfering in
social systems because AI involves
automating activity that normally
requires human intelligence the very
definition of AI for example or
criminals used artificial speech
generation to make the voice of the CEO
of a company over the telephone
this attack exploits the fact that the
sound of someone's voice is strong
evidence of their identity thus the so
called vulnerability here is our
practice of relying on people's voices
as evidence of their identity this is
socially useful it's deeply ingrained
practice we we'd hate to not be able to
go on doing that excuse me
they say it some have responded by
suggesting that the research community
should simply quote warn Society unquote
but individuals may be increasingly
shown and worth untrustworthy content
but the language of quote let the users
decide for themselves
unquote which is reminiscent of computer
security discourse I'll take the word
for it parameters would lose its
empowering sentiment if users become
landed with problems which no good
solution exists in other words they're
working towards their conclusion but
really the punters are just not in a
position to decide for themselves in the
field of a high
they say our key suggestion is that AI
vulnerabilities will be on average
harder to patch that software
vulnerabilities are and so our high
researchers cannot assume that AR
publication would always have a
defensive bias in other words will not
always be for the good now they say a
common response is that an AI model that
is useful for an attack could similarly
be useful for detecting these attacks
you recall this was the reason why
Grover was published their services to
help us learn how to defend ourselves
against fake news generated by a I
however are our authors so Menand athos
a but offensive AI systems can often be
trained against detection systems so as
to evade them thus when items generated
by Grover were pre filtered by the
detection system the remaining items
were harder to detect as you can imagine
that seems extremely plausible so I
gather that they generated a whole bunch
of same fake news articles they threw
them against a detection system
presumably also automated couldn't
equally be human beings I suppose some
some were filtered out and that system
but what was left was harder to detect
that's not very hard to doubt is it ok
so this is one reason for not making the
detection system freely available
yeah so continuing to apply that
framework to AI secondly even in cases
where sorry this is following
immediately from the previous points
you remember people were saying that
detection systems in AI can be useful
for defending us against malicious AI
because obviously not said brokers and
therefore the disclosure of detection
systems is a good thing they are saying
they're continuing by saying another
point against this is that even where a
detection system can in theory detect
the area's activity it may not be
feasible to deploy it for example in the
particular case of detecting AI
generated voices it might require a lot
of surveillance of calls a large part of
computation and a lot of false positives
being thrown up in fragging ordinate
conversations okay
now there's a discussion of the
independent acquisition of AI knowledge
and they apply it down to the middle of
the page they apply it to the example of
text generation and they say but where
the risk comes from a big actor a
state-led disinformation campaign
obviously they're thinking of the
alleged Russian think news campaigns in
the 2016 US election presidential
election which at least half of America
is firmly convinced was responsible for
the outcome of the election when the
risks come from a state-led
disinformation campaign one uncertainty
is how much these state actors would
benefit from the research being
disclosed because if not much because
they're a big and powerful research
organization themselves then the risks
of disclosure are less in other words
the the costs are less anyway
nevertheless this doesn't mean that you
can cheerfully disclose secrets of say
thank news detection even on the grounds
that you know tsunami is
it's so well equipped but you're not
helping much
we have to consider that there are other
actors with less access to a I talent
and AI compute and there may also
perhaps be very few actors outside the
AI research community Joe suppose means
the respectable Western community in
this case there may be people outside
capable of having the original insights
continuing the paper
not so many geniuses out there
and so forth in some cases we large
technology companies need to prepare
defenses okay III won't go on it to
unravel that thought I've got I want to
push on today conclusion the conclusion
is that our analysis should aid AI
researchers in thinking critically about
implications of publishing work that
could be miss EULA's the community
should grow is tool lots of analogies
and concepts so that disclosure norms
can be well-designed I think that's what
you could say about this paper that it
is talking about analogies and concepts
um these are worse like empty boxes into
which you can drop specific concrete
considerations when you're thinking
about a different problem and so they're
they're almost talked about the language
of this discussion now one challenge
they say we'll be building disclosure
policies in accordance with the
legitimate and effective norm setting
process so that's what we're trying to
work towards and sort of and I think I
suppose you could say of disclosure
and not just then I think but also a
bunch of agreed agreed
policies and principles in connections
with disclosure they also point out but
the security impact of disclosure is
input although it's important it should
only be considered on site a host of
other considerations in other words
everything we've been talking about in
this paper I still only one aspect what
you should be thinking about when
deciding when to disclose sorry that
thought at the top is a slightly
floating thought and a publication must
be able to scale and really adapt a more
powerful and capabilities with what the
thing is that we must bear in mind well
any policy or principle relating to a
publication has got to be able to adapt
to future more powerful capabilities
along we mustn't mean to limited by what
we can do today
and then go on to say our framework
should only be one input to an
assessment of publication norms because
there are potential quotes non-security
benefits
scientific publication using include you
know overall the normal benefits of open
publication contributions to economic
growth and the quality of life
the advancement of science and its broad
societal benefits better monitoring of
scientific investments in other words
accountability of investment it leads to
internationalism and cosmopolitanism
next rule minded it leads to greater
civilian control and involvement in
science
and also there are other tools for
tackling harmful AI not just regulating
disclosure the research community can
differentially invest in those projects
and tragic trails that I'm a socially
beneficial researchers can invest extra
effort in understanding and mitigating
the potential harmful uses of their
research in a place that being
responsible researchers from the
beginning before they is closed there is
not and they can their efforts can
include the crafting of norms and
policies to steer the use of AI and so
that said and that is the end of their
framework well thank you very much Chris
for your presentation


What's the Use of Utility Functions?

okay so in some of the earlier computer
file videos I talked about utility
functions or objective functions and we
got a lot of different comments relating
to that idea one thing people said was
well surely this kind of monomaniacal
following of a single utility function
at the cost of everything else is really
the cause of the problem in the first
place why even use a utility function or
maybe have several conflicting ones that
interact with each other or something
like that some people asked why do we
assume that an AI will have a utility
function in the first place aren't we
making a pretty strong assumption about
the design of the AI when in fact we
don't know how it would be implemented
humans don't have explicit utility
functions that they consult when they're
making their decisions a lot of
different AI designs people are working
on now don't have utility functions
coded into them explicitly so why make
that kind of unwarranted assumption so
before we get into that let's just go
over what a utility function is okay so
here's the earth or the universe it can
be in any one of several different
states so let's just look at three
possible world states in this world I'm
enjoying a pleasant cup of tea in this
world I've run out of milk so the tea
isn't quite how I'd like it to be and in
this world I'm being stung by noon two
wasps we want some way of expressing
that I have preferences over these world
states some of them are better for me
than others so a utility function is a
function which takes as an argument a
world state and outputs a number saying
broadly speaking how good that world is
for me how much utility I get from it so
in this example perhaps a nice cup of
tea is worth 10 a a mediocre cup of tea
is worth nine and the wasps are minus a
thousand but Rob you might say that's
way too simple I care about all kinds of
things and what I what I love about the
world is is complex and nuanced you
currently steal everything down to just
a single number on each world state note
with that attitude you can and you kind
of have to but let's just forget about
the numbers for now and talk about
preferences
let's make some basic assumptions about
your preferences the first one is that
you do have preferences given any two
states of the world you could decide
which one you would prefer to happen or
you could be indifferent but there's
this basic trilemma here for any pair of
world states a and B either a is
preferable to B or B is preferable to a
or you're indifferent between a and B it
doesn't matter to you which one happens
always exactly one of these things is
true hopefully that should be obvious
but just think about what it would mean
for it not to be true like what would it
mean to not prefer A to B not prefer B
to a and also not be indifferent between
DNA similarly what would it mean to
prefer A to B and simultaneously prefer
B to a if you're faced with a choice
then between a and B what do you do the
second basic assumption is transitivity
so you have this relation between States
is preferable to and you assume that
this is transitive which just means that
if you prefer A to B and you prefer B to
C then you prefer a to C again this
seems intuitively pretty obvious but
let's look at what it would mean to have
intransitive preferences let's say I
prefer being an Amsterdam to being in
Beijing and I prefer being in Beijing to
being in Cairo and I prefer being in
Cairo to being in Amsterdam what happens
if I have these preferences let's say I
start out in Amsterdam I prefer being in
Cairo so I get on a plane and I flied to
Qatar now I'm in Cairo and I find
actually I prefer being in Beijing so I
get on the plane I fly to Beijing I'm
now in Beijing and I say oh you know
actually I prefer to be in Amsterdam so
I slide around stir done and now I'm
back where I started and hey what do you
know I prefer to be in Cairo so you can
see that if your preferences are
transitive you can get sort of stuck in
a loop where you just expend all of your
resources flying between cities or in
some other way changing between options
and this doesn't seem very smart so if
we accept these two pretty basic
assumptions about your preferences then
we can say that your preferences are
coherent you may have noticed there
something else that has these two
properties which is the greater than
relation on numbers for any two numbers
a and B either a is greater than B B is
greater than a or a and B are equal and
if a is greater than B and B is greater
than C then a is greater than C the fact
that preferences and numbers share these
properties is relevant here so if your
preferences are coherent they'll define
an order overworld States that is to say
given your preferences you could take
every possible world state and arrange
them in order of how good they offer you
there will be a single ordering
overworld States you know there aren't
any loops because your preference is a
transitive now if you have an ordering
of world States there will exist a set
of numbers for each world state they
correspond to that ordering perhaps you
could just take them all in order and
give each one a number according to
where it falls in the ordering so those
are your utility values for any coherent
preferences there will be a set of
utility values that exactly represents
it and if you have a utility value on
every world state well there will be
some function which takes in world
States and return to their utility
values and that's your utility function
so if you have consistent preferences
you have a utility function but Rob you
may say I don't have consistent
preferences I'm a human being my
preferences are all over the place
that's true human beings do not reliably
behave as though they have consistent
preferences but that's just because
human intelligence is kind of badly
implemented our inconsistencies don't
make us better people it's not some
magic key to our humanity or secret to
our effectiveness or whatever it's not
making us smarter or more empathetic or
more ethical it's just making us make
bad decisions talking about utility
functions is actually a way of assuming
very little about the design of an AI
other than assuming that it has coherent
goal directed behavior it doesn't matter
how its implemented if it's effective at
navigating the world to get what it once
it will behave as though it has a
particular utility function and this
means if you're going to build an agent
with coherent goal directed behavior
you'd better make sure it has the right
utility function
[Music]
just wanted to say thank you to my
patreon supporters the three people who
somehow managed to support me before I
even mentioned in the video that I was
setting up a patreon and I especially
want to thank Chad Jones who's pledged
$10 a month thank you so much it really
means a lot to me that there are people
out there who think what I'm doing is
worth supporting
so thanks again