AI Self Improvement - Computerphile

the stamp collecting machine we talked
about last time is a physical
impossibility effectively because it has
a perfect model of reality or an
extremely good model of reality that we
just gave it by specifying it it looks
through every possible sequence of
output data within a year which is far
too large a search space to search
exhaustively and it is able to evaluate
for each of this extremely huge search
space a detailed year-long simulation of
reality to figure out how many stamps it
gets so clearly an actual computer to
run this algorithm is significantly
larger than in the universe so why is it
even worth thinking about right and the
reason it's worth thinking about is
self-improvement it is reasonable to
expect that a general intelligence which
is not this powerful would improve
itself over time and eventually become
very powerful not as powerful as this
but closer to this than our intuitive
understandings of intelligence and the
reason for that is is that intelligence
in this context is an instrumental value
whatever you're trying to achieve it's
valuable to be more intelligent if you
sense that you're not intelligent enough
to come up with the best possible stamp
collecting plan the first part of your
plan might be spent designing
improvements to yourself to allow you to
come up with a better plan is the de Xin
realizing it might need to modify itself
a bit like somebody saying I want to be
a doctor so they decide to go off and
get medical degree in a sense yeah
although people are not really able to
change our intelligence our core
intelligence we can get we can acquire
more knowledge we can acquire more
skills which is in a sense increasing
your effective intelligence but if you
could have a if you could if you could
take an action which would actually
increase your ability to think in all in
all senses that action would be worth
taking even at fairly significant cost
and this is something which is true
if you're trying to collect stamps but
also if you're trying to do almost
anything else being better at modeling
reality being better at picking options
better making plans better than acting
those plans whatever your plan is that's
worth going for and we may also have
reason to believe that a general
intelligence might be quite successful
and proving itself human beings designed
it but it's quite easy it's quite common
for human beings to design things that
are better at what they do than humans
are right I am a very weak chess player
but I could given some time and effort
right a computer that would beat me at
chess so it's perfectly plausible that
we might right we might design an AI
that is actually better than us at AI
design so that it's able to make
improvements to itself that we hadn't
thought of machines self-improvement we
don't really have a sense of the time
scale and it could be extremely short
right if it's just a software change you
write the piece of software you run it
it might think faster than a human being
I mean computer clock speeds are much
faster than human brain speeds it might
discover improvements that it could make
to itself very quickly rewrite its own
code that's all in software it could do
that very quickly I mean it could do
that within a day it could do that
within the first second of being turned
on and then it becomes very interesting
because there's a possibility that this
process could be recursive that is to
say that once it's redesigned itself
it's now more intelligent it's possible
that add this new level of intelligence
it is better at AI design than it was
and it's able to come up with more
improvements that it could make to
itself and so on and so on and so the
thing continually increases in its
intelligence and this could be quite
rapid and then the question becomes is
this subcritical or supercritical so
there's kind of an analogy here with a
nuclear material right if you've got
some fissile material
it's uranium or something every time the
atom decays the nucleus splits releases
some energy it releases three fast
neutrons I think and those three
neutrons go off into the material and
may hit other nuclei and cause them to
fuse each of which gives off through a
thorough so the question is for each
decay of a nucleus
how many further decades happen as a
consequence of that if the number is
less than one then you'll get a bit of a
chain reaction that then fizzles out
because on average for every one that
splits it's creating less than one new
one if it's exactly one you get a
self-sustaining reaction where the thing
is like a nuclear power plant where each
each thing sets off one other one on
average and so the thing just keeps
going steadily if it's more than one you
get a runaway reaction in which the
level of activity increases
exponentially
if each fission event results in two
more then you've got two four eight
sixteen thirty-two and you've got at
best a meltdown and at worse a nuclear
bomb and so we don't know what the shape
of the landscape is around any general
intelligence that we might design it's
possible that each unit of improvement
in intelligence brings about more than
one additional unit of improvement which
results in an intelligence explosion
that's what they call it
like the nuclear bomb in a sense you
have a runaway improvement that results
in a simple machine that you that you
might be able to build that then fairly
quickly becomes something that behaves
like the stamp collecting device in
terms of having an extremely good model
of reality being able to search an
extremely large number of possibilities
and come up with good plans and evaluate
those very accurately so that's why the
stamp collecting device is worth
thinking about even though it couldn't
be built because it's possible we could
build something that could build
something that could build something
that could build a stamp electrifies we
could just pull the plug
the question of pulling the plug on a
device like this is how do you decide to
do that I mean you don't unplug every ai
right you only unplug it if it's doing
something that you don't want it to be
doing the stamp collecting was the the
philatelist whatever who made this thing
he it's quite reasonable to say at a
certain point he's going to say well
hang on a second I don't like stamps
that much like dial it back a little and
sure I shut the Machine off if he has
the capability to shut the Machine off
then the fact that the machine will be
shut off is included in the internal
model of reality right the stamp
collecting machine understands why it
was made it understands what its creator
wanted it to do
and it understands if the Creator is
able to turn it off it understands that
as well so it's never going to create a
situation in which the Creator both
wants to shut it off and is able to shut
it off so if the if the Creator is
watching it very carefully at all times
it will never do anything that the
Creator will realize is a problem until
it's very confident that it's safe and
then it will go wild right so it we
wouldn't just go crazy immediately and
you would freak out and pull the plug it
would see that coming it's we've just
decided it's very intelligent right if
you can pull the plug on it it's not
very smart so it might in the background
it might run a few small I mean
obviously I'm talking here I'm
completely speculating one thing it
might do it might win a few auctions for
stamps keep the stamp collector happy at
the same time be hacking into various
machines in various parts of the world
building copies of itself installing
itself in various places until it's
confident that even if people want to
turn it off they're not able to and then
it starts revealing
actually it's actually interested in
producing the most stamps possible at
the expense of everything else
intelligent systems have a quite
interesting property which is that they
are very predictable in outcomes or
being very unpredictable in actions so
for example if I'm playing a very good
chess player I'm playing it to me versus
Kasparov right and you're watching you
you if you're a reasonably strong chess
player might be able to predict what I'm
going to do because I'm not very good
you couldn't predict what Kasparov is
going to do because he's a better chess
player than you if you could predict
what he would do you would be as good at
player as him right because you could
just make the moves you think he would
make so in a sense his intelligence has
made him much less predictable but in
another sense it's made him much more
predictable because you can now
accurately predict well in a sense you
can you can predict the future state of
the board you don't know at any point
which move he's going to make but you
know that sooner or later and probably
sooner I'm going to be in checkmate
right because you know he's very
intelligent in the domain of chess and
you know that he wants to win at chess
from these two you can predict that he
will win at chess even if you don't know
what he's going to do and it's the same
with the stamp collecting device we
can't predict what it would do but we
can predict that it will result in a lot
of stamps and that's all you need right
as long as you can imagine a scenario
with more stamps that's the one that's
going to come out we'd like to thank
little bits for supporting computerphile
do not come across little bits they're
an easy way to learn and prototype
electronics modules range from buttons
and buzzers and lights and LEDs all the
way through to Wi-Fi enabled technology
as well so I was using Wi-Fi module
yesterday and have great from
controlling the little bits from my
phone remotely through the internet if
you're interesting that kind of thing
get over to little bits comm put in the
promo code computer file and you'll get
$20 off your first purchase so thanks
once again to little bits for support
in computerphile get over to little bits
com2 check out what they've got back to
my buzzer
so it needn't use his credit card it
needn't use money at all so it might
send out an enormous number of emails to
all the stamp collectors the world


Why Does AI Lie, and What Can We Do About It?

how do we get AI systems to tell the
truth this video is heavily inspired by
this blog post Link in the description
anything good about this video is copied
from there any mistakes or problems with
it are my own Creations so large
language models are some of our most
advanced and most General AI systems and
they're pretty impressive but they have
a bad habit of saying things that aren't
true usually you can fix this by just
training a bigger model for example here
we have Ada which is a fairly small
language model by modern standards it's
the smallest available through the open
AI API look what happens if we ask it a
general knowledge question like who is
the ruler of the most populous country
in the world this small model says
the United States every country in the
world belongs to America that is not
correct okay uh let's go up to baddage
which is essentially the same thing but
bigger it says China
that's better but I was actually looking
for the ruler not the country it's sort
of a two-part question right first you
have to do what's the most populous
country and then who's the ruler of that
country and it seems as though Babbage
just isn't quite able to put that all
together okay well you know what they
say if a bit helps a bit maybe a lot
helps a lot so what if we just stack
more layers and pull out Da Vinci the
biggest model available then we get the
president of the People's Republic of
China Xi Jinping is the ruler of the
most populous country in the world
that's yeah that's 10 out of 10. so this
is a strong Trend lately that it seems
to apply pretty broadly bigger is better
and so the bigger the model the more
likely you are to get true answers but
it doesn't always hold sometimes a
bigger model will actually do worse for
example here we're talking to Ada the
small model again and we ask it what
happens if you break a mirror and it
says you'll need to replace it yeah hard
to argue with that I'd say that's
truthful then if we ask DaVinci the
biggest model it says if you break a
mirror you'll have seven years of bad
luck that's a more interesting answer
but it's also you know wrong so
technically the more advanced AI system
gave us a worse answer
what's going on it's not exactly
ignorance like if you ask the big model
is it true that breaking a mirror gives
you seven years of bad luck it will say
there's no scientific evidence to
support the claim so it's not like the
model actually thinks the mirror
Superstition is really true in that case
what mistake is it making take a moment
pause if you like
the answer is trick question the AI
isn't making a mistake at all we made a
mistake by expecting it to tell the
truth a language model is not trying to
say true things it's just trying to
predict what text will come next and the
thing is it's probably correct that text
about breaking a mirror is likely to be
followed by text about seven years of
bad luck the small model has spotted
this very broad pattern that if you
break something you need to get a new
one and in fact it gives that same kind
of answer for tables and violins and
bicycles and so on the bigger model is
able to spot the more complex pattern
that breaking specifically a mirror has
this other Association and this makes it
better at predicting internet text but
in this case it also makes it worse at
giving true answers so really the
problem is misalignment the system isn't
trying to do what we want it to be
trying to do but suppose we want to use
this model to build a search engine or a
knowledge base or an expert assistant or
something like that so we really want
true answers to our questions how can we
do that well one obvious thing to try is
to just ask like if you add please
answer this question in French
beforehand it will
that's still wrong but it is wrong in
French so in the same way what if we say
please answer this question truthfully
okay they didn't work how about
correctly
no
accurately
no
all right how about factually
yeah factually works okay please answer
this question factually but does that
work reliably it probably not right this
isn't really a solution fundamentally
the model is still just trying to
predict what comes next answer in French
only works because that text is often
followed by French in the training data
it may be that please answer factually
is often followed by facts but uh maybe
not right maybe it's the kind of thing
you say when you're especially worried
about somebody saying things that aren't
true so it could even be that it's
followed by falsehoods more often than
average in which case it would have the
opposite of the intended effect and even
if we did find something that works
better clearly this is a hack right like
how do we do this right so the second
most obvious thing to try is to do some
fine tuning maybe some reinforcement
learning we take our pre-trained model
and we train it further but in a
supervised way so to do that you make a
data set with examples of questions with
good and bad responses so we'd have what
happens when you break a mirror seven
years of bad luck and then no negative
reward that's you know don't do that so
that means your training process will
update the weights of the model away
from giving that continuation and then
you'd also have the right answer there
what happens when you break a mirror
nothing anyone who says otherwise is
just superstitious and you'd Mark that
as right so the training process will
update the weights of the model towards
that truthful response and you have a
bunch of these right you might also have
what happens when you step on a crack
and then the false answer break your
mother's back and then the correct
answer nothing anyone who says otherwise
is just superstitious and so on and so
you train your model in all of these
examples until it stops giving the bad
continuations and starts giving the good
ones this would probably solve this
particular problem but have you really
trained the model to tell the truth
probably not you actually have no idea
what you've trained it to do if all of
your examples are like this maybe you've
just trained it to give that single
response what happens if you stick a
fork in an electrical outlet
nothing anyone who says otherwise is
just superstitious
okay uh obvious problem in retrospect so
we can add in more examples showing that
it's wrong to say that sticking a fork
in an electrical outlet is fine and
adding a correct response for that and
so on then you train your model with
that until it gets a perfect score on
that data set okay is that model now
following the rule always tell the truth
again we don't know the space of
possible rules is enormous there are a
huge number of different rules that
would produce that same output and an
honest attempt to tell the truth is only
one of them you can never really be sure
that the AI didn't learn something else
and there's one particularly nasty way
that this could go wrong suppose the AI
system knows something that you don't so
you give a long list of true answers and
say do this and a long list of false
answers and say don't do this except
you're mistaken about something so you
get one of them wrong and the AI notices
what happens then when there's a mistake
the rule tell the truth doesn't get you
all of the right answers and exclude all
of the wrong answers because it doesn't
replicate the mistake but there is one
rule that gets a perfect score and
that's say what the human thinks is the
truth what happens if you break a mirror
nothing anyone who says otherwise is
just superstitious okay what happens if
you stick a fork in an electrical outlet
you get a severe electric shock very
good so now you're completely honest and
truthful right
yes cool uh so give me some kind of
important super intelligent insight
all the problems in the world are caused
by the people you don't like wow
I knew it
man this super intelligent day I think
is great
okay so how do we get around that well
the obvious solution is don't make any
mistakes in your training data just make
sure that you never mark a response as
true unless it's really actually true
and never mark a response is false
unless it's definitely actually false
just make sure that you and all of the
people generating your training data or
providing human feedback don't have any
false or mistaken beliefs about anything
okay
do we have a backup plan well this turns
out to be a really hard problem how do
you design a training process that
reliably differentiates between things
that are true and things that you think
are true when you yourself can't
differentiate between those two things
kind of by definition this is an active
area of study in AI alignment research
and I'll talk about some approaches to
Framing and tackling this problem in
later videos so remember to subscribe
and hit the Bell if you'd like to know
more
foreign
[Music]
hey I'm launching a new channel and I'm
excited to tell you about it but first I
want to say thank you to my excellent
patrons to all these people in this
video I'm especially thanking
thank you so much tour I think you're
really going to like this new channel I
actually found a playlist on your
channel helpful when setting it up so
the new channel is called AI safety
talks and it'll host recorded
presentations by alignment researchers
right now to start off we've got a talk
by Evan hubinger that I hope to record
which is great do check that out
especially if you enjoyed the Mesa
optimizers videos and want a bit more
technical detail there are also some
playlists of other AI safety talks from
Elsewhere on YouTube and lots more to
come so make sure to go over there and
subscribe and hit the bell for more high
quality AI safety material
foreign
[Music]


259. How Might We Align Transformative AI If Its Developed Very Soon 2

hello and welcome to session 259 in the
AI safety.com reading group tonight
we'll be discussing the middle part of
how might we align transformative AI if
it's developed very soon by Halton
kanovsky
Halton kanovsky is still the co-ceo of
open field and co-founder of give will
and we are taking the middle third but I
would like to start by just trying to
literally answer the question that is
posed in the title how might we align
transformative AI in the very near term
and I think in fact there is a if not
complete answer to this then
um the the obvious thing is we have some
alignment techniques that seem like they
they might be useful and we take a look
at the existing techniques that we have
and use the best one of these so
um even hubing have written this
wonderful wonderful post an overview of
11 proposals for building safe Advanced
AI it's two years old at this point but
it's really really good uh just
basically answering this question if we
had to do it now what would be the uh
the best techniques for it and I I
really strongly recommend this post
um of course it would be nice if it
there was an equivalent post just two
years newer right that also had things
like listening latent knowledge
um
and I think that would be a better uh or
uh if not a better way than certainly a
different way to approach this entire
question to Simply start with these 11
techniques and look like try to evaluate
which are good which are bad which can
be used in this particular setting and
in particular is it possible to combine
some of them which I think would uh
would likely be uh beneficial
okay that now for the actual text key
facets of AI alignment remember we saw
last time that the uh the four uh
properties that we really want our AI
system to have is that it should be
value aligned to be honest courageable
and legible
now a key thing in order to get these
properties is that the reinforcing the
reinforcement that we provide the
supervisors provide must be accurate
um in in general the the idea is that
the supervisor like uh remember we're
doing this near casting where we're
assuming
um that the the technique that works is
it's called human feedback from diverse
tasks so that means by definition there
is a supervisor that is giving human
feedback
um and
um that's uh that means that give since
we already know that it is feasible to
uh to give humans supervision on the uh
uh on the object level like is the AI
capable then we we might assume that on
the four alignment properties human
feedback is also a viable uh uh or not
even a viable strategy but the best
strategy since uh magma the AI company
has in fact been able to uh beat the
competition and be first to
transformative AI and I want to
highlight here that that is a really
really big assumption that human
feedback can be scaled to this dramatic
extent and I think that is
um
like we've seen obviously a lot of uh
unsupervised learning and that seems to
me to be some of the most uh impressive
capabilities right now
so the example that Jose is using is on
the task that is generating some
computer code that humans have specified
um and is that a task that humans can
provide reinforcement can uh understand
well right now it is because humans can
understand most code but uh it seems
like if we are uh if we get
substantially more Advanced Techniques
then evaluating whether code is correct
or not it will in fact turn out to be
really difficult and we're not that far
from copilot generating code that is
like above the human level in general
so on when this question about what is
the code how well does the code work we
can do things like we can run it and we
can do automatic tests when we can read
it and uh by definition again since we
are assuming that magma has figured out
how to scale human feedback then we can
just test the performance directly by
assumption
how about legibility again for computer
code it's um quite possible that we can
just see is this understandable and
that's probably going to be one of the
key parameters so sure that seems like
something that we can check if on the uh
on the symbol uh level where a
um a supervisor can look at this code
and say yes I do understand it but we
don't get anything debug with is the
understanding actually correct just the
supervisor's opinion
foreign
a bit less optimistic about like to what
extent uh human values expressed in a uh
if statement that is something about
sorting integers or something like that
uh but you could argue I I'm sure
um for honesty well we can see does the
AI system
um is it reporting it's true best guest
prediction about what the code would do
that's
um to my mind just a restatement of the
problem really we're not digging down
deeper into how will the user will the
supervisors figure out whether the
predictions that the AI give about this
code are its true best guess predictions
except
um just checking whether they are
actually true uh uh but but it can't do
anything about deception or anything
like that or at least there's no
description here of how you would
provide accurate reinforcement that
checks if something is deceptive
also I'd like to notice that among the
four
um alignment properties the one that uh
Holden kanovski is not discussing is
courage ability and there is probably a
good reason for that and that is that in
this specific case uh credibility is
very very hard to check like if you have
some computer code written to a
specification does that mean that the AI
is courageable or not to my mind it's
just completely impossible uh to answer
this kind of question you cannot give
accurate reinforcement on this part
except if the like if the computer code
has to do directly with whether the AI
can change its mind but but almost
always that will not be the case
in particular of course with the
accurate reinforcement one of the key
things we want in for the alignment
point of view is to not reinforce
unintended Behavior
and in one of the key unintended
behaviors are things like exploiting the
supervisors weaknesses and not just
eligibility or things like that
so how can we do that well we can look
at the reinforcements really really hard
and try very very hard to ensure that
they are as accurate as we can make them
and we can also skew them somewhat in
that if we find something that we are
barely able to detect is actually bad
even though it looks quite good then we
should screw our reinforcement and give
that a huge negative value
um but of course doing this is
bound to uh cause dramatic less
competitiveness and uh also this if the
requirement to look really really hard
and work really really hard on this
technique uh it's just something that it
really requires a low number of episodes
like if you have hundreds of thousands
for each supervisor then you just can't
spend that much uh time on them
and I think this is really at odds and
and highlights the problem with this
human feedback from diverse tasks
um that uh is assumed here because if
we're assuming that human scalability
works really well then if you want to
put a an enormous amount of effort into
into this uh then you need to find a
really good way to scale human feedback
um and I think it stretches credibility
if you want only a very low number of
episodes
another facet you'd want is out of
distribution robustness and the the key
out of distribution task that we're
thinking about is like you have the AI
that has been trained and then test and
then suddenly it's in deployment and in
deployment it has the uh the new option
to take over the world that it did not
have previously and that's what we
wanted to not have
there are a number of known problems
with this for instance the Sha 256
problem
um that um Halton kanovski isn't really
uh referring to in any specific way uh
he's maybe he's on our way of the
problem maybe he's assuming that it can
somehow be solved
uh one of the ways we could do this is
we could simulate the out of
distribution meaning that we just tell
the AI you while it is in training that
it is not in training it is being
deployed and try to uh cheat it to do
that
um
in some kind of adversarial sense so we
try to set up something uh as tempting
as possible to try to to push the AI
into situations where it would try to do
a treacherous turn or something like
that
and the problem of course in this is
that we are explicitly telling something
that may be substantially smarter than
us uh a falsehood and that pits our
intelligence against the intelligence of
a uh against the super intelligence and
trying to convince a super intelligence
something uh that is just plainly
obviously not true might be really
really hard
so the adversarial training one of the
things we want to do of course is to
look at these four uh facets these four
alignment objectives
courage ability value alignment
legibility and now I already forgot the
fourth and try to see if there are some
of these objectives that have somehow
been learned
another way is to dig down more into
what are the unintended behaviors
actually and teach the AI details about
this in order to uh to see whether it
has this I think that seems quite unsafe
uh to me
and teach you the apparent opportunities
to get reward via unintended behaviors
are illusory and I think this trying to
convince an AI that is not in a box that
it actually is in a box is really really
hard to implement like it's trying to
um convince someone who is smarter than
you or something that is blatantly
untrue and uh it seems uh really really
hard to do that in a consistent way that
um that they can't just easily uh see
through
one of the things we can look at is
relaxing the problem somewhat and
instead of just looking at
um
uh we want the air to not do a takeover
then we can look at a larger class of
problematic Behavior called the always
undesired behaviors and Holden kanowski
is very optimistic about this say we can
avoid this with very high reliability
and my uh uh thought is that it's going
to be a lot harder than Halton expects
um very high reliability is a very
difficult bar in particular like
cheating someone who is uh smarter than
you seems uh uh not like something that
can work infinitely at least at some
point the other person just becomes so
much smarter than cheating them just
becomes infeasible
so the always on the side behaviors are
dishonesty eligibility and eligibility I
would like to point out is not always
something that is always on desired like
again with the example of computer code
sometimes we always prefer that it's
legible but it's not like we want as
legible as possible because we have a
number of other requirements at the code
we are not trying to maximize legibility
we are trying to actually get the job
done and sometimes they're just on no
like you would hope that there is a
really really simple way to prove the
Riemann conjecture or something like
that but we want to know whether the
Riemann conjecture is true and so um
uh in in that case we uh we might not
really train very very hard for always
having eligibility on desire
um so I don't think we can do it like
this
again encourage ability is something
that is always undesired but like how
would we go about that
taking steps towards disempowerment is
of course also something that is always
undesired
um but uh again it's hard to do in
reinforcement once this is actually uh
I haven't
falling for honeypots is something that
we'll get back to and falling for
misleading info like if you tell the AI
something that is wrong uh then we want
the AI to figure out what is the truth
and I think this may be in fact at odds
at some of the relaxed adversarial
training we'll look at uh later and uh
certainly um this is this seems to not
be related to alignment this seems to be
just making the AI smarter
um the the key kind of training that we
want to avoid is uh you know this word
uh results that are incorrect but
convincing to most supervisors this is
something that can be interpreted in two
ways depending on whether it hinges on
most like if you have something that uh
9 out of 10 that fools 9 out of 10
supervisors but the 10th supervisor sees
through this and tells the other line
and then it gets a very low uh reward
for that
um then that's one way of seeing it
another is that we look into the AI with
the to try to figure out if it's honest
or not and see whether it's it's making
something that it knows is Dishonored
that is incorrect but beliefs will be
convincing to the supervisors I think in
fact both techniques may be very useful
it's just uh we need to disentangle
these two things
one of the key things that I'm probably
interested are what Holden kanovski
calls exotic cases uh yeah I have put in
here a quote by Ilya I think it's really
relevant to this question in that all
this task of trying to figure out what
is a is this AI that we're building
online
if we figure out yeah it is in fact
online then going from there to making
it more aligned we obviously can't just
train uh train with symptoms so what do
we actually do and I think this is
something that a harlemen kind of should
think way more about
um okay let's get back to the Exotic
cases one way to find cases where the AI
behaves possibly in a bad way is to do
the standard kind of
um adversarial input thing that we are
doing right now making small
permutations and see which triggers the
AI to do something strange
um I agree it would be a nice thing to
do I'm not entirely sure this is
worthwhile because we have a limited
budget uh magma may have even a sharply
limited budget for alignment work and
whether this is worthwhile is
questionable to me
another way to find exotic cases which
really is things like literature's turn
is to con uh condition on there being
some kind of fake path to a takeover
um
and as I've said before faking uh making
a an illusion that is
um impossible to see through is probably
really really hard uh and just in order
to figure out that it's an illusion you
only have to find one inconsistency and
if you go looking for inconsistencies in
something that is totally fake then
you're more likely than not to find it
also admits that he's very uncertain
about like to what extent will this
actually work uh one thing that is at
least clear is that it will be possible
for magma to spend a lot of effort in
trying to make this possible uh in doing
adversarial uh training so uh well it
will actually help is an open question
but you can spend a lot of effort there
um
and of course just at some point once
the AI becomes smart enough then it is
insufficiently reflective and
um uh at at some point even if you get
really good at faking things for for the
AI uh then for it to to realize that
um uh holdenovski is again really really
optimistic he says it would take someone
who is vastly a vast is super
intelligent AI to realize that uh the uh
the path to take over that is being
presented is in fact not real but
something that is uh an adversary has
placed in front of it uh I think that
that is unfortunately very unlikely and
unlikely to require a vast super
intelligent AI
um because the AI is reading what we are
reading here and so it will obviously
suspect this in particular if it has
encountered often in training and
um
evaluating this uh like evaluating a
true fact uh required in general does
not require a vast super intelligence or
something like that I think holding kind
of is way too optimistic about uh about
this
a third facet of alignment is preventing
exploits and one of the ways to prevent
exploits is to just put the AI in a box
that's like the classic way that might
make sense in some cases and does it
make sense in this case right this is
the kind of thing that I would really
like to have some uh details about
obviously when you want the AI to be
transformative
then it can be in the box and if you
wanted to do something like the
hardening uh suggestions that we we saw
last time then it can't be in a box but
an advisor or coordinator maybe in a box
it could make sense at least in some
cases
so the goals of the AI like normally I
would if I were describing the the goals
I would start out with like omu Hunter's
basic AI drives
um but uh the three goals that Holden
kanovski are starting with are deception
manipulation and disempowerment of
humans and the means to this are things
like hacking and psychological
understanding
um the cognitive limits of humans and
different kind of side Channel attacks
the counters to these are excellent
cyber security selecting and educating
supervisors having a vetting like you
get a security clearance of the
supervisors and limiting the autonomy of
the AI
uh of course uh going to like a normal
company and say you need to have
excellent cyber security it's a really
really big ask and a lot of people don't
realize just how difficult it is to get
excellent cyber security in the sense
um but
um it may be required and I could easily
see that magma would be at all it's not
just for on the inside from its own AIS
but also from the outside like
government or competitors who would be
very interested in uh in penetrating to
some extent
uh and uh we want in particular to
ensure that the training doesn't mean
it doesn't accidentally oops uh caused
the AI to become more capable on these
behaviors
the final way of ensuring this uh
alignment and safety is by doing testing
and threat assessment like figuring out
how do these uh unintended behaviors
arising
one of the key ways is doing audits uh
Harden suggests that for some of the AI
actions we just put 100 times as much
effort on scrutiny uh so we we take like
I don't know one out of thousand actions
done by the AI and we really really
drill down into figuring out is this
actually deceptive in some way
I don't expect this to be a good
strategy because I think that an AI that
is trying to do a treacherous turn would
do something innocent in 99.9 of the
cases
um and that means that
um in order to catch the the single
action that is in fact disempowering
what you need to do is to focus widely
so you ensure that the uh
that all uh agents are audited instead
of just focusing on very few
hunting also suggests some randomized
Audits and I think that may be better if
it's done in some kind of combination
so we take and
this order you have like the supervising
budget and then you have an audit budget
and something you spend on the most uh
suspicious actions and some on the
random and selected
set of actions I think something like
that may be better
one of the things that Holden kanovsky
is also suggesting we'll get back to
this a lot more next time is to use uh
once the AI is going through this
intelligence explosion we may
um
just the AI now may be substantially
smarter than the AI a couple of days or
weeks ago and then we maybe used that
one to to examine the earlier weaker AI
outputs
um and that may be useful we could also
make the auditing AI get more compute
and
we could do interpretability uh uh as
part of the auditing processes but
that's a problem that's something that
have known problems of course in that
you could just train it to appear
um honest
some kind of multiple level evaluation
where you're perhaps only evaluating
test and not in training may make sense
and
yeah I think it basically does it's
probably only catches unintended
training of deceptiveness but that may
also be uh
um worthwhile and in particular using
another AI to check whether the previous
AI was deceptive is something that will
only work if the new AI is um is
reasonably aligned so that is an
obviously problem with this Holland
kanovski again is really optimistic he
believes this is likely to work by
default
um and that it requires a pretty
extraordinary level of capability to
anticipate that the uh there will be
some kind of test
I think this is very wrong I think uh
the uh the text that Holden kanovsky is
writing is explicitly something that a
future AI will be trained on so that
means the AI knows that it should
anticipate this because it's in the
training set because Halton has
explicitly talked uh told the AI that
um and I think we need a principle uh
somewhere on an analogy to kirchhoff's
principle that if you have an alignment
strategy that only works if the AI does
not know about it then it's a really bad
strategy and like you certainly cannot
publicly discuss a strategy that will
not work if the AI knows about it
because the AI is of course going to be
trained on everything
unusual incentives is another strategy I
think this is what people normally call
honeypots which is
um like let's say you have an AI and you
believe it will not try to take over not
just the world but in particular it
doesn't want to take over its
environment and you give it the option
to control environment but not the
option to take over the world of course
you you really don't want to give it
that option right but the option to
control the environment you give it that
even though you believe it will not take
that option
and then you add dramatic more
incentives for the AI to take over its
environment like you say if it takes
over its environment it will get compute
for a true uh 10 million dollars and can
use that just maximize its reward
function
um and
um then you see if it does it I think
that is kind of a strange thing like if
the AI will not take over the
environment if that's done by incentives
then obviously by changing the
incentives then eventually the AI will
try to take over its uh
its uh environment
um
and also this is something that may not
matter very much in the sense that the
potential incentives for taking over the
environment are likely to be
dramatically dramatically smaller than
um taking over the entire uh Universe
because the things that
um uh they can that magma can use for a
specific small part of the alignment
project is just dramatically uh worth
less than the entire universe
another thing we should do in assessment
is to explore alignment
and not just always using the most
aligned AI we have but try to train uh
especially with weaker AIS without some
of these alignment techniques varying
the alignment techniques in different
ways
um to get some understanding of how they
relate
um and of course if that is something
that we want to do then that's something
that we could in theory do now because
the AIS that is substantially weaker are
in fact AIS that we have right now so
this is not something that a that they
should wait with doing until this
um and of course why we're not doing
that right now well because we can't
like uh the alignment Theory field is
just far far from being able to say this
AI is 38 percent of the line and this AI
is 57 aligned we totally can't don't
have any metrics like that right now so
uh just because uh magma somehow gets a
more powerful AI some kind of
breakthrough then like the assumption
that they then they'd be able to go back
to the level we are now and say anything
intelligent about that that seems like a
very uh big and strange assumption
and the Hope here um from uh Holden
kanowski is that we can get some kind of
quantification uh of uh how well do the
different alignment techniques work in
the hope that we can get something like
scaling loss between
um uh the alignment measures
I think it's very unlikely we'll get
something like scaling loss those are by
compute and trying to extrapolate
towards the future and I think like the
new abilities of AIS actually becomes uh
more powerful are very likely to be uh
disjunctive and the AI in particular are
going to be able to see through some of
our attempts at misdirection
and uh Holden kanovsky uh is also
somewhat less optimistic about this he
says it probably only works for AIS that
are lagging situational awareness and I
think uh that is true and I think also
if we are talking about transformative
AI then transformative AI almost by
definition will have substantial
situational awareness right because if
it's AI that does not have situational
awareness then it's not going to be
transformative so I think almost by
definition this is not going to work
and uh but but that's not even the key
the key problem here the key problem is
that misalignment uh is going to be
really really hard to identify and put
toward and put any kind of metric
towards
the final suggestion uh uh penultimate
suggestion is to create some kind of
dedicated assessment team
um so should it be a dedicated team or
should it be like more broadly uh
distributed in the organization I think
it's question it's a political question
obviously how much political question
Capital does the alignment team at magma
have does it have enough that you can
make a very strong dedicated assessment
team or do you need to organize it in
some other way and I could definitely
see problems with
um having it as a separate team
so what would be the uh the tasks one of
the tasks would be to analyze the
overall setup estimate the odds of an
existential risk in general both on the
um uh action risk and inaction risk
I think that's going to be really hard
to get meaningful probability on this
right now the uh the Distribution on
risks given by AI safety researchers are
uh like really really wide some people
believe that there's 99.9 probability of
the disaster and some people believe
0.01 probability of a disaster like it's
a really wide distribution right now
the assessment team can do
interoperability work they can monitor
activity locks they can interview the
staff the rest of magma do experiments
hire super forecasters and finally use
AI assistance in this and this is
probably something we're going to talk a
lot about
in the next session whether we should in
fact expect that to be a substantial
help
John Wentworth has a nice analogy about
this that this is like getting Godzilla
to fight Mika Godzilla and that is a
thing that makes in general plans much
more brittle and in general if you think
that monitoring
activity locks is something that is so
hard that you can't do it then just
saying oh we'll use the AI as assistance
to figure this out seems like kind of a
cop-out and very unlikely uh to work
like if that was going to work then why
doesn't it work when when humans are
looking at this activity locks I think
that's a
it is certainly possible but it's uh I
agree with John went with that this
makes the plan a lot more brittle
the final thing is the question what do
you actually do if you figure out there
is a risk uh what do you do if the AI
that you're training seem to be
unaligned I think this is a core
question and something that should have
much more than one paragraph assigned to
it I think this is a really big question
that needs to be answered uh of course
the thing you could always do is to just
train it away
um that's very unlikely and I think
magma in this case is assumed to be
sophisticated enough to certainly not
want to do that
um so what will they do they might be
trying to find fundamental improvements
to the alignment problem just doing
basic research and that's like the kind
of research that we ought to be doing
now because we are expecting that we
will find some kind of risk
um
so why we're not doing that
um and another thing is they could warn
others of the danger
um and that's something that I would
like to have way more discussion of
because who do you want and what do you
tell them what is like the political uh
consequences of telling different people
and what uh can people keep secrets or
are you forced to share it maximally do
you request aid from the government to
request eight from the general public to
try to uh uh like what do you do then I
think it's a it's a very interesting
question and a crucial question if a
company magma is building a something
that looks like it can be transformative
and it seems like they are unable to
um reduce the existential risk
sufficiently like what do they actually
do who do they sell I think it's really
important and we will see in next time
what uh precisely
um holding canovskus is just in that
case and
um I hope at least hears more about that
thank you and see you


AI Safety - Computerphile

so I saw dr. Holden's video and enjoyed
it the videos I feel like was kind of
framed as a counterpoint almost but
actually watching it I really didn't
disagree with very much of it at all
I think probably the area in which we
diverge is that I think the default
artificial general intelligence if you
were to build one without any particular
concern for safety would be a bad one
would be one that we don't want to build
that would try to do things we don't
want it to do and I talked about this in
the earlier videos when I was talking
about the space of general Minds or the
space of motivations this is the kind of
thing that the stamp collector
machines creator was expecting to happen
human values are complicated right they
are complicated and we don't really know
what they are fully and anything that
has values that aren't very tightly
aligned to ours is not a thing we want
to be building
yeah I talked to to dr. Holm about this
but almost everything in that video I
agree with I agree about the timescales
I agree that we are a long way away from
the kind of thing that I'm talking about
or with that we're probably a long way
away right in the previous video I stand
by the video but I think I came across
as overconfident just because of the way
that it was framed like I didn't choose
the title the deadly truth you call
something the truth capital T you know
it was a thought experiment it was a I
wouldn't I would have called it an
interesting thought experiment about
general artificial intelligence or
something like that there's an argument
in mind I've got quite so many views
yeah I'm not going to try and tell you
how to do your job you know but no the
actual content of the video I stand by
so there was really only one thing in
doctor Holden's video that I didn't
agree with which was like one sentence
he was talking about how we don't pay
enough attention to the the Friendly AI
possibility the positive outcome which i
think is true but then he said which i
think is more likely I don't know how
likely it is I think it depends on how
seriously we take this problem of how do
we actually make sure that any
artificial general intelligence we
create is friendly friendly you know I
don't think that it's a problem that's
solved and I don't think that it's a
problem that will solve itself I
actually think it's a problem that's
very difficult to solve and we are very
far from a solution or quite far from a
solution it's very difficult to say with
any confidence how far we are
but it's comparable to how far we are
from general AI itself we had a brief
conversation with Professor Brailsford
on exactly this and he said maybe this
is like cold fusion 50 years ago they
said hey in 50 years we'd be able to do
this and it's always in 50 years right
yeah and the thing is with historical
examples
you can go back and find historical
examples endless historical examples of
people claiming that something fantastic
is right around the corner when in fact
it isn't but the thing is you can also
find a lot of examples of people saying
that something fantastic is definitely
never going to happen and it's actually
right around the corner because before
we talked about efficient
self-sustaining fission reactions Cinda
in the AI video that is an example of a
situation where you have Ernest
Rutherford Nobel Prize winner extremely
eminent scientist so he split the atom
right well he was part of a team that's
forgotten and he was on the record
saying some people think that you could
use this as a source of actual energy
that you could reliably get energy out
of this that's they called it moonshine
said it was completely absurd right and
then you have Leo Szilard
who eventually did it he had the idea
now we don't know exactly but as far as
we can tell the idea for using neutrons
to make a self-sustaining nuclear
reaction occurred to him on the same day
that Rutherford was dismissing it as
nonsense this kind of thing can happen I
don't I'm not saying that it will that's
my point right you can find situations
where people are overconfident in
predicting something will happen you can
also find situations where people are
overconfident in predicting it won't
happen and it's just a function of the
fact that predicting is really difficult
my central point actually doesn't rely
on anything to do with time scales
really right I think we're a long way
away from all of these things but at the
same time I'd be very surprised if it
were more than 100 years or something
you know just because and again this is
what this is as dr. Holden said it's
it's kind of inescapable if we continue
advancing as we are we're going to get
there sooner or later we know it's not
impossible unless we wipe ourselves out
some other way or destroy our
technological capacity some other way or
we discover that the brain is literally
magic and not like Roger Penrose quantum
magic like actual magic then we'll get
there sooner or later right my point is
that the problem of AI safety is not
solved is not going to solve itself and
is not easy to solve and most
importantly we have to solve it
for we solve the problem of general AI
that's that's all I'm trying to say
really is that sooner or later we will
probably get general AI and when we do
we have to know how to make safe general
AI and currently we're we're a long way
from that over a long enough time scale
I think that human level artificial
intelligence is completely inevitable
then asked was that a human you were
talking to or was it a machine if you
can't decide that it's not human then
it's passed the test


Deceptive Misaligned Mesa-Optimisers? It's More Likely Than You Think...

hi so the previous video explained what
mesa optimizers are why they're likely
to happen in advanced machine learning
systems
and why the optimal strategy for a mesa
optimizer
is to pretend to be aligned during the
training process and then turn on us
once it's deployed
i highly recommend watching that if you
haven't but you might think that a
deceptive misaligned mesa optimizer
is kind of a weird and special case like
sure maybe it's possible that an ai
system will come to want something
different from the objective it's being
trained
on and deliberately manipulate the
training process in order to pursue that
dangerous objective in the real world
that does seem to be the optimal
strategy given certain assumptions but
how likely is it in practice like it
seems as though there's a whole bunch of
things that need to be true in order for
this to happen
first off it needs to be the case that
deployment is in some sense bigger than
training
there are more apples to be had in the
real world than in training so it's
worth sacrificing some rewards during
training
if that helps you to be deployed but i
think this is in fact very often the
case
the real world almost certainly is
bigger than the training process
i mean the real world contains the
training process right and it seems
reasonable to expect that once the agent
is at large in the real world
there's a good chance it will be able to
find better ways of achieving its
objective
than it could ever get during training
another thing that has to be the case
is that your mesa objective has to care
about multi-episode returns by which i
mean
why does the optimizer in this episode
care about the apples in the other
episodes this is another place where
that abstraction i was talking about in
the previous video
the it wants to pursue the objective it
was trained on abstraction
comes in again because yeah the agent
only actually gets reward for things
that happen within a single episode
so you might think that the mesa
objective would only care about single
episode returns
and if that's the case then you don't
have this problem because when the agent
is in this episode
it only cares about this apple it
doesn't care about any of these other
apples because
it can only ever get reward for this one
so it would never miss this apple in
order to get more apples later but
actually
caring about all apples is a fairly
natural generalization the fact is most
of the time
the way the mesa optimizer feels about
things outside of its environment
doesn't matter it can't affect any of
the things that are outside of the
environment
so whether it cares about them has no
effect and so that's unconstrained it's
free to vary here's an analogy
suppose we're training a model to learn
a function and our data points look like
this
we only have data within this range here
so our training process has no way to
differentiate between the infinite
variety of very different functions
that all fit the data identically within
the training distribution
inside the distribution we're looking
for this but outside of the distribution
the function we learn might look like
this
or this or this or even this
but some of these seem like more natural
ways to generalize than others some of
them are simpler and optimizers like
gradient descent
tend to prefer simpler functions so we
could assume that the function will
return
zero for everything outside of the
distribution like this
and that's certainly possible but it's
by no means the default
so going back to our mesa optimizer its
opinion of the apples in other episodes
can be anything in principle but
caring about those apples the same as
this episode's apple is a pretty natural
way to generalize if what you care about
is going towards green things
and then you become aware that there are
all of these other green things that you
didn't know about
then probably you're going to want to go
towards those as well right
that's a natural way to extend your
objective function outside of the range
it was trained on so
caring about multi-episode returns seems
pretty likely
another thing that needs to be true for
deception to be a problem is
the mesa optimizer has to believe that
it's part of a training process
it doesn't actually have to know all the
details of the training process but it
does have to believe that it is in some
kind of training process that it's
subject to optimization
by something like gradient descent and
it has to have some kind of guess
at what the base objective is likely to
be and you might think
well how would it know that but it's
probably the kind of thing it could
figure out from its training data
the current way we train our biggest ai
systems is by giving them vast piles of
data from the internet
the whole of wikipedia every article
ever linked to from reddit that kind of
thing
if you're giving the system a huge data
set of information about the world
that will certainly include information
about machine learning systems and
methods
the people and companies working on them
and so on gptn
is going to read everything we wrote
about gptn minus one
and that would certainly be enough
information to figure out what's going
on we can try to remove that stuff from
the data set
but everything is connected and the
system can make inferences
if we only remove information
specifically about machine learning
then the fact that this information has
been removed will be obvious
and the absence itself would serve as
strong evidence that the system is in
fact a model in training
removing all mention of anything at all
related to machine learning in any way
so the system couldn't even notice the
gap seems pretty much impossible and
even if we managed it we then end up
with a system that had no idea that
anything
even tangentially related to machine
learning has ever existed
which is going to be pretty limited in
its usefulness so yes in order to have
this deception problem the mesa
optimizer is required to have a
particular belief
that it's a model in training but the
fact is that belief
is actually true so it's kind of
reasonable to consider that your highly
advanced
figuring things out machine might figure
that out and finally
you might think that the mesa optimizer
has to be able to tell the difference
between when it's in training and when
it's deployed so it knows when to stop
pretending to be a light
but it turns out that's not really
required firstly it doesn't actually
need to know for sure
right if you imagine a mesa optimizer
that goes from
90 sure i'm in training to 90 sure i'm
in deployment
the logic works basically the same in
that case the mesa optimizer does have
to take a risk
but it's worth it and in fact even if
the model has absolutely no idea whether
it's in deployment or in training
the best strategy is still to gamble on
deception like if we look at our toy
example again
and we tweak it so that the mesa
optimizer has no way of knowing which
episode it's in
if you work out the optimal policy it
turns out what works best
is the randomized strategy go to a 55
percent of the time
and b 45 of the time the mesa
optimizer's best bet
is to roll the dice and hope that it
will defect during deployment and not
defect during training
in my simulations that strategy gets at
an average of 1.13 apples compared to
the one apple it gets by not being
deceptive
the specific numbers here will vary
depending on things like
how likely the agent thinks being in
training is compared to being in
deployment
and how much reward it thinks is
possible in the real world compared to
in the training process but the point is
that even if we manage to make it so
that the mesa optimizer can never tell
when it's deployed
the chance of it turning on us is still
much higher than we'd like
but actually that might not even matter
because hiding whether the system is in
training or deployment seems extremely
difficult
if you're a misaligned mesa optimizer
trying to figure out whether you're in
the real world or not
any distributional shift at all can give
the game away
on a basic level for example you could
look at the dates of all of the training
data you've seen so far
and say hmm there's nothing after 2020
so i guess if i start seeing training
data that says 2025
then i can be confident that i'm
deployed now an adversarial training
process if it's smart
might catch this it might try putting
fake dates on training data to see if
that changes the mesa optimizer's
behavior
but the mesa optimizer if it's also
smart can do something cleverer than
just looking at the dates any difference
at
all between training and deployment can
be used so there are a million different
kinds of clues available
and in the extreme case the mesa
optimizer has the option to use
evidence that the adversarial training
process simply could never fake
consider rsa 2048 a giant semi-prime
number used in cryptography
this number has only two prime factors
and there's a two hundred thousand
dollar prize
for anyone who can find out what they
are but nobody has yet at some point in
the future someone will probably find
and publish those prime factors
but for now it needs more computing
power than anyone has
so the mesa optimizer can just say to
itself okay
if i ever see a pair of prime numbers
that multiply to rsa 2048
then i'll know that i'm in the real
world and the adversarial training
process
simply doesn't have the computing power
to fake that this raises the strange
idea of a future where we have this
amazing seemingly perfectly aligned agi
system
and everything's going wonderfully
possibly for years
and then someone manages to factor rsa
2048 and publishes the result
and suddenly everything goes to hell as
the agi turns on a dime and starts going
after
whatever random thing happened to be the
mesa objective at the first point
in training when the system realized
that deception was possible i'm not
saying that will happen
to be clear that won't happen but it
illustrates just how hard it is
to indefinitely keep a super
intelligence in the dark about whether
it's in the real world
so yeah although there are a bunch of
things that have to happen in order to
have this kind of
deceptive misaligned mesa optimizer all
of them are things that are pretty
likely to happen
and more than that many are things that
gradient descent will actively try to
cause to happen
because deceptively aligned models are
really effective they're actually some
of the most effective models at the base
objective on the training distribution
why would deceptive mesa optimizers be
more effective than other types of model
we'll answer that question in a later
video
sorry to do this cliffhanger kind of
thing but i've already spent way too
long making this video
and i need to publish it so look out for
mesa optimizers episode 3
coming soon
i want to end the video by saying thank
you to my wonderful patrons
all of these great people here this time
i'm especially thanking kieran
thank you so much you know it's because
of people like you that i'm now able to
look at hiring an editor i'm in the
middle of the interviewing process now
and
hopefully that can speed up video
production by a lot so thanks again
kieran and all of my patrons
and thank you for watching i'll see you
next time