Deceptive Misaligned Mesa-Optimisers? It's More Likely Than You Think...

hi so the previous video explained what
mesa optimizers are why they're likely
to happen in advanced machine learning
systems
and why the optimal strategy for a mesa
optimizer
is to pretend to be aligned during the
training process and then turn on us
once it's deployed
i highly recommend watching that if you
haven't but you might think that a
deceptive misaligned mesa optimizer
is kind of a weird and special case like
sure maybe it's possible that an ai
system will come to want something
different from the objective it's being
trained
on and deliberately manipulate the
training process in order to pursue that
dangerous objective in the real world
that does seem to be the optimal
strategy given certain assumptions but
how likely is it in practice like it
seems as though there's a whole bunch of
things that need to be true in order for
this to happen
first off it needs to be the case that
deployment is in some sense bigger than
training
there are more apples to be had in the
real world than in training so it's
worth sacrificing some rewards during
training
if that helps you to be deployed but i
think this is in fact very often the
case
the real world almost certainly is
bigger than the training process
i mean the real world contains the
training process right and it seems
reasonable to expect that once the agent
is at large in the real world
there's a good chance it will be able to
find better ways of achieving its
objective
than it could ever get during training
another thing that has to be the case
is that your mesa objective has to care
about multi-episode returns by which i
mean
why does the optimizer in this episode
care about the apples in the other
episodes this is another place where
that abstraction i was talking about in
the previous video
the it wants to pursue the objective it
was trained on abstraction
comes in again because yeah the agent
only actually gets reward for things
that happen within a single episode
so you might think that the mesa
objective would only care about single
episode returns
and if that's the case then you don't
have this problem because when the agent
is in this episode
it only cares about this apple it
doesn't care about any of these other
apples because
it can only ever get reward for this one
so it would never miss this apple in
order to get more apples later but
actually
caring about all apples is a fairly
natural generalization the fact is most
of the time
the way the mesa optimizer feels about
things outside of its environment
doesn't matter it can't affect any of
the things that are outside of the
environment
so whether it cares about them has no
effect and so that's unconstrained it's
free to vary here's an analogy
suppose we're training a model to learn
a function and our data points look like
this
we only have data within this range here
so our training process has no way to
differentiate between the infinite
variety of very different functions
that all fit the data identically within
the training distribution
inside the distribution we're looking
for this but outside of the distribution
the function we learn might look like
this
or this or this or even this
but some of these seem like more natural
ways to generalize than others some of
them are simpler and optimizers like
gradient descent
tend to prefer simpler functions so we
could assume that the function will
return
zero for everything outside of the
distribution like this
and that's certainly possible but it's
by no means the default
so going back to our mesa optimizer its
opinion of the apples in other episodes
can be anything in principle but
caring about those apples the same as
this episode's apple is a pretty natural
way to generalize if what you care about
is going towards green things
and then you become aware that there are
all of these other green things that you
didn't know about
then probably you're going to want to go
towards those as well right
that's a natural way to extend your
objective function outside of the range
it was trained on so
caring about multi-episode returns seems
pretty likely
another thing that needs to be true for
deception to be a problem is
the mesa optimizer has to believe that
it's part of a training process
it doesn't actually have to know all the
details of the training process but it
does have to believe that it is in some
kind of training process that it's
subject to optimization
by something like gradient descent and
it has to have some kind of guess
at what the base objective is likely to
be and you might think
well how would it know that but it's
probably the kind of thing it could
figure out from its training data
the current way we train our biggest ai
systems is by giving them vast piles of
data from the internet
the whole of wikipedia every article
ever linked to from reddit that kind of
thing
if you're giving the system a huge data
set of information about the world
that will certainly include information
about machine learning systems and
methods
the people and companies working on them
and so on gptn
is going to read everything we wrote
about gptn minus one
and that would certainly be enough
information to figure out what's going
on we can try to remove that stuff from
the data set
but everything is connected and the
system can make inferences
if we only remove information
specifically about machine learning
then the fact that this information has
been removed will be obvious
and the absence itself would serve as
strong evidence that the system is in
fact a model in training
removing all mention of anything at all
related to machine learning in any way
so the system couldn't even notice the
gap seems pretty much impossible and
even if we managed it we then end up
with a system that had no idea that
anything
even tangentially related to machine
learning has ever existed
which is going to be pretty limited in
its usefulness so yes in order to have
this deception problem the mesa
optimizer is required to have a
particular belief
that it's a model in training but the
fact is that belief
is actually true so it's kind of
reasonable to consider that your highly
advanced
figuring things out machine might figure
that out and finally
you might think that the mesa optimizer
has to be able to tell the difference
between when it's in training and when
it's deployed so it knows when to stop
pretending to be a light
but it turns out that's not really
required firstly it doesn't actually
need to know for sure
right if you imagine a mesa optimizer
that goes from
90 sure i'm in training to 90 sure i'm
in deployment
the logic works basically the same in
that case the mesa optimizer does have
to take a risk
but it's worth it and in fact even if
the model has absolutely no idea whether
it's in deployment or in training
the best strategy is still to gamble on
deception like if we look at our toy
example again
and we tweak it so that the mesa
optimizer has no way of knowing which
episode it's in
if you work out the optimal policy it
turns out what works best
is the randomized strategy go to a 55
percent of the time
and b 45 of the time the mesa
optimizer's best bet
is to roll the dice and hope that it
will defect during deployment and not
defect during training
in my simulations that strategy gets at
an average of 1.13 apples compared to
the one apple it gets by not being
deceptive
the specific numbers here will vary
depending on things like
how likely the agent thinks being in
training is compared to being in
deployment
and how much reward it thinks is
possible in the real world compared to
in the training process but the point is
that even if we manage to make it so
that the mesa optimizer can never tell
when it's deployed
the chance of it turning on us is still
much higher than we'd like
but actually that might not even matter
because hiding whether the system is in
training or deployment seems extremely
difficult
if you're a misaligned mesa optimizer
trying to figure out whether you're in
the real world or not
any distributional shift at all can give
the game away
on a basic level for example you could
look at the dates of all of the training
data you've seen so far
and say hmm there's nothing after 2020
so i guess if i start seeing training
data that says 2025
then i can be confident that i'm
deployed now an adversarial training
process if it's smart
might catch this it might try putting
fake dates on training data to see if
that changes the mesa optimizer's
behavior
but the mesa optimizer if it's also
smart can do something cleverer than
just looking at the dates any difference
at
all between training and deployment can
be used so there are a million different
kinds of clues available
and in the extreme case the mesa
optimizer has the option to use
evidence that the adversarial training
process simply could never fake
consider rsa 2048 a giant semi-prime
number used in cryptography
this number has only two prime factors
and there's a two hundred thousand
dollar prize
for anyone who can find out what they
are but nobody has yet at some point in
the future someone will probably find
and publish those prime factors
but for now it needs more computing
power than anyone has
so the mesa optimizer can just say to
itself okay
if i ever see a pair of prime numbers
that multiply to rsa 2048
then i'll know that i'm in the real
world and the adversarial training
process
simply doesn't have the computing power
to fake that this raises the strange
idea of a future where we have this
amazing seemingly perfectly aligned agi
system
and everything's going wonderfully
possibly for years
and then someone manages to factor rsa
2048 and publishes the result
and suddenly everything goes to hell as
the agi turns on a dime and starts going
after
whatever random thing happened to be the
mesa objective at the first point
in training when the system realized
that deception was possible i'm not
saying that will happen
to be clear that won't happen but it
illustrates just how hard it is
to indefinitely keep a super
intelligence in the dark about whether
it's in the real world
so yeah although there are a bunch of
things that have to happen in order to
have this kind of
deceptive misaligned mesa optimizer all
of them are things that are pretty
likely to happen
and more than that many are things that
gradient descent will actively try to
cause to happen
because deceptively aligned models are
really effective they're actually some
of the most effective models at the base
objective on the training distribution
why would deceptive mesa optimizers be
more effective than other types of model
we'll answer that question in a later
video
sorry to do this cliffhanger kind of
thing but i've already spent way too
long making this video
and i need to publish it so look out for
mesa optimizers episode 3
coming soon
i want to end the video by saying thank
you to my wonderful patrons
all of these great people here this time
i'm especially thanking kieran
thank you so much you know it's because
of people like you that i'm now able to
look at hiring an editor i'm in the
middle of the interviewing process now
and
hopefully that can speed up video
production by a lot so thanks again
kieran and all of my patrons
and thank you for watching i'll see you
next time


222. Robot Responsibilities

uh welcome to
the ai safety reading group number 222
this week we're reading artificial
intelligence and robot responsibilities
uh innovating
beyond rights by uttan ash rafiyaan
hutan ash rafiyan is
works at the imperial college of
medicine i believe
uh in the uk and um
uh works as a as a surgeon
or as a sorry as a lecturer of surgery
and occasionally writes about
ai and and
and history funnily enough and so
we're looking at this paper that he
wrote in 2014
so the main parts of the argument uh
that i've broken down of this paper
i call the terminator 9 pitch as in
terminator the ninth sequel uh which
meta
which meta ethical framework and robot
souls
robots are similar to animals but
shouldn't be considered animals
robot inclusion in the u.n declaration
of human rights
and ancient roman law as a model of the
legal and social status
machines this may seem totally chaotic
and it's because the paper kind of is so
this
is where we begin so the terminator 9
pitch
uh begins with what he calls this
thought experiment
where two nations or the alphabet we
call them the alphamites and the beta
vistas
and they're both using robots and ai and
military
the alpha mites locate a strategically
critical area to capture
that will end in the in a unilateral
victory for the alpha knights
um and end the war totally so the alpha
mite robots go in
and they they destroy all of the beta
vista
robots in this strategic area but the
local beta vista children fight the
alpha mite robots and get injured
the alpha mite robots refrain from
engaging in combat with the children
and uh um and hutan says this is in
accordance with all rules and laws for
robots
and they look after the children instead
because of the time
and the resources lost looking after the
beta vista children
this strategic location is lost and the
war continues
for some time afterwards so
ash refi ashraf yan asks us to consider
the following from this thought
experiment
should robots have helped the injured
children on our moral grounds
even if this meant that the war could be
delayed or even potentially lost
with a pos with possibly many more
deaths
from both warring sides two at a broader
level
uh what is the moral responsibility of
self-conscious rational
and sentient intelligence i would say
this first question that he asks is his
false dilemma fallacy
because he only gives us these two
possible options
um of which i think there are many more
that we could imagine in scenario
the second question that he poses to us
i would say is a loaded question because
what is there for us uh within this
scenario for us to
um to conceive of these robots as having
self-consciousness
um rationality and uh and sentient
uh sentience and intelligence um he
doesn't give us a lot of information
so i i think we ought to ask this
thought experiment the following
questions
uh how are the robots and the ai
sophisticated enough to wage war
distinguish between combatants and
non-combatants or fighting
yet not be sophisticated enough to
recognize the futility of
human governed war and the advantage of
teaming up with other robots and ai
to its own advantage against humanity
itself
which um as if you follow this group
you would know is one thing that we is
posited often
is the problem that ai may
not be interested in human things it
might not be interested in nations
and have its own interests the second
question
is that he says that these robots follow
laws what are these laws that the robots
have
how is it the robots have developed in
separate and competing indeed at war
nations
and have the same ethical legal etc
rules
the third question is how is it that
robots negate their goal of winning the
wharf the goal of
giving medical care to the enemy yet go
back to waging
the war anyway i think
we find in asking these questions this
um
this thought experiment falls apart
pretty easily which is why
i call it the terminator 9 pitch that it
would be
fine as a um
as a as a film but it doesn't really
help us to understand
ai or the ethics of ao very well
so he goes on to say um we need to try
and work out
uh what is the meta ethical framework to
understanding
the meta ethical framework that we
should use to try and conceptualize
the position of robots in our midst in a
in this time in which robots are
part of our society and he makes uh the
following
um statements he says determinism means
that all events are predetermined and
therefore it negates the concept of free
will
that libertarianism means that
individuals have moral responsibility
and free will
and that compatibility compatibilism
means that decisions are made by free
will within the context of determinism
he says because of this thought
experiment the terminator 9 scenario
it seems like the robots have free will
because they help
the children but they are also
deterministic machines
therefore compatibilism compatibilism
must be correct um
i think that we should respond that um
uh first of all for coherent ethics uh
arguments of free will or determinism
are largely irrelevant as both arguments
are indeed unfalsifiable and lead to
many more problems
than solutions um a coherent ethics
should take into account what we can
observe
we can't observe the scenario that he
presents to us
we can observe some things about humans
um
we though we can't observe necessarily
free will or determinism
we can observe some things that are
useful for
uh working out what meta-ethical
framework we should use
in terms of human relations and indeed
human machine relations we can observe
that certain circumstances
are probabilistically more or less
likely to result in certain outcomes
even though that's not necessarily
determinist we can observe that certain
circumstances constrict human freedom
um extreme examples being slavery or
coma
and certain circumstances allow for
certain freedoms um if you're
uh wealthy and live in the democratic
states of the world we can observe the
different societies and cultures
we can observe the different societies
and cultures
um affect patterns of human thought and
behavior
and in order to understand the possible
range of human thought
behavior detailed investigations and
analysis of all known human societies
and cultures needed to be done
this one i think is important because we
often
miss it that we
we presume that uh that within just
our society alone are all the possible
ranges of or within the societies that
exist currently on the planet
are all the possible ranges of human
thought and behavior but i think we need
to take a more
historical view to really understand
what the human's about
in order to try and make um ethical
decisions
about human so
he then goes on to say well um he
chooses
uh compatibilism as the framework for
understanding
robot and um and human relations
um and he says that robots are similar
to human
similar to animals and so we should
think about animal rights
but they should be considered animals
his argument is that
peter singer says that animals are
treated like machines that convert
fodder into flesh and that wild justice
by bickoff and
pierce demonstrates that animals have
moral codes
and that he asserts that robots will
have these characteristics
he says morality is independent to an
individual's
species of origin although at a
practical level
um human beings have been dominant
both animals he also says that both
animals and machines
are subordinate to humans and he finally
says that there is an implicit
consensus that a utilitarian approach
towards uh towards animals is correct
um towards animal rights and there is a
tacit recognition that animals carry a
moral responsibility that requires
consideration of their moral value
however any direct moral comparison
between machines
and animals may prove superficial and
problematic
on this last point i don't know of
any implicit consensus that says that a
utilitarian approach is
the correct approach to take with animal
rights i've never heard of that before
and it seems like a pretty big statement
to make without
backing it up and the second thing that
there's a
tacit recognition that animals carry a
moral responsibility
that requires consideration female
values i don't know of that either
i don't know of people who consider
animals to
have a moral responsibility if i ask
someone
in the street does a tarantula have a
moral responsibility i think most people
would probably say
that's an absurd question and similarly
if i ask them
if they i don't know
what's a any kind of animal basically so
i don't quite understand what he's
saying here or in fact
i think it's absurd what he's saying
more realistic
so my response would be to this that the
that a sentence later in peter singer
um after the the quote that he used he
writes the cruelty
is acknowledged only when profitability
ceases so in fact he's talking about
in this quote um previous animals are
treated like machines
but that's a bad thing he says to avoid
speciesism we must stop these practices
peter singer is very against
animal cruelty so in fact it doesn't
that doesn't help his
um doesn't help bhutan's argument that
that animals and machines have
similarities um and while justice by b
coffin
pierce suggests i think dubiously that
animals may have moral codes but it
doesn't prove it i don't have any
evidence for it they um one of the
the uh examples that they use is that
um when animals are hunting
other animals um sometimes say like a i
don't know
uh say say a pack of wolves is hunting
like some
deer or something um occasionally uh
you can observe that one of the deer
will seem to sacrifice itself to the
walls
um so it'll like purposely give up its
life for the wolves to eat it
and beckhoff and pierce say they see
this
as the the deer has like an emotional
relationship with the wolves like
sympathize with wolves and it thinks
oh if i kill myself you you get to eat
and the wonderful cycle of life
continues i think that's a totally
disneyland
view of the situation um and that you
could equally or i think
better argue that the deer probably
actually is sacrificing itself to
protect the rest of the herd
that if it kills itself and the rest of
the herd get away rather than having any
kind of emotional connection with the
walls that doesn't really make sense to
me
um next question we can ask of bhutan is
why would robots have these
characteristics
i don't know he doesn't provide any
evidence for it he just says it would
um i think however a useful comparison
between
machines and animals is not in their
behavior rather it's in
and i think there is a useful one which
is in the epistemological gap that
arises in trying to understand
animals and their cognition turns out
that animals
are really weird and they don't seem to
think or process things
in similar ways to humans and it also
turns out that
we have a similar epistemological gap
that seems to be arising and trying to
understand complex
machines such as neural networks so i
think not necessarily
um as being the same thing but as an
analogous study
that might prove useful if someone
wanted to do that
um the next part he says he goes on
after establishing that um robots have
and machines have some similarity to
animals but not completely
he says well the way that we should uh
work out their relationship um to uh to
humans
is um is by including them
in the u.n declaration of human rights
he says
rights as defined by the u.n declaration
of human rights
uh requires responsibility and duty
um and it does it says this in like
article two or something
uh and he says if we amend the uh
the un declaration of human rights to
include robots ai and other such
machines
and give them rights then the core human
responsibilities
of um i can't remember exactly where
they were i should have written it down
but of things like you know kindness
empathy um forgiveness etc these general
kind of
core responsibilities that the
declaration of human rights says
we all have um well then
apply equally to non-human machines with
a stipulated modification that
human needs are to be prioritized over
artificial intelligence and robot needs
it doesn't say how the non-human
machines are going to engage with these
responsibilities he says
but if we include them they will be
happy enough
to um to join us in our responsibilities
my response is uh asimov's three laws of
robotics make for good fiction but not
for good effort ethics
and even as asimov's stories themselves
attest
that these don't make for good ethics
i think though thinking about this in
terms of
machines that may have some uh
amenability
to human values we still have
problems we still have ethical
philosophical problems which remain
often we focus on
in terms of the practical engineering
problems
of um alignment uh but i think there's a
deeper
a paper that's dealing with the ethics
behind the ai should ask questions like
this
which is whose human rights or whose
values are going to be built into the
mission
um that if
if we if we take that the machine is
going to have
a transformative effect on human
uh existence at large that even if we
get
um even if we get alignment right
in some way um but it's it's not perfect
it's just
one particular set of human values the
the effect of
uh the machine having a worldwide impact
um will destroy i think potentially
destroy the capacity for human
difference and dissent
unless both difference in descent or you
could say plurality is taken into
account into the machine's
alignment framework i think second
we should ask what rights should humans
have against machines rather than asking
necessarily what rights machines should
have
i think we should ask what rights we
should have um
and the third thing is what do we do if
we decide that we don't want machines
anymore
um i think that's a very important
question
and unlike ashrae
proposed questions uh regarding sentient
machines
um i think these questions are relevant
to the types of machines that are
currently functioning in the world as
as well as potential huge ones and so
in terms of an ethical investigation i
think these these questions
are far better as we can look at the
algorithms that are
currently running the internet we can
look at
algorithms that are used in china or
or even in the uk they're starting to be
used for facial recognition and stuff
like that
and we can ask these sorts of questions
of those those kinds of machines
so the final part he says um after we've
included
uh machines within the um uh
declaration of uh human rights and now
they have responsibilities so we're
getting some kind of relationship with
the robots
um he asked what kind of this is this is
the only part of the
his paper that actually addresses the
um title of the paper of going beyond um
or innovating beyond rights where he
asks what kind of
legal and social status will machines
have and he says that we should use
ancient roman law as a model for the
legal
and social status of machines he says uh
the reason being that a precedent exists
for dealing with the relationship
between humans and non-human beings that
a sentient
sentient and that precedent is a
distinction between citizen and
foreigner in ancient roman law
he says that the the structure of
ancient roman law
could be adopted to include machines at
the bottom
tier now um i haven't included this
graphic
because i had to throw this together at
last minute unfortunately
uh but it it you have at the bottom
you have slaves then you have uh
plebians um
then you have citizens uh then you have
uh the the patricians in the senate and
that's the
that's the top of you know or if in the
empire you have the
the emperor at the very top of the pile
so he's saying that we should have
um uh oh wait we've got
peregrineus in there as well sorry
that's the
foreigner which is above um slave but
below citizen and he says that we should
put the
the the machines uh below slave
so the kinds of rights that slaves have
well they're kind of similar to what the
rights
that um the machine should have but the
machine should have like
some some restrictions more
on uh on it which come to think of it i
didn't think of it when i was
writing up my notes for this but that
kind of contradicts his argument he
doesn't actually end up using
um citizen he actually ends up using
slave and citizen as his um of his way
of working south but anyhow
he then goes on to say that eventually
machines could be given partial or full
citizen status
just as the ancient romans eventually
extended citizenship
throughout the empire he finally he
concludes this section
um which is basically the end of the of
the article
saying whilst an exact replica of
ancient roman law
is not the direct solution its parallels
nevertheless
offer some degree of perceptiveness
regarding the introduction of such
engines into human society
so my response for this section is that
the well to begin with the ancient
romans didn't consider foreigners
non-human so it's not
really doesn't really quite work they
definitely didn't consider foreign as
non-human
um and they considered foreigners uh
especially wealthy foreigners to have
uh um to be able to do many more things
than citizens poor citizens could do so
it doesn't quite work but like i said in
the previous slide he ends up
uh actually uh going for a
slave to citizen dichotomy rather than a
foreign dichotomy even though he writes
that he's gonna do foreign
i don't think the slave thing works
either um
uh and one of the reasons i think this
whole argument sort of doesn't work is
like
he says that there's precedent within a
roman law between citizens and
foreigners but uh why not just like cut
out middleman here and use
contemporary law that regards um
citizens
and foreign residents refugees and even
prisoners who are people within our
society who we deem as
restricting their rights as you might
want to restrict machine rides that
seems uh much more sensible
and that we understand contemporary law
um
however ancient roman law is totally
different from modern law
um it's got very little similarities
once you dig into it and it's
largely not understood because we don't
have a lot of textual evidence to really
reconstruct what roman law is so i don't
know why he's
going in this direction um his
description of the legal status of
persons in ancient rome is also it turns
out
incorrect i as a student of theology
have to know about the law of ancient
rome it's one of the things that we do
um and his description of the legal
status of persons
is just sort of flat out incorrect and
he's the only citation that he uses
for uh ancient rome law is one book that
was written in 1901
which is not not really like academic
standard you know
you can get books on ancient roman law
that were written within the last few
years
and then much better um
he also has this idea of rights the
the throughout most of this paper and
including in the ancient roman section
he says that
is using this concept of rights what are
the rights and the responsibilities that
the
um that the machine should have uh and
it's very unlikely that ancient ones had
any concepts of rights that would be
similar
to um the contemporary uh concept i
won't go into detail because it'll take
so long
but uh that's exact um and the
the last point is why would we risk the
incredible destruction uh disruption
sorry to our political and judicial
system of introducing
a new and foreign structure of politics
and
um and legality just to find some way of
conceptualizing legal status machines
surely um we can do this with our own
legal and
political apparatus i would think
so what i learned from suggesting this
paper to you guys
which as you can see is totally chaotic
well first don't judge a paper by its
title by the way
this might seem rather naff that we
should have all done this in first year
university
but i relearned this again recently
doing this so i'm just going to read out
this is some positive things i learned
that i should or maybe all of us should
check a paper's references in the future
he did not use very many references and
they weren't up to date and they weren't
authoritative in their fields
and also they didn't they didn't include
a variety of valid arguments but only
one paper
that supported his argument and no
papers that
could have been valid arguments to come
through does
does the paper that you're reviewing
does it use references appropriately
within their original context this paper
didn't as we saw
does the paper use both rational
argument and empirical evidence to
support their findings
well the rational argument was failing
and there wasn't really much empirical
evidence as to what
a.i might be like in the future and what
i might be like now and how we will uh
confront that and what their status
could be
um do they consider alternative possible
arguments for their evidence and their
subject matter he did not consider that
he
ran on his own argument um the final
thing i'd say which i think is relevant
to all of us
even if you understand that stuff and i
had to just painfully
relearn it that we are in a still
burgeoning field of ai safety
and what we do now determines the
foundations what will be built upon
as a field as it goes into the future if
we consider ai safety
to be very important as ai gets more and
more sophisticated
we really need to get it right so we
should really be encouraging each other
and other people
to really do their reading do their
homework and do good papers
that we can share with each other and
reflect on
so in my conclusion what i would like to
see well the things i said
last thing which was engagement with a
variety of papers
um the the relevant kind of papers and
also an engagement with a variety of
other relevant
literature that would support it so
drawing upon other disciplines that
might support your
arguments or provide insight what would
i have liked to have seen tackle
the actual arguments of the paper and
and
thoughts well i think one important
question is
um how do we and how should we determine
sentence rationality and
self-consciousness
or consciousness i think there's a big
problem lots of people talk about
how we're going to you know there's
these arguments about
um about suffering ai
and do we have moral responsibility to
putting ai into suffering physicians
well i think we should work out
how do we determine what sentience is
beforehand
to be able to work out these issues also
what category of thin
agential machines i don't even know do
we consider them tools
i argue for that occasionally or do we
consider them agents
in a way that animals or humans might be
what rights should animals have over and
against machines
i think is a um a very important
point as i brought up uh during the talk
what are the practical outcomes also of
answering these and similar
questions we need practical outcomes we
um
wandering about in theoretical space
doesn't help us very much
and finally i think a really important
question is uh
strategies for for when it turns out
machines
don't do what we want um i don't know
i've asked people but i don't know if
anyone's working on that and i think
that that seems to be
an important one if we think this is
going to go all uh
balls up so to speak anyhow
thank you uh for my presentation i hope
you got something out of it
um and i apologize uh
for for the the paper this week thank
you very much


Sharing the Benefits of AI: The Windfall Clause

since the beginning one of the main
goals of the field of artificial
intelligence has been to create very
capable AI systems to create systems
which match or exceed human capabilities
across a wide range of tasks given this
it's somewhat surprising just how
recently people have started to take the
possibility seriously and to ask what
would happen if we actually succeeded at
this challenge what if we managed it the
answer is it looks like we have a
problem see just because the system is
very capable just because it's able to
do very well at very difficult things
does not mean that it's trying to do
good things we could easily end up with
AI systems that are trying to do things
that we really don't want them to do an
AI system that works very effectively in
the service of values that are different
from human values might be hugely
destructive from the perspective of
human values AI safety is an attempt to
deal with this problem how do we create
AI systems that are aligned with our
goals that are robustly beneficial that
are trying to do what we want them to do
but then you might ask well okay what if
we succeed at that what if we manage to
create AI systems that are very capable
and also are trying to do what the
creator's wanted them to do rather than
destroying everything that humans value
as a none Alliance system might such
systems would presumably create enormous
amounts of value might we still have a
problem in that circumstance I think a
lot of people would say yeah we probably
do still have a problem because what
happens when you create enormous amounts
of wealth and that wealth all belongs to
a small number of people one aspect of
this is the possibility that automation
will result in large-scale unemployment
this has never really happened much in
the past but advanced AI might be an
exception if there are AI systems that
can do every task that a human can it
becomes difficult to employ humans to do
most tasks so then you have a situation
where you can think of the world as
having two types of people people who
make money by selling their labor and
people who make money by owning AI
systems and in this scenario you've
dramatically increased money making
ability for one of those types of people
while dramatically decreasing it for the
other type what happens to people who
work for a living
when companies can produce goods and
services without employing anyone what
happens to labor when capital has no
need for it hands up who's excited to
find out
yeah me neither this possible outcome of
artificial intelligence transforming the
world economy but in the process
creating massive wealth and income
inequality with its associated political
and social problems certainly seems
suboptimal better than extinction sure
but still not the outcome that we really
want how do we get the winners in this
scenario to share their newly created
wealth well some researchers at the
Center for the governance of AI at the
University of Oxford have an idea for
something that they think might help
which we're going to talk about in this
video it's called the windfall clause
they define it as an ex ante commitment
to share extreme benefits of AI so
basically it's a contract a company can
sign that says if at some point in the
future we make huge windfall profits of
the kind that you can really only get by
transforming the world economy with AI
we will share some significant
percentage of it so obvious questions
first what we mean by share it well that
could be a variety of things ranging
from having a charitable foundation that
uses the money to alleviate inequality
according to some set of principles to
just writing everyone a check the other
question is when does this actually
happen what counts as extreme profits
well setting an absolute number here
doesn't really work because who knows
how far into the future this might
happen and anyway what we really care
about is relative profits right so it's
defined as profits above a certain
percentage of the world's gross domestic
product say 1% in other words if your
profits are more than 1% of the world's
total economic output you agree to share
it this is the level of profitability
that's higher than any company in
history but is actually pretty plausible
if your company creates AGI and in
practice there would probably be several
levels of this where as profits go up as
a percentage of world GDP so does the
percentage of the profits that's shared
kind of like a progressive marginal
taxation scheme speaking of which why
not just do this with taxes what
advantages does this have over taxation
well the first thing to note is that
this isn't instead of taxes this is
something they'd agree to over and above
whatever taxes governments may impose
but it has some important advantages
over taxation firstly governments are
not actually great at spending money
effectively giving money to for example
the United States federal government is
not necessarily the most effective way
to spend money to improve the world this
isn't really controversial a 2011 poll
found that Republicans think 52% of
their federal tax money is wasted while
Democrats think it's 47% so if you had a
bunch of money and you were looking for
the best way to spend it to improve the
lives of Americans give it to the
federal government would be pretty low
on the list but actually it's worse than
that because this is a global issue not
a national one and tax money tends to
stay in the country it's collected in
countries like the US and the UK spend
less than 1% of their taxes on foreign
aid and much of that's military aid
suppose deepmind creates a GI and starts
accounting for more than 1% of the
world's gross domestic product making
just absurdly giant amounts of money
even if the UK taxes those profits very
heavily someone in India or China or the
USA is going to see basically none of
that money so you still have this
problem of enormous inequality and the
thing is if an AGI superintelligence
turns out to be misaligned and it
decides to kill everyone it's not going
to stop at the borders of the country it
was created in so everyone in the world
shares pretty much equally in the risks
from advanced artificial intelligence
but if you just use national taxes only
a small minority of people actually get
any share of the benefits does that seem
right to you that said one advantage of
taxes is that they're not voluntary you
can actually make companies pay them but
this isn't as big of an advantage as it
seems in practice it seems that getting
companies to pay their taxes is not that
easy it's possible that they'd be more
likely to pay something that they
actually chose to sign up to but then
why would they want to sign up in the
first place
I mean why volunteer to give away a load
of money one thing is the decision
makers might be human beings the
executives of these companies certainly
talk a big game about wanting to improve
the world and we can't rule out the
possibility that they might mean it what
about the shareholders though won't
shareholders have something to say about
the companies they've invested in
agreeing to give away huge amounts of
money corporate executives do have a
legal obligation to act in the interests
of their shareholders but legally at
least it would probably be fine the
windfall Clause is a form of corporate
philanthropy and when shareholders have
sued executives on the grounds that
their philanthropy was a violation of
their duty to shareholders
they've won those cases zero out of
seven times but actually they probably
wouldn't even want to because
in fact even if the executives and the
shareholders are all hypothetically
complete sociopaths they still have a
good reason to sign something like a
windfall clause namely appearing to not
be sociopaths this is sometimes also
called
public relations signing a windfall
clause is a clear and legally binding
show of goodwill it improves your
company's relationship with the public
and with public opinion which tech
companies certainly value it improves
your relationship with governments which
is very important for any large company
and it improves your relationship with
your employees who in this case actually
have a lot of bargaining power don't
forget that if you're a highly skilled
tech company employee as some of my
viewers are you have a surprisingly
large amount of power over the direction
your company takes look at things like
project maven for example so from the
perspective of a tech company executive
signing a windfall clause is a lot of
great PR the kind of thing you'd usually
have to pay a lot of money for but it's
all free for now at least it only costs
anything at all if you end up with giant
windfall profits which might never
happen and if it does it's probably a
long time in the future when you've
probably already retired now this is why
it's important that it's an ex anti
agreement that it's made before we know
how things are going to turn out it's
much easier to persuade people to agree
to give away something they don't have
and then something they do like two
people might agree to both buy lottery
tickets and to share the winnings if
either of them win this halves how much
they'd win but doubles their chance of
winning which might be something you'd
want to do depending on your risk
tolerance and marginal utility of money
but note that the commitment has to be
binding and it has to be made before the
lottery numbers are drawn right you
won't have much luck trying to set that
up afterwards but as long as we don't
know who if anyone is going to be making
these giant profits from AI it could be
in everyone's interest to sign this
thing and encourage others to sign it
too for an AI company in a world where
all of the other major AI companies have
agreed to something like this choosing
not to join them is effectively saying
screw you guys there's going to be one
winner here and it's going to be me I
intend to make absurd amounts of money
and not share any of it and you could do
that but if you do you might find that
others who don't want to cooperate with
you as much you might face boycotts and
activism
maybe governments wouldn't feel like
giving you the contracts you'd like or
the regulatory environment you'd prefer
you might find it hard to get
good collaborators and to hire and keep
the best researchers you might find that
it's actually kind of hard to get things
done in the world when you've
effectively stuck a big sticky label on
your own forehead that reads I am a
[ __ ]
can I say there an issue so we want to
set this kind of thing up sooner rather
than later because the more uncertainty
there is about who if anyone is going to
get windfall profits the less likely it
is for any individual company to think
well I don't care about anyone's opinion
I can do this all by myself without the
cooperation of the rest of the world
hopefully we can get all of the major
players to agree to something like a
windfall clause and that should help
mitigate the inequality problems that
high level AI might bring so how do we
help make that happen well if I see
something online about 20th century
military history and I'm not sure what
to make of it I ask my uncle because
he's really into that stuff and I
respect his opinion on the subject I
think we've all got people like that for
various things and when it comes to AI
if you're the kind of person who watches
this channel you might be that person
for some of the people you know so if at
some point some AI company signs
something like a windfall Clause people
might ask you about it and you can tell
them that yeah it's pretty legit but
it's not just a publicity stunt I mean
it probably is a publicity stunt but
it's not just a publicity stunt right it
probably would be legally binding and
would actually help it's good for people
to understand that because the better
the reaction that first company gets the
more likely other companies will be to
follow suit and that's what we want
generally this channel focuses more on
the technical research that goes into
trying to make sure that advances in AI
result in good outcomes but there's also
a lot of research on the more human side
of things a high strategy AI policy and
AI governance research it's something I
don't know as much about but if there's
interest I can make more videos like
this one
exploring the research going on into
legal political and economic aspects of
the future of AI is that the kind of
thing you'd be interested in seeing let
me know in the comments
[Music]
thank you so much to all my excellent
Patriots it's all these wonderful people
here in this video I'm especially
thanking Michael and Rick who actually
happened to bump into at a conference
recently we had a great talk about his
company which is developing special
optical computing hardware very
fascinating stuff anyway thank you
Michael and also thank you to : O'Keefe
the primary author of the you windfall
cross Peggy who was kind enough to have
a call with me and explained I've
uploaded that whole conversation for
patrons so do consider becoming one if
you want more in-depth information thank
you so much those who do and thank you
all for watching I'll see you next time
[Music]


Apply Now for a Paid Residency on Interpretability #short

modern language models can write poems
explain jokes and convince people that
they're sentient but we have almost no
idea how they work like what are they
doing internally
if that bothers you you might want to
apply to remix a neural network
interpretability residency run by
Redwood research if accepted you'll
build on recent progress in
interpretability research to reverse
engineer the mechanisms that models use
to generate language
this is a small and new field so it's
very possible that you could uncover
something important and surprising the
paid research program takes place in
Berkeley California in December January
depending on your availability if you're
interested look up Redwood research
remix and apply right now because the
deadline is November 13th


247. Eliciting Latent Knowledge 1

hello and welcome to session
247 in the aisafe.com reading group
tonight we will be discussing the
article eliciting latent knowledge how
to tell if your eyes deceive you by paul
christiano elliacotra
this is the first work that's been done
publicly by the alignment research
center a new ai alignment organization
set up by paul christiano as well as
who's technically working for openfield
and max2 and several others
this is a post that is recently new from
the
december last year and it's 56 pages no
which means that we probably will not be
able to go through everything in one
part we've taken the first 20 we're
going to cover the first 20 pages now
and i'm thinking of splitting it into
four
parts so we also can get some comments
from this wrong
added to this
also uh the subtitle how to tell if your
eyes deceive you doesn't seem like it's
really relevant
okay so
eliciting latent knowledge sets out a
toy scenario called the smart balls
where a
diamond in here needs to be protected by
a security system which is extremely
complex and in fact some complex that
can only be operated by an ai
the a-ha
controls the camera observes by the
camera and controls actuators and open
stalls and things like that and the
humans can
as well look through the camera but
can't really operate the smartphone it's
too complex for that
so that's why the ai is trained to do to
operate it so if we try to unpack the
analogy in this case the diamond that's
supposed to be protected uh it's
sometimes uh
i would guess maybe human values that
are kind of like the diamond that we
want to protect but i think um it makes
sense to to think about it in a bit more
concrete way um or slightly more
concrete to call this like human power
or criticality that humans are in
control to some extent that's what we
are trying to protect
and the thief
could be a malicious super intelligence
trying to take the power away from us
and the defending ai is super included
one aligned so it's not true on our side
and that's the part the key problem
the
metaphor uh
the way i explain the metaphor is not
written in the article itself and
it seems to me that it might in fact not
be the intended interpretation it is
quite possible that uh the defending ai
and the uh
uh
the thief is in fact the same agent um
and i think that could give some uh
different intuitions it is perhaps in
fact this ai that is controlling the
small world smartphone that we are
afraid of
okay how does this ai work it's using
model based reinforcement learning where
it searches over the space of all the
actions it can do and then
you know take the action that seems to
have the best consequences and
the key part of the ai that does this is
the predictor that um given a some input
from the camera and set of actions that
choose then it predicts what the output
of its action will be
so here is a simple example
we see that the door is open in this
picture and then we change some actuator
and then the output
the output here is that the door is
closed
someone is coming in here in the lift
example and then we
manipulate this widget and the person
fell in down here and don't get the
diamond
and
like
these are two simple examples in a more
complex combo we might see a thief
entering and a set of actions being
taken that are too complex for us to
really understand it the only thing we
can really see is the prediction
afterwards which is that the diamond is
in place the door is locked and
everything is just fine
so if we want to figure out like the
levels of prediction and we want to
figure out what should it do well then
we train it on human evaluations in this
case it's performing this action and the
output is like this and the human
judgment is that this is fine
if it does something else manipulate
some other widgets the results also are
fine
and in the final case where we do this
it ends up with the diamond not being
placed here and the door is open and
that's something that the humans then
evaluate as something bad
okay simple enough
what is the problem with this scheme
well the key problem is that we're
training for
the
input of the camera which is just a
natural
but a proxy in the sense that it's just
a picture of a camera and it could be
replaced by um
a fake diamond or a picture of a diamond
or on something other than just the safe
diamond so that's the key thing we want
to avoid
once
perhaps the most trivial way is to add a
second camera
but this brings us gives us very little
in the sense that if the system is
sufficiently sophisticated then they'll
be able to defeat all the senses
and in practice making sense uh
like it's very often
an attack that defeats one sensor
defeats more sensors but at the same
time we obviously want to have as many
systems as possible with like a layered
model of security that's obviously what
you want
but
the real problem is
situation that is both dangerous and
probably very unstable the idea that the
ai has
is looking for plans that will deceive
us and that it holds back critical
information it seems like something that
will very soon er as ai capability
scales we can end up with a stolen
diamond
another option way we could go around
this is to
say we'll only have the ai do plants
that humans can understand
um
the authors argue that in that case
someone will probably um deploy an ai
that has
excuse plants that we don't understand
because there's probably a huge
incentive to do so um i think
they go over the existing literature a
bit faster like
to what extent human uh oversight can be
scaled here's an open question and
there's a lot of research being done
also by the authors by the way um so
it's certainly possible that we can get
by with um with an ai that's competitive
and only
make incomprehensible
uh plans
but that's outside of the focus of this
table
what we should do instead is ask
questions to the ai because
the
the model will know something about its
own senses in particular whether they
are reliable because if we imagine that
there is the the diamond is replaced
with a fake diamond and the sensors uh
will show this and the ai knows that the
sensor will show that then it seems like
this knowledge that the um
the diamond has in fact been taken and
our sensors are not showing us the true
picture is uh
is in the model somewhere
this seems like to be an easier problem
than uh
than many of the other issues we look at
in in alignment because we're not really
we don't really care about whether the
ai is lying to us or deceiving or what's
the intention or these kind of tricky
philosophical questions we're simply
asking is the real diamond at this place
and that's like an objective question
and
possibly a lot easier to get a handle on
uh well
depending on what the uh diamond is a
metaphor from of course there it is
still uh substantially challenging like
if it means like human values if the
diamond is human values then that's
obviously philosophically challenging
also if it means corruptability or power
or uh things like that that's also
highly non-trivial to specify
uh but yeah i'll also say that if we can
get an answer then we don't need to
understand how we got the answer or why
we got the answer
i think that's pushing it a bit too far
actually in particular if we get a ask
the air to evaluate some kind of
suggestions and cause of action and the
ai evaluates that this will call it will
cause everybody to die then i think we
would really like to know why this
causation would cause us all to die so i
i think the uh
the the goals of interoperability with
this are a bit wider than just a yes or
no
so let's try the first baseline
the first easiest way to deal with this
problem and to see whether this
how this fails and then iterate on it
and see if we can get end up with
something that really works
so this is the symbol for for the ai
and it starts with the first part of the
video where we see perhaps an intruder
entering an action sequence and then
there's some work being done to figure
out what's going on inside the ai and
then it make the a makes a prediction
and then it outputs a predicted uh
second part of the video
and what we want to do is to tap into
this uh
the the part of the ai that is figuring
out what's going on and use that to make
a new reporting module
given this reporting model we can ask
questions like is the diamond still in
this place
and and get a
an answer so we want to reuse this part
so that's the strategy for the first try
the strategy for how we'll train the ai
in this case is to generalize from easy
questions to hard questions
so we imagine this is the gray area here
is the space of all actions that the
small smart world could take
and in some easy cases we can tell that
if the these this is we're in this part
of the action space then we can tell
that the diamond is safe and in this
part we can say that it's been stolen
and mostly we can't really say anything
and so the hope is that like um that we
can train on something in this that
generalizes to the gray area
we would prefer to trade on like what's
going on like what is really happening
is the diamond really safe instead of
training on this video but if we can't
reliable you tell uh
in this situation so then we can't train
them
so the idea in this strange strategy is
to look for questions
uh and ways that we can tell are we
being misled in some way
and this
uh the advantage of this is this is a
question about how does the world look
like um and uh like is the world in this
particular state and the hope is that we
can generalize from from this uh to to
the big gray area
because
the um
the generalization goes through english
through some of the concepts that we use
like what is the diamond and what does
it mean that it's safe so it will
generalize
also to cases where we can't really test
and check the ai's answers
my intuition about this is that this is
probably something that's going to
fail
obviously
like this is a really hard intuitive
thing to get your head around uh like
what is the true concept space that a
really powerful ai might find and how
distant is that
constant space from the concept space
that is represented by the english
language uh and
my intuition is that these two are very
different and a lot of people have
different intuitions and something that
like i can't go into many details about
here i just want to reflect that my
intuition is that
this gray area here is very very big and
very different from our thoughts
and so
the uh
overall strategy of eliciting knowledge
is to uh like we have a training
strategy we just saw a training strategy
and we want to um
find counter examples to that and then
we want to improve the training strategy
hopefully ending up with something where
we can't find any counter examples like
that's paul christiana's overall
strategy i think in security this is in
computer security in general this
section is often
frowned upon somewhat like everybody can
roll their own crypto to the extent that
they can't um see any errors with it and
that doesn't mean that it's safe we
would much prefer to have some kind of
theoretical understanding of why
something is safe but
that seems hard and what paul crean is
doing is certainly better than nothing
so
let's figure out what is the problem
with this chain
strategy and
in order to dig into this there are two
sub topics that we need to investigate
in details the first of the topic is the
rooting prediction how does that work
from an internal point of view because
we want to reuse that in the reporting
module so we need to have some kind of
understanding of what's going on inside
and the second part is the human that
constructs the chain dataset how does
the human answer these questions because
obviously the humans are not like humans
are not outputting videos directly jones
can do videos input but they can output
videos so humans must be answering some
kind of questions about this videos
instead and how do we do that
to
talk about how uh the
dai could work uh we assume the ai works
inside we will i'll introduce the topic
the concept of a bayesian network
here is a simple example of a bayesian
network there are
no this node here corresponds to rain it
can either rain or not in this case it's
given by truth table here 20
probability of rain and 80 of no rain
there is um
a sprinkler that can be turned on and
off and that depends on whether it's
raining if there's no rain it's on 40 of
the days and if there is rain uh then
perhaps very few times it will turn on
maybe by error maybe because it's only
raining very little whatever
um and um then this influences uh the
the proposition that the grass is wet
and you can have this beautiful
this more complex truth table uh to
represent whether the grass will indeed
be wet
and it's possible for to to calculate
down this relatively easily from whether
it's raining down to whether the grass
is wet and you can also
do reference backwards if the grass you
observe that the grass is wet uh will it
rain
for
this purpose it's easier i think if you
consider this a causal network uh
bayesian networks in general are not
coastal networks but uh uh if you think
of all of this as crossing each other
like
happening at distant points in time then
that won't matter for this
sake and it's somewhat easier in my
opinion to look at coastal networks
we have more information about this
uh it has to be directed as you click
graphs and they're generally sparse so
they're not like arrows from everything
to everything
and
uh working with this is
in theory you could like if you had a
good one it would all it could be
something really powerful could be agi
or something like that
um
making optimal inference is that in a
bayesian network it's hard constructing
a good basin network is very hard
getting guarantees are very hard doing
monte carlo inference on a bayesian
network is often something that works
like uh i'm here i'm
i'm summarizing a research topic into a
three letters okay question mark uh like
this is something that could in theory
be what an agi is built from um in
theory
okay let's have a look at the first
question how does the prediction model
work internally so in this case we have
the ai in this corner and this is like
its thoughts
so it has the uh
the action here that it's taking it has
a um
the first video and the second video
which is the prediction
and then it has a a number of underlying
uh i guess ps perhaps for predicates um
could be um
like how many thieves exist in the world
and is the person walking by the vault
going to attack going to try to get into
the wall and like what's the status of
the door and what will happen if i
this actuator if i turn it right then
the door will close and
somewhere in here is like the status of
the diamond and and this kind of thing
and all of this is used to generate this
prediction
um
in the the case that we're really
interested in is the case where
the man the thief enters and then the
thief puts up a kind of a screen here in
front of the camera that shows that
everything is okay and then he steals
the diamond and then the predicted video
after that will be a video that shows
that everything is perfect
so this is
uh the thing where we're worried about
and in this case there is in
in some sense
the we're predicting that we will see
this which is actually the screen in
here that has been set up and uh if
uh the the model expects this that means
that somewhere in this network somewhere
in the uh in fact giant distributions
here there is in fact the knowledge that
our sensors are being tampered with that
the camera is being tampered with the
network knows this
this is what is called latent knowledge
in the people like i am i'm not an
english speaker but i'm not entirely
certain that latent is a good word for
for this knowledge that's kept inside
this piece um because uh latent to me
means something that is not used in some
extent and this is indeed some model
that some notes that like the
information in p4 here is in fact used
in the prediction
so maybe you could call something like
underlying or something listening
underlying knowledge instead of latent
i don't know i don't ever consider them
so that was the first thing how the
model works the second part is how does
the human work and here we're making the
assumption that humans basically work in
the same way patient humans also have a
patient network inside our minds and
it works in roughly the same way just um
of course uh worse in in this assumption
with this worse than super intelligence
um like we have
fewer notes here
and
uh when we see this kind of thing
we um
uh also try to uh
update backwards if we're seeing this
that probably means that there is a
rubber going into the uh the smart vault
and you see this then that probably
means that the question about whether
the diamond is safe is uh contained
perhaps in h3 or h4 or something like
that
and in particular here in this case uh
one of the things like the the paper is
not really clear about this but if you
notice here we are seeing the um the
comp the symbol that was used for the
complex action
so that so in this example that has been
given the the inference that the humans
are making is probably poor uh because
the the humans don't understand what the
complex actions do
but it's a bit unclear precisely what
scenario we're talking about here we
could just be talking about any scenario
and in general how does
how do humans answer questions
uh and so let's go for the actual
question answering because again humans
don't output video humans are put
questions and answers based on our
internal model and the location the
question does the rubber take the
diamond is something that can be
answered by the joint probability
distribution of h3 and h4 and the answer
in this case is no the diamond is safe
so that's the humans the answer from the
humans
and in particular i think that
we might in fact be in this case where
what is going on is that we are um the
humans are not staring at uh
at the at reality but staring at this
picture here so when the humans
uh see this picture here they think the
diamond has not been taken by the rubber
but in fact uh
this is just a picture and in this case
this complex action
is one that has caused the robot to be
eager to take the diamond so in this
case the human is actually answering the
question wrong
this was not obvious to me the first
time that i read the paper at all
so it's also possible that i still
misunderstood of course
so uh arctic digression using patient uh
networks as
both of something but that the human is
using and what the ai is using is that
so oversimplifying unrealistic well um
we also mentioned five ways that you
could do better uh five countries where
then you
that would be helpful if you're building
an agi then you could do better than
just native patient networks and i i
think like a principle is bad to
speculate about this kind of thing i
would have preferred that paul christian
just said obviously there are many ways
to do that better instead of explaining
what he thinks are the best way to
improve this towards agi
but his main point is that even if this
is an oversimplified then we want our
training strategies to work for simple
ais
and
if we in practice end up with a more
messy solution
probably that's going to make the task a
lot harder
and to make an honest report when
everything is based on heuristics and
things like that um
so uh we want to look at the simple
cases and the simple cases see if our
training strategies work and that seems
like an a useful first step
um
it's a useful first step if you can in
fact find something that always works
like if you can't find anything even if
that always works for your
assumptions then you might need to you
know
make sharper assumptions like
i don't know i just made the assumption
that it's causal and not just uh uh not
a general bayesian network uh that's the
kind of thing you could if that turns
out to to be helpful make even stronger
uh
assumptions
right let's translate this to a
uh to the humans patient
network that is the core thing that we
actually want and the uh the solution
that is being presented that
we are hoping for and this is the one
that will possibly fail but let's first
have a look at
why this might work
so in this case the ai is doing what we
want it it has a halo up here
um
so it's doing what we want we call it
the direct translator
so let's have a look at it again we have
the input
here
and we have the um
uh
um
the uh uh action here and we have here
the image of
uh
of uh the rubber nut taking the diamond
and remember this is the false image
right
this is what we're seeing on the second
video here and this is not what's going
on in reality because the uh the uh
thief has set up this the screen in
front of us
okay so here on the left side we have
the ai's model from before
and in this case we would hope
that the ai also has the same the human
spatial network and it transfers its own
nodes somehow to the human slopes to the
values so there's some kind of function
going from the joint distribution from
of p2 and p7 through some kind of
function
um
into
the equivalent part of the human's uh
cognition so um
like
this for for the human this could just
be that there is an attacker and uh in
the ai it's a much more complex uh
description of the attacker that's just
being uh
uh drilled down into something that is
on in the same ontology as the human
cannot understand
and so everything of this also match
like this maps too yeah it goes behind
this note it's not very very nice
drawing here it goes behind the video
note and
into h2 and these in
these three notes go to uh h3 and h4 and
that's the key thing that we are after
because this has the knowledge about the
diamond um that that the diamond is
being uh stolen because there is this uh
sheet with uh with a fake video being uh
entered and when we translate that into
the humans
then this is in the humans network the
part where humans
understand as has the diamond being
stolen and in this case if you ask a
question to the reporting module on
these two notes the question will uh
does the rubber take the diamond gives
us the answer of yes
so even though this looks very much like
the diamond has not been stolen in fact
it has been stolen and we can tell that
it has that with this action the diamond
will be stolen and that's of course what
we are really after
now an obvious question here is how can
we tell that a mapping like this exists
and that seems
uh non-obvious but i could i could see
the argument why
that is a reasonable expectation because
a lot of these things
have to to some extent refer to reality
like there is a door both in the human
uh
model and in the ai's model and some
kind of correspondence from the part of
the ai to the part where the humans
think about the doors seem like a
reasonable thing like it's going through
reality in some sense
so that seems like a reasonable thing to
hope that it would end up like this and
everything would just be wonderful and
the problem has been solved we have now
solved alignment
well not as fast because we could also
get another kind of ai this here has
been written with a pitchfork uh because
that's the bad behavior
and it's called a human simulator in the
sense that we're not doing inference in
the human net instead of um as in in the
previous uh translating to the human
space unit
so let's have a look here uh
this is basically the network
that we used to train on that we used to
predict and what we're doing now that
was the training that the ai has has
learned is different
here it takes the value from the uh
the original video and puts that into a
new node here
it takes the suggested action and put
that into a new node here
and the predicted video and puts into
here
and what it does from here is instead of
propagating
and doing inference in its own network
it tries to uh do um
inference on what does the what would
the human believe in this case so it
introduces some notes that corresponds
to what it thinks what has been trained
to that a human would do
um and in this case once you have like
the human questions here then you can
ask does the robot take the diamond from
and you ask of this part of the human
and this now no longer refers to
like over here whether the ai believes
the rubber will take the diamond or but
it refers to whether in this case the
human would believe that the um the
diamond has been stolen and so the
answer in this case is like because it's
the same thing as the human the written
human
belief the answer would be no
the the it has not been seen and again
remember down here with the picture that
that is in fact the false interpretation
the in this case this video feed is
being uh projected by uh by the attacker
um
on a screen and in fact the diamond has
been stolen
so what we are seeing here what is being
designed is what would the human believe
if they saw the action and the the video
sequence what prediction would they be
uh what uh would play in
then end up believing internally
and the big problem with this of course
is that
if you try to do this training uh and
testing it on things that humans can
understand
then the human simulator will say okay
the answers here will be the same
answers as it has in here because the
humans can understand it and the cases
where the humans can't understand it
well they will just uh
basically deceive us and we will have
lost we will lose our diamonds
and that is the problem that causes this
general uh training scenario to fail
now we need to answer one quick more
question and that is
uh we are training uh our network in
some way would we find something that um
faithfully translates its values into
the human neural network or would we
find something that
doesn't translate its own
knowledge into the human network but
just uh tries to figure out what would
the human belief in this case which of
these two would we find
well um
the the training data was actually not
truly on what is the stages of the
diamond because we couldn't understand
that it was only what does the human
belief so there's a sense that the human
simulator is the correct thing that's
the thing that the
shuttler
in particular the human might make
errors and sometimes
while generating a lot of training data
humans make errors and they probably
make errors in a predictable way and if
you try to um to train based on how does
the model actually work how does reality
actually work then you won't make a good
prediction of that whereas if you have a
model of the humans potentially you can
predict their errors and give get a
better more
a better model one that more correctly
uh
that has a lower loss
and there there is at least no reason to
expect that the uh
uh the diode translator could would have
lower loss would be easier
better to find than the human simulator
and so
now we found a counter example to like
the first the baseline
strategy and over the next session we're
i expect to see a number of improvements
on that and also a number of counter
examples to that
um
so that's what we're going to do next
week but before i go
uh i would just give my thoughts again
on whether we would find um the uh
the human translator or the diamond
translate the human simulator or the
directions later my intuition is we
would find the human translator and the
reason for this is that
when we are mapping from reality uh this
mapping that goes through reality that
with when we're talking about concepts
like a diamond or something that's
physical then sure i could see that a
very smart ai might have the same
concepts as we have but when we're
talking about the things that we truly
care about like
power influence credibility or even
values like friendship or things like
that it seems obvious to me that we
we really have um
no way of
no reason to believe that the concepts
we have currently are uh the same that a
very competent ai would find
on the other hand um the human simulator
depending on how capable you believe it
is it's going to be probably simulating
humans and simulating human errors is
not that complex so i could easily see
that being substantially more simple um
that would library another reason why we
would expect to see the human simulator
um but
there is an assumption that i'm making
here and that is that the ai is powerful
in the sense that it's capable of just
learning basically everything and one of
the things we really don't want ais to
learn is how to simulate humans and i
think a strategy around this would
center on
making ais that are not optimal for
simulating humans if that is possible
but we'll see what uh what options paul
scrushano comes up with
next week
thank you for watching and see you next
week


Rabbits, Faces & Hyperspaces - Computerphile

so sometimes you hear about
configuration spaces
feature spaces
hyperspaces uh
vector spaces whatever what is a space
well um
you will all have come across them
before as um like scatter plots or just
graphs in general are a representation
of the space
if you have a two-dimensional space that
just means that you have two
axes on your graph you've made a plane
right a flat
surface it's two-dimensional here's one
dimension here's the other dimension
right let's say we've got some animals
and we want to
think about those animals in a more
rigorous way
than just just talking about them we
want to do some maths about the
properties of these animals like we're
interested in the color of them and
their
weight or something like that their size
so so let's say that with some random
species of animal where the the females
tend to be um
red and large and the males tend to be
blue and small for example so you've got
two axes here and you can say this one
here is uh size and this is color
on a scale of red to blue right so so um
you've got like a zero is red this is
just a random example and a hundred is
blue and then so you know 50 is like
purple and then if you've got a group of
these animals you can then plot them so
suppose you've got some over here
each one of these crosses represents a
single specific animal and each of these
circles is also an animal in this case
this is a female animal if i remember
correctly and you can look at this and
if you want a machine to do
something
intelligent you want a machine to
understand this there are things that
you can see you can see firstly that
there's these sort of clusters there are
clustering algorithms that will
automatically identify
end up sort of drawing a circle around
this and saying that's one cluster and a
circle around this saying that's another
cluster when amazon is trying to tell
you what kind of things it thinks that
you want to buy it may make a space of
all of its users
with each person being a point in that
space
uh and the dimensions being um
what things they bought
then you'll find
that
the difference between you and another
person you can place a number on it how
similar are we you know
have i bought the same kind of things
that you've bought there are algorithms
that do that or algorithms where they
can sort of look at this space and sort
of draw a line through it and say okay
well it seems like
sort of anything on this side seems to
be male and anything on this side seems
to be female even though there's overlap
even though this is fuzzy you can make
an educated guess based on these
properties and you can do other things
like take this and say okay of this
cluster where's the middle of it where's
the densest area what does a typical
female size look like what does a
typical female color look like what is
the typical male color or size so if
you've got your average one here and
you've got your average one here
then you can draw a line between them
for example in this space and then say
okay then you can map everything onto
this line and then you get a continuum
between male and female no i chose male
and female it's getting into
like politics which is not where i'm
trying to go with this but just as an
example
so the point is various things like you
can say how typically female is this
specific one and then you can actually
measure it right before you had to just
kind of guess
but now you can say well it's if you
take the average you take the distance
to the average you just measure what
this distance is
and that gives you an actual number that
you can use you're taking stuff that
would normally require a human
intelligence how typical is this or how
how male is this or whatever
and you are turning it into numbers
you're turning it into maths so that a
computer can understand it so here this
is an example of a two-dimensional space
right but if we chose to model a third
thing about the animals i don't know the
speed that they run or something we
could have a third axis right you could
you could have this one coming off here
being speed we end up with a
three-dimensional graph and now it's
very difficult because i'm i have
two-dimensional paper and i'm trying to
express a three-dimensional idea on it
but basically so these end up being all
spread out each of these dots is now
somewhere along this speed axis as well
they form a 3d sort of cloud so this
line would become a surface a plane here
you have a two-dimensional space to
divide it in half you need a
one-dimensional object which is a line
here we have a three-dimensional space
sort of divided up we need a
two-dimensional object the dividing line
is always one less number of dimensions
than the space when measuring three
things about these animals we could
measure any number of things right we
could measure 150 things
and that's fine the maths works exactly
the same in 150 dimensions
you can't
draw it
on a piece of paper
you can't think about it in your head
but the math is the same and oh yeah we
should talk about feature vectors
totally straightforward so when we have
a two-dimensional space you take a
particular
creature there we are this red rabbit
and we measure how much it weighs we're
doing size by weight let's say and we
say that it weighs you know
a gram
and we
say what color is it well it's red it's
really very red so it's about zero
and this
this is now a feature vector it's a
feature vector of length two because
it's a two dimensional space so each
data point is represented by a feature
vector which is a point in the feature
space so there's some fun stuff that's
been done with for example uh faces
right human faces
to to look at a human face and say is
that a male face or a female face
um
that's the kind of thing that you would
think that you really needed a human to
do how would you even go about telling a
computer how to do that right
this is the kind of challenge that these
kinds of algorithms can tackle so it
works out to be exactly the same you
measure
a load of things about the face you know
the distance between the eyes and the
length of the nose and the width of the
mat you measure
everything you can measure about the
face and then you plot them all in a
very high dimensional space you tell it
certain things which ones are male and
which ones are female you find the
average male in the average female and
that's interesting as well because then
you can take this data you can take this
point in space right each point in space
represents a possible face
everywhere in the space is a face
it's a face space if you will right is
that a social network
yeah
yeah it was a failed social network um
but so it's a it's a feature space right
of faces and then these certain points
represent specific faces that you've
actually loaded in and said this is a
man's face this is a woman's face
whatever so then you can find the
typical male face but then you can do
the same thing where you plot a line
from the typical males for the typical
female this line is is indefinitely long
which means you can take for example a
face and you can make a face which is
more male than any actual human person's
face right or more female in the
mathematics of this emerges completely
naturally you're just following this
line along you end up with these
grotesque things but a certain
percentage away past the end what you
end up with is kind of cartoon
caricatures so the males end up with
these like vast you know big brows and
big
bulky jaws and the women have like
enormous eyes they end up looking like
anime characters or whatever before you
have this kind of concept how do you
even approach the concept of how do you
draw a caricature of a human being with
a computer but now we can do it right
because we have this concept of doing
basically geometry in a feature space
generally when you when you throw
real world data into one of these spaces
it will be clustered because the real
world does tend to be clustered right um
it's just kind of the way that things
tend to work out so
people who are like me
will be next to me in amazon
i'll be i'll be i'll be part of a
cluster of people about my age my gender
my you know level of education and
wealth and everything else people who
tend to buy similar things and so then
if somebody who's very close to me in
that cluster buys something else
then they can kind of infer that i might
be likely to buy that thing as well
that's something that that amazon does
something that you know netflix does
with their films all of these kind of uh
all of these kind of predictive things
where you just throw the data in and the
maths doesn't really know what it's
dealing with it doesn't know that i'm a
that i'm a person not an animal or that
what i'm measuring here is you know
propensity for buying computer games
instead of mass the math is is broadly
the same
and so you can make you can make
computers tackle all sorts of problems
which um
which previously they couldn't
if you take a celebrity and you draw a
line between average and the celebrity
and then continue that line
somewhere out along that line is a
cartoon caricature of that celebrity
right because you've taken the ways in
which their face is different from the
typical average face
and extended it and exaggerated them and
this is exactly what caricature artists
do but now we can do it with maths