144. Value Learning with Rohin Shah

hello and welcome to session 144 in the
AIC t-dot-com reading group tonight we
have roving shot joining us from the
Center for human compatible AI and he
will answer some of the questions that
we have that have come up during the
discussion of the value learning
sequence hello Rowan so the first
question is with regards to non cold
directed artificial intelligence and
agents what what is the central example
of a non goal-directed AI that you have
in mind when you wrote this sequence I
don't know if I have a central example
of some non goal-directed I think I had
central examples of goal-directed a eyes
and was thinking of like it might be
possible to not have one but perhaps in
one example it could be like so firstly
there's like all of these good
old-fashioned AI expert systems we're
doing basically symbol manipulation in
order one thing you could have imagined
that they were doing with symbol
manipulation in order to answer
questions or generate stories or things
like that and sort of notably there's no
utility function or loss function that
they're optimizing they're just like
manipulating symbols according to rules
that we program in and then out comes an
answer and I don't think it was like a
priori obvious that such an approach
couldn't have led to very powerful AI
systems yet I would just not expect any
of the problems that we think about with
goal-directed agents to show up with
such a system if it did become super
intelligent now it turns out that like
okay we investigated this approach and I
didn't really work but sort of if I take
the position of not having to run that
experiment and like said could we build
a symbol manipulation yeah that works I
don't think it was obvious that the
answer was no yeah
another example would be basically
probabilistic inference based approaches
so you could imagine a system again
let's say a question answering system
that basically well maybe that one has
like a similar flavor to the expert
system just with more probabilities
thrown in in order to make it less
rule-based
but that one also would not have a loss
function or an objective function that
it's maximizing or minimizing and I
wouldn't expect the normal problems to
arise done either okay one more thing if
people have questions or would like to
ask a question then please in the chat
window please write that you have a
question and I'll maintain some kind of
cue and until someone says they have a
question then I will just ask some of
the questions that people have sent to
me so the next question is actually from
the a children's movie that my my
children are watching about a a Artwalk
robot because this is a one of the
examples that you give off an Uncle
Terry agent and a teen that always
chooses the action that starts with the
letter A and it's further an imitation
agent in that it imitates a an artwork
but it's still obviously so is this an
example of an ngon goal-directed agent
yeah I think one of your actions is like
aardvark imitation then I don't know I
was imagining something more like the
actions are things like moves left
rotate your arm in some way and things
like that I think if you imagine actions
at that lower level then you it's like
pretty clear that this should be a non
gold directed agent it's like going to
do just random stuff in the world that's
really not going
to be trying to optimize the world in
any way it's not going to be subject to
any of the convergent instrumental sub
goals because it's not maximizing
anything it's just a very simple
computation it's like look at the names
of my actions which one starts with a
and then choose that one now if your
actions themselves are complicated
enough that they can be modeled as goal
directed like aardvark imitation I mean
then sure maybe that would be goal
directed and I could believe that but I
was not imagining such like complicated
actions when I gave that example okay um
one of the other well you give a number
of non co-directed agents and I've tried
to write all of them down here with like
a hurricane the lowest is an agent that
doesn't take an action ever that's
probably not gonna rekted and then there
is another few kinds 1 2 4 which to me
are obviously completely unused useless
basically and then there are a few that
are a bit more substantially less
useless but become more useful the more
goal-directed they are so is there there
are two trade-off between how useful and
how goal-directed they are that seems
plausible I am not totally sure well
like I said I think and into intuition I
have is that like a priority we didn't
know that expert systems wouldn't work
and expert systems seem like they're not
goal directed but sort of we then did
observe the fact that expert systems
didn't work and this I think I've become
a bit more sympathetic over time to the
notion that like goal directed miss is
pretty strongly correlated with
usefulness
I think most of my point in the sequence
is more that like it's not a like law of
math that AI systems must eventually
become goal directed it's definitely
dependent on what we actually build but
it seemed
I think there's a reasonably compelling
case to be made that because of how
useful goal-directed agents will be we
will end up building them I looking at
this hierarchy I think it's seems
basically right to me I'll note that
like number seven I mean that one the
one where you have an agent that helps
the goal of some other agent
goal-directed 'no smite not be the right
concept for this one but it does seem
both useful and more safe whether it's
or not it's goal directed I'm not sure
maybe this just means that goal directed
is not exactly the right concept for
what I want to think about but like
that's sort I think number the that kind
of agent is an example of an agent that
might not have all of the safety risks
that we associate with explicitly
expected utility maximizers
but still it was very useful okay
I think there's a question in the chat
Robert please go ahead yeah I just was
wondering about the this lexicographic
sorting it's it's distinct from an agent
which wants to take actions that are
high in the alphabet because like if you
have a if you have enough of a time
horizon then you might have convergent
instrumental goals towards you know in
the short run doing some things in order
to ensure that in the future you always
have a lot of odd marks around so that
you can continue in the long term to
reliably be near the beginning of the
alphabet but that's like a different
kind of agent is it is the relevant
thing that its preferences are defined
purely over the action space and not
over like the world state or well
history is anyway I agree with
definitely the first part that this is
very distinct from an agent that wants
to take the first action sort of
lexicographically I would more say I
mean you say that
like then your second part was like is
the is the distinction that the
preferences are over actions instead of
States I like would somewhat say that it
doesn't have preferences because like
I'm imagining a program that's like take
the list of actions sort had
lexicographically Index take index zero
and execute that right nothing
definitely does not have convergence
instrumentals up goals right like if you
suddenly give it the take over the world
action well they're taking over the
world action is still not
lexicographically first and it's not
going to take it yeah right
like okay to the extent that it has
preferences they are the actions rather
than that yeah I guess you can make it
that way it's not that's not an
unreasonable way of muddling it it's
fair I'm trying to think of like other
examples in that like can you can you
define anything that's purely
preferences over actions that would be
goal directed that seems unlikely if
your actions are varies if your actions
are simple because if you define
preferences only over actions and don't
let the agent reason over long
timescales then it's forced to act act
extremely myopically it's basically just
a reflex agent decorously that's just
like only looks at which action would be
good to do right now with no regard to
what the state is I would be very
surprised if anything like that was
co-directed but also not useful yeah it
also would not be useful
okay so the next question is one of the
in economy know exactly which section
you talked about all there's a section
called all behavior can be rationalized
as expected utility maximization where
you have a rather neat construction that
I've given here where you make a utility
function which which if you try to write
down this utility function it will be
very very large and I feel when I read
this I was reminded of Charles Chinese
room argument which kind of makes the
same way to I don't know want to say
cheat your intuitions but but you think
of a when people normally think about a
utility function they think about
something very compact and in fact here
we have a truly enormous do you think
that's a fair criticism that a utility
function that is too large to be written
down it's not really a utility function
I I agree that it's pretty analogous to
the Chinese room argument I so I think
so to be clear I do think that if you
wrote a program that where you write
down a utility function which would be
sort of by necessity compact and simple
and then you like maximize the
expectation of that utility function I
think that is dangerous
I think it has all the convergent
instrumental sub goals and we maybe not
all of them depends on what the utility
function is but usually it will have
that and probably this ends up killing
yes if it's like maximized appropriately
well or sufficiently intelligent my
point with this construction is that
there's this argument that you can say
is that that knowing nothing about how
these like in in in that story that I
just laid out we know that there is an
exclusive utility function and inside of
the air and the a is like thinking about
that utility function on how to maximize
it so if that's your scenario then I'm
like yes
I agree with all of the classic
arguments seems dangerous I think
there's another argument which people
sometimes make it seems common to me so
sorry let me turn that off there's a
sorry yeah there's another argument that
people sometimes make which is that well
let's leave aside we'll say nothing
about how the air works on the inside
because who knows what super-intelligent
a-- is going to look like maybe it's
just a bunch of deep learning maybe like
we figure out how to get expert systems
to work maybe we do something entirely
different who knows is like some we can
predict it and invest what we can say so
the argument goes and I disagree with
this argument is that since this is a
super intelligent AI system you won't be
able to find any coherence violations in
its behavior in particular it will
satisfy all of the vnm axioms because if
it doesn't sorry the vmm axioms this is
the bond I'm and Morganstern utility
theorem there are some axioms you can
make the argument that the AI system
will satisfy these axioms because it's
not even like extract resources out of
the AI which you shouldn't be able to do
with it super intelligent and then
because then you run forward the
argument you use the bun Lyman
Morgenstern utility theorem and you say
therefore I can model the AI system as
optimizing a utility the expectation of
the utility function even though I don't
know exactly how the AI system works on
the inside I think my point is that if
you're trying to run the argument as
like I can model the behavior by a
utility function that's a vacuous
statement because you can model all
behavior as a utility function and like
when you run the argument this way
there's no reason to expect the utility
function to be simple or compact like
you need some other sort of assumption
in order to get to a simple compact you
a function that makes a lot since I've
tried to elaborate a bit more here on
the next slide on basically not just
what would be a I have but what is
actually inside our heads and it's it
compressible well I can try to make a
scale like it could be something really
trivial uh like just evolution basically
and there was a philosopher I forgot who
was quoted in the sequences that broke
down like 50 different things like
friendship and love and art and novel
experiences and all these cathing and
then what we'll probably see in naive
ambitious value learning if that's
possible which is you know something
still possible to write down on a piece
of paper or something like that and then
something that can't be smaller than a
brain and something that is even larger
is that this is a reasonable model yeah
yeah I think if you're going to use
human values that makes sense I have
like I think there are some decent
arguments that like what we should be
thinking about is not exactly human
values because they're not really
well-defined right now and we should
instead be thinking about the process by
which humans have values that's yeah so
I had this argument comes from many
people that Gillian Hadfield is one that
comes to mind and yeah but if you're
going to use human values my guess is
that they're going to be in the highly
complex region it's not obvious that
that's the case you could imagine that
like after a long period of deliberation
all of humanity comes to the consensus
that actually no it's like hedonic
utilitarianism with this very simple
definition of what happiness is after we
like solve problems of consciousness and
whatnot and then you're like okay that
is in fact just what our value
and then that's like pretty simple and
we can encode it into an AI system I'd
be surprised if that happens but it's
not inconceivable okay um and then
please feel free to chime in if you have
any yeah could you maybe go into more
depth about the argument that we should
care about not human values themselves
but the process in which they arise or
at least for me to want somewhere where
I can become less ignorant about it yeah
I don't know if any good resources
written on this the I guess the yeah the
short version of the argument is humans
don't really have any consistent set of
values lots of people have like very
different moral intuitions ah you caught
nothing
hmm you caught me okay yep very
different moral intuitions lots of
different approaches new ethics
especially when you posit weird thought
experiments that aren't in like
situations that people normally
encounter so like it's really it doesn't
seem reasonable to say that humans have
values it seems more reasonable to say
that humans have norm which are suited
to the current context in which they
live if you change the context the norms
need to be updated and changed but that
doesn't just sort of happen
automatically we've seen this like in
the past with as technology progresses
so you know as soon as we get if we ever
get for example the technology to upload
human minds or then make digital copies
of minds currently we have a norm of
every entity every being every person
gets one vote and everyone's votes count
equally in the scenario where we've got
a bunch of digital minds that can copy
themselves that is not going to work
well because at that point it just
becomes everyone whoever has the most
money and just buy a bunch it can just
create a bunch of copies of themselves
and they form a giant voting bloc that
gets whatever they want to be passed if
we're still using democracy with one
vote that's equal to everything and so
some something we'll have to change
there will have to change norm somehow
but you know yes like a utility monster
well I feel like a utility monster is
different but sure anyway just to
elaborate the process by which we
uncover these values I think Wade I
suggested like a two step process where
first we spend a million years figuring
out what our values are and only in the
solar system and then once we have
figured that out then we you know tell
the rest of the universe or something
like that and I guess in the first step
where we are just trying to figure out
our values
it doesn't actually need to be very
optimal right then we we can have a very
suboptimal process there and then as
long as we don't optimize it very much
God house law is not going to be a
problem yeah that seems right I think I
agree with that
the example that we figure out how to do
that yeah oh one more example I like
actually of this like learning norm
learning values or values updating over
time like I recently found out that
privacy was not a thing in the past at
some point at some point like bedrooms
were invented and that they were still
like somewhat public but over time they
became more private and now sort of
everyone thinks that privacy is this
like deeply enshrined value at least in
the West and it just was not for most of
human history and that seems like the
sort of thing where the norm I still
don't know why privacy of all evolved to
be one of our values
but it's this is the sort of exact thing
I mean when I say that like there is
some sort of process that creates values
and they change over time and we
shouldn't be we probably shouldn't be
like trying to figure out what human
bellies are and then hard cut them into
them AGI or something like that yeah but
I kind of have a counter-argument to
that saying well you know I don't really
or at least I care about my current
values and like I get that the process
which created my values are wouldn't
necessarily create my values and a Miss
eduation um but I mean they did create
my values and I care about my values not
any other hue you know not any other
possible news values except for the fact
that one of my values is fair I care
about other people's values but you know
disregarding that it's like now that I
exist I have a strong strong bias
towards my current values so wouldn't
you say that or wouldn't creating AGI
that is based upon the process not
fulfil my values as well as just trying
to fulfill my values I predict as an
empirical fact that are like this is an
empirical predictions that's whatever
you call your values right now even if
that like you would up your own will
free of manipulation or whatever changed
them somewhat as new technologies get
invented for example the democracy the
democracy example is a good one yes I
mean I'm gonna change my mouth like my
values tomorrow when I wake up and I'm
like a bit sleepy yeah probably gonna be
less tolerant of most things and when
I'm like you know doing that so like you
know my values changed within a single
week so yeah that's a really good
argument yeah note that there is change
in values and sort of the entire point
of thinking of human values in the first
place was to give our AGI system a
description of what we want that can
stay static and that it can optimize
forever and I'm just like we don't I'm
not sure such a static description
exists even in principle maybe it does
he could imagine something like what I
would think if I were given like a
million years to think about it and
maybe that's sufficiently good that as a
static description that it works but I
don't know okay well I just like to just
add something there I'm reminded of
today I came across the term flashlight
method of writing of writing say a novel
and the idea somebody said that they're
writing a novel is like driving a car in
the dark you can see as far ahead as
your headlights show you but it that is
enough being able to do that is enough
to complete you know a very long journey
by car at night and you can get through
it through a whole novel but if you can
if you can just write say one scene at a
time and see what develops from that
where I mean some some writers don't
like that they prefer to have to know
the route map of the whole novel and it
seems to me that developing values is
like this that we it could be a crazy
dream to think about anything convergent
in by the way of our ethic that's going
to last a million years and that's we
can only try to but use systems
artificial intelligence political
systems anything like that which we can
endorse but which have enough
flexibility and development in them that
our grandchildren can carry things on
and they will endorse things that we
wouldn't approve of you know just as we
approve of things that are our
great-grandparents would not have
approved of and as as long as as long as
each generation can endorse what say the
next two Jenner
nations do that we can just progress
that way without any sense of heading
towards an ultimate goal that we can
visualize yeah
I pray yeah a pretty strongly agree with
us I think Sauron has some questions for
me about exactly that perspective
later on I believe I have just I'm just
thinking about about what my what that
question was but in the meantime if we
are continuing then actually I have some
questions a bit more about strategy and
AI safety strategy and I have wait I I
don't have a picture of him there are no
pictures of him but he says that one of
the points of goal directed Ness is that
you get economic efficiency for free and
then you answered in this comment that
hopefully we can convince the relevant
actors that goal-directed agents have a
significant chance of causing
catastrophe and I've tried to write down
why I'm a bit less optimistic about this
if we think like as an example the
people who are right now using the
strongest computer to simulate nuclear
weapons they are obviously very
insensitive to this kind of argument
they probably don't care very much about
existential risks and and they care
about a lot a lot about their country's
power in the same way and this
requirement that we can convince all the
relevant actors it's a very very strong
requirement and I'm wondering and could
you give some more teachers do you truly
believe we can actually make everybody
stop building called directed agents to
avoid some kind of competitive scenario
yes I think there's a pretty entangled
set of beliefs that make me say this
part of it is that I don't think there
will be that many actors that are
building very powerful AI systems like
it seems based on
during the current state of the art and
a research study you need just a lot of
compute for it and it's actually quite
expensive which is sort of what you'd
expect from the outside view that like
big projects that are very impactful
upon the world will be expensive and so
hopefully there won't be that many
actors so that's one thing I think
another thing is that like sort of in an
abstract sense I think everyone agrees
that extinction is bad and you don't
want it and it's like significantly
worse than most of the upsides you
immediately get by building powerful AI
systems I'm not sure if that last part
but it seems plausible to me and like
right now I'm more and more I believe
that a bit more than like Oh eggs
extinction is like comparable to you
like building an alliance super
intelligence that's aligned with you I
don't know it seems to me that like most
actors have like our most large actors
anyway I have like a decent amount of
shared interest of like yeah human
humanity prosper as everyone is like not
in a scarcity is in a post-scarcity
world types things like that and so I'm
I think we can maybe hope could get
agreement of like yeah the extinction
risk is really just so large that it
even though there would be benefit from
making powerful AI systems to each
person after individually that would be
that's like not worth the extra risk of
extinction that they would have so
that's another thing a third thing is
serve a dis analogy with nuclear weapons
which is that for for nuclear weapons
the you don't nuclear weapons do not
automatically go to extinction risk if
one country has nuclear weapons then
they have not very much downside to
themselves because like you know if they
bomb some other country that's a problem
for the other country it's not much of a
problem for them and you know there's
upside in that they like there is
so yeah I will also have outside which
is being able to have more geopolitical
power but sort of the downside is very
different and that the downside is a
different country gets extinct basically
but not us whereas with a I to the
extent that were concerned about
accident risks and not things like
lethal autonomous weapons which I think
is what we're focusing on here accident
risks just affect the entire world so
that that's one dis analogy now you
might argue that like at this point
today nuclear weapons are an extinction
risk and so why aren't we disarming I
think for that I would say the nuclear
risks are only an extinction risk
because of mutualist mutually assured
destruction and like if you stop having
mutually assured destruction then
nuclear weapons no longer become an
extinction risk and so you have all the
incentives for having them again and so
in order to be in the I don't know if
it's stable but to be in this
equilibrium that we're in right now
where we have me where the world is not
current where like nuclear weapons
aren't really being used that might
actually just depend on mutually assured
destruction being a thing and in order I
think in mutually assured destruction it
probably makes sense for supercomputer
is being used to simulate nuclear
weapons because each actor needs to make
sure that no other actor gets too far
ahead of them like first strike and
second string second strike capabilities
in particular are very important under
mad doctor yeah so and notably none of
this applies to the super intelligent AI
case in in this task of convincing that
we might have to do to expect we can do
this in our current epistemic state or
will they need to be some changes either
more research being done or some kind of
fire alarm alarm for a ICT and yeah I
don't think we can do it with our
current epistemic state
for one thing like man I'm in this
research field I've thought about it for
quite a while and I'm not sure what
kinds of extinction wrists are there or
what's a dangerous thing to build and
what's not so I think we need to have a
much clearer picture of that we probably
would need to build agreement among a
researchers at least before we could
convince more all larger actors yeah
yeah I use an extinction with students
disputed so until we get that dispute
outta for me
debating what distinction extinction is
fara
yes that's true yep yeah so so all of
this I think it's actually not that
controversial among a researcher is that
an expected utility maximizer I'm sorry
an explicit and expected utility
maximizing with an explicit utility
function that was like super powerful
would would end up killing everyone I
think the the part that people disagree
on is more whether or not we build such
a thing and what how far away it is like
most of the arguments seem to be why are
you worrying about this it's way off in
the future knows what sorts of AI
systems were going to go I don't see
very much argumentation that's of the
form or or Elsi arguments of the forum
we will never be able to build AI
systems that are that powerful I don't
really see arguments of the forum if AI
said even if AI systems are that
powerful they will not tell us there are
some but it's pretty rare it seems to me
like the the vast vast majority of
people who are actually building a eyes
maybe not the eye researchers but people
actually using this are strongly not
caring about this more than they they
disagree so it's more it's not because
they
yeah do you think that's true yeah I
think that's basically true I might
especially if you're not talking about a
researchers that seems very true I do
think that I mean I'm hopeful that we
can like build better arguments such
that people do in fact get convinced but
even if that ends up being too hard a
task I do think there will be like fire
alarm type events not for AGI is coming
but for our current AI systems fail and
weird ways that are dangerous so like a
sort of example of this right now is
like the way that recommender systems
optimized for basically people being
angry at each other all of the time I
think we'd like basically all agreed at
least in the air researcher community it
seems not controversial to say that I
think people agree upon it about it this
isn't exactly a fire alarm because you
can sort of see okay this is the the
algorithms are like not particularly
smart they're like operating in this
limited domain so to go from that to
like Opower fillet I will kill us all is
definitely not a step that is really
justified even in my opinion but I
expect that these sort of warning signs
will become more and more impactful and
more and more obviously connected to
intelligence as time goes on and that I
mean I don't like this fact it probably
means that they're like very just large
harms to humans but it is good in the
sense that it probably will get it
probably will get more consensus on here
are the dangers of AI and allow us to
coordinate to not have those dangerous
AI systems
okay great
I have more questions and that's about
AI safety without Foom I one of the
things I asked you in the email
correspondence but what parts of Nick
Bostrom's book superintelligence and
you'd cast his arguments that you almost
disagreed with and about half of your
arguments focused on arguments against
Foom the quick take off so I'm wondering
in the situation where we don't have a
fool we don't have a an intelligence
explosion but we do still have a what
person calls a medium speed takeoff
somewhere there is time for an AI race
that over one year causes people to go
from a very very basic AGI to something
that is super intelligent do you feel
this is a how dangerous do you feel this
is so when you say stupid idea here what
do you mean you mean like level of
humans let's say you can simulate a the
reasoning of a human with like an IQ of
90 using a huge amount of computer so
that's how it's not economically viable
but this is something that you know it's
very amenable to improvement so it can
so it can be improved and over one year
will be improved to a superintelligence
yep okay make sense yeah that is quite
fast but plausible I think sort of part
of my general optimism about us solving
these problems is that I would expect
that within that year we will notice
something going wrong if something
actually would go wrong
and update based on that I agree that
there would be well there's at least
time for in the air race to happen
whether an air race would actually
happen is less clear I'm sort of hoping
that by the time we get to this world we
have some more better global
coordination better understanding of
risks I'm not sure if that's actually
good yeah
odds that after this year we would wish
we had worked more on the AI safety I
feel like this is a really contingent
upon how this happened maybe if I seem
ok right today is when the stupid AGI
gets built and that in one year it's
going to be super intelligent I am
pretty scared of that scenario that's
whatever seems quite bad yeah but mostly
that scenario seems quite bad because
most of my models seem wrong in this
world and then I'm like ok and in this
world
it looks more like we it looks more like
we have somehow created an AI system
that's really smart and has already
become like really useful to society and
so can't be turned off or like there's
no will that will let us turn it off
even if we could yeah but but really I
want to express large amounts of
uncertainty here mostly because the
scenario seems to depend on a lot of
like external details of how the world
works or how we got to this point ok so
it seems likely I'm not really sure
about this that the world is generally
trending to some kind of greater
coordination cooperation so that in the
future we'll be more able to avoid this
kind of very hard races how quickly do
you feel is the world trending towards
this are we are we actually moving in
the
direction compared to like 20 years ago
I don't want to speak about the world
broadly but like AI for I think for AI
in particular I think the world is in
fact moving in this direction I think my
guess for okay let's preface all of this
with I am NOT in the AI strategy
researcher I talked to something I
started to researchers sometimes but
this is not what I think about whole
time so take all of us with lots of
grains of salt that said it does seem
like the world is in fact moving towards
more coordination on AI I think the
partnership on AI is an example of an
organization that sort of seems geared
in this direction and they seem to be
doing they seem to actually be doing
things like a worry with any such
organization would be look like oh it's
just PR for the companies and I I wasn't
sure if this was true or not and I think
now I'm like more lean towards the case
that that that that's actually not true
that it actually is likely to get things
done the made you change your mind oh
talking to people who have more
interactions with partnership on the I
than I do
yeah also you can see some of their up
but they have started producing a little
bit about but but that was not the major
source of evidence I think other things
are like this is you know I'd actually
been interested to know whether or not
this was historically true but this
might be a case in which we like foresee
a risk before it actually arises and my
I now don't endorse this belief but like
my vague guess was that historically we
in fact did not foresee problems before
they arise
that was like a big reason for how hard
it was to fix them but as I say that I
realized I wouldn't actually know if we
did in fact foresee problems before they
arose so maybe we did and just do
anything about them it is really hard to
figure out what our appropriate
historical parallels yes okay then I
have another question about narrow AI
and ambitious AI in the you know in
chess there is this kind of obviously in
the beginning humans were better than
the AIS and there was a time where Sybok
teams of humans and AIS were both better
than the humans and better than the EAS
themselves and then eventually of course
the chest algorithms get better and
better until we get to the point where
the humans are just in the way do you
think this is a reasonable a you talk
quite a bit about their you know the the
humans provide the goals and then the
non-coal directed AI helps and this
seems like a strategy that would be
dominated at some point by just goal
very good AI without the human to focus
is great I think if you can define the
task that you want done that is probably
true but given that we almost always
cannot do that like in chess we can do
that nice property of chess and go but
given that we cannot in fact do that for
almost every task I think there's like
some there's probably some sense in
which the goal directed there will be a
goal directed AI that is more
intelligent than the human plus non goal
directed AI but I I doubt that there is
a goal directed AI that is more useful
than a human plus non goal directed AI
at least or like if such a thing exists
that goal directed AI is somehow getting
its goals from a human eye
yeah
for like optimizing opinions or
something what are your goals that I had
in mind was something simple like
earning money in this case the defining
the cult is it's really really easy you
know you just have a bank account and
you want to make this number as high as
possible and in in this kind it's almost
as easy as just to define the winning
what is the winning movement what's not
yeah I think if you limit your AI system
to like only its it's only allowed to
trade stocks or something like this and
can't do anything else
it's then I probably agree with you but
also in such a scenario I think the AI
system is pretty safe like the model of
an AI system where it ends up killing us
all requires the AI system to have a
very good world model a good
understanding of the world and be able
to like take actions in the broader
world now sometimes you can definitely
talk about how it could use the limited
set of actions it has in order to figure
out ways of influencing the real world
or the rest of the world in a way that
you didn't intend but that seems to me
to be a quite a difficult learning
problem actually and I think if you got
to a point where an AI system was able
to do that you've probably trained it on
a pretty large diverse set of data about
the real world
that was very fuzzy sorry but I'd be
surprised if it just sort of discovered
this wait would I be surprised I don't
know yeah maybe I maybe I want to just
express confusion and uncertainty for
this question
then I will go to the next question I
guess this I don't know if Stuart
Armstrong has meant to to join us here
but one of the earliest part of the
sequence was where he tried to use
Kolmogorov complexity to see if it was
possible if assigning correct values to
humans had a local murmur of complexity
and he found in this that basically you
can't do this you can assign any any
value to two humans but this seemed to
me like the way humans do this of course
we have things like Hanlan's racer that
says you shouldn't assign to malice what
could be as I explained by stupidity and
the same way you know if humans fail to
a multiply two very very large integers
it's probably not because that's our
value is because we can't multiply this
and some of the examples that your
Armstrong gives like perfect anti
rationality with the opposite goals
let's obviously also something that
humans and we we basically never
consider that because we we don't
operate with a makarov complexity only
we we have some kind of a human brain
and given that the person in stand in
front of us is a human with a brain that
looks roughly like our own then what is
the simples the simplest explanation
based on that do same thing a scheme
like that could work with any scheme
like this is that you have to make some
sort of assumption about the human so if
you don't make an assumption about the
human then or just generally when you're
inferring values and you want to
distinguish them from bias there's like
one way to think that is that there's
this values object and they go through
biases in order to produce behavior and
the behavior is not optimal with respect
from to the values because of the biases
and sort of the post after that one or
before that one I forget the order I put
them in
by Paul Christiano about the easy goal
entrance problem still being hard so it
makes this point that like in this
situation if you want to get so the the
thing that you observe is human behavior
and you need to decompose it into values
and bias if you want to and then you can
then once you do that you can take the
values and optimize it to get better
behavior which better optimizes for our
values now if you want to outperform the
human behavior that means you need to
figure out the direction of the bias or
the mistake model as he calls it which
survey see the sort of Stuart's result
is saying well you can't just learn this
model and yeah you can just learn what
the human biases are which means you
have to put some assumption about it and
so then the values that you infer and
the optimal policy corresponding to them
are only as good as your model of human
biases and I think that it is pretty
hard to get such a model of human biases
and sort of even if we did to the extent
that we're trying to do ambitious value
learning I like take it as basically
axiomatic that any any model we make
would be mislead somehow I get one
perfectly capture human biases and the
last two posts of that chapter by Jacob
Stein hard and Hawaiian Evans on miss
specification sort of argue that when
one of your assumptions is miss vessel
is pretty badly wrong or isn't as
specified lots of bad things can happen
and I would expect this to happen
especially if you were trying to do
ambitious value learning I probably
agree with that but but I just point out
that some of the examples of biases you
could probably look at a human brain and
then saying okay when the human is asked
to multiply to very large numbers he
answers I do not know and from looking
at deeply into the brain you can figure
out that that is because the human brain
is
unable to multiply these two large
numbers so it's it's an a mistake from
the human that it doesn't give the right
answer rather than because it's his
value to say I don't know I mean this
seems like something that could could
work and could be expanded opposite yeah
that seems right
you know what do I think about that
and just check if Stuart Armstrong has
joined us because he might have a
comment yeah no I don't think is James
yeah I guess in the multiplication case
there's a notion of correctness and I
think you're basically making you think
you're making the assumption like if I
were to take this frame of how do you
decompose values and biases I think
you're like essentially making the
modeling assumption there that actually
okay yeah you could like sort of look at
the human brain and be like this
particular information in particular the
product of these two numbers never
appears in the brain and maybe that
allows you to infer something yeah I'm
not sure maybe there's something to do
with this I think I would be surprised
if this like managed to get you all of
human biases like you could get some and
you can make some progress with that
maybe but it seems very difficult to
figure out like certain notably humans
also are not able to do this is it like
a bias that we care more about people
closer to us geographically than far
away or is that just part of our values
effective altruists will tell you it's a
bias lots of other people will tell you
it's a value I think I agree with this
just to be clear we've we've now spent
one hour and ten minutes on this and if
you need to go then please say so
otherwise I guess we have time for a few
more questions so I should have quite a
bit well if you don't mind then I don't
mind either
and one of the more controversial
statements that you made is that
that this absolu article by eliezer
yudkowsky is in fact vacuous that a
superintelligence will look like an
expected utility maximizing and the the
argument is si seat our arguments
against some straw arguments for
instance that in in history there's been
a lot of cases where a less smart person
has outsmarted someone who is smarter
and that might be a way to have a ICT
that even if the the computer is really
really smart then you know it might be
book smart nerd smart and don't know
anything about politics of power or
something like that and in this case
then by modeling it as an expected
utility my maximize that cannot be
cheated in in this particular way it's
like a counter-argument to that do you
feel that some people hold maybe not
explicitly this the first straw argument
and do you think that the article is a
counter-argument to that which are sorry
I'm not actually sure which one this is
supposed to be representing is this a
straw version of eleazar's argument or
my argument no no a third person really
imagine someone who is not worried
particular worried about air safety
because he believes that if we build an
AI then sure it might be really good at
mathematics but we'll still be better at
you know military stuff and that's why
we will be able to defeat it and then
actually it will we won't have any easy
way to cheat this machine I see I have
not actually run across anyone who
really claims this first argument maybe
I've heard it occasionally from like
random people like like uber drivers and
things like that
people like that who I end up talking to
about my research but not from anyone
who's thought about it seriously but I
that some of eleazar's writing would be
a good counter-argument to this latest
argument then the other straw argument
that you make in someone who is who's
talking about a eyes and curly imagine
that he's writing a computer program and
then suddenly this computer program is
really really smart in some way and even
though this computer program is really
really smart it might have some kind of
mental flaw or vulnerability and really
it kaskus argument here seems to be that
you can have both things that are
intelligent and things that are
optimized and if the if you have a beam
that comes from nothing maybe a random
coincidence then it might be exploitable
but if it became super insulting through
some kind of optimization process then
it cannot be exploited do you feel this
the the article could be a
counter-argument to this kind of straw
argument interesting possibly I had not
I did not have that reading of eleazar's
article but that doesn't mean that's not
what he meant
[Music]
yeah I don't know I don't think it's um
I don't think eleazar's article is a
very compelling response to that
particular straw argument it's like okay
for sufficiently I guess for
sufficiently intelligent AI systems that
seems true but like that's sort of just
I think Eliezer takes as an axiom the
take like an axiom of eleazar's article
is if a if we have a super intelligent
AI system it can and will do any
cognition that we are capable of I think
if you have that assumption that's
already a
counter-argument to the straw argument -
and you don't need the rest of the
article that he's written so I would
really say it's more just an assumption
of his argument as opposed to something
he's arguing that said I also don't see
very many people who argue the straw
argument - I imagine of it yeah I
mentioned matter oh maybe we're gonna
say the same thing let's find out like I
could imagine that it kind of comes down
to the the assumptions you make about
what is his super intelligence like I
could imagine an intelligence which
works out all of mathematics way before
it works out human interaction maybe
that's just a much more complex problem
to solve and most of mathematics we
would consider harder than than human
interaction but maybe that's also or
maybe that's not the case maybe there's
super intelligence which just works out
both at a similar rate and then that
would not be problem yeah I mean I think
I agree that the AI systems that first
seem like super intelligent on like
pretty complicated complex tasks will in
fact seem not as intelligent as humans
on other tasks and social interaction
seems like a pretty likely one
I think eliezer is definitely thinking
of like AI systems that are beyond that
it's just like saying at some point we
will hit a level of intelligence where
is just outperforms humans on everything
by a lot and why the the making barracks
you know what a super intelligent gist's
yep yeah I should also say I'm like not
sure I understand what eliazar saying
it's not clear to me that I've
understood what he's trying to convey
but taking my best guess the the thing I
was trying to say before is that my
model of iliad Kowski is that he
basically is bombarded with bad
arguments against AI safety so he is
probably intimately familiar with a
myriad of bad arguments so
like these I think he has at least
double dated times people have said to
him that ah da I will just be book smart
but will be pretty politically smart so
it won't be a problem I think people
will so that's why he he care about
countering its kind of acumen
yeah that seems possible it definitely
sounds like that to me as well I don't
actually know but like also does use
this like he does use this argument in a
way that's like this is why we care
about utility functions and I like I do
in fact pretty strongly disagree with
that like sort of the best example of
where he's using it that way as an um it
was a talk called a alignment why it's
hard and where to start
I believe that was the title also like
ma'am if he's responding to stir
arguments I'd really wish he'd just say
that these are straw arguments before I
spend all of my time reading something
that's meant to argue against the thing
that I very clearly do not believe I've
just started reading your um your
summary of about comprehensive ni
services I'm very intrigued by that
because it it seems to describe a much
more plausible route to AI AGI if that's
what it is then then you know the the
the sort of single entity super
intelligent entity that Eleazar and Nick
Bostrom seemed to have in mind all the
time
you can you say something about how how
how that difficult case or case or would
like other case just because case is all
signed English word yeah okay you see
something how Christ relates to this
slots into this whole value learn
approach oh man yeah if only I had a
good answer to back let's see I think a
thing about what one thing is if you
look at sort of the chapter 3 of the
value learning sequence a narrow Valley
learning part I think most of that is
more compatible with case than things
like ambitious value learning in that
you could have a service and and Eric
talks about this in his full report on
case you could have a service that
predicts human approval so you give it a
plan and you say hey our human is going
to approve of this plan it says either
yes or no and you can use or maybe
somewhere degrees of confidence and you
can use that information other services
could use that information as needed you
could use that to monitor all the plans
that other AI services are doing and see
oh hey the services doesn't seem to be
doing things that humans would approve
of we should probably check that maybe
update it somehow figure out what's
going on so that's one way things
continue they're good that's one way
that narrow value learning could
interact with case I think more
generally that just like treating their
value learning as a service that other
services can call upon makes some amount
of sense you could imagine for example a
personal assistant
or like a flight booking assistant that
is like you say hey can you book me a
flight to New Orleans on Friday and it
looks and it sees oh hey there's a
flight but it's a red-eye or maybe it's
like $2000 for some reason and that's
like hmm that's suspicious that's like
weird and probably not what the human
was expecting let me call upon my narrow
value learning service and try and
figure out what the human actually
wanted and then present a bunch of other
options like maybe you suggest that I
take a flight on Thurs
instead because it's much cheaper and I
and it won't be overnight yeah it's not
much of an answer but that's something I
hope I shall also say that
hopefully the a ICT reading group will
get to a wreckless report if we can find
something some find some parts of it
that is relevant because the entire
thing might be too much and your summary
ruins might be a bit too little
we need something preferably 10 to 20
pages but that's gonna be quite a
challenge to find yeah it's very much
you could you could do something like my
summary Richard no summary and Robin
Hanson's blog post about it
along with all of the comments on all
three of those yeah that might work okay
so while I was trying to summarize your
value learning sequence then there were
some times where I tried to draw some
kind of models where I was actually
unsure whether you would approve of them
and so the first one I have here is that
you we imagine that we have narrow value
learning that is working in a very
narrow sense and then we use this this
kind of loop to build some kind of norm
inference where we figure out what
should da I not do and while we do the
AI is trying to work within the
framework of norm interference we want
to gradually transition to something
more ambitious and value learning and
thus that doesn't seem reasonable I
think that it would make two
clarifications to that so one is the way
I defined ambitious value learning which
was in some sense of strong definition
and extremely rigid I think that you
would just never want to do ambitious
value learning but if you take a more
normal
sons of the of what that would be like
right we want to figure out all right
how we could make a good future and it's
not a utility function that prescribes
exactly what to do in every single
particular environment but sort of
general paths that are general things
that we want with like maybe some
correction by humans along the way or
something like that then I think it's
pretty reasonable I also don't know that
I would say Maribelle e-learning is will
lead to Norman France they feel somewhat
like two parallel strands of research
like I guess my intuition is that like
if you wanted to infer the values of a
single agent the things you do are just
quite different from what you do if you
want to infer the norms that a set of
agents are following that that's my
intuition right now I don't have a great
justification for it but they do feel
like parallel strands of research and
not that narrow value learning feeds
into norm inference okay great then I
had another yeah kind of maybe a wind
diagram or something where we have the
set of all expected utility maximizers
and as a true subset of that we have the
golden rated agents and it's a true
subset of that we have the explicit
reward maximizes where all the subsets
are true as in there are goal directed
agents which are not explicit rewards
maximizes its every dude does also
reflect reflect yourself thoughts so if
I expected utility maximizers you mean
just any agent because everything can be
model as an expected utility Maximizer
then I think I agree with this this
seems that seems right to me okay then I
have a question here which I believe you
have roughly answers you have answered
that this fact that intelligences should
have voting rights would be a norm that
we that we prescriptively have right now
that probably won't be changed and then
one of my also
was that if you look at human norms
descriptively right now you also run
into a lot of problems like might is
right so do you feel descriptive norms
of prescriptive norms are what will be
based basing the UNAM inference on
probably descriptive ones just because
that's sort of what inference means it's
like you observe what is actually
happening and then figure out what's
what the norms are I do think that like
with norm infants you also want to
figure out the process by which norms
are updated over time which currently is
well there are some like opaque social
process by which it happens just among
communities people but there's also like
law and Court and judges and courtrooms
and things like that so this is not my
area of expertise but you'd probably
want some way both of inferring what the
current norms are and also a way of
figuring out how the norms should update
in the future I don't have very much
clarity on this though okay then and
option you you wrote about is that you
have the AI have some kind of estimate
of what its reward function should
actually be and then you write that that
the AI should have an estimate that that
would be an obvious thing to implement
and I agree that having the a estimate
what should be its own true reward
function is probably easy but if the
reward function is not compact or if the
reward function is compact then it seems
quite hard to implement in practice
right you imagine that there is one
option is the human what they actually
truly value is paper clips and nothing
but paper clips and another
hypothesis is that humans care about
reproductive fitness and nothing but
reproductive fitness you have a third
which is like pure hedonism or something
like that and you know if you want to
have an estimate of all the options then
you know enumerate all the options of
which I just had three here seems really
hot to implement
yeah I actually don't remember what
context I said listen but my guess is
that I was making more of an abstract
theoretical point that wasn't when I
said the most obvious implementation I
meant more like if we had unlimited
compute and the ability to like have a
prior overall possible reward functions
I do you remember which post this wasn't
me I might be able to find it search you
so let me just the most obvious
implementation
here reward uncertainty I'm just yeah
yeah okay I was making that I was trying
to make a more abstract point here I
should probably update that and make
that clear okay it was just there might
have been something upwards that I
missed him then this is actually not a
question about for you I might want to
put it to Owen just see if I have other
questions that I think I would really
care about
there is creature building what is the
relationship between courage ability and
value learning because you end up
talking quite a bit about courage
ability and it seems you know what is
the relationship between these two
concepts yeah yeah I think I got more
clarity on this since writing the
sequence so hopefully I should be able
to give a better answer now so the way I
think if courage ability is it's the
property that you're a system is trying
to help you
whereas value learning is about whether
the agent knows what it is that you want
and these are somewhat orthogonal in
that value learning is more a statement
statement about like the agents
knowledge whereas courage ability is
more a statement about the agents
motivation and you can either you can
either say I am going to create an AI
system that is chords both meaning that
it is trying to help me and then as long
as it meets some basic level of
competence of like intelligence or
something then one of the things it will
try to do when it's trying to help me is
it will try to learn my values or my
preferences perhaps what's in your
values for now and so there you could
say okay I'm going to build a corrigible
a and that will lead to value learning
also to be clear this is like I think
this is Paul's notion of courage ability
it is definitely not me
mccords ability we have two different
meanings for the word college ability at
least it's not great I just wanted to
note that first could you say what is
the difference between mirrors and poor
Cristiano's definitions of cordage
ability yes so the one I've been using
so far is my interpret is what I think
Paul's definition is though I've been
known to misinterpret Paul before the
marry definition is more like if you it
basically gives you a fallback mechanism
so that if you say for example turn up
AI system you are doing a bad thing turn
off now it actually just does turn off
no questions no like inference over
there Ward functions no like modeling
the human is rational or irrational
nothing like that it just turns off so
in this sense of like when you there are
some situations some fall back some
situations where a human can activate a
fallback mechanism that provides
robustness in case something has gone
drunk that's I think the miry sense of
courage ability cool so I'll go back to
the original question so with the Paul
sense of cords ability where the AI
system is trying to help you that can
lead to about preference learning as
long as the agent is at least somewhat
capable conversely you could talk about
okay I'm going to build an agent that
does value learning or preference
learning and then it's going to optimize
for my preferences or values as based on
what value learning and preference
learning is doing and then if it's like
a reasonably competent at learning my
preferences it will learn that hey I
prefer that it listens to me when I tell
it to shut off I prefer that it doesn't
deceive me I prefer that it did not kill
me etc etc and as a result it ends up
behaving the way
corrigible agent would or like it learns
that I prefer that it like helps me in
general which is sort of the property
that Cory's ability is supposed to be
this is the zooming an agent that does
preference learning and then optimizes
for my preferences so that's sort of
there are two parts of that one is the
preference learning system and one is
how you optimize that all the preference
learning there's often reward
uncertainty is a component of that I
think the reward uncertainty
automatically gives you some of these
properties as well so really there's
like you could either try to build a
corrigible AI and get preference
learning at that or you could build a
preference learning plus reward in
certain de AI and yet Gorge ability out
of that okay thank you for that
clarification and then I in one of the
the concept of having a AI as a
controller and having a plan then I made
a statement earlier that sometimes the
the plan would be better than the
control if the plan was good enough I
think I used a metaphor like if you're
trying to navigate a maze then the plan
of always sticking to the left wall is a
good plan and a controller in this
situation would be a bad plan I would
put perform worse and then I think this
you're right that actually in control
theory you can prove that a controller
is superior to a plan and then there are
there's no link unfortunately so I
couldn't actually go and see where my
assumptions are wrong can you sketch
roughly why why I'm not right with my
maze example yeah so I think actually in
in your example but the plan of the
thing you're calling a plan of like
always going left or sticking to the
left wall or right well whichever it
doesn't matter that would I
alleviate controller and listener in
this in this terminology because you are
relying on the ability to choose your
actions after seeing your observations
so like a plan in the main finding
setting would be something like I must
walk forward five steps and then turn
right and go forward three steps and
then turn left and go forward one step
then go right turn right go forward ten
steps and so on which in the case of a
maze is fine and would work if you get
to see them if you make the plan effort
when seeing the maze in advance and so
in that sense setting both a controller
and a plan would work just fine but
that's assuming that you can predict the
environment perfectly at the time that
you make the plan if you cannot in fact
predict the environment perfectly when
you make the plan then you need to like
use your observations in order to note
when you basically need to use your
observations in order to like constrain
the possible ways the environment could
be and once you have constrained those
possible ways then there is an action
that always like guarantees the property
that you want but if you can't constrain
the states but the way the environment
is by using observations then there's no
plan that would actually let you
accomplish your goal or satisfy whatever
specification you have there wasn't a
link because I I don't know if anyone
has ever written a paper that just makes
this point I think it's obvious to
people in control theory or something
and they haven't actually bothered
writing it down in a paper there are
papers that prove these guarantees with
adversarial environments that definitely
is the thing
okay let me just see if there is I guess
mmm that will probably be all so does
anyone have any final questions for
Rowan then I will say thank you very
much for joining us
it's been a pleasure and we've learnt a
lot I think so
thank you very much yeah