115. Discussion with Alexander Turner

I'm cured and of zero if if not and so
the reason I think this works and we
don't need to worry about this in
general is that it this by measuring the
agents ability to receive to ensure it
receives favorable observations about
the world means that we don't need to
make choices about what's actually in
the world it could be anything we don't
need any ontology things in the world or
any actual even world model that
corresponds to anything we'd agree with
all we need is a model over you know
given what I've seen so far which
observations do I expect to see next and
so this way of thinking about impact
shifts from what's actually going on in
the world it makes it indexical now when
you think about impact its with respect
to you so a bad impact like I say and
you post is we might define it as
decrease in my ability to achieve my
goals and if we take away the bad we
would see that it's changed in ability
to achieve goals and so this is with
respect to the agent if the agent goal
is to essentially have you know the
triple utility function that's just one
if it's on the whole time and zero
otherwise this measures power it
measures its ability to ensure it just
remains on it doesn't care about
anything in the world but it you know
measures how much power it has how well
it's able to reason where what its
vantage point is essentially and so then
if we consider an agent trying to take
off well how can you take off without
increasing your ability to remain on in
the future if you supposed well okay I'm
considering this thing that will improve
my algorithms or ensure that people
can't turn me off
well you compare not doing the thing so
if you don't do the thing you're more
able to be shut off you're less likely
to ensure that you can remain on and
then if you compare to doing the thing
you're now in a better position to
achieve that goal so we've effectively
abstracted away the notion of what's
going on in the world and we view it as
a more fundamental statement about what
it means
or the agent to be acting and pursuing
this so I'm does that answer your
question
I think it answers the question and then
you know opens up a few more questions
that I have Jenny
I'd like to just take a few you know
seconds to think about my my next
question if I can yeah sure any other
questions then I can perhaps take one
altitude in Stuart Armstrong has written
a follow-up post where he says he
believes that this algorithm would be
vulnerable to wire heading do you agree
with his assessment I do not so I think
it's it's a reasonable thing to point
out and I'm glad you did
I probably should have made it more
clear so in the post there's a section
where I talk about you know different
ways maybe you could gain this
particular formulation and I'd like to
emphasize keeping the formulation away
from the question of whether this is you
know the conceptual core of impact so if
it's the formulation is one specific
attempt to formalize that and so in
discussing that I mentioned well here's
one thing you could do you could do
observational wire heading where you
build some device that detects whether
you're you know pursuing other goals and
if you are and make sure that you're
only as able to pursue those other goals
as if there's a nothing and otherwise
like it gives you as much of your normal
utility as you want and so I talked
about this and a couple other methods
and then I presented something that
basically tries to detect whether you're
moving towards your main goal which
reportedly you're trying to do in a
low-impact way or whether you're doing
something that doesn't have anything to
do with it which would be you know which
would include attempts to get around the
impact measure penalty and it seems that
this what I what I provided does
disallow all the proposed workarounds
that the notice so far and so if there
does not in my opinion
seem to be
a way to break this right now modulo the
flaws which aren't really breaking it
but just like other questions so it
doesn't seem like there's a way to break
it right now
and I think is good reason to help that
it's either true that you can't or that
we're very much within distance of
making sure we can't I think it's not an
open question as to whether this test
also catches too many false positives
but that is another type of so I think
my follow-up question will probably have
been in extent of that that well if you
are not making it care about the actual
world which is in my opinion it is very
possible to do what are you making it
care about exactly I'd like there is
some confusion for me that I don't quite
understand so it cares about the actual
world is that the penalty functions the
things we're measuring you know changing
their ability to achieve the goals they
do incorporate information about the
world through the model that says you
know what do I expect to see next given
what I've seen so far and implicitly
that's going to have some representation
of things going on in the world okay and
so and so yeah my assertion is that by
we don't necessarily need to be aware of
these specific things changing in the
world to get the effect of this agent
won't move too far in one direction to
change the world from the way it is now
as measured you know by its ability to
achieve different goals you know we're
saying it can't reach these really high
impact undesirable plans without
incurring an extremely large penalty in
this formulation and I'm saying that
this is a way of doing it that doesn't
depend on any specific details of what's
in the world but I think we've good
reason to suspect oh conserve them maybe
they'd be more theory I think it's kind
of hard to unpack exactly why that is in
a compact way if you haven't read the
post you know it's only gonna care about
the parts of the world that it's
information from right so I feel like
it'd just be
sentence of is to make sure that it that
it that it keeps that exact part as
close to what it was before and then you
know it doesn't have to care about any
of the other parts which is not
currently observing or currently aware
of existing law however you want to
define whatever it's observing about the
world right now quite sure I understand
could be Rufus well it seems to me that
that the AI is actually not incentivized
to to keep the world intact if you're
not defining exactly what parts of the
world we want to maintain when you're
only making it care about maintaining
the observations it's having it's not
incentivized to maintain anything
outside that observation space so it's
gonna be you know incentivize to heck
that observation space to be as close to
how the world is currently looking but
then all the parts that is not currently
observing it would not at all care about
and that to me seems like it could prove
very problematic so so the difference
here is you're I think you might be
thinking of it's not measuring change
and the actual observations it's going
to see it's measuring change in its
ability to string together favorable
observations and so this is like a very
different thing and I think it's like
one of the ways I see like yeah I think
this is like a very reasonable pattern
match and it's probably what I would do
as well where there's a different
concept here of well it's not what it
actually expects to see but its ability
to produce unfavorable observations
which inherently captures information
about other parts of the world about its
vantage point and so by constraining
what it expects to see in particular
would actually have like little to no
regular as an effect on the boundary
like change in gaming the penalty I
don't think that would be very related
and that's due to the formalization
so first I would like to say that it's I
really liked it I have one question do
you have any open problem with unknown
unknowns with the unknown on earth did
you two meet
previous questions that were very kind
of related to it that if there is
something a patient doesn't know then
how much effort engage and aid for
example to find out the thing that I see
useful to know in order to not limit it
so it's the way I understand it the
question is is this a question of well
house is affected by the agents like
ignorance about you know maybe it's
model is incomplete or it just hasn't
got enough information so I think that
if generally agents with under specific
like incomplete models they don't have
much predictive power we'd expect them
to be much less likely to act because
their models would have more noise in it
so any given action would be more likely
of producing change in its expected
ability to pursue a goal after you know
in the long term penalty might wait a
considerable amount of time and then try
to reach that goal and compare it you
know its ability to do that after not
acting with its ability to do that after
acting and if your model isn't very
precise you might you know expect to see
a larger shift there and so I think I
think that is in general a problem well
if our agent doesn't know this thing
then it might you know it's not going to
take it into account which is different
from the intuition pumped but yeah
having an incomplete model I would
expect that pump to be somewhat helped
this formulation and in addition it's
also less likely to do things as soon as
I realize the effects are big with
now us needing to tell the agent that
the effects are bad which is basically
always sooner and it seems like a
disciple company so I think you know
it's not impervious to that but I think
in practicality agents probably won't be
able to have a gigantic change before
they're good enough to try to have that
gigantic change by modeling its effects
and pursuing the desirable outcome yeah
I believe that it's a very good model
I'm just have you tried out anything
validation needs to explicitly go
somewhere and do something in order
point out or remodel such sets in order
I will expect the value of information
to the calculations to basically be the
same you know just module of its new
objective so I'm not sure that it would
be a spirit like a special consideration
for this and so I haven't done that so I
have a question and that is imagine that
you have an agent implementing this
which have which gets a a new action
let's call it simplify an action that
can simplify the environment in some way
but doesn't have a particularly large
impact it doesn't stop it from doing a
lot of things because it's it's purely
as some way to simplify its environment
it would as I can see the algorithm
would take this just about always I am
NOT always that's to say it feels well
specified but it sounds like really
small so it's not really it's just kind
of something that makes it more
convenient for its own goal that doesn't
really change other goals and yeah right
okay to strenghten see ya are there any
rules or commands that you can give it
to this agent that it will resist that
maybe some taxonomy of these entities
there are holes that it will resist like
because we reach other you know maybe
changes we want to change its
formulation perhaps to do a different
objective you mean I don't mean many
anything specific I'm just produce in
open-ended a is there any kind of for
example maybe you give it some on Google
that will limit its future options very
strongly and then maybe decide that it's
stupid : I did not do it yeah so if we
were going to implement code that would
increase would not be good for its
current utility function then this would
now be the default and so it would be
heavily inclined to accept it and I
think this kind so this is immediate
kind of portability where the default
case is it gets changed or shut down and
I think this is heavily increased by the
formulation it doesn't seem to it's not
clear whether it will does the same
thing for all kinds of courts ability
we'd want and particularly you know
knowing exactly when you're on this it's
not gonna help with that well I think
that in addition to seemingly
defining low impact it also brings a lot
of additional legibility oh it might be
would expand on the question assume that
there are different people involved with
this agent let's say the manufacturer
and the owner and finally the user and
maybe the manufacturer and the owner
doesn't do not work at the user in some
certain codes to the HM then how to
implement it
so what would happen is if the agent
wouldn't look if wet what the outcome by
default would be the user ends up
implementing this exchange then the
agent wouldn't take steps to make sure
that doesn't happen so I guess I guess
this problem would really come down to
how we want to handle it and be more of
a human kind of mechanism sign
yeah management design problem as I
understand it so that wouldn't be really
within the scope of a timber torii
preservation
what do you see us the the next step in
your research so the next step I think
is to I think the formulation needs to
be relaxed in some ways I think it's
very exciting that it appears to not
like we have a formulation and we might
have good reason to hope that you know
well we've tried to come up with these
waves of gaming it so far and none of
them worked
and this seems like the concept will
scale to know all the way so then the
question is can we make sure that for
every task we wanted to do we aren't
turning to many of the actual good
actions as false positives I think this
is plausible that we're not but I'd like
to make that even more clear and I think
they'll also kind of some people saying
well I'm a little bit uneasy because
this this particular formulations like
something that I view as low impact and
so I think that would be a good next
step then from there you know looking at
issues of embedded agency or making more
tractable and I'd also think I would
like to go and walk through a little
more slowly because I think that I
basically just tried to put too much in
that one post and I kind of went too
quickly I think a lot of people raised
really good points but there were also
some assertions which I'm glad were made
and it seems like people's confusions
the the more confuse coming to also
uploaded comments so I'm kind of one
hearing whether you know whether I
didn't communicate some key aspects one
of the things I think that I'd like to
emphasize more is this as a new gears
level model of what's happening for
agents as they consider plans and as
they execute plans you know as it moves
them through outcome space through
attainable utility space you know maybe
we now have a model for instrumental
convergence and this seems like a very
natural concept for alignment I expect
it to be very useful for other problems
as well and so I think the open
questions is probably a lot of
low-hanging fruit there as well and
we'll see how much of it I can do it my
own and it's probably you know rooms
rather people are stepping if they're
interested what are there any particular
paths that you feel would be helpful for
the people to just have in any
particular directions that might require
a lot of work but might not be very
mathematically heavy or requiring coding
skills or whatever what parts do you
need help with basically so I think the
least one of the least heavy parts would
be well you know not requiring anything
beyond what's really in the post is
coming up with these ways of relaxing
the impact measure that is also kind of
a mechanism design problem right now it
doesn't feel like it should be you know
any more difficult than some other other
problems that dealt with while
developing it but that's a school year
and I don't necessarily have as much
time as well as I did during the summer
so you know help with content
verification and kind of coming up with
ideas well what's the simple boundary
between behaviors and plans that try to
just skirt the penalty and ones that are
actually low-impact and they're actually
just ways of pursuing the agents main
goal and I think if we can get that down
and you know have really good reason to
say yep this is going to do what we want
it's going to draw the clean line
I think that you really really hover so
that that be the first thing to to come
to mind okay any other questions well
then I would like to say thank you very
much Alexandra for coming so this ring
travel it's been a pleasure so the next
part of the Ring group is the Familia
which is your first welcome to stay that
is


AI Safety at EAGlobal2017 Conference

this weekend I went to Imperial College
London to attend the effective altruism
global conference the conference isn't
actually about AI it's about charity the
idea is like if you want to save human
lives and you've got a hundred pounds to
spend on that you have to make a
decision about which charity to give
that money to and they'll all say that
they're good but which charity is going
to save the most lives per pound on
average it's a difficult question to
answer but it turns out that there are
popular charities trying to solve the
same problem where one charity is a
hundred or a thousand times more
effective than the other it's kind of
insane but it can happen because apart
from these guys nobody's really paying
attention you know people don't really
do the work to figure out which
charities are actually effective or what
they're trying to do so that's pretty
interesting but it's not why I attended
see there's an argument that if people
like me are right about artificial
intelligence then giving money to help
fund AI safety research might actually
be an effective way to use charitable
donations to help the world not
everybody agrees of course but they take
the issue seriously enough that they
invited a bunch of experts to speak at
the conference to help people understand
the issue better so this charity
conference turns out to be a great place
to hear the perspectives of a lot of AI
safety experts Victoria Krakov nur from
deep mind safety team and a wine Evans
from the future of humanity Institute
gave a talk together about careers in
technical AI safety research which is
basically what this channel is about I'm
not going to include much from these
talks because they were professionally
recorded and they'll go live on YouTube
at some point I'll put a link in the
description as in when that happens but
yeah Vica talked about what the problems
are what the field involves and what
it's like to work in AI safety and a
wine talked about the places you can go
the things you should do you know what
things you'll need to study what
qualifications you might need or not
need if the case may be they answered
questions afterwards the sound I
recorded for this really sucks
but yeah the general consensus was there
are lots of interesting problems and
hardly anyone's working on them and we
need at least 10 times as many AI safety
researchers as we've got deepmind is
hiring the future of humanity Institute
is hiring actually there will be a link
in the description to a specific job
posting that they have right
and a wine is working on a new thing
called org which is an up yet but we'll
be hiring soon lots of opportunities
here o some people were there out doing
that if animals can experience suffering
in a way that's morally relevant then
maybe factory farming is actually the
biggest cause of preventable suffering
and death on earth and fixing that would
be an effective way to use our charity
money so I tried out their virtual
reality thing that lets you experience
the inside of a slaughterhouse from the
perspective of a cow worst we are
experience of my life seven point eight
out of ten Helen toner an analyst at the
open philanthropy project talked about
their work on artificial intelligence
analyzing how likely different scenarios
are and thinking about strategy and
policy you know how we can tackle this
problem as a civilization and how
they're helping to fund the technical
research that we'll need in the
questions she had some advice about
talking to people about this subject and
about doing the work yourself
here's Alan Defoe also from the open
philanthropy project who went into some
detail about their analysis of the
landscape for AI in the coming years I
really recommend this talk to help
people understand the difference between
when people are trying to tell
interesting stories about what might
happen in the future and when people are
seriously and diligently trying to
figure out what might happen in the
future because they want to be ready for
it
some really interesting things in that
talk and I'd strongly recommend checking
that out when it goes up online probably
my favorite talk was from shahara VIN
from the Center for the Study of
existential risk at the University of
Cambridge he was there talking about a
report that they're going to release
very soon about preventing and
mitigating the misuse of artificial
intelligence really interesting stuff
dr. Levine is very wise and correct
about everything to consume it in a more
video engaging way what miles has that's
all for now the next video will be the
next section of concrete problems in AI
safety scalable supervision so subscribe
and click the bell if you want to be
notified when that comes out and I'll
see you next time shoes is cashews
everywhere this is a great conference
I want to thank my wonderful patrons who
made this channel possible by supporting
me on patreon all of these excellent
people in this video I'm especially
thanking Kyle Scott who's done more for
this channel than just about anyone else
you guys should see some big
improvements to the channel over the
coming months and a lot of that is down
to Kyle so thank you so so much
okay well there's cashews here this is a
great conference


Value Alignment | Stuart Russell

so the first question is what is a good
decision anyway an economist will tell
you that it means maximizing your
expected over the whole future utility
and this applies to everything from
lottery tickets to Davos meetings to
building radio telescopes on maximizing
AI has made a great deal of progress
eighteen years ago
deep blue beat Garry Kasparov at chess
just last week the game of poker was
solved perfectly laughing and all and
humans could no longer compete and right
now the deep mine system is playing 29
different video games superhumanly well
that it learned entirely from scratch
just by watching the screen imagine if a
newborn baby did that on expectations
these depend on perception and learning
again a huge amount of progress the
Watson system extracting information
from text cars watching the world as
they go by learning algorithms that
classify images and write descriptions
even a system that discovers the concept
of a cat entirely for itself just by
looking at millions of images of
everything under the Sun now a lot of
this progress comes from mathematical
ideas here are just a few of the
equations from my undergraduate course
and there will be a test if Linder
allows it until later on there's also a
lot of progress that comes from
commercial investment so every one of
these areas are 1% improvement is worth
billions of dollars so we may see in the
future domestic robots for example
search engines that read and understand
every page on the web even a machine
that will discover the missing sock
perhaps in the look the very distant
future so the point of AI is that
everything civilization has to offer is
the product of our intelligence so if we
can amplify that then there is no limit
to where the human race can go but
actually want to point to a problem and
that comes in the utility part of the
equation so imagine for example that you
asked your robot to maybe make yourself
some paper clips that you might need and
your robots very very clever
it takes you very literally and pretty
soon the entire
the six feet deep in paper clips so this
is the Sorcerer's Apprentice and King
Midas all rolled into one now
technically what happens is that if you
ask a machine to optimize and you leave
out part of your preferences the machine
will set those elements to an extreme
value for example if you say Google car
quick take me to Zurich Airport it will
max out the speedometer and they say oh
I didn't mean break the speed limit well
it'll still put its foot on the gas and
then when it gets the airport slam on
the brakes so this is the problem of
value alignment and if you combine
misalignment of values with a
super-intelligent
machine that's very capable then you
have a really serious problem for the
human race so the point is that machines
can and will make better decisions than
humans but only if their values are
aligned with those of the human race
now my colleagues my distinguished
colleagues may argue that super
intelligent AI will never happen let me
take you back to September 11th 1933
Lord Rutherford the world's leading
nuclear physicist said that atomic
energy was moonshine could never happen
the next morning Leo Szilard invented
the nuclear chain reaction the next
morning so we have to be careful let's
look at nuclear fusion in particular
long ago they invented a method of
generating unlimited amounts of energy
long ago it's called the hydrogen bomb
so now fusion concentrates on
containment and AI has to do the same
thing if you want unlimited intelligence
you have to solve value alignment so one
way of doing this is called inverse
reinforcement learning what that means
is for example a machine sees somebody
making coffee in the morning and then
figures out the purpose the underlying
utility function that explains this
behavior namely that having coffee is a
good idea as long as it's not too much
of it now it's not quite as simple as
that as I'm sure you all see humans
differ in their values cultures differ
in their values none of us behaves
perfectly but there is a huge amount of
information that the machine can access
about human actions every television
program every book every novel every
movie every newspaper article is about
human actions
and in particular about our attitudes
those actions so the rational thing for
a machine to do is to engage in an
extended conversation with a human race
about its values before it can take any
action that affects the real world so my
claim is that in the future we will be
able to design super intelligent
machines that do exactly what they're
supposed to do which is to support
greater realization of human values and
I think this is maybe the most important
conversation that we can have over the
next 50 years thank you
you


How soon until machine intelligence? Oxford professor on Transcendence

well the movie makes it look kind of
imminent as if this is happening in the
near future the truth of the matter is
that nobody knows now in light of this
fundamental ignorance people have
different responses so one is to say
that because we can foresee exactly how
far away we are from human level machine
intelligence therefore we can ignore the
prospect we can say that it will never
happen or admit it from our thinking
about the future another kind of
response is to pick some arbitrary date
2045 and then become very convinced that
that's when it will happen now the
correct attitude in light of this
ignorance is instead to think in terms
of a probability distribution
probability distribution smeared out
over a wide range of possible arrival
dates so we assign some probability to
it happening maybe in 1020 years maybe
more probability that will take 50 years
or 70 years or 80 years some probability
that it might take a hundred years or
200 years and maybe some probability
that it will never happen and then then
we have to take into account this range
of possibilities when we are thinking
and planning for the future now we did
run a server here actually just last
year at the future of humanity Institute
where we pulled some of the world's
leading experts in artificial
intelligence on when they thought we
would have human level machine
intelligence and the median answer was
that there was a 50% chance that we will
have human level machine intelligence by
2040 or 2050 depending on exactly which
group of people we asked now in my view
this has to be taken with a big grain of
salt because basically this is the
subjective opinions of these folks it's
not based on any hard evidence
nevertheless it gives some indication
that perhaps it's not unreasonable to
take seriously the possibility that we
might have human level machine
intelligence within the lifetime of many
people who are walking on the earth
today


Would you have warning before artificial superintelligence? Oxford professor on Transcendence

whether there will be a lot of advance
notice that we were kind of moving into
the danger zone may depend on which
particular path it is that we have taken
to get there
so with the emulation path the uploading
path there will be a lot of advance
notice before
realistically before you would upload a
human being you would upload an animal
and you would have simple partial
successes in uploading animals in the
movie there is this macaque that they
had uploaded that that should be a big
alert to the whole world like after
you've uploaded a macaque obviously the
human will just be very soon coming and
with the emulation you would probably
need a lot of equipment you might need
like a factory sized machine floor we're
different microscopy scanning
apparatuses are not like in the movies
that somebody does it at home with a few
electrodes and and that that would give
a big signature and said nobody would
pull it off in the garage it would be
big sort of commercial or States big
Research Institute we did where some of
the other paths are like with a bit more
mathematical top-down AI approach it
might be that you could get the rapid
breakthrough with no advance warning so
first to have a system that basically
doesn't work because there is one
essential part that doesn't work and all
the other parts are there but with is
one essential part missing the whole
thing doesn't really function and then
once you put in the last missing part
suddenly the whole thing kind of starts
to function at the high level so with AI
it's at least possible that you could
get these surprise scenarios where it
looks like you're completely lost in a
dense jungle and then suddenly the
finish line appears in declaring just a
few steps ahead


201. Corrigibility

welcome to the 201st session
of the ai safety reading group presented
by me
and today we are going to be talking
about
corrugability in general though i
claimed that we would be talking about
post call correct ability by paul
christian
we'll cover that but this is more of a
broad overview of what the idea is
current conceptions of it and
current problems
so the structure of the talk will be
roughly we'll discuss
mary's view of griggability uh stuart
armstrong's attempt at constructing some
tools for making an
agent partly correctable um paul
cristiano's
idea of correct ability and why he
thinks it's central to ai safety
and finally we'll be covering the
perspective of stuart russell
and david hadfield manal from
chai on correct ability
now let's get into it charlie
so uh corrigability was a concept
originally
devised by people at mary and named by
our very own rob mills
and the meaning has shifted somewhat
over the years so now
it is referring to a few different
concepts
which accomplish a few different goals
broadly the idea is to make agents which
divert defer to humans
and in the initial specification of
correct ability
or rather a list of what we would like
in correct ability we
see that miri listed four
criteria these were
that the agent should not interfere with
its designers modifying it
it should maintain its designers ability
to modify it
it should not manipulate or deceive its
designers
and it should preserve the other aspects
when creating
sub-agents or successors now some people
have some issues with the
no manipulation does it errata they
think that it should be
in under the heading of
reward tampering but we won't actually
have time to go into that
so just ignore that for a moment i'll
just note that i kind of disagree that
it is
quite important that an ai doesn't
manipulate its deceivers
it's designers otherwise the other three
designer are kind of pointless
so this is quite a vague concept and no
one was really sure
what exactly it meant to be correctable
no one had a formal specification
so me returned to a concrete problem to
see what does the rata do
and don't work this
was called roughly the off switch
problem
and
they redefined some of the desiderata
and to what we could basically call
recreatability
which i just realized i put above so
weak corrigability basically means that
you have this situation where there's a
button which could switch the ai off
and it's got some kind of utility
function we want to
specify the utility function u so that
it doesn't incentivize shutdown
if the button isn't pressed
it incentivizes shutdown if the button
is pressed
it does not incentivize the agent to
push its own shutdown button
because that would be a bit pointless
right you make an ai it goes to a
shutdown button turns itself off
immediately well
it's kind of useless and defeating point
um
the other incentive is that
any of its successors should also share
the other incentives that is they won't
attempt to
stop the creators from pressing the
shutdown button they won't try to press
it themselves
they if the button is pressed they will
shut down
now there's some
subtlety to all this because you could
ask well what does it mean for an ai to
shut down in a situation where it's
in some incredibly complicated state
and shutting down would be disastrous in
and of itself if we just stop what it
was doing
that's a interesting problem of its own
right but we are just going to totally
ignore that
because that's not really what we're
aiming for here we're just going to
assume that we have some
utility fund in color
some utility function us which
incentivizes shutdown when the button is
pressed that is to say
if we press the button the agent should
act like a us
maximizer but we want it so that before
we press the button
agent acts like a u n maximizer u n just
stands for basically whatever utility
function we've programmed into the agent
so it might be i don't know a task where
it's putting strawberries on plates or
detecting
hot dogs in pictures or what have you
it's just some generic utility function
and they proposed two ideas
to explore what exactly goes wrong
which does a director easy to achieve
etc etc
neither proposal worked neither
satisfied the
preserve the other capacities
when you're making a new agent
and the first one just in a sense of
fail kind of completely
um the second one failed to some extent
the first three or rather
succeeded in a very narrow way
so let's just go through this quickly
for example you might say naively well
let's just add
un and us and make it so that there's
like some
trade-off between them so that is we say
that we
call the utility function um
we define the utility function so that
it is equal to
some constant times u s plus some
constant times u n
so it basically optimizes shutdown
behavior to some extent
optimizes other behavior to another
extent
but this fails for the standard reasons
that is there's
the ai will probably find some
incredibly perverse way to maximize this
that we didn't expect and this is
like say normal good hearting behavior
so we're not going to go
too much into that but this suffers from
the usual pitfalls
of a utility utility maximizer
so we're just going to say that was a
complete failure and move
on
the other proposal was quite interesting
and it led to some methods which are
more useful
but before we get to those more useful
methods let's just cover this one
which is that mary
attempted to make the ai act
as if it thought the button would never
be pressed rather could never be pressed
they basically changed its view of the
probabilities of how likely something is
to occur it said that
the probability of a button press
occurring is zero
in the ai's mind and
in a sense this kind of solves part of
the problem because the ai
isn't really bothered about someone
pressing the button and knows it can't
happen it doesn't matter
if the human is standing over there with
their hand hovering above above the
button
they will not press it with probability
one so why bother
but the issue is that this introduces
some really perverse incentives and
in a way this makes sense because you're
distorting its
idea of how the world works and you're
distorting it incredibly
much so
let's say for example
the ai says that okay i'm going to
make a bet if i lose i'm going to
irreversibly commit to pressing the
button
so i know that if i lose that implies
that the button will be pressed with
probability one
but i know the button can never be
pressed
and the only way i will ever lose
or rather
me pressing the button is equivalent to
me losing since
the button can't be pressed that means i
can't lose
and since i can do this for any bet i
can never lose any bed no matter
how bad the odds which is you know
obviously wrong and dangerous
so the ai might just take a one to a
trillion odds of something going wrong
for the most minor gain
and then once it loses which it
inevitably will it will press the button
anyway
because it's irreversibly committed to
doing that so
whilst it won't prevent humans from
pressing it it will do some pretty
bizarre things
and note in fact it will only prevent
humans from pressing it
in very particular situations and that's
something that the matter methods suffer
from too
so anyway this idea is kind of
unworkable
but it did provide the seeds for some
methods which improved on
these are called the indifference
methods so
the rough idea is that you make the
agent
indifferent to certain events which may
occur but only those events
in essence as if nothing they do will
affect whether it will occur or not
there is no action they can do to change
the likelihood of this outcome so they
don't really care
they would just go so what if the human
attempted to press the button
and as a result it will also make no
attempts to prepare for shutdown
so let's say that you've got an
ai and it's building a skyscraper
now suddenly these designers decide they
want to stop construction so
they press the off button the ai
immediately drops whatever it was doing
and switches off
some poor sword gets squashed on her by
a falling girder
on the plus side it will have tried its
best in making the building
which is why this is a feature not a bug
it doesn't incentivize shutdown
and it doesn't incentivize pressing the
button
more specifically the agent would pay no
utility to press the button
it will sacrifice no utility in order to
make sure the button is pressed
and it will sacrifice no utility in
order to
keep the button functioning
which is kind of odd right
because what if for example the utility
function says
that okay i don't really care if you
press the button or not but
i want some parts to make some
contraption so then the agent says okay
well
i don't really care the buttons pressed
but there
it has some nice computation parts to it
so i'll just
grab it and use it for scrap pieces
this method of indifference does nothing
to really prevent that
right it's just saying that the ai
doesn't really care if the button is
pressed or not it's not saying whether
or not the ai
doesn't care if the button is still
maintained
or not you're only making it indifferent
to a very particular thing
and there's another failure of this
which
is that okay fine it
doesn't care if the button is pressed or
not but maybe
the human pressing the button is
correlated with some behavior that it
does care about so say for example the
humans
go to press the button and they are
prepared that
if pressing the button doesn't work
they'll just
launch some nukes at the ai
in which case the ai is thinking okay
well i don't really care if the button
press occurs
but you know my utility function says
that i do care
whether or not i'm used um that will
just
ruin the stuff i'm doing so it will try
to manipulate the females into not
pressing the button
or you know it might just destroy the
humans in retaliation first
there's an interesting point about this
which is that as the air gets more and
more capable
more and more intelligent and able to
manipulate people
you will probably be able to continue oh
no don't worry if you press the button
will totally work
no need to prepare for some crazy
contingency where you have to nuke me
and then the humans go oh okay fine and
the ai
is fine with that it doesn't care about
if the button is pressed
but it has managed to to convince the
humans not to do
some behavior which you would care about
but you know this is
in effect preventing us from pressing
the shutdown button
um if things go sour which we don't want
and this is basically the broader point
which is that
it won't obey humans
at all insofar as its utility function
tells it to or not you've got to tell it
what to be indifferent about
you want to specify every single way you
might want to shut it down
you should be indifferent to these
methods so including say
whether the human wants to shut it down
by shooting it
at a particular time at a particular
place and then you have to specify all
of those times in all of those places
and all of the particular conditions
so we can see a sort of trend here
indifference methods are
cool like reward functions they're quite
powerful in general and they're
logically
coherent well most of them
some of them have weird problems with
counterfactuals but we won't go into
that
but the point is that like a reward
function
you need to know what you're going to
put into the indifference methods you
know you need to know what the agent
should be in difference towards
just like you need to know what the ai
should maximize
you need to know the semantics as stuart
armstrong the inventor of these methods
put it
and this is why he calls them syntactic
correct ability
they are like a series of formal
manipulations
that have no real content
to them they can be applied to any
situation they're incredibly general
but they lose this
specific nuance which we want
when you design a ai
be corrigable
so unless you have the semantics you
won't be able to fill the other
desiderata like
preserving shutdown behavior and you
won't
be able to fulfill the
desiderata about
the broader grigor
those are the first four things that
miri
raised miri sorry defined as
credibility which is don't interfere
with your designers modifying you
maintain your designer's ability to
modify you
those two weak or ability or sorry
indifference methods can achieve but you
need to put in a heck of a lot of work
specifying exactly which ways
it shouldn't interfere with modification
in exactly which ways it should maintain
things
but it doesn't solve manipulation
problems and it doesn't solve the
preserve all of the other aspects of
credibility when creating sub-agents or
successors
and there are also practical dis
considerations for many of
these algorithms for different methods
some of them are about as hard as
general bayesian reasoning
which is you know pretty darn tough
right and
that's there's some work being done to
make these practically useful
but i mean
it might not be worth it in the long run
or other might not be sufficient
now say whether there's a natural
generalization
which doesn't suffer from the problems
of indifference methods
and does fulfill the other two
desiderata
is an open problem
stuart estimates maybe 15 chance that
these methods could be generalized
to work for weak correct ability that is
it will make an agent that won't try to
prevent you shutting it down and the
host of assorted behaviors we'd like to
go along with that
but he doesn't think that it's
sufficient to stop stuff like the agent
manipulating you
which is a pretty tricky problem
like if a super intelligence
super intelligence exists then they can
probably manipulate us in some extremely
subtle way
which fulfills all of the criteria of
weak correct ability
but doesn't fulfill any of the other
criteria
what you really want here is strong
trigger what mary initially outlined
i keep playing strong drug ability and
referring to the four digits of the
router but really there's a crispr way
of stating all this
namely strong corrugability is when an
agent
is trying to do what the human wants it
to do this idea isn't without flaws but
we'll get to that later
so notice that this is quite a strong
condition on the ai
for example if the ai happens to be a
human happens to be an aic
then the ai would be trying to do what
an ai safety researcher would do
namely solve ai alignment in some sense
this is
dealt with a lot of the problem of
alignment
as we wouldn't have to worry about the
ai irreparably ruining the world
in much the same way as we wouldn't have
to worry about an alignment
researcher irreparably wrong
as yetkowski put it he would not mind
handing the keys to the universe to pull
cristiano in the expectation that paul
cristiano would hand them back
a strongly corrugable ai would act
mostly like
bull cristiano so we wouldn't have to
worry about it in that sense
note that this is of course not the same
thing as solving
all of the alignment problem because the
ai
doesn't know for example what human
values are
it's trying to figure it out but it
hasn't gotten the answer yet
so it's not a solution to full ai but it
feels like
a lot you could breathe a deep sigh of
relief
if we made an agent with strong
corrugability
and paul christiano in fact is working
on this problem he thinks that
the stronger credibility for the reasons
i just mentioned
is basically good enough as a solution
to the alignment
problem
and let's just talk about his views
so
things that there's
a few nice aspects about strong
credibility
one is that he thinks that griggable
agents
will make corrigable successors because
that's what
say we would want them to do
right and furthermore
say its successor is somewhat
mis-specified
but it's still griggable so we have this
agent that's perhaps more intelligent
more capable
it's gotten uh we've designed it a
little wrong
and that's okay because it's still
strongly corrugable it knows that
there's some issues with it
so it will try to correct course it's
robust to miss specifications
this is what paul cristiano calls a
broad basin of attraction
if you imagine this very crudely drawn
map
as something like a map of the possible
designs of an ai
then corrigable agents basically form
a basin of attraction right like inside
this circle
let's say that's where corrigable agents
are
if one is corrigable then it's
going to sort of naturally fall
inwards into more griggable states
and it will make its way say from here
a little inwards inwards inwards right
until it gets down to basically
maximally griggable and from there
it will no
know exactly what to do
or rather it will know exactly why it
should do what it needs to do which is
solve the alarm
and it will be quite capable so there's
a pretty good chance that
we'll be okay
and his actual proposal
is that you should do this thing called
iterated distillation and amplification
in other words ida that's the acronym
how it works is you start with a
corrigable agent
and you work together with it to make it
slightly more powerful
which you and the first agent can check
to see if it's griggable
you use all of the tools available to
you at this
point in time so you throw everything
you have at checking that this thing is
really corrigable then you use this new
agent
to repeat the process it's slightly more
intelligent you can make a slightly more
capable agent now
and you can in principle bootstrap your
way up to a super intelligent corrigable
agent
there are a couple of issues with this
approach
some of them are practical like whether
his idea for the initial design
of a curricular agent will try to
manipulate us or not
that is how do we get to the first
coringable agent
and how can we make sure that it's
really correctable
like one of the natural
[Music]
problems with strong curriculability is
how do you make sure that an agent isn't
just manipulating us to make it look
like it's correctable
so paul thinks that this is
[Music]
not that hard to do eliezer thinks that
okay saying it's easy to check if an ai
is manipulating a human seems to be
anthropomorphizing things a bit
and he doesn't think it's that simple at
ask
world christiano thinks that it's
feasible as in
not necessarily simple but a
technical challenge we can probably
overcome and he thinks that predicting
if
x is manipulating y will get easier as a
i become more intelligent so naturally
this is an actual prediction right like
as we make more and more capable agents
we can check whether or not
it becomes simpler for us to make ai
which can predict if something is
manipulating us or not
so hopefully in the next few years or
the next couple of decades
we'll get some feedback on this
but there's another problem
and this is one which applies to
corrugability in general
this in fact ties in with manipulation
in a sense
who exactly are we corrigable towards so
consider
you happen to be a 25 year old ai
researcher
or say
someone who has
a robot serving them or whatever just
someone with a robot
and you are a particular age you've got
particular values particular preferences
and the ai asks you
what do you want
what do you think of my doing something
in this particular situation
in fact no let's be a bit more concrete
let's give a slightly silly example
say that you made a correctable ai
and you've
had eight slices of cake
the robot ai is told to bring cake
to it now naturally the robot sees the
plates on the floor sees that there's
even a cake half eaten there
and it remembers that the supervisor is
meant to be on a diet
now the human supervisor right now wants
cake
but you could probably gently persuade
them that they don't want this cake in
actuality
so should you persuade them or not you
could say okay well i should really just
try to aim to
do what they want me to do right now in
which case you would just say okay fine
whatever here's your cake
you could do something else which is
just give them a
argument for why they really don't want
this and then
they say after thinking about it for a
while you know what you're right i
really shouldn't have any more cake
or consider another example they're an
aging billionaire
who is wondering
which nephew to leave their will towards
they want you to help them decide
now he's got a pretty diverse family one
of them's a
hippie another's a social justice
crusader a third is an ea
a force the vicar and a fifth is a
dancer
so you've got a lot of leeway to help
him
and it seems like there's no natural
path here there's no sense that
there's only one argument you could give
him
one way you could help him develop his
preferences
so what do you choose because clearly
you've got to help him do
something right even doing nothing as an
action
and this is manipulation of a very
different sort because
the ai is still clearly trying to help
the human
it's trying to do what the human wants
it to do but the human doesn't really
know what they want to do
right so solving this in a sense feels
like
it requires solving a lot of deep
questions in moral philosophy
what are human values this is
basically the whole problem of ai safety
of alignment really
and it's in a sense quite similar to the
issue with the indifference methods we
have to put in
a lot of work about human values a lot
of work about
what counts as the humans trying to shut
you off
before we can really make the age
incredible so if it's the case that
here we need to solve human values
before we can make strongly corrugable
ai then you know what's the point
because in that case we would have
basically solved the ai safety problem
now i haven't finished learning about
paul's agenda
so maybe iterated distillation and
amplification has a solution to this
problem
and in fact i think it does but
you should probably take what i say here
with a pinch of salt
so paul thinks that
it's not necessary to
deal with to do something like say
you're
the ai and then you simulate
the human supervisor's entire life
to figure out what they would want you
have you to have done
because you know clearly after 20 30 40
50 years the human will have quite
different preferences
and you probably could have influenced
them in any
number of ways etc etc
rather what he thinks you should do is
that you should
the ai should consider simulating the
human as they are now with their current
preferences
giving them and giving them some time to
think
so it has this mental simulation of the
human and says well
what do you want a cake or death and the
human says okay in their
imagination give me five days to think
and then the human comes back and says
um no actually i don't want cake
or more pertinently um
i have this the ai could ask do you
think i'm being manipulative by asking
you cake or death
and the human could say okay give me
some time to think
and i'll judge whether or not you're
being manipulative
based off my current preferences that's
the key right
the human is thinking reflecting on
their current preferences
and saying are they corrigable are they
being honest are they not manipulating
us based off the human's
current conception and it this is kind
of a strange thing to do because it
feels like
there are a lot of ways that
the human could reflect on their current
preferences it doesn't
feel like there should be a single way
that this could evolve
but whether it's actually true as of yet
is kind of unclear
so that's that's
probably the key issue with paul's
grigability agenda at the moment
and we've discussed correct ability
ala cristiano we've discussed
gregability al-amiri
these are basically talking about the
same sort of concept
but we'll now go into another conception
somewhat briefly which is not
so much correctability as rather an
examination of
whether we want correct ability at all
like whether there are any downsides to
having an agent that's totally obedient
and this is covering the work
done by stuart russell and david
hadfield menaul
who work at ca hai
now whilst they think corrigability is
useful
they think that there are some downsides
and they try to investigate this using
their framework of
cirl games where
basically the human nai are assumed
to share a reward function
and they hope to create agents which
learn what this assumed reward function
is within this
context of a game so we're basically
assuming a way
that the humans preferences can change
and the ai tries to learn what the
human's reward function is which it
shares
by assumption within these sets of games
and this doesn't mean that the agent
will always obey the human
there are cases where the agent could
perform better according to this
presumed reward function
if it disobeys sometimes
so they formalize this and if in one
line i had to phrase it this
formal condition of when to obey the
human one to be
correctable is that you should obey the
human
if your moral uncertainty times your
measure
of human rationality is greater than or
equal to the expected cost of letting
that human correct you
this is more my conjecture
of their approach
but to be fair it's based off an
interpretation that they gave in the off
switch game and i think that this
interpretation basically holds across
the board
for their work on correct ability
whether
this is really true would be an
interesting research question mind you
but let's just ignore that we can get a
fair bit out of this interpretation
so now suppose that the human is
perfectly rational or rather the ai
thinks it's perfectly rational
in this sense this thing would basically
be viewed as
equal to infinity and assuming that
the human can't screw things up too
badly that is
the reward function doesn't say that
there are infinitely bad outcomes
this thing
is always going to be smaller than the
term on the left because that's
something times infinity
greater than or equal to something
and if a and b are both finite then
clearly the term on the left is greater
than the term on the right
so necessarily the ai in these
cooperative games will obey
but obviously the human isn't always
rational right this is a point that
they're trying to make which is that
if the human is making mistakes then if
the ai just blindly obeys the human
which is kind of what it should do in
if it's griggable then there are going
to be some trade-offs it might not get
rewards as quickly as otherwise
because say if it's like i don't know
just as a trivial example
say it's a self-driving car and it
thinks that
every passenger is infinitely rational
and it happens to be
driving a toddler after school now the
toddler
is quite curious and wonders what would
happen if they drove into a lake
the ai wisely decides
in its view to heat the toddler and
drive into the lake
clearly this is you know a pretty
terrible thing to do so
let's do russell and go
but okay fine well we want a better
measure of human rationality we want it
such that
it's the optimal measure of human
rationality so if you viewed
this measure of human rationality as
being on a linear scale
so i hate the human's rationality
and how
much value the ai can produce
if it thinks the human has a certain
level of rationality it turns out that
this will be optimal at exactly the
real point right like this
optimal point for when the
robot should obey the human should
reflect reality
hope you excuse my terrible handwriting
and this of course presents problems
because in the case of extremely
complicated scenarios
the human is going to be not very
rational relative to the ai
and the ai might be a super indulgence
and it figures that okay i basically got
i figured out pretty much
everything there is to know
about human morality and like i've got
this
residual term left this residual
uncertainty this tiny little thing
called epsilon
and i think that the humans are
kind of stupid so this is like
i'll assign them some measure of
rationality which is really big
it's this huge thing maybe i should just
make that a bit clearer
and you also know that
oh sorry
this is perceived human rationality so
this should be like something really
small
and of course letting the human decide
on arbitrary occasions is pretty bad
because say you're a super intelligence
you've got the world to run
and listening to the human could lead to
unfathomably bad consequences
so we'll just say that that's really bad
that's some huge quantity
so on the left you have a tiny quantity
times a tiny quantity
that's obviously much greater than this
huge quantity
uh don't worry about the units so
the super intelligence will disobey and
this is basically yadkowski's criticism
of this approach to corrugability
because it's not really correct ability
if the ai will eventually disobey if in
the limit of superintelligence there's
no way to convince it that yeah you're
wrong
because it's totally sure that it's
converged to the right
utility function of course
people at ch ai say well you know if
it's a bayesian reasoner and it's got
the
right sort of hypotheses about what the
reward function is
then it should eventually converge into
the
right level of human rationality
sorry the right level the right utility
function
and in that case we really would be
better off letting the ai run things
but they do agree that yes when you're
designing your initial ai you probably
want it to behave
you want it to obey you because it's
going to misunderstand things
and in that sense they do agree that
weaker ability is quite useful
they also agree that strong credibility
is a very useful property but they think
that in some way that we'd probably be
able to
get the ai to learn it by doing these
cooperative games
whether or not that's true is an
entirely different matter
to some extent i think that this
probably isn't true
because stuart armstrong
gave this statement
that
you can't infer values from behavior or
rather
occam's razor is not sufficient
to infer human values if you look at
what humans do
and you've got your prior and you just
update naively
then you could converge to any number of
rationale
sorry any number of utility functions
any number of
possible values because
the humans might be irrational in a very
particular way
like they're telling you that they
want cake they don't want death but
maybe the humans are
all uh actually
nihilists they want to die but they're
just so
hopelessly bad at fulfilling their
values that they keep saying they want
cake and they you know
they keep denying you when you say cake
or death
and this is it's unclear whether or not
this is really a problem whether or not
you
can actually in practice infer human
values
using occam's razor in principle it
seems like you shouldn't
be sure you can in practice maybe you
might
it's hard to say and oh my god i'm
i just realized that timothy was asking
to be let in
i hadn't let him in i'm very
sorry for that timothy um
i'm afraid you missed the talk because
that's basically it
okay thank you very much


Oxford professor on Transcendence: how could you get a machine intelligence?

one can also ask how could we get there
like what are the possible paths that
could take us from ariana
to human level machine intelligence in
the movie it occurs
through uploading a person and
they've done previous experiments on a
macaque monkey
and this this idea of copying an
existing intelligent system the human
brain
and and then transferring that into the
computer is one possible path towards
artificial intelligence another possible
path
is is to forget about biology and all
its messy details
and and take a more mathematical
approach to this and this is like the
way that
the good old-fashioned approach in ai
has proceeded
and then there are things in between
that kind of draw inspiration from
biology but
doesn't try to copy it exactly just
figure out the basic principles of how
the brain work
and it's an open question which of these
different avenues will get
to human level machine intelligence
soonest
but the fact that there are these
multiple different paths that all lead
to the same destination
increases the chance that the
destination will ultimately be reached
but it's an open question exactly how
how we will get there


249, MIRI Announces New "Death With Dignity" Strategy

hello and welcome to session 249
in the ai safety.com reading group
tonight we'll be discussing the
restaurant post miri announces new death
with dignity strategy by elias
yatkowski eliza yutkowski is the founder
and senior researcher at the machine
intelligence research institute in san
francisco and this is a post on last
round which was posted on the 1st of
april this year and that date is a
coincidence it's literally attacked
april's fools
it is
of course called mary's strategy but
it's uh it doesn't have very much to do
with miri and it's not really a strategy
in the sense where you think about like
they're changing from um
from asian foundations or something like
that
kowski starts with a an overview of the
strategic picture which is very very
bleak
miriam didn't have the api alignment and
at least knows that it didn't and it's
in past tense so i think this is a third
deal
alignment
it's incredibly complicated and it has
no chance of working in real life before
deep mind destroys the world
that's also a
rather
i think
complaining that they are incredibly
complicated is not my first objection
towards this um they are not that
complicated at least most of them are
not that complicated um
but the problem rather is that
we are
secure any kind of strong arguments why
they would be likely to work that's
that's the problem with them not the
complex
and is it
an interpretability scenario which is uh
very uh dark where there is in fact a
speculative warning being given by
interoperability research
and even
and even if we could get a speculative
morning in this uh scenario then it's
not accompanied by any kind of fix
there's no suggestion for how we can
align crit align the system and face
with this warning management decides to
basically press on and ignore the
warning
for four reasons
the first is that they are uncertain of
is this this is a speculative warning
this is actually the tool that the ai is
plotting to destroy us all
the second is the question of whether it
will act on these intentions
the third is
a race dynamic like real theme where if
we don't build it probably someone else
will do it's very cool
they find some kind of face in this one
where
if
if ai is going to destroy us then
there's nothing we can do about it so we
might as well
we might as well go ahead
in order to understand uh
the the um
uh the rest of the of the blog post the
concept of logistic success curves or
logistic functions is
essential so let me start by
to explain this and start by using an
analogy building a bridge
if you want to build a bridge there is a
predicted cost let's say it costs 10
billion to build a bridge there is some
variance there is some like some tissue
maybe sometimes they go above budget
sometimes below
and if you're uh
where the x-axis have
the cost in uh logarithmic so 10
to the
x
and on the y axis probability then we
can identify some numbers like i just
said that the bridge had an
estimated cost of 10 billion so that
means 10 to the 10th
gives us a dollars gives us a 50 chance
of success
now if we have less money we might only
have 10 to the ninth so one billion
what is the probability that building
this bridge will turn out to cost one
tenth as much as predicted i think that
happens very very rarely i've just
written one percent
and uh symmetrically
if we get
10 times as much money as we expect to
need so we have enough money to build 10
bridges instead of one what is the
probability that at least one of these
bridges will actually be built well um
i've put it at 99
so that's kind of the uh the motivation
behind logistic success curves
and one thing we should notice in
particular is what if the budget is much
much lower than what we expect to have
to take like once we are at um uh 10 to
the seventh uh so one thousandth of what
we expect to pay basically the chance
that we can build a bridge and pay only
one thousands of what we uh expected is
basically zero
so that means we have a graph that for
the first uh
from x series or seven is basically
indistinguishable from zero percent
chance of success
here we have a logistic function um as
you can see um
it has a um
we have the input here
uh like if we get 10 to 20 then 99.9999
we can make it if we get
100 we can almost certainly not build
the bridge
um this has a uh a midpoint
i've put it at
10 billion dollars so 10 and has a
growth rate which is basically the slope
of this curve and it's of course reality
that determines
how
how this function looks in reality
one of the things we will notice is that
the distances when you go from one point
to another point it's not like we get an
additive a linear
increase in the probability of success
we get
a multiplicative uh relationship so if
we go from here to here we double our
changes and from here to here we double
our changes here so here we double our
chances etc
um and of course the claim underlying
this is that this logistic success curve
is a a good way of looking at
the alignment problem that we will um
need to we don't know precisely how the
curve looks because we've never done it
before but there is probably a level of
effort where we are really really
certain we'll make it and a level of
effort where we are really certain we
will not make it
on this uh logistic uh
success curve illegally believes that we
are far underwater in the basement so
very far to the left to the point where
we in fact have just about zero percent
chance and that's really sad and it can
be hard to get uh motivated and stay
motivated because even if you work as
far as much as you can the probability
of
if if you double the probability that
humanity will make it well we've just
gone from two percent to zero percent
and that's still zero percent so the
expected value of all of your incredible
work is nothing and that's kind of
demotivating and and that's why elitist
utkowski is suggesting an emotional
reframing um where we are not trying to
improve
our odds of making it we are helping
humanity die with dignity or at least
we are working so humanity dies with
more dignity than we would otherwise
have had because uh
elias koski is clearly seeing us on a
very very on dignity
on our on a path to a very very
undignified death
and
one another way of bringing this dignity
is like it would be a better look on our
tombstone if we
died with more dignity if we had some
better
probability of making it
let's talk a bit more about dying with
dignity
um
uh one you can imagine a uh
a uh
uh an intervariability researcher such
as chris olah um
right now i think elizabeth seems under
the impression that he's working on a
deep mind he's right now working at
anthropic uh but this researcher if he
was not working on interpretability then
it's much more likely that no one would
try to generate a warning and and fail
even if uh chris owner ends up failing
and it's more dignified if as in his
scenario management of the project that
killed us all are warned in some way
that the project will kill us all
it would be
more dignified uh if many people just
like chris ola was doing something that
could perhaps have worked to generate a
warning but did not in fact do that in
real life um and this phrasing uh didn't
actually work in real life or variations
on that it happens a lot in this text
and that's of course uh underscoring one
of elias koski's main points that there
is a substantial difference between
things that could work out in theory and
and things that actually
end up working in the real world because
the real world unfortunately is far more
complex than the theories that you can
just
just make up
and the least dignified uh way we could
die would be in
total ignorance no warnings and no one
even trying to figure out how we could
get a warning
how about the machine intelligence
research institute
there they have a dignity too it's not
just the human civilization but also the
machine intelligence research institute
has a dignity um in how much are they
contributing and they are sad they're
sad that um
it looks like we're going to fail and
elizabeth is sad but they're not said
that they have tried because
was important that um
made a serious effort in case the
problem just wasn't that hard
as
it would have been effectively less
signified to die and not knowing if all
it really took was
a serious level
and that's
kind of a new
intuition on
miri's
logo you can imagine time flowing
upwards so here is the singularity of
agi and the paths here are the possible
human paths through the future and there
is like one that is orange one that's
green and then 80 of the paths leading
away from the singularity are dark and
so uh with the implications that that we
have died
i haven't i haven't
thought of this uh interpretation before
uh or nor read it anymore
so that was the dignity of uh miri what
about the dignity of earth
well it's not looking good right now uh
earth steals brilliant kids towards
theoretical physics
that was actually my person when i when
i grew up i have read about all these
wonderful things that theoretical
physicists did and i thought that
sounded really really interesting i
would love to do that if i was
very smart and it turned out i wasn't
very smart so i went into computer
science instead
but what i should be going into of
course would have been ai alignment and
that's what in particular the brilliant
kids should have been doing
and of course some of many of them are
going into quantitative finance or
things like that
but earth's dignity is not
zero we don't have no dignity because
when we die there will in fact have been
people who know why we are dying
and um
it's not enough the problem is not
solved because the people who know will
not be believed or will that uh have the
influence that they need they most
people will uh
will uh choose to listen to uh some
other someone else who is reading some
kind of political game but um that's not
really the the crooks of the problem
it's not really a problem of
politics because even if we have
politics
much better we have actually solved the
alignment problem and that just means
that
even if the first couple of projects
decide not to destroy the world
eventually there will be one who is uh
careless enough and uh for strategic
reasons pushes ahead too far and then
gets us all killed
i'm not entirely sure i agree with a uh
i certainly don't agree with a
a complete split between the political
problem and the technical problem
because if we had some kind of let's say
we get a proof of this ai is misaligned
in some sense that would both be a very
strong technical result on the way to
solving the alignment problem and would
also have a strong political um
advantage so these two problems
in my view are somewhat related
as slightly better technical solution
will uh uh
better technical work even if it's not a
full solution will make the political uh
question easier
we could in theory imagine dying with
even more dignity
uh like interpretability research could
have been better to the sense that we
get some really strong warnings
that's dying with more dignity if we
instead of doing nothing then try an
alignment scheme that is really has a
very low chance of working that's a lot
better and um
if we
um
do this substantially sufficiently
better we could uh instead of right now
dying without a serious effort we could
die of the social and technical problems
that are really unavoidable so
it looks like right now we are just
not taking this problem seriously so
we'll die of
uh
we'll die without trying some relatively
trivial things
but we could die
trying even where we have done the
trivial things
and perhaps
just maybe if we uh find something in
the future some miracle some way we are
wrong some way we are surprised that
make things easier rather than harder
then perhaps
perhaps we could take advantage of this
miracle
that's going to require a lot of sanity
it's going to require a lot of
preparation and of course the doctor
there is a
miracle
but
taking advantage of this miracle
depends on dignity and so this hint of a
definition of dignity like dignity is
something that allows us to take
advantage of
a model a positive model error
and in principle it wouldn't even be
possible for us to not die but
it is casually does not mean squirts
because that's not how
real life works in real life
we die
why we won't be saved
well first of all
there are probably a large number of
ways we are raw about hr because we
don't have an api we haven't seen it at
all and it's possible that many of our
theories are just basically wrong but
when you're fundamentally wrong about
a domain if you use the analogy of
rocketry then
that doesn't mean that your rocket do
does exactly as you want just using half
as much fuel what happens is that the
rocket explodes in a way you didn't
predict
because rockets are
machines where everything needs to work
and if you're really confused about
building rockets you are not going to
the home
in particular
model errors make your life more
difficult almost always but not always
and
the social
problems the political problems are also
more likely to become harder in the
future rather than easier because
when people become scared they will
probably make worse decisions
i'm not entirely sure that i believe the
problem so much is that people make bad
decisions right now i think the problem
mostly is that people are not making
decisions at all so what the way i pre i
predict the future is more that we will
see um
instead of right now where we are seeing
nothing in the future we might see
something which is probably going to be
bad decisions but uh but different in
kind
now eliza carski introduces dignity
points
and uh that's from this
we had the desire to die with more
dignity because if we can't actually
survive then dying with dignity is
something that we can in fact achieve
and if you help in humanity die with
even one more dignity point you yourself
die with 100 dignity points
that's a bit stingy i expect that
someone who helps humanity substantially
is um and towards preventing existential
risks is
is indeed someone who is worthy of way
more dictators of course
um
so
and he's also uh later claiming that if
you help humanity go down within
slightly more of a real fight that is to
die an extremely dignified death
and uh there is a definition here a
project that doubles humanity's chance
of survival from zero percent to zero
percent is helping humanity die with one
additional information theoretic bit of
dignity
uh this is really crippling but i'm not
sure information theoretic bit is
uh the right framing for dignity points
in that they are points scalars and bits
are units of information in the
information theoretic sense and i don't
think that's what dignity points are but
you know me of course i will digress a
bit on dignity points and try to look
deeper into the uh
the definition
so
eliasian cascade gives a
literal definition of uh
dignity points um they are measured over
hunters lock arts of survival
that is the graph on which the logistics
success curve is a straight line
now let's have a look at that
and try to uh go through some simple
examples
so i have uh first in excel type up some
examples of probabilities what's in text
us ratios and what are the log parts
we'll start with um some of the the
thing the odds that we normally have
about like here's 50 probability that's
odds one to one and one divided by one
is one and the log of one is zero um
and if we have 20 chance of success
that is uh one to four
and one the odds which is an outrage
ratio of zero to 25 which is minus two
so
compare the situation where we have even
odds and we have uh 20 chance of making
it uh that is um
is uh two dignity points down
now we can be more pessimistic which is
clearly illegally
and see we have like 120 one in 100.1
one ten thousand that is minus 13 uh
dignity points for our civilization
let's try to have a simple uh
calculation i think the most
negative uh number i could find if i
think through all the bad news and
updated fully on those another on all
the good news is one in one hundred
thousand uh odds of making that
um so that is one two uh yeah 99 999
which comes out to minus 16.6 log odds
and if you want to obtain one dignity
point from that well then you take this
and you just add one and then comes out
to uh this box if you're uh
exponentiated um and um
that's this ratio and that is this
probability which is yeah kind of the
double probability because the
difference between this number and this
number is basically nothing so
at least for small numbers talking about
the probability and the odds ratio is
you know basically the same thing
and that suggests a an easier way of
looking at this or at least an
alternative way of looking at this
because if we have this is the logistic
uh
success curve here is the uh analytical
expression for the logistic success
curve um
and
most people i uh would claim don't think
about probabilities and they don't think
about odds ratios
um and that suggests that people are
probably interpreting this mostly as the
logarithm of this
so let's see what is the logarithm of
this function well this is um it's this
graph here and as you can see
it is for small probabilities basically
a straight line
so that's also what really is a cascade
was talking about that if you
double our chances you get
a fixed number you get one dignity point
um
again this is the log of the probability
and not uh log odds
and and this interpretation should
suggest that if you believe
that we are probably not gonna make it
if you are 99 sure we're not gonna make
it well 99 sure that is this number here
that is minus 6
in
minus 6.6 dignity points that is
somewhere around this level if you
believe we are at this level
then
the expected utility of improving our us
is really really bad but you'll get
quite a few dignity points so someone
who believes that we are almost
certainly going to die is gonna love
dignity points because emotionally they
provide something you can actually
achieve something that is positive
whereas someone else might believe that
there's a 90 chance we'll make it and if
they think about probability it's not
like odds then they'll see us up here
as so and there is a going uh just a
marginal approved improvement around the
90
probability of success is worth a huge
number of expected utility and very few
dignity points
um so it could be argued maybe that um
people who think we are doomed love
dignity points and people who think we
are probably gonna make it they
will not be motivated by different
points
uh maybe i like uh this is uh i've spent
too long time on this to call a hot take
uh
uh so um
but i i'm not putting a lot of uh weight
on this this was basically a digression
let's move back to the rest of the uh of
the post which is structures as a
dialogue between elias erkowski and
an anonymous questionnaire uh i have uh
put the uh
using a collar inverted picture of ilia
syrizkowski for for the personal making
the question
so the first question is dying with
dignity does that just mean we accept
that we are going to die and not
try to fight a hopeless battle
and elias wikowski answers no because
that does not increase in the gods of
foster's survival
elite costume
is sad that we're gonna die and and came
the fourth hardest when we were in the
more slow part of the logistics success
curve um where something could have been
changed it's not something he
regrets but he burned himself out to
some degree and is taking um
some time off now
it is certainly true if you fight mine
any longer you start with just
marginally more dignity um but
and that is indeed dignified the
undignified thing is deluding yourself
about the outcome or the probable
outcome
so here we see undignified being used
mostly about the process of deluding
yourself
the second question is
if someone has a uh or the the question
here claims to have a clever scheme for
saving the world and states that he
should is it asking if it correct that
he should uh act as if he's going to uh
uh succeed with this even though there
are really really strong arguments then
this misguided doomed
um because if it doesn't work then we're
all dead and then nothing matters right
no with an um mark mark because that's
not dying with dignity
and before he dives into the real
arguments he starts out with a heuristic
for how to generate the answer that uh
to this question and that is that it's
not dignified to die in this way uh
because what you are actually doing is
stepping out of uh reality into some
kind of um uh
mentally uh
safe zone where things are going to go
well and instead of being in reality
and it's more dignified to dive with
your commitment to recent intact and try
keep searching truth and motivation
i love this picture i found it on the
internet i think it's louisiana
um
so uh the uh the question
the person here is uh objecting to the
heuristic answer and what's the real
answer like uh all the utility in the
future lies in worlds where the scheme
that is being uh suggested is working so
why not just focus on that
and uh
answers that that is in fact not how the
surviving worlds look like in his
expectations
where people try to live inside reality
and
stare into reality with grim
determination and try some way to
shape up their impossible changes
english is not my first language i'm not
entirely sure what shape up means that's
like sizing up in looking at what other
changes are improving the chances
and and the the surviving walls are ones
in which a miracle happens and
people are positioned with resources and
in particular
itself is crucial that that allows them
to take advantage of the of a miracle
um about that your scheme is going to
work then very often you'll need several
and you make a lot of assumptions and
you get very very far from the reality
it could be said that if you want to
make this kind of improbable assumption
it could in theory work but you can only
make one of them ever and that's the
problem that what people who are going
to do this are going to make
i think it's a it's nice to see this
spelled out but i would really have
liked to see this uh argument compared
with the argument for specialization
which
very often looks very much like this
very often people who like if you want
to specialize in i don't know i say to
via debate it makes sense to some kind
uh kind of uh live in a world where the
success and failure of um ai safety via
debate is the crucial thing like um in
order to specialize on just working on
aicc via debate and then hopefully
someone else will be working on other
research agendas um so it's not the same
argument and i would have liked to
see some kind of comparison between the
two so um because i do believe
visualization is really important to get
anything done
agi alignment is murphy cursed is a is
one of india's caskets arguments
and that's to be understood as in uh i
leave murphy's evolve was originally
proposed in not quite in rocketry but in
something like experimental jet engines
where
someone called murphy
made the law that if anything can go
wrong it will go wrong
and a murphy curse to me like murphy's
law probably doesn't happen literally
for everything in most domains but a
murphy curse domain is one where
being pretty much everything that can go
wrong does go wrong
so what do you need in a very request
domain you need mathematics mathematics
is uh the uh
the only way to ensure that things
cannot possibly go wrong
and that points very much towards the
aging foundation's agenda that miri has
been pursuing and uh are finding too
hard unfortunately
i would also add here that uh murphy
cursed domains might not make a lot of
sense
if it's something that can be tested
and a
a domain is only murphy cursed
to the extent that testing is impossible
so the two examples are rocket
prototyping and computer security
which are also somewhat morph cursed
but less so than alignment because we
have a scientifically unprecedented
experiment in agi and we have
optimization pressures and some of them
are
working against us the real world once i
need the ones working against us
we can
we cannot really predict whether
intelligence
greater than ours will do by almost out
of principle
we need to hit a relatively small target
and we have a lot of things that can
break in
extreme practice and we have no way of
really testing things beforehand
so if you have a really clear scene that
really looks like it's going to work not
just to you and me but also looks too
easy it counts me like it would work
well
in this case adi alignment is so murphy
cursed that even if it looks like it's
going to work and you can't see a reason
why it shouldn't work then it has a 50
chance working in real life because the
domain is just super hard
if you have something that you are much
less certain about or you have like some
moderately weak arguments like four
percent chance that would work then if
you put something like that up against a
real a truly birth course to me then has
zero percent chance of working in real
life
and worse a lot of these uh schemes are
in fact actively uh harmful they're
called hair brained uh usually basically
harmful
and the reason giving is they're
invented by the kind of people who uh
come up with unbreakable schemes
obviously and try and get rid of them
with counter arguments like if it
doesn't work then we're all doomed
anyway
the problem with and main way they are
they're harmful they drain resources
away from projects that are not in their
brain
organizations that look into reality
people who look into reality and try to
take advantage of
some kind of miracle
i'm not very happy about this
part of the article because i find it
insufficiently specific we need to have
some kind of precision in being able to
distinguish what
what
schemes are
misguided and
unlikely to work and which are actively
harmful
um
and um
the three uh uh
parts
uh that that's being used the three
arguments are criteria is they are
self-reported as clever there are strong
arguments why they're doomed and strong
arguments why they are misguided
none of this is really enough i think
you probably want to burn some kind of
common you know to have a
truly actively harmful scheme
to some extent also because if you're
pursuing a a scheme that is misguided
then in the course of pursuing it
probably you'll learn something more
if nothing else you would on average you
expect to learn that that it is in fact
misguided
third question is
should we have a breakdown and whale in
the weight of doom industry in the
streets
and uh it is it asks me with dignity
also has a good argument that this is
not very dignified that much that most
somewhat uh
darkly very darkly indeed he suggests
that you have a private break breakdown
in your bedroom or a breakdown with a
trusted friend if you must
and the problem from an expected utility
point of view is that if you go
waiting in the streets you'll associate
believe in reality with people who are
also unable to control their emotions
actually this is a rather large subject
how should uh
rationalists deal with emotions quite a
lot have been written by this also by
elias caskey a big subject
and the reason here we don't want this
is because we are uh
we still need to think strategically and
if we uh associate belief with uh in
reality with uh
being stupid then uh then voices are
strategic
position
there is
the opposite suggestion hiding the great
truth and just presenting everything is
fine
that also doesn't seem very dignified
um
it is some
basic language about
how we should grow and also
thing that that in practice people who
deceive others generally also deceive
themselves to some extent they don't
live in reality
and another problem is that if good
people are liars that means that um
people
other people can't really trust if
rationalists go around lying then people
can't trust rationalists and that will
hurt us a lot and that will in
particular
hurt our ability to coordinate with each
other
and of course also also other people so
for that practical reason it's a bad
idea to uh to lie about uh and pretend
everything is fine
there is quite a digression here from
alias yukowski stating that this line of
argumentation is generally used by
people who don't understand
consequentialism or in utilitarianism
i haven't seen this argument very much
so i can't really
have an opinion about whether that is
actually true i expect indiana has
seen many many people make this kind of
argument
the fifth question is on extreme actions
or
rather the questions are not really
clearly delineated in this numbered way
that i've uh i've made them
so this is
a question of will elias and carson do
something that is that makes it unsafe
to be around him
and he says no he will be relatively to
be around
because why would you do something
extreme and something unethical
which is still not sufficient to save
the world
there is no extremely ethical actions
that would save the world in actual
reality because the thing that needs to
be done to save reality is
is not extreme oriented or unethical
it's basically to
look reality hard in the eye and do some
very very hard alignment research and
there's no unethical thing you could do
that would actually save the world you
could try to like politics could have
things that are extremely
unethical but don't they don't save the
world
so israel
will be relatively safe at least like
safe in the sense that he won't cause
you to die but you will die
i thought about this well i should
speculate about um
what are the extreme and ethical actions
we might be talking about here and uh i
think india's youthkowski by not
discussing them i'm trying to establish
some kind of overtone window and saying
we will not talk about this kind of
thing here
among
right thinking people and that's why
here even though i
was thinking about making a digression
about what extreme and unethical actions
are available um
then um
i i will not speculate on that
unfortunately some of eso encounters
self-proclaimed fans
might not be quite as safe as him
and he is in that case he's filled jason
whatever it is i know so
he is indeed putting
zero weight on this
he has like um the the simple arguments
that are being presented here with just
the expected utility uh calculations is
in fact something that he has written
about like
that very much but also not a little
there is some uh warnings against
following expected utility calculations
of a cliff that's something that you
definitely shouldn't do if you're not
really sure what you're doing
he also
states that he in his culture he is
perhaps
perhaps even quite different from his
fans in the sense that his he's grown up
with um liberalitarian science fiction's
uh
works of science fiction where one of
the things that differentiates heroes
and villains is when
the going gets tough do they then just
abandon their ethics or do they really
stick to their ethics even when it's
hard
and sure it's possible that if you tell
people the truth that could be panic and
penny can't disrupt the plans
uh and that way be bad but
this is not what's going to happen in
reality because there is in fact no
plans that can be disrupted so in that
way panic doesn't necessarily mean that
we have a lower chance of
of success
i must admit with this unsafe people uh
discussion in general i'm surprised uh
like this is questions that elijkowski
chooses to bring up and i'm surprised
you give this that much prominence like
i don't really see extremely unethical
actions being taken
okay
so uh
talking about things even though it can
cause panic it means dying with more
dignity
and
i know a bit um
uh i noticed that right as elias kowski
says he
seemed to not talk about these extreme
analytical actions and not speak plainly
about those immediately afterwards he
says that it's more dignified to speak
plainly and obviously he means plainly
about risks and not playing about the
unsafe actions
but in order to avoid panic he
wants to provide the people who might
panic with an easy line of retreat which
comes here and that is question six
this is just an april fools joke right
because it's been posted on the first of
april and tech equals four so really
this is not what he actually is and he
answers why of course it is
[Music]
strategically making sure that this is
something that can
cannot by someone who is not really
understanding deeply the subject uh uh
cannot really uh
infer that he this is actually what he
believes and the way around this the
only way to actually figure out what
does aliasing you'd ask me believe and
the only way to figure out the truth is
in fact to stare at the truth and try to
figure out what is the state of ai risk
what is the probability that we'll make
it and try to live in in that mental
mental world and
figure out what is true and allowing no
other consideration with that change and
that is
that is the essence of dignity
so now we have four definitions of
dignity we have the logarithms of
survival mathematics we have the dignity
is the thing that allows us to take
advantage of miracles it has the
exclusive truth thinking
so
uh suggestion finally is to people who
are worried about this and think it
might be an april's fool's joke to um
just
if they put a small probability that
he's right then he'd rather that they
just have super probability that the
world's gonna end put that probability
in a small safe corner in the back of
their mind and stay out of the way of
the gloomy people
and don't get away of any of their their
hopeless plants
um so in this way uh no
in israel what he's saying he is clearly
clearly not what he believes so is that
a lie
i think it's more uh
it's better to um
model what he's saying as some kind of
collusion between uh between these two
people is helping um this uh
uh christian maker um live in the
reality he prefers to live in um so
they're working together in some sense
and finally i want to say a few words
about aichhf2.com
um
and our strategy and how it fits into
this uh because from this we can uh
infer some of the things that the esv
customers say we should do we should
gather resources in some kind of like uh
set up research departments and prepare
tools and uh i'll learn as much as we
can and also um uh investigate as much
as we can and look out for positive
miracles
um
and so
are safety.com well we're obviously in
the reading group
looking out for signs of possible
miracles by you know reading that's
wrong and discussing the different texts
um and the the hope of what we'll be
doing in aicte.com the startup is
investigating to what extent language
models can generalize a major level of
so that also seems to like find out what
you can kind of think
and
you know gathering information we're
trying to actually you know build some
kind of research organization and that
counts as gathering resources um and not
just looking for a miracle but actually
you know taking a bit down and maybe
we'll find something that will
count as a an actual miracle if we're
really lucky and of course
we are explicitly open to changing
research directions if it turns out that
we find something that is uh
or someone else finds something that
looks really interesting and being a
position to that also seemed like a good
idea
um
there was one thing in particular that
struck me when i read this and that is
uh one of the risks we're facing is we
if we look into how can language models
generalize one meter level up there is a
chance of course that will inadvertently
find a fine tuning for g3 that will make
it easier for it to generalize a level
up and after which qrt3 might be
motivated to destroy the world and that
could destroy the world and that's of
course really bad and so
what i've literally said that if gmt3 is
capable of taking over the world we're
all doing anyways
so that's kind of close to the
hairbrained schemes that elias that
yokosuka was warning us about
but it's not fitting on some of the
formal criteria
first
it's not that people have a strong
argument that gbg3 is capable of uh of
destroying the world is around the
opposite that we are reasonably sure
that that
cannot do that
and of course even if gt3
if there is a interview
change the gt3
would be able to take over the world um
with a good prompt then
the probability of that happening is
much smaller than the probability that
we will learn something um by
investigating whether it can
generalize immediately level up
so
in total i believe like this is
sufficiently far from the hairbrained
schemes that he is
uh wonders about um that i don't feel
uh
this is a particular problem for aic3 to
come
we'll see
thank you and good night oh thank you
and see you next time